JP2020122891A - Manipulation system - Google Patents

Abstract

To provide a manipulation system capable of successfully detecting a position of a tubular instrument.SOLUTION: A manipulation system 100 for collecting minute objects with a tubular instrument 10 comprises: the tubular instrument 10; a manipulator 20 mounted with the tubular instrument 10; a sample stage 30 on which a container 38 for housing the minute object is placed; a first microscope 41 arranged above the sample stage 30; a first imaging device 45 for imaging a first image via the first microscope 41; and a controller 50. The controller 50 has: an image processing unit for creating a first binary image that displays a dark part having a brightness of a threshold value or less on the basis of the first image; and a position detecting unit for detecting a position of the tubular instrument on the basis of the first binary image.SELECTED DRAWING: Figure 3

Description

The present invention relates to a manipulation system.

Biopharmaceuticals are manufactured from cells. The cell used as a raw material is required to be derived from a single cell. Therefore, it is necessary to select and collect specific cells from the culture medium. Patent Document 1 describes a cell collection device having a manipulator for sucking cells.

JP, 2013-169185, A

In the cell collecting device having the manipulator, an error may occur between the position of the end of the tubular instrument detected by image processing and the actual position of the end of the tubular instrument. Therefore, it may be difficult to reliably collect cells based on the detected positional information of the tubular device.

The present invention has been made in view of the above, and an object of the present invention is to provide a manipulation system capable of satisfactorily detecting the position of a tubular instrument.

A manipulation system according to one aspect is a manipulation system for collecting a microscopic object using a tubular instrument, wherein the tubular instrument, a manipulator to which the tubular instrument is attached, and a container for accommodating the microscopic object are The controller includes a sample stage to be placed, a first microscope arranged above the sample stage, a first imaging device that captures a first image via the first microscope, and a controller. An image processing unit that creates a first binary image that displays a dark part having a brightness equal to or less than a threshold value based on the first image; and a position that detects the position of the tubular device based on the first binary image. And a detection unit.

According to this, the manipulation system can easily perform image processing such as removal of a detection portion such as a microscopic object other than the tubular instrument based on the first binary image. In addition, the manipulation system can detect the position of the tubular device satisfactorily by performing edge detection and pattern matching based on the first binary image.

As a desirable mode, the controller includes a storage unit that stores a reference binary image of the tubular device, and the position detection unit includes a detection portion formed from the dark portion of the tubular device in the first binary image, The tubular device is detected by comparing with the reference binary image. According to this, the manipulation system can suppress the influence of the minute object imaged in the first image and the liquid layer inside the tubular instrument, and can detect the position of the tubular instrument in a favorable manner.

As a desirable mode, the controller includes a threshold value setting unit that sets the threshold value based on the brightness of the first image. According to this, even when the brightness in the visual field of the first microscope changes due to the measurement environment or the like, the manipulation system can favorably detect the bright part and the dark part of the tubular instrument. That is, the manipulation system can improve robustness against changes in the environment.

As a desirable mode, the image processing unit converts the first image into a grayscale image displayed only with luminance, and creates the first binary image based on the grayscale image. According to this, the load of the image processing performed by the image processing unit can be suppressed as compared with the case of using a color image.

As a desirable mode, the image processing unit enlarges a detection portion formed from the dark portion of the tubular device in the first binary image, and arranges the region corresponding to the space inside the tubular device so as to be sandwiched therebetween. The two said dark parts which were made are connected to one. According to this, the manipulation system can suppress erroneous detection due to the difference in the position of the boundary between the liquid layer and the air layer inside the tubular device.

As a desirable mode, the image processing unit removes the dark portion indicating the minute object from the first binary image. According to this, the manipulation system can suppress erroneous detection of the tubular instrument due to the presence of the minute object.

As a desirable aspect, the image processing unit creates an image cut out in the partial region surrounding the detected tubular instrument from the first binary image, and the position detection unit from the image cut out in the partial region, Based on the difference in brightness between the detection portion formed of the dark portion of the tubular device and the background, the outer shape of the portion of the tubular device formed along the extending direction of the detection portion formed of the dark portion is detected. .. According to this, the manipulation system can accurately detect the outer shape of the tubular instrument along the extending direction.

As a desirable aspect, the image processing unit includes, from the first image, an outer shape of a portion along the extension direction of a detection portion formed from the dark portion of the tubular instrument, and an end of the tubular instrument in the extension direction. A region surrounded by the outer periphery of the partial region of the portion adjacent to the section, to create a tubular instrument image extracted from the tubular instrument, the position detection unit, based on the tubular instrument image, the tubular The position of the end of the device in the extending direction is detected. According to this, the manipulation system can accurately detect the position of the end of the tubular instrument in the extending direction.

As a desirable mode, the image processing unit creates a tubular instrument binary image that displays a bright portion having a brightness equal to or higher than a threshold value based on the tubular instrument image, and the position detection unit causes the tubular instrument binary image. Based on the above, the air layer and the liquid layer inside the tubular device are detected. According to this, the manipulation system can detect whether an arbitrary region in the tubular device is an air layer or a liquid layer.

As a desirable aspect, the image processing unit, from the first image, by cutting an image of a portion where the microscopic object does not exist inside the tubular instrument, to create a reference tubular instrument image, the position detection unit, The minute object inside the tubular instrument is detected based on an image showing a difference in brightness between the detected image of the tubular instrument and the reference tubular instrument image. According to this, the manipulation system can reliably detect that the minute object is collected in the tubular instrument.

As a desirable mode, the manipulator further includes a drive device that relatively moves the tubular instrument with respect to the sample stage, and the controller includes the micro object detected by the position detection unit and the tubular object. A distance detection unit that automatically detects a separation distance from an instrument, wherein the controller operates the drive device so that the distal end of the tubular instrument detects the minute target object detected by the position detection unit. The position detecting unit detects the position of the tubular device a plurality of times when the distance is moved closer to the position and the separation distance is larger than a preset value. According to this, the manipulation system can correct the position of the end of the tubular instrument to an appropriate position for collecting the minute object by repeatedly detecting the position of the tubular instrument.

In a preferred embodiment, the inside of the tubular device is surface-treated. According to this, the manipulation system can suppress the adhesion of the minute object to the inside of the tubular device.

According to the present invention, it is possible to provide a manipulation system capable of satisfactorily detecting the position of a tubular instrument.

FIG. 1 is a perspective view showing a configuration example of a manipulation system according to the first embodiment. FIG. 2 is an enlarged perspective view showing a part of the manipulation system shown in FIG. FIG. 3 is a schematic diagram showing a configuration example of the manipulation system according to the first embodiment. FIG. 4 is a block diagram showing a configuration example of the manipulation system according to the first embodiment. FIG. 5 is a block diagram showing a configuration example of the detection unit. FIG. 6 is a block diagram showing a configuration example of the storage unit. FIG. 7 is a diagram showing an example of the screen of the display unit. FIG. 8 is a side view showing a configuration example of the sampling pipette according to the first embodiment. FIG. 9 is an enlarged view of the tip of the sampling pipette. FIG. 10 is a flowchart showing an example of a detection sequence for detecting the sampling pipette of the manipulation system according to the first embodiment. FIG. 11 is a schematic diagram showing an example of a grayscale image in which the first image is displayed only with luminance. FIG. 12 is a graph schematically showing the distribution of brightness of cells and a pipette for collection. FIG. 13 is a schematic diagram showing an example of a binary image. FIG. 14 is a schematic diagram showing an enlarged dark portion of cells in the area A2 shown in FIG. FIG. 15 is a schematic diagram for explaining the process of complementing the shape of the dark part of the cell shown in FIG. FIG. 16 is a schematic diagram showing a binary image that has been subjected to a process of complementing the shape of a dark part. FIG. 17 is a schematic diagram showing a binary image in which dark areas of cells are removed. FIG. 18 is a schematic diagram showing a binary image in which the detection portion of the sampling pipette is enlarged. FIG. 19 is a schematic diagram showing an example of the reference binary image. FIG. 20 is a schematic diagram showing an example of the first image displaying the detection result of the sampling pipette. FIG. 21 is a flow chart showing an example of a detection sequence for detecting the shape of the sampling pipette in the manipulation system according to the second embodiment. FIG. 22 is a schematic diagram showing an example of a grayscale image in which the partial area shown in FIG. 20 is cut out and displayed only with luminance. FIG. 23 is a schematic diagram showing a binary image of a partial area. FIG. 24 is a schematic diagram showing a binary image obtained by performing a process of complementing the shape of a dark portion in the binary image of the partial region. FIG. 25 is a schematic diagram showing a binary image in which a dark part of a cell is removed from the binary image of the partial region. FIG. 26 is a schematic diagram showing an enlarged binary image of the detection portion of the sampling pipette in the binary image of the partial region. FIG. 27 is an explanatory diagram for explaining a method of detecting the edge of the detection portion of the sampling pipette. FIG. 28 is a flowchart showing an example of a detection sequence for detecting the end portion of the sampling pipette in the manipulation system according to the third embodiment. FIG. 29 is an explanatory diagram for explaining the position of the end portion of the detection portion in the Y-axis direction. FIG. 30 is a schematic diagram showing an example of a pipette image obtained by cutting the sampling pipette from the first image. FIG. 31 is an explanatory diagram for explaining a method of detecting the end portion of the sampling pipette. FIG. 32 is a flowchart showing an example of a detection sequence for detecting the inside of the sampling pipette of the manipulation system according to the fourth embodiment. FIG. 33 is a schematic diagram showing an example of a grayscale image in which a pipette image is displayed only with brightness. FIG. 34 is a pipette binary image showing the bright part of the collection pipette. FIG. 35 is a flowchart showing an example of a detection sequence for detecting cells inside a sampling pipette in the manipulation system according to the fifth embodiment. FIG. 36 is a schematic diagram showing an example of a reference image created by cutting out an image of a portion where cells do not exist inside the sampling pipette. FIG. 37 is a schematic diagram showing an example of an image obtained by cutting out a part of the detected pipette image. FIG. 38 is a schematic diagram showing an example of the difference image between the detected pipette image and the reference image. FIG. 39 is a schematic diagram showing an example of a binary image showing the bright portion of the difference image. FIG. 40 is a flowchart showing an example of the operation sequence of the manipulation system according to the sixth embodiment. FIG. 41 is a schematic diagram schematically showing a first image according to the sixth embodiment.

Hereinafter, modes for carrying out the invention (hereinafter, referred to as embodiments) will be described in detail with reference to the drawings. The present invention is not limited to the embodiments described below. The constituent elements in the following embodiments include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the constituent elements disclosed in the following embodiments can be combined appropriately.

(First embodiment)
FIG. 1 is a perspective view showing a configuration example of a manipulation system according to the first embodiment. FIG. 2 is an enlarged perspective view showing a part of the manipulation system shown in FIG. FIG. 3 is a schematic diagram showing a configuration example of the manipulation system according to the first embodiment. The manipulation system 100 shown in FIG. 1 to FIG. 3 is an apparatus that sorts out desired microscopic objects one by one from among a plurality of microscopic objects contained in a container 38. The minute object is, for example, a cell.

As shown in FIGS. 1 to 3, the manipulation system 100 includes a base 1, a sampling pipette 10, a pipette holding unit 15, a manipulator 20, a sample stage 30, and a first imaging device 45. The microscope unit 40, the controller 50, the second microscope unit 60 having the second imaging device 65, the third imaging device 75, the joystick 57, the input unit 58, and the display unit 80 are provided. In this embodiment, one direction parallel to the mounting surface 30a of the sample stage 30 is the X-axis direction. The direction parallel to the mounting surface 30a and orthogonal to the X-axis direction is defined as the Y-axis direction. The normal line direction of the mounting surface 30a is the Z-axis direction. For example, the arrangement of the base 1 is adjusted so that the mounting surface 30a is a horizontal plane orthogonal to the vertical direction.

The collection pipette 10 is a tubular instrument for collecting cells. For example, the sampling pipette 10 is needle-shaped, and its material is, for example, glass. The tip of the collection pipette 10 is provided with an opening for collecting cells. Details of the sampling pipette 10 will be described later with reference to FIGS. 8 and 9.

The pipette holding portion 15 is a tubular instrument for holding the sampling pipette 10. The material of the pipette holding portion 15 is, for example, glass or metal. One end of the pipette holding portion 15 is connected to the sampling pipette 10. Further, the other end of the pipette holding section 15 is connected to an electric micro pump 29 included in the manipulator 20. The internal pressure of the pipette holding unit 15 and the sampling pipette 10 is reduced or increased by the pressure P supplied from the electric micropump 29. When the internal pressure of the collection pipette 10 is lower than the atmospheric pressure, the collection pipette 10 can suction and collect cells from the opening at the tip. Further, when the internal pressure of the collection pipette 10 is higher than the atmospheric pressure, the collection pipette 10 can discharge (release) the collected cells to the outside from the opening portion at the tip of the collection pipette 10. The pipette holding portion 15 is connected to the manipulator 20 via a connecting portion 28 described below.

The manipulator 20 is a device for moving the pipette holding unit 15 and the sampling pipette 10 in the X axis direction, the Y axis direction, and the Z axis direction.

As shown in FIG. 3, the manipulator 20 includes an X-axis table 21, a Y-axis table 22, a Z-axis table 23, driving devices 26 and 27 (for example, a motor), and connecting portions 28 and 71 (see FIG. 1 ). ) And an electric micro pump 29. The X-axis table 21 moves in the X-axis direction when driven by the drive device 26. The Y-axis table 22 moves in the Y-axis direction when driven by the drive device 26. The Z-axis table 23 moves in the Z-axis direction when driven by the drive device 27. The drive devices 26 and 27 and the electric micropump 29 are connected to the controller 50.

In the manipulator 20, the Z-axis table 23 is mounted on the Y-axis table 22. As a result, the pipette holding unit 15 and the sampling pipette 10 can move in the Y-axis direction by the same distance as the Y-axis table 22 as the Y-axis table 22 moves. Further, the Y-axis table 22 is mounted on the X-axis table 21. As a result, the pipette holding unit 15 and the sampling pipette 10 can move in the X-axis direction by the same distance as the X-axis table 21 as the X-axis table 21 moves. Further, the pipette holding unit 15 and the sampling pipette 10 can move in the Z-axis direction by the same distance as the Z-axis table 23 as the Z-axis table 23 moves.

As shown in FIG. 3, the sample stage 30 supports the container 38. For example, the container 38 is mounted on the mounting surface 30a of the sample stage 30. The container 38 is, for example, a dish or a well plate. The sample stage 30 includes an X-axis stage 31, a Y-axis stage 32, and a driving device 36. The X-axis stage 31 moves in the X-axis direction when driven by the drive device 36. The Y-axis stage 32 moves in the Y-axis direction when driven by the driving device 36. The X-axis stage 31 is mounted on the Y-axis stage 32. The drive device 36 is connected to the controller 50.

Although FIG. 3 shows a case where the shape of the sample stage 30 in plan view (hereinafter, planar shape) is circular, the planar shape of the sample stage 30 is not limited to circular, and may be rectangular, for example. Further, although FIG. 3 shows the case where the planar shape of the container 38 is circular, the planar shape of the container 38 is not limited to circular. As shown in FIG. 1, the planar shape of the container 38 may be rectangular, for example. 1 and 3 show the case where one container 38 is placed on the sample stage 30, the number of containers 38 placed on the sample stage 30 is limited to one. However, it may be plural.

The first microscope unit 40 is arranged above the sample stage 30. The first microscope unit 40 includes a first microscope 41, a first imaging device 45, and a light source (not shown) that emits light toward the mounting surface 30 a of the sample stage 30. As shown in FIG. 2, the first microscope 41 has a lens barrel 411, an objective lens 412, and a drive device 414 (see FIG. 3). The first microscope 41 is a stereoscopic microscope in which the objective lens 412 is located above the container 38. The lens barrel 411 of the first microscope unit 40 moves in the Z-axis direction when driven by the driving device 414. Thereby, the first microscope 41 can adjust the focus position. A plurality of types of objective lenses 412 may be prepared according to a desired magnification. The first imaging device 45 can image the tip of the sampling pipette 10 from the Z-axis direction via the first microscope 41. The first microscope unit 40 may include an eyepiece lens (not shown).

The connecting portion 28 shown in FIG. 1 connects the pipette holding portion 15 to the manipulator 20. The lens barrel 411 of the first microscope unit 40 is connected to the manipulator 20 by a connecting portion 71. As a result, when the lens barrel 411 moves in the Z-axis direction, the manipulator 20, the pipette holding unit 15, and the sampling pipette 10 also move in the Z-axis direction together with this. The connecting portions 28 and 71 are made of metal, for example. The connecting portions 28 and 71 are attached to the Z-axis table 23, for example.

The second microscope unit 60 is arranged laterally of the sample stage 30. The second microscope unit 60 has a second microscope 61 and a second imaging device 65. As shown in FIG. 2, the second microscope 61 has a lens barrel 611, an objective lens 612, and a drive device 613 (see FIG. 1). The objective lens 612 moves in the Y-axis direction when driven by the driving device 613. Thereby, the second microscope 61 can adjust the focus position. The second imaging device 65 can image the tip of the sampling pipette 10 from the Y-axis direction via the second microscope 61. The second microscope unit 60 is fixed to the base 1 via the fixture 3. The first imaging device 45 and the second imaging device 65 have solid-state imaging elements such as a CMOS image sensor or a CCD image sensor, for example.

The third imaging device 75 is fixed to the base 1 via the fixture 4. The fixture 4 can move in the X-axis direction and the Y-axis direction, and can extend in the Z-axis direction. Accordingly, the third imaging device 75 can image the sample stage 30 side from the obliquely upper direction of the sample stage 30 that intersects the X-axis direction, the Y-axis direction, and the Z-axis direction, respectively.

The input unit 58 shown in FIG. 3 is a keyboard, a touch panel, or the like. The joystick 57 and the input unit 58 are connected to the controller 50. The operator can input a command to the controller 50 via the joystick 57 and the input unit 58.

Next, the function of the controller 50 will be described with reference to FIG. FIG. 4 is a block diagram showing a configuration example of the manipulation system according to the first embodiment. The controller 50 includes hardware resources such as a CPU (central processing unit) as a calculation unit and a hard disk as a storage unit, a RAM (Random Access Memory), and a ROM (Read Only Memory).

As shown in FIG. 4, the controller 50 has an image input unit 51a, an image output unit 51b, an image processing unit 52, a detection unit 53, an image editing unit 54, a control unit 55, and a storage unit 56 as its functions. The image input unit 51a, the image output unit 51b, the image processing unit 52, the detection unit 53, the image editing unit 54, and the control unit 55 are realized by the above calculation means. The storage unit 56 is realized by the storage unit described above. The controller 50 performs various calculations based on the programs stored in the storage unit 56, and outputs drive signals so that the control unit 55 performs various controls according to the calculation results.

The control unit 55 includes a drive unit 414 of the first microscope unit 40, drive units 26 and 27 of the manipulator 20 and an electric micropump 29, a drive unit 36 of the sample stage 30, and a drive unit 613 of the second microscope unit 60. To control. The control unit 55 supplies drive signals Vz1, Vxy2, Vz2, Vxy3, Vy4 (see FIG. 3) to the drive devices 414, 26, 27, 36, 613, respectively. The control unit 55 also supplies a drive signal Vmp (see FIG. 3) to the electric micropump 29. The control unit 55 may supply the drive signals Vz1, Vxy2, Vz2, Vxy3, Vy4, and Vmp via drivers and amplifiers provided as necessary.

The first image signal Vpix1 (see FIG. 3) output from the first imaging device 45, the second image signal Vpix2 (see FIG. 3) output from the second imaging device 65, and the third imaging device 75 are output. The third image signal Vpix3 (see FIG. 3) is input to the image input unit 51a. The image processing unit 52 receives the first image signal Vpix1, the second image signal Vpix2, and the third image signal Vpix3 from the image input unit 51a, and performs image processing. The image output unit 51b outputs the image information image-processed by the image processing unit 52 to the storage unit 56 and the display unit 80.

For example, the first image signal Vpix1 includes the first image 811 (see FIG. 7) captured by the first image capturing device 45 through the first microscope 41 and the capturing time thereof. The first image 811 is a moving image. Similarly, the second image signal Vpix2 includes the second image 812 (see FIG. 7) captured by the second image capturing device 65 through the second microscope 61 and the capturing time thereof. The second image 812 is also a moving image. The third image signal Vpix3 includes the third image 813 (see FIG. 7) captured by the third image capturing device 75 and the capturing time thereof. The third image 813 is also a moving image.

The first image 811, the second image 812, and the third image 813 are color images or gray images, respectively. The gray image is an image including white and black and gray which is an intermediate color between white and black. The gray image has a plurality of gray levels in gray. The gradation means the number of gradations of color and brightness.

The image processing unit 52 performs image processing such as image enlargement or binarization on at least one of the first image 811 and the second image 812 in order to facilitate detection of cells. The binarization of an image is to convert a color image or a gray image (hereinafter referred to as an original image) into a binary image (binary image) that has no shading and has only white and black. The image processing unit 52 creates, for at least one of the first image 811 and the second image 812, an enlarged image obtained by enlarging the original image and a binary image obtained by binarizing the original image. The image processing unit 52 may enlarge the original image and binarize it to create an enlarged binary image. The image processing unit 52 outputs at least one kind of the enlarged image, the binary image, and the enlarged binary image to the detecting unit 53 and the image editing unit 54 as image information.

The detection unit 53 receives the image information from the image processing unit 52, and automatically detects the position and number of cells based on the received image information. Then, the detection unit 53 outputs the detection result to the image editing unit 54 and the control unit 55. In the present disclosure, “automatic” means that the device operates without the operator's judgment.

FIG. 5 is a block diagram showing a configuration example of the detection unit. As shown in FIG. 5, the detection unit 53 has, as its functions, a position detection unit 531, a distance detection unit 532, a number detection unit 533, a brightness detection unit 534, and a threshold value setting unit 535. The position detection unit 531 automatically detects the position of the cell ce (see FIG. 7) based on the first image 811 or the second image 812 image-processed by the image processing unit 52. In addition, the position detection unit 531 detects the position of the tip portion 103 of the sampling pipette 10 based on the first image 811 or the second image 812 image-processed by the image processing unit 52.

The distance detection unit 532 automatically detects the separation distance between the cell ce detected by the position detection unit 531 and the opening 103a (see FIG. 9 described later) of the tip 103. The number detection unit 533 automatically detects the number of cells ce based on the first image 811 or the second image 812 image-processed by the image processing unit 52. Examples of the image processed by the image processing unit 52 include at least one kind of an enlarged image, a binary image, and an enlarged binary image. The detection results of the position detection unit 531, the distance detection unit 532, and the number detection unit 533 are output to the image editing unit 54 and the control unit 55, respectively.

The brightness detection unit 534 detects the brightness L-BG of the entire first image 811 based on the first image 811 image-processed by the image processing unit 52. The brightness L-BG is an average value of brightness in the entire first image 811. The threshold setting unit 535 sets the threshold L-TH based on the brightness L-BG detected by the brightness detecting unit 534 (see FIG. 12). The threshold value L-TH is a reference value of luminance when the image processing unit 52 creates a binary image (binary image 811A (see FIG. 13)). The brightness L-BG and the threshold L-TH are automatically set by, for example, the Otsu binarization process.

The image editing unit 54 creates the edited image signal Vpix4 by associating the first image signal Vpix1, the second image signal Vpix2, and the third image signal Vpix3 with each other based on the image capturing time. The edited image signal Vpix4 includes an edited image. The edited image is a moving image in which the first image 811, the second image 812, and the third image 813 captured at the same time are displayed side by side. In the edited image, each of the first image 811, the second image 812, and the third image 813 may be an original image, or may be an enlarged image, a binary image, or an enlarged binary image obtained by performing image processing on the original image. The image output unit 51b outputs the first image signal Vpix1, the second image signal Vpix2, the third image signal Vpix3, and the edited image signal Vpix4 to the storage unit 56.

The image editing unit 54 also receives the detection result of the cell ce from the detection unit 53. When the detection unit 53 detects the position of the cell ce, the image editing unit 54 may reflect the detection result on the edited image. For example, the image editing unit 54 may automatically indicate the position of the cell ce detected by the detecting unit 53 with an arrow in the enlarged image, the binary image, or the enlarged binary image received from the image processing unit 52, or the detecting unit 53 may The detected cell position may be automatically surrounded by a frame line.

FIG. 6 is a block diagram showing a configuration example of the storage unit. As shown in FIG. 6, the storage unit 56 has, as its functions, a program storage unit 56a that stores a program for operating the manipulation system 100 and an image storage unit 56b that stores an image signal. The image storage unit 56b includes a first image storage unit 561 that stores the first image signal Vpix1, a second image storage unit 562 that stores the second image signal Vpix2, and a third image storage that stores the third image signal Vpix3. It includes a unit 563, an edited image storage unit 564 that stores the edited image signal Vpix4, and a reference image storage unit 565 that stores the reference binary image 103r1 (see FIG. 19). The first image storage unit 561 may temporarily store at least one kind of an enlarged image, a binary image, and an enlarged binary image created by the image processing unit 52.

The image output unit 51b outputs at least one image signal of the first image signal Vpix1, the second image signal Vpix2, the third image signal Vpix3, and the edited image signal Vpix4 to the display unit 80.

The display unit 80 is, for example, a liquid crystal panel or the like. The display unit 80 is connected to the controller 50. The display unit 80 displays various character information, images, etc. on the screen. FIG. 7 is a diagram showing an example of the screen of the display unit. FIG. 7 illustrates a case where an edited image is displayed on the screen 81 of the display unit 80. In the edited image, the first image 811, the second image 812, and the third image 813 captured at the same timing are arranged side by side. The display unit 80 may display the edited image in real time or almost in real time, or may read the edited image stored in the edited image storage unit 564 and display it for reproduction.

The operator can operate the joystick 57 or the input section 58 to switch the image displayed on the screen 81. Further, the operator can operate the joystick 57 or the input unit 58 to temporarily stop the edited image (moving image) displayed on the screen 81. Further, based on the program stored in the program storage unit 56a (see FIG. 6), the control unit 55 (see FIG. 4) automatically displays a predetermined image on the screen 81 or an image displayed on the screen 81. May be switched automatically.

Further, the image displayed on the screen 81 is not limited to the edited image and may be only the first image 811, the second image 812 or the third image 813. Further, the image displayed on the screen is not limited to the original image, and the image processed by the image processing unit 52 (for example, at least one of an enlarged image, a binary image, and an enlarged binary image). May be For example, the image displayed on the screen 81 may be the grayscale image 811G shown in FIG. 11, the binary image 811A shown in FIG. 13 or the like, the binary image 814A of the partial area BB shown in FIG. 23, or the like.

FIG. 8 is a side view showing a configuration example of the sampling pipette according to the first embodiment. As shown in FIG. 8, the sampling pipette 10 is a glass needle that is bent in two stages. Specifically, the collecting pipette 10 is, in plan view, a tubular central portion 101, a tubular rear portion 102 connected to one end of the central portion 101, and a tubular tip portion connected to the other end of the central portion 101. And 103. The rear part 102 is a part on the side held by the pipette holding part 15. The tip portion 103 is a portion on the side where a minute object such as a cell is collected.

A first bent portion 104 exists between the central portion 101 and the rear portion 102. A second bent portion 105 exists between the central portion 101 and the tip portion 103. The longitudinal direction of the central portion 101 and the longitudinal direction of the rear portion 102 intersect each other. The longitudinal direction of the central portion 101 and the longitudinal direction of the tip portion 103 also intersect each other. The longitudinal direction of the rear part 102 and the longitudinal direction of the tip part 103 are parallel or substantially parallel to each other. For example, an obtuse angle formed by the longitudinal direction of the central portion 101 and the longitudinal direction of the rear portion 102 (hereinafter, the bending angle of the first bending portion 104) is θ1. An obtuse angle formed by the longitudinal direction of the central portion 101 and the longitudinal direction of the tip portion 103 (hereinafter, the bending angle of the second bending portion 105) is θ2. The absolute value |θ1-θ2| of the difference between the bending angle θ1 of the first bending portion 104 and the bending angle θ2 of the second bending portion 105 is 0° or more and less than 5°.

When the length of the central portion 101 in the longitudinal direction is L1, the length of the rear portion 102 in the longitudinal direction is L2, and the length of the tip portion 103 in the longitudinal direction is L3, L3<L1 and L3 <L2. According to this, it is easy to dispose the tip portion 103, which is the tip portion of the sampling pipette 10, in the container 38.

The shape obtained by cutting the central portion 101 along a plane orthogonal to the longitudinal direction of the central portion 101 is circular. Similarly, the shape obtained by cutting the rear portion 102 along a plane orthogonal to the longitudinal direction of the rear portion 102 is circular. The shape obtained by cutting the tip portion 103 along a plane orthogonal to the longitudinal direction of the tip portion 103 is circular. When the outer diameter of the rear portion 102 is φ21 and the outer diameter of the tip portion 103 is φ31, φ21>φ31. In addition, the outer diameter φ11 of the central portion 101 decreases from the first bent portion 104 side toward the second bent portion 105 side.

For example, the central portion 101 has a narrowed portion 106 having a large change in outer diameter between the first bent portion 104 and the second bent portion 105. In the central portion 101, the second portion 101b located between the narrowed portion 106 and the second bent portion 105 is smaller than the first portion 101a located between the narrowed portion 106 and the first bent portion 104. The outer diameter φ11 is small. The length L1' in the longitudinal direction of the second portion 101b is longer than the length L3 in the longitudinal direction of the tip portion 103.

FIG. 9 is an enlarged view showing the tip of the sampling pipette according to the first embodiment. As shown in FIG. 9, an opening 103a is provided at the tip of the tip 103. Between the opening 103a and the second bent portion 105, the size of the inner diameter φ32 of the tip portion 103 is substantially constant. When the diameter of the cell ce to be collected by the collection pipette 10 is φce, φ32 is preferably larger than φce by about several μm. As a result, the collection pipette 10 can introduce the cells ce into the tip portion 103.

The inside of the sampling pipette 10 is surface-treated. For example, the inner peripheral surface of the sampling pipette 10 is coated with polyvinylpyrrolidone (hereinafter, PVP (Polyvinylpyrrolidone)). When coating the inner peripheral surface of the sampling pipette 10, for example, the sampling pipette 10 is immersed in the PVP solution, and suction and discharge are repeated a plurality of times. Then, rinsing is performed by repeating suction and discharge a plurality of times in the liquid 39 (see FIG. 7). As a result, the inner peripheral surface of the sampling pipette 10 is coated with the PVP solution. As a result, the manipulation system 100 can prevent the cells ce from adhering to the inside of the sampling pipette 10 when the cells ce are collected.

Next, the detection operation of the sampling pipette 10 of the manipulation system 100 will be described. FIG. 10 is a flowchart showing an example of a detection sequence for detecting the sampling pipette of the manipulation system according to the first embodiment.

In the operation sequence shown in FIG. 10, the container 38 contains the liquid 39 and a plurality of cells ce (see FIG. 7). As shown in FIG. 10, the manipulation system 100 automatically focuses the first microscope 41 on the tip 103 of the sampling pipette 10 (step ST11). For example, the operator operates the joystick 57 or the input unit 58 (see FIG. 3) to instruct the controller 50 (see FIG. 3) to automatically detect the cells ce.

In response to this instruction, the controller 55 (see FIG. 3) of the controller 50 outputs the drive signal Vz1 (see FIG. 3) to the drive device 414 to move the objective lens 412 in the Z-axis direction. As a result, the first microscope unit 40 focuses the first microscope 41 on the tip portion 103 of the sampling pipette 10.

Next, the first imaging device 45 captures the first image 811 via the first microscope 41. The image processing unit 52 receives the first image signal Vpix1 that is the information of the first image 811, and converts the first image signal Vpix1 into an image (grayscale image 811G) in which only the brightness parameter is extracted from the first image 811 (step ST12). Thereby, the load of the image processing performed by the image processing unit 52 can be suppressed, and the binarization processing, the enlargement processing, and the like can be favorably performed.

FIG. 11 is a schematic diagram showing an example of a grayscale image in which the first image is displayed only with luminance. As shown in FIG. 11, when the focus of the first microscope 41 is on the sampling pipette 10, the tip 103 of the sampling pipette 10 is located inside the container 38. A large number of cells ce and the tip portion 103 of the sampling pipette 10 are imaged in the grayscale image 811G. The tip portion 103 has a liquid layer 109 in which liquid is present and an air layer 108 in which liquid is not present.

Next, the brightness detection unit 534 and the threshold setting unit 535 (see FIG. 5) set the brightness threshold L-TH (step ST13).

The brightness detection unit 534 detects the brightness L-BG (see FIG. 12) of the entire grayscale image 811G. The brightness L-BG is an average value of the brightness of the entire gray scale image 811G. The threshold setting unit 535 receives the information of the brightness L-BG detected by the brightness detection unit 534 and sets the threshold L-TH. The brightness L-BG and the threshold L-TH are automatically set by, for example, the Otsu binarization process.

FIG. 12 is a graph schematically showing the distribution of brightness of cells and a pipette for collection. FIG. 12 schematically shows the luminance distribution along XII-XII′ shown in FIG. As shown in FIG. 12, the threshold setting unit 535 sets the brightness L-BG of the entire gray scale image 811G as the threshold L-TH. However, the present invention is not limited to this, and the threshold value L-TH may be a brightness different from the background brightness L-BG. In this way, the threshold setting unit 535 automatically sets the threshold L-TH, so that even if the brightness in the field of view of the first microscope 41 changes due to the measurement environment or the like, the manipulation system 100 can: The dark portion T-a and the light portion T-b of the sampling pipette 10 can be satisfactorily detected. That is, the manipulation system 100 can improve robustness against changes in the environment.

The cell ce observed by the first microscope 41 has a shadow depending on the state of the light emitted from the light source. Therefore, one cell ce has a dark portion ce-a and a light portion ce-b. The dark portion ce-a has a luminance smaller than the background luminance L-BG. The bright portion ce-b has a luminance higher than the background luminance L-BG. Further, as shown in FIG. 11, the tip portion 103 of the sampling pipette 10 has a bright portion T-b corresponding to the air layer 108 and a dark portion T-a corresponding to the glass tube portion sandwiching the air layer 108. Have. The bright portion T-b has a higher luminance than the dark portion T-a, and the dark portion T-a has a lower luminance than the threshold value L-TH (luminance L-BG).

Next, the image processing unit 52 receives the information of the threshold value L-TH and performs image processing of the grayscale image 811G. Specifically, the image processing unit 52 creates a binary image 811A by extracting a portion having a luminance equal to or lower than the threshold value L-TH from the gray scale image 811G (step ST14). FIG. 13 is a schematic diagram showing an example of a binary image. As shown in FIG. 13, in the binary image 811A, in the cell ce, the dark portion ce-a having the luminance equal to or lower than the threshold L-TH is displayed in white, and the light having the luminance higher than the threshold L-TH is displayed. The part ce-b and the background are displayed in black. Further, at the tip portion 103 of the sampling pipette 10, the dark portion T-a having the brightness equal to or lower than the threshold value L-TH is displayed in white, and the bright portion T-b of the air layer 108 and the liquid layer 109 is displayed in black. .. That is, the binary image 811A is created as a reverse image.

In the following description, the portion formed by the two dark portions Ta of the sampling pipette 10 will be referred to as the detection portion 103g. The one dark portion T-a and the other dark portion T-a are arranged so as to sandwich the bright portion T-b corresponding to the air layer 108 and the liquid layer 109.

Next, the image processing unit 52 complements the shape of the dark portion ce-a of the cell ce (step ST15). FIG. 14 is a schematic diagram showing an enlarged dark portion of cells in the area A2 shown in FIG. FIG. 15 is a schematic diagram for explaining the process of complementing the shape of the dark part of the cell shown in FIG. FIG. 16 is a schematic diagram showing a binary image that has been subjected to a process of complementing the shape of a dark part. Specifically, as shown in FIG. 14, the image processing unit 52 extracts, from the dark portions ce-a of the plurality of cells ce, the dark portion ce-a having the concave portion ce-x on the outer periphery. The determination of the dark portion ce-a having the concave portion ce-x and the dark portion ce-a having no concave portion ce-x can be made based on the Haywood circular factor.

As illustrated in FIG. 15, the image processing unit 52 complements the shape of the dark portion ce-a so as to fill the concave portion ce-x of the dark portion ce-a. As a result, the outer diameter shape of the plurality of dark portions ce-a becomes a smooth shape having no concave portions ce-x. Although FIG. 15 shows three dark portions ce-a, as shown in FIG. 16, the image processing unit 52 forms the concave portions ce-x with respect to all the dark portions ce-a included in the binary image 811A. The shape of the dark portion ce-a is complemented so as to be filled.

Next, the image processing unit 52 removes the dark portion ce-a of the cell ce from the binary image 811A (step ST16). FIG. 17 is a schematic diagram showing a binary image in which dark areas of cells are removed. As illustrated in FIG. 17, the image processing unit 52 removes a small round dark portion ce-a (see FIG. 16) indicating the cell ce from the binary image 811A. As a result, only the detection portion 103g of the sampling pipette 10 is displayed on the binary image 811A. Note that, in FIG. 17, the dark portion shown between the two dark portions T-a corresponds to the interface 109 a between the air layer 108 and the liquid layer 109.

The image processing unit 52 removes the small dark portion ce-a from the binary image 811A, so that the dark portion ce-a detected overlapping with the sampling pipette 10 is also removed. Thereby, the manipulation system 100 can suppress erroneous detection of the cells ce overlapping the sampling pipette 10 or the cells ce adjacent to the sampling pipette 10 as a part of the sampling pipette 10. Therefore, the manipulation system 100 can suppress the occurrence of an error in the position or shape of the sampling pipette 10 due to the presence of the cells ce.

Further, since the image processing unit 52 performs a process of complementing the shape of the dark portion ce-a in advance, the variation in the shape of the dark portion ce-a becomes small. Therefore, the image processing unit 52 can reliably extract and remove the dark portion ce-a from the binary image 811A.

Next, the image processing unit 52 enlarges the detection portion 103g of the sampling pipette 10 in the binary image 811A (step ST17). FIG. 18 is a schematic diagram showing a binary image in which the detection portion of the sampling pipette is enlarged. As illustrated in FIG. 18, the image processing unit 52 magnifies the plurality of dark portions Ta of the sampling pipette 10 in the binary image 811A. As a result, the two dark portions T-a arranged with the bright portion T-b corresponding to the air layer 108 and the liquid layer 109 of the sampling pipette 10 interposed therebetween are connected.

In the binary image 811A shown in FIG. 18, the interface 109a between the air layer 108 and the liquid layer 109 is removed from the detection portion 103g shown in FIG. Therefore, the manipulation system 100 can detect the sampling pipette 10 based on only the dark portion T-a wherever the position of the interface 109a inside the sampling pipette 10 is. Therefore, the manipulation system 100 can suppress erroneous detection of the sampling pipette 10 due to the difference in the position of the interface 109a.

The enlarged dark portion T-a is enlarged about 6 times in area ratio with respect to the dark portion T-a before the enlargement. In addition, the barycentric position of the enlarged dark portion T-a matches the barycentric position of the dark portion T-a before the enlargement. The magnification of the dark portion T-a can be appropriately set according to the shape and size of the sampling pipette 10 (glass tube thickness), the threshold value L-TH, or the observation environment.

Next, the position detection unit 531 receives the image information of the binary image 811A shown in FIG. 18, compares the image of the detection portion 103g with the reference binary image, and detects the position of the sampling pipette 10 (step ST18). FIG. 19 is a schematic diagram showing an example of the reference binary image. The reference binary image 103r1 is a binary image when the same image processing as that in steps ST11 to ST17 described above is performed on the standard sampling pipette 10. The reference binary image 103r1 is stored in the reference image storage unit 565 of the storage unit 56 in advance. Further, the storage unit 56 may hold a plurality of reference binary images 103r1 for each type of the sampling pipette 10.

The position detection unit 531 compares the detection portion 103g of the binary image 811A with the reference binary image 103r1 by, for example, template matching or shape matching. Then, the position detection unit 531 calculates the degree of similarity between the detection portion 103g and the reference binary image 103r1. The position detection unit 531 detects the position of the detection portion 103g whose degree of similarity is equal to or higher than a predetermined reference value.

FIG. 20 is a schematic diagram showing an example of the first image displaying the detection result of the sampling pipette. As shown in FIG. 20, the image editing unit 54 includes the sampling pipette 10 in the first image 811 based on the first image 811 received from the image processing unit 52 and the detection result received from the detection unit 53. The partial area BB is shown surrounded by a bounding box.

The position detection unit 531 receives the information about the bounding box from the image editing unit 54. When the bounding box is smaller than a predetermined size, for example, when one side of the bounding box is smaller than 20 pixels, the position detecting unit 531 controls the image processing unit 52 and the control unit to detect the detection result indicating that the sampling pipette 10 is in the non-detecting state. It is output to the unit 55. When the sampling pipette 10 is not detected, the image processing unit 52 ends the image processing based on the detection result.

The detection operation shown in FIGS. 10 and 11 to 20 can be changed as appropriate. For example, in step ST15 shown in FIG. 10 and FIGS. 14 to 16, the image processing unit 52 omits the process of complementing the shape of the dark portion ce-a of the cell ce, and the dark portion ce-a smaller than the predetermined area. May be removed from the binary image 811A.

As described above, the manipulation system 100 according to the first embodiment is a manipulation system 100 that collects cells ce (microscopic objects) using the collection pipette 10 (tubular instrument). The manipulator 20, the sample stage 30, the first microscope 41, the first imaging device 45, and the controller 50 are provided. The sampling pipette 10 is attached to the manipulator 20. A container 38 for containing the cells ce is placed on the sample stage 30. The first microscope 41 is arranged above the sample stage 30. The first imaging device 45 captures the first image 811 via the first microscope 41. The controller 50 has an image processing unit 52 and a position detection unit 531. The image processing unit 52 creates, based on the first image 811, a binary image 811A (first binary image) that displays the dark portion T-a having a luminance equal to or lower than the threshold value L-TH. The position detection unit 531 detects the position of the sampling pipette 10 based on the binary image 811A.

According to this, the manipulation system 100 can easily perform image processing such as removing the detection portion of the cells ce and the like other than the sampling pipette 10 based on the binary image 811A. Further, the manipulation system 100 can satisfactorily detect the position of the tubular instrument by performing edge detection and pattern matching based on the binary image 811A.

The controller 50 also includes a storage unit 56 that stores the reference binary image 103r1 of the sampling pipette 10. The position detection unit 531 detects the sampling pipette 10 by comparing the detection portion 103g formed from the dark portion Ta of the sampling pipette 10 in the binary image 811A with the reference binary image 103r1. According to this, the manipulation system 100 suppresses the influence of the cells ce imaged in the first image 811 and the liquid layer 109 inside the sampling pipette 10 and detects the position of the sampling pipette 10 well. be able to.

The controller 50 also has a threshold value setting unit 535. The threshold setting unit 535 sets the threshold L-TH based on the brightness L-BG of the entire first image 811. According to this, even when the brightness in the field of view of the first microscope 41 changes due to the measurement environment or the like, the manipulation system 100 can detect the dark portion Ta well. That is, the manipulation system 100 can improve robustness against changes in the environment.

In addition, the image processing unit 52 converts the first image 811 into a grayscale image 811G that is displayed only with luminance, and creates a binary image 811A based on the grayscale image 811G. According to this, the load of the image processing performed by the image processing unit 52 can be suppressed as compared with the case of using a color image.

In addition, the image processing unit 52 enlarges the detection portion 103g in the binary image 811A, thereby bringing together two dark portions T-a arranged across the region corresponding to the space inside the sampling pipette 10. Connect. According to this, the manipulation system 100 can suppress erroneous detection due to the difference in position of the interface 109a between the liquid layer 109 and the air layer 108 inside the sampling pipette 10.

The image processing unit 52 also removes the dark portion ce-a indicating the cell ce from the binary image 811A. According to this, the manipulation system 100 can suppress erroneous detection of the sampling pipette 10 due to the presence of the cells ce.

(Second embodiment)
FIG. 21 is a flow chart showing an example of a detection sequence for detecting the shape of the sampling pipette in the manipulation system according to the second embodiment. In the second embodiment, the image processing unit 52 creates a pipette image by cutting out the partial region BB including the sampling pipette 10 from the first image 811 shown in FIG. 20 (step ST21). In this case, the image processing unit 52 may set the area larger than the bounding box shown in FIG. 20 as the partial area BB. For example, the image processing unit 52 may expand the boundary box shown in FIG. 20 by 10 pixels in one direction in the Y-axis direction and expand it by 10 pixels in the other direction in the Y-axis direction as the partial area BB.

Next, the image processing unit 52 performs the same image processing on the clipped pipette image as in steps ST12 and ST14 to ST15 of the first embodiment. Therefore, detailed description of the same processing as the above-described image processing will be omitted.

The image processing unit 52 converts the extracted pipette image into an image (grayscale image 814G) in which only the brightness parameter is extracted (step ST22). FIG. 22 is a schematic diagram showing an example of a grayscale image in which the partial area shown in FIG. 20 is cut out and displayed only with luminance. As shown in FIG. 22, the grayscale image 814G includes the tip portion 103 of the sampling pipette 10. Note that in FIG. 22, the cells ce are not included, but when the cells ce are present overlapping or close to the tip 103, the grayscale image 814G shows the tip 103 of the sampling pipette 10. And a plurality of cells ce may be included.

Next, the image processing unit 52 creates a binary image 814A by extracting a portion having a luminance equal to or lower than the threshold value L-TH from the gray scale image 814G (step ST23). FIG. 23 is a schematic diagram showing a binary image of a partial area. As shown in FIG. 23, in the binary image 814A of the partial region, the detection portion 103g of the sampling pipette 10 is displayed in white.

Next, the image processing unit 52 complements the shape of the dark portion ce-a of the cell ce and removes the dark portion ce-a of the cell ce from the binary image 814A (step ST24). FIG. 24 is a schematic diagram showing a binary image obtained by performing a process of complementing the shape of a dark portion in the binary image of the partial region. FIG. 25 is a schematic diagram showing a binary image in which a dark part of a cell is removed from the binary image of the partial region. As shown in FIG. 24, the image processing unit 52 complements the shape of the dark portion ce-a located on the right side of the binary image 814A. As shown in FIG. 25, the image processing unit 52 removes the dark part ce-a from the binary image 814A. Accordingly, the image processing unit 52 creates the binary image 814A including only the detection portion 103g of the sampling pipette 10.

Next, the image processing unit 52 enlarges the detection portion 103g of the sampling pipette 10 in the binary image 814A (step ST25). FIG. 26 is a schematic diagram showing an enlarged binary image of the detection portion of the sampling pipette in the binary image of the partial region. As shown in FIG. 26, the detection portion 103g is configured by one dark portion T-a by connecting two dark portions Ta in the Y-axis direction.

Next, the position detection unit 531 detects the edge of the detection portion 103g based on the binary image 814A (step ST26). FIG. 27 is an explanatory diagram for explaining a method of detecting the edge of the detection portion of the sampling pipette. As shown in FIG. 27, the position detection unit 531 detects the edge 103e of the detection portion 103g along the D1 direction from the other side of the Y-axis direction to the one side, and also along the D2 direction opposite to the D1 direction. Then, the edge 103e is detected. The position detector 531 detects the edge 103e based on the difference in brightness in the Y-axis direction.

The position detection unit 531 detects the D1 direction and the D2 direction at a plurality of locations along the X-axis direction. Accordingly, the position detection unit 531 can detect the outer shape of the portion along the extending direction of the detection portion 103g based on the difference in brightness between the detection portion 103g and the background. The position detection unit 531 detects the edge 103e along the extending direction of the detection portion 103g by approximating it as two linear lines.

When the distance between the two edges 103ge in the Y-axis direction is greater than or equal to a predetermined value, for example, 15 based on the information of the detected two linear edges 103ge (see FIG. 29), the position detection unit 531 When the number of pixels is equal to or larger than the number of pixels, it is determined that the outer diameter shape of the sampling pipette 10 can be detected. On the other hand, the position detection unit 531 does not detect the outer diameter shape of the sampling pipette 10 when the distance between the two edges 103ge in the Y-axis direction is smaller than a predetermined size, for example, smaller than 15 pixels. To judge.

The position detection unit 531 detects the angle between the two edges 103ge, and when the difference in angle between the two edges 103ge is less than or equal to a predetermined angle, for example, 2° or less, the outer diameter shape of the sampling pipette 10 Is determined to have been detected. On the other hand, the position detection unit 531 determines that the outer diameter shape of the sampling pipette 10 is not detected when the difference in angle between the two edges 103ge is larger than a predetermined angle, for example, larger than 2°. To do. Further, the position detection unit 531 moves the two edges 103ge by 8 pixels, moving the upper edge 103ge downward in the Y-axis direction and the lower edge 103ge upward in the Y-axis direction. Thereby, the position detection unit 531 can approach the edge of the actual size of the sampling pipette 10 from the position of the edge 103ge of the enlarged dark portion T-a.

By the method as described above, in the manipulation system 100 of the second embodiment, the image processing unit 52 creates an image cut from the first image 811 in the partial region BB surrounding the detected sampling pipette 10, and the position The detection unit 531 detects the outer shape of the portion along the extending direction of the detection portion 103g based on the difference in brightness between the detection portion 103g and the background from the image cut out in the partial region BB. Thereby, the manipulation system 100 suppresses the detection error due to the cell ce located in the vicinity of the sampling pipette 10 and the interface 109a inside the sampling pipette 10, and the outer shape along the extending direction of the sampling pipette 10. The shape can be accurately detected.

(Third Embodiment)
FIG. 28 is a flowchart showing an example of a detection sequence for detecting the end portion of the sampling pipette in the manipulation system according to the third embodiment. The position detection unit 531 detects the position of the end portion 103f of the sampling pipette 10 in the Y-axis direction based on the binary image 814A of the partial region BB created in the second embodiment (step ST31).

FIG. 29 is an explanatory diagram for explaining the position of the end portion of the detection portion in the Y-axis direction. As shown in FIG. 29, the position detection unit 531 detects positions Y1 and Y2 where the outer periphery of the binary image 814A and the extension line of the edge 103ge on the end side (left side of FIG. 29) of the detection portion 103g intersect. .. Then, the position detection unit 531 calculates the midpoint between the positions Y1 and Y2 in the Y-axis direction as the Y-axis direction position 103y of the end portion 103f of the sampling pipette 10.

Next, the image processing unit 52 receives the information on the edge 103ge and creates a pipette image 814B cut from the first image 811 at the edge 103ge of the detection portion 103g (step ST32). FIG. 30 is a schematic diagram showing an example of a pipette image obtained by cutting the sampling pipette from the first image. The pipette image 814B is an image obtained by performing only the process of cutting the image without performing the image processing such as the binarization processing and the gray scale processing described above. As illustrated in FIG. 30, the image processing unit 52 cuts and creates the upper end and the lower end of the pipette image 814B along the edge 103ge. Further, the image processing unit 52 cuts and creates the left end of the pipette image 814B along the left end of the outer circumference of the partial region BB. The image processing unit 52 cuts and creates the right end of the pipette image 814B along a line connecting the right end points of the two edges 103ge. Accordingly, the pipette image 814B includes the end portion 103f of the sampling pipette 10 (tip portion 103).

Next, the position detection unit 531 receives the information of the pipette image 814B and detects the position of the end 103f of the sampling pipette 10 based on the pipette image 814B (step ST33). FIG. 31 is an explanatory diagram for explaining a method of detecting the end portion of the sampling pipette. As shown in FIG. 31, the position detection unit 531 detects the end portion 103f of the sampling pipette 10 along the D3 direction from the right end to the left end of the pipette image 814B. Accordingly, the position detection unit 531 can detect the position of the end portion 103f in the extending direction of the sampling pipette 10 based on the difference in brightness between the sampling pipette 10 and the background.

As described above, in the manipulation system 100 according to the third embodiment, the image processing unit 52 causes the external shape of the portion along the extending direction of the detection portion 103g and the extension of the sampling pipette 10 from the first image 811. A pipette image in which the sampling pipette 10 is extracted by cutting out the region surrounded by the outer circumference of the partial region BB adjacent to the end portion 103f in the present direction and the line connecting the right end points of the two edges 103ge. 814B (tubular instrument image) is created, and the position detection unit 531 detects the position of the end 103f in the extending direction of the sampling pipette 10 based on the pipette image 814B. According to this, the manipulation system 100 can suppress the detection of the cells ce and the like arranged around the sampling pipette 10. Further, since the pipette image 814B is subjected to image processing only by clipping the image, it is possible to accurately detect the position of the end portion 103f in the extending direction of the sampling pipette 10.

(Fourth Embodiment)
FIG. 32 is a flowchart showing an example of a detection sequence for detecting the inside of the sampling pipette of the manipulation system according to the fourth embodiment. As shown in FIG. 32, the image processing unit 52 creates an image (grayscale image 814BG) in which only the brightness parameter is extracted from the pipette image 814B created in the third embodiment (step ST41). FIG. 33 is a schematic diagram showing an example of a grayscale image in which a pipette image is displayed only with brightness.

As shown in FIG. 33, in the grayscale image 814BG, the difference in brightness between the air layer 108 and the liquid layer 109 inside the sampling pipette 10 is displayed. The brightness of the air layer 108 is higher than the brightness of the liquid layer 109.

Next, the image processing unit 52 extracts a portion having a brightness equal to or higher than a threshold value from the gray scale image 814BG to create a binary image (pipette binary image 814BB) (step ST42). FIG. 34 is a pipette binary image showing the bright part of the collection pipette. The image processing unit 52 uses the discriminant analysis method to automatically set the threshold value from the information on the luminance distribution of the grayscale image 814BG. The image processing unit 52 automatically converts the grayscale image 814BG into the pipette binary image 814BB shown in FIG. 34 based on the set threshold value.

As shown in FIG. 34, in the pipette binary image 814BB, the bright portion 103h of the sampling pipette 10 is displayed in white and the dark portion 103i is displayed in black. The bright portion 103h is a region corresponding to the air layer 108 of the sampling pipette 10, and the dark portion 103i is a region corresponding to the liquid layer 109 and the glass tube portion of the sampling pipette 10.

Next, the luminance detection unit 534 receives the information of the pipette binary image 814BB, detects the luminance in an arbitrary area A3 of the pipette binary image 814BB, and calculates the number of pixels of the bright portion 103h (step ST43). The position detection unit 531 receives the information of the pipette binary image 814BB and the information of the calculation result (the number of pixels of the bright portion 103h), and compares the calculation result with a predetermined threshold value. Accordingly, the position detection unit 531 determines whether the arbitrary area A3 is the air layer 108 or the liquid layer 109 (step ST44). The position detection unit 531 can detect that the arbitrary region A3 is the air layer 108 when the number of pixels of the bright portion 103h is equal to or larger than the threshold value. Further, the position detection unit 531 can detect that the arbitrary region A3 is the liquid layer 109 when the number of pixels of the bright portion 103h is smaller than the threshold value.

As described above, in the manipulation system 100 of the fourth embodiment, the image processing unit 52 creates the pipette binary image 814BB that displays the bright portion 103h having the brightness equal to or higher than the threshold value based on the pipette image 814B. The position detection unit 531 detects the air layer 108 and the liquid layer 109 inside the sampling pipette 10 based on the pipette binary image 814BB. According to this, the manipulation system 100 can detect whether the arbitrary region A3 in the sampling pipette 10 is the air layer 108 or the liquid layer 109. The control unit 55 can automatically control the electric micropump 29 by detecting whether the arbitrary region A3 is the air layer 108 or the liquid layer 109.

(Fifth Embodiment)
FIG. 35 is a flowchart showing an example of a detection sequence for detecting cells inside a sampling pipette in the manipulation system according to the fifth embodiment. As shown in FIG. 35, the manipulation system 100 executes the detection sequence of the first to third embodiments described above to detect the shape of the sampling pipette 10 (step ST51).

Next, the image processing unit 52 cuts out a part of the region from the detected pipette image 814B (see FIG. 30) and creates a reference image in a state where there are no cells ce (step ST52). FIG. 36 is a schematic diagram showing an example of a reference image created by cutting out an image of a portion where cells do not exist inside the sampling pipette. As shown in FIG. 36, the cells ce do not exist in the air layer 108 of the sampling pipette 10 of the reference image 103r2. The reference image 103r2 is an image that has not undergone image processing such as binarization processing and gray scale processing.

Next, the control unit 55 (see FIG. 4) drives the electric micropump 29 (step ST53). As a result, the internal pressure of the collection pipette 10 becomes lower than normal pressure, and the cells ce are sucked and collected from the opening 103a of the collection pipette 10.

Similar to step ST51 and step ST52, the image processing unit 52 images the pipette image 814B of the collection pipette 10 in the state in which the cells ce are collected, and creates the detection image 814Bs by cutting a part of the pipette image 814B ( Step ST54). The image processing unit 52 cuts out the same region as the reference image 103r2 shown in FIG. 36 from the pipette image 814B.

FIG. 37 is a schematic diagram showing an example of an image obtained by cutting out a part of the detected pipette image. As shown in FIG. 37, in the detected image 814Bs, the collected cells ce are present in the air layer 108 of the collection pipette 10.

Next, the image processing unit 52 creates an image of the difference between the detected image 814Bs and the reference image 103r2 (step ST55). FIG. 38 is a schematic diagram showing an example of the difference image between the detected pipette image and the reference image. As shown in FIG. 38, in the difference image 814C, the air layer 108 and the glass tube portion of the sampling pipette 10 are displayed in black because the difference between the detection image 814Bs and the reference image 103r2 is small, and the cell ce is in the air. It is displayed with a higher brightness than the layer 108.

Next, the image processing unit 52 creates a binary image 814D by extracting a portion having a brightness equal to or higher than a predetermined threshold value from the difference image 814C (step ST56). FIG. 39 is a schematic diagram showing an example of a binary image showing the bright portion of the difference image. As shown in FIG. 39, in the binary image 814D, the bright portion ce-b of the cell ce is displayed in white and the portion other than the bright portion ce-b is displayed in black.

The position detection unit 531 can receive the information of the binary image 814D and detect that the cell ce has been collected by the sampling pipette 10 when the binary image 814D includes the bright portion ce-b. Further, the number detection unit 533 (see FIG. 5) may detect the number of cells ce located in a partial region of the sampling pipette 10 based on the binary image 814D. Thereby, the manipulation system 100 can collect the cells ce one by one. Further, when the cell ce is not detected, the manipulation system 100 repeatedly executes the operations of step ST53 to step ST56.

As described above, in the manipulation system 100 of the fifth embodiment, the image processing unit 52 cuts out an image of a portion where the cell ce does not exist inside the sampling pipette 10 from the pipette image 814B, and the reference image 103r2( (Reference tubular instrument image), and the position detecting unit 531 determines the cells inside the sampling pipette 10 based on the image showing the difference in brightness between the detected image of the sampling pipette 10 and the reference image 103r2. detect ce. According to this, the manipulation system 100 can reliably detect that the cells ce have been collected by the collection pipette 10.

(Sixth Embodiment)
FIG. 40 is a flowchart showing an example of the operation sequence of the manipulation system according to the sixth embodiment. As shown in FIG. 40, the manipulation system 100 executes the detection sequence of the first to third embodiments described above, and detects the position of the end 103f of the sampling pipette 10 (step ST61).

Next, the manipulation system 100 lowers the first microscope unit 40 toward the bottom surface of the container 38 and focuses the first microscope 41 on the plurality of cells ce. The first imaging device 45 captures the first image 811 via the first microscope 41. The position detection unit 531 detects the position of the cell ce based on the information of the first image 811 imaged by the first imaging device 45 (step ST62). Note that in step ST62, the image processing unit 52 may perform image processing such as binarization on the first image 811 in order to facilitate detection of the cells ce.

FIG. 41 is a schematic diagram schematically showing a first image according to the sixth embodiment. As shown in FIG. 41, a plurality of cells ce and the tip portion 103 of the sampling pipette 10 are imaged in the first image 811. The control unit 55 (see FIG. 5) selects the cell ce1 existing in a specific region from a plurality of cells ce as a collection target based on the position information of the cell ce from the position detection unit 531. The specific area can be set arbitrarily and is, for example, a rectangular area on the left side of the end portion 103f.

Next, the control unit 55 moves the sampling pipette 10 (step ST63). Specifically, the controller 55 moves the X-axis table 21, the Y-axis table 22, and the Z-axis table 23 by operating the drive devices 26 and 27 (see FIG. 3). The sampling pipette 10 and the pipette holding unit 15 move as the X-axis table 21, the Y-axis table 22, and the Z-axis table 23 move. Thereby, the control unit 55 can bring the end 103f of the tip 103 closer to the cell ce1. The control unit 55 may move the sample stage 30 in at least one of the X-axis direction and the Y-axis direction (hereinafter, horizontal direction). The movement of the sample stage 30 in the horizontal direction is realized by the control unit 55 outputting the drive signal Vxy3 to the drive device 36.

Next, the manipulation system 100 executes the detection sequence of the above-described first to third embodiments in a state where the sampling pipette 10 is close to the cells ce1, and determines the position of the end portion 103f of the sampling pipette 10. It is detected again (step ST64).

Next, the distance detection unit 532 (see FIG. 5) detects an error between the position of the cell ce detected by the position detection unit 531 and the position of the end 103f of the sampling pipette 10 (step ST65). Here, the error between the position of the cell ce and the position of the end 103f of the sampling pipette 10 is the distance between the end of the cell ce and the end 103f, the position shift in the X axis direction, and the Y axis direction. It includes information such as displacement.

The distance detection unit 532 determines whether there is an error between the position of the cell ce and the position of the end 103f of the sampling pipette 10 by comparing with a preset threshold value (step ST66). The threshold information regarding the error is stored in the storage unit 56 in advance.

When there is an error larger than the threshold value (Yes in step ST66), for example, when the separation distance is larger than the threshold value, the process returns to step ST63, and the control unit 55 receives the information from the distance detection unit 532. The sampling pipette 10 is moved. Thereby, the manipulation system 100 can correct the position of the sampling pipette 10, and can reliably collect the cells ce.

When the error is less than or equal to the threshold value (step ST66, No), for example, when the separation distance is less than or equal to the threshold value, the manipulation system 100 collects the cells ce (step ST67). Specifically, the control unit 55 causes the sampling pipette 10 to suck the cells ce1. For the suction of the cells ce, the control unit 55 outputs the drive signal Vmp (see FIG. 3) to the electric micropump 29 to drive the electric micropump 29, and the internal pressure of the sampling pipette 10 is changed to the external pressure of the sampling pipette 10. It is realized by making it lower than.

The manipulation system 100 may execute the detection sequence of the sixth embodiment in combination with the detection sequences of the above-described fourth and fifth embodiments.

As described above, the manipulator 20 further includes the driving devices 26 and 27 that move the sampling pipette 10 relative to the sample stage 30, and the controller 50 detects the cells detected by the position detection unit 531. ce has a distance detection unit 532 that automatically detects the separation distance between the sampling pipette 10 and the controller 50 operates the drive devices 26 and 27 so that the end 103f of the sampling pipette 10 detects the position. When the cell ce is moved closer to the position of the cell ce detected by the unit 531 and the separation distance is larger than a preset value, the position detection unit 531 detects the position of the sampling pipette 10 multiple times. According to this, the manipulation system 100 can correct the position of the end 103f of the collecting pipette 10 to an appropriate position for collecting the cells ce by repeatedly detecting the position of the collecting pipette 10. it can.

10 Sampling Pipette 15 Pipette Holding Section 20 Manipulators 26, 27, 36, 414, 613 Driving Device 29 Electric Micropump 30 Sample Stage 38 Container 40 First Microscope Unit 41 First Microscope 45 First Imaging Device 50 Controller 52 Image Processing Section 53 detection unit 531 position detection unit 535 threshold value setting unit 80 display unit 100 manipulation system 103 tip portion 103g detection portion 103r1 reference binary image 103r2 reference image 811 first image 811A, 814A binary image 811G, 814G grayscale image 812 second Image 813 Third image ce Cell T-a, ce-a Dark part T-b, ce-b Bright part L-TH Threshold value L-BG Background brightness

Claims (12)

A manipulation system for collecting microscopic objects using a tubular instrument,
The tubular device,
A manipulator to which the tubular instrument is attached,
A sample stage on which a container for containing the minute object is placed,
A first microscope arranged above the sample stage;
A first imaging device that captures a first image through the first microscope;
And a controller,
The controller is
An image processing unit that creates a first binary image that displays a dark portion having a brightness equal to or less than a threshold value based on the first image;
A position detection unit that detects the position of the tubular instrument based on the first binary image. The controller has a storage unit for storing a reference binary image of the tubular device,
The manipulation system according to claim 1, wherein the position detection unit detects the tubular instrument by comparing a detection portion formed from the dark portion of the tubular instrument in the first binary image with the reference binary image. .. The manipulation system according to claim 1, wherein the controller has a threshold value setting unit that sets the threshold value based on the brightness of the first image. The image processing unit converts the first image into a grayscale image that is displayed only with luminance, and creates the first binary image based on the grayscale image. Manipulation system described in. In the first binary image, the image processing unit enlarges a detection portion formed from the dark portion of the tubular device, so that two regions are arranged with an area corresponding to a space inside the tubular device interposed therebetween. The manipulation system according to claim 1, wherein the dark portions are connected to each other. The manipulation system according to any one of claims 1 to 5, wherein the image processing unit removes the dark portion indicating the minute object from the first binary image. The image processing unit creates an image cut from the first image in a partial region surrounding the detected tubular device,
The position detection unit, from the image cut out in the partial region, based on the difference in brightness between the detection portion formed from the dark portion of the tubular device and the background, the detection portion formed from the dark portion of the tubular device. The manipulation system according to any one of claims 1 to 6, which detects an outer shape of a portion along the extending direction of the. The image processing unit is adjacent to the outer shape of a portion along the extension direction of the detection portion formed from the dark portion of the tubular device from the first image, and the end of the tubular device in the extension direction. A region surrounded by the outer periphery of the partial region of the part is cut out to create a tubular instrument image in which the tubular instrument is extracted,
The manipulation system according to claim 7, wherein the position detection unit detects the position of the end portion of the tubular instrument in the extending direction based on the tubular instrument image. The image processing unit, based on the tubular instrument image, creates a tubular instrument binary image that displays a bright portion having a brightness equal to or higher than a threshold value,
The manipulation system according to claim 8, wherein the position detection unit detects an air layer and a liquid layer inside the tubular instrument based on the tubular instrument binary image. The image processing unit, from the tubular instrument image, cut out an image of a portion where the microscopic object does not exist inside the tubular instrument to create a reference tubular instrument image,
The position detection unit detects the minute target object inside the tubular device based on an image showing a difference in brightness between the detected image of the tubular device and the reference tubular device image. Alternatively, the manipulation system according to claim 9. Further, the manipulator has a drive device for moving the tubular instrument relative to the sample stage,
The controller is
A distance detection unit that automatically detects a separation distance between the micro object and the tubular device detected by the position detection unit,
The controller is
By operating the drive device, the distal end of the tubular instrument is moved so as to approach the position of the minute object detected by the position detection unit,
The manipulation system according to any one of claims 1 to 10, wherein the position detection unit detects the position of the tubular instrument a plurality of times when the separation distance is larger than a preset value. The manipulation system according to any one of claims 1 to 11, wherein a surface treatment is applied to the inside of the tubular instrument.