WO2021190420A1

WO2021190420A1 - Method and system to perform nasal swabbing based on image matching

Info

Publication number: WO2021190420A1
Application number: PCT/CN2021/081896
Authority: WO
Inventors: Chieh Hsiao CHEN
Original assignee: Brain Navi Biotechnology Co., Ltd.
Priority date: 2020-03-21
Filing date: 2021-03-19
Publication date: 2021-09-30
Also published as: TW202216076A; TW202145963A; TWI773202B

Abstract

A method to determine a pathway in a patient for an apparatus to collect a specimen from the patient. The method includes obtaining two-dimensional image information and three-dimensional image information associated with the patient with one or more optical devices; selecting a set of two-dimensional feature points from the two-dimensional image information; transforming the set of two-dimensional feature points to a set of three-dimensional feature points in the three-dimensional image information; determining points associated with the pathway based on the set of three-dimensional feature points; and transforming the points from a first coordinate system associated with the one or more optical devices to a second coordinate system associated with the apparatus.

Description

METHOD AND SYSTEM TO PERFORM NASAL SWABBING BASED ON IMAGE MATCHING

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No. 62/992,922, filed March 21, 2020 and U.S. Provisional Application No. 63/093,221, filed October 18, 2020, which are incorporated by reference in their entireties.

BACKGROUND OF THE INVENTION Field of the Invention

Embodiments of the present invention relate generally to methods and systems to collect specimens from patients. More specifically, embodiments of the present invention relate generally to methods and systems to collect specimens from a safe distance.

Description of the Related Art

Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

Collection of specimens from surface of the respiratory mucosa of a patient with nasopharyngeal swabs is a procedure for a diagnosis of Covid-19. The procedure is also commonly used to evaluate patients with suspected respiratory infection caused by pathogens, such as other viruses and some bacteria. However, the procedure is usually carried out by a medical specialist. The medical specialist may be exposed from the threat of Covid-19 or pathogens.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is an example figure showing the spatial relationships among several points that may be encountered on a pathway to collect a specimen from a patient;

Fig. 2 is a flow diagram illustrating an example process to determine a pathway to collect a specimen from a patient;

Fig. 3 illustrates an example system to collect a specimen from a patient;

Figs. 4A, 4B, 4C, 4D and 4E illustrate top views of an example system in various states during a specimen collection from a patient;

Figs. 5A and 5B illustrate top views of an example specimen processing device in various states;

Fig. 6 illustrates example components in a specimen processing device; and

Figs. 7A and 7B illustrate an example laboratory including one or more systems to collect specimens from one or more patients, all arranged in accordance with some embodiments of the present disclosure.

DETAILED DESCRIPTION

In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented here. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the Figures, can be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

Fig. 1 is an example figure showing the spatial relationships among several points that may be encountered on a pathway to collect a specimen from a patient, arranged in accordance with some embodiments of the present disclosure. In Fig. 1, a pathway 100 may include entry point 110 and target point 120. In some embodiments, pathway 100 corresponds to a natural path in the nasal cavity of the patient. Point 110 may correspond to the nostril of the patient and point 120 may be adjacent to the nasopharynx of the patient. In some embodiments, sampling tool 150 is disposed on the patient to facilitate a robotic arm to collect the specimen from the patient. Sampling tool 150 may be curved or have a specific configuration to fit pathway 100. In some embodiments, sampling tool 150 includes first feature 151 at or around one end of sampling tool 150. First feature 151 includes one or more feature points and may facilitate the robotic arm to move to point 110. The curved structure or the specific configuration of sampling tool 150 may also facilitate the robotic arm to be guided from point 110 to point 120 through pathway 100.

Fig. 2 is a flow diagram illustrating an example process 200 to determine a pathway to collect a specimen from a patient, arranged in accordance with some embodiments of the present disclosure. Process 200 may include one or more operations, functions, or actions as illustrated by

blocks

210, 220, 230, 240, 250, 260, 270, 280, and/or 290, which may be performed by hardware, software and/or firmware. The various blocks are not intended to be limiting to the described embodiments. The outlined steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments. Although the blocks are illustrated in a sequential order, these blocks may also be performed in parallel, and/or in a different order than those described herein.

Process 200 may begin at block 210, “obtain patient’s two-dimensional (2D) and three-dimensional (3D) images. ” In some embodiments, patient’s images may correspond to surface anatomy information of the patient. The set of images may be a set of pictures taken by a three-dimensional camera (e.g., a camera with a depth sensor) or a set of two-dimensional cameras. For example, the camera (s) may be configured to take pictures of the patient’s head to capture the head appearance and contours (e.g., eyes, ears, nose, nostril opening, earlobe, etc. ) of the patient. In some embodiments that a sampling tool is disposed on the patient (e.g., on patient’s nose) , the camera (s) may be configured to take pictures of the patient’s head to capture images of the head appearance, contours of the patient and the sampling tool disposed on the patient. The cameras may be fixed at a stationary device (e.g., a robotic arm) of a system to collect a specimen from a patient. In some embodiments, the camera (s) may take pictures from different positions while the robotic arm is moving. For example, the pictures may include pictures having depth information taken from different angles between the left ear, the nose, the month and the right ear of the patient.

Block 210 may be followed by block 220 “select 2D feature points from 2D images. ” In some embodiments, an artificial intelligence engine may be employed to select a set of two-dimensional feature points from the two-dimensional images obtained in block 210. In conjunction with Fig. 1, the set of two-dimensional feature points include first feature 151 and some other facial features (e.g., nostril openings, earlobe, etc. ) of the patient.

Block 220 may be followed by block 230 “transform 2D feature points to 3D feature points in 3D images. ” As set forth above, in conjunction with Fig. 1, the depth information associated with the patient and first feature 151 may be collected by the three-dimensional camera or the set of two-dimensional cameras. In some embodiments, the depth information may be added to the set of two-dimensional feature points to transform the set of two-dimensional feature points to a set of three-dimensional feature points in 3D images obtained in block 210.

Block 230 may be followed by block 240 “determine pathway to collect a specimen based on 3D feature points. ” The pathway may include an entry and a destination. In some embodiments, the entry may correspond to any of nostril openings of the patient. In conjunction with Fig. 1, sampling tool 150 is designed to have a specific spatial relationship with nostril openings of the patient. Therefore, based on feature points of first feature 151 and feature points of nostril openings of the patient. A point and an angle associated with entering the entry of the pathway may be determined.

In some other embodiments, the destination may be determined based on a distance away from the entry. The distance may correspond to a distance from any of the nostril openings to corresponding earlobe of the patient because this distance generally corresponds to a distance from a nostril opening to the nasopharynx according to statistic anatomy relationships. Images of nostril openings and earlobes are surface anatomy information of the patient and may be obtained in block 210. Therefore, the destination of the pathway may be determined.

In some embodiments, the determined angle entering the entry is maintained until reaching the destination. In conjunction with Fig. 1, although the determined angle is maintained, the curved structure or the specific configuration of sampling tool 150 may guide the robotic arm from the entry to the destination in the pathway.

Block 240 may be followed by block 250, “transform coordinates. ” Block 250 includes, but not limited to, transforming the entry, the angle and the destination of the pathway from their original coordinate system (i.e., three-dimensional camera coordinate system) to the coordinates of the robotic arm (i.e., robotic arm coordinate system) . The transformation may be based on known spatial relationship between the camera (s) and the robotic arm.

Block 250 may be followed by block 260, “drive robotic arm to points on pathway. ” In block 260, the coordinates of the pathway to collect the specimen in the three-dimensional camera coordinate system may be transformed to the robotic arm coordinate system. Therefore, in conjunction with Fig. 1, the robotic arm may move to the point 110, and move and rotate along the pathway 100 to point 120. In addition, after reaching point 120, the robotic arm is configured to obtain the specimen from the patient.

Fig. 3 illustrates an example system 300 to collect a specimen from a patient. In some embodiments, example system 300 includes, but not limited to, robotic arm 320, specimen processing device 330, base 340 and protective screen 350. In conjunction with Fig. 2, robotic arm 320 may be driven to rotate and collect specimens from the nasopharynx region or the oral-pharynx region from patient 310. Base 340 is configured to provide a support for robotic arm 320. Protective screen 350 is configured to divide patient 310 from a health care worker (not illustrated) who operates system 300. Robotic arm 320 is configured to place collected specimens to specimen processing device 300 for further processing.

Fig. 4A illustrates a top view of an example system 400 in a first state during a specimen collection from a patient. In some embodiments, example system 400 includes, but not limited to, robotic arm 420, specimen processing device 430, base 440, protective screen 450 and monitor 460. In the first state, robotic arm 420 is in a first initial position. Base 440 is configured to provide a support for robotic arm 420. Protective screen 450 is configured to divide patent 410 from a user (not illustrated) who operates system 400 through monitor 460.

Fig. 4B illustrates a top view of an example system 400 in a second state during a specimen collection from a patient. In some embodiments, in the second state, specimen processing device 430 is configured to expose tube assembly rack 431. Robotic arm 420 is driven to move from the first initial position to a second position at tube assembly rack 431 to hold a swab placed in tube assembly rack 431.

Fig. 4C illustrates a top view of an example system 400 in a third state during a specimen collection from a patient. In some embodiments, in the third state, robotic arm 420 is driven to move from the second position to a third position at the nasopharynx region or the oral-pharynx region of patient 410 to collect specimens from patient 410.

Fig. 4D illustrates a top view of an example system 400 in a fourth state during a specimen collection from a patient. In some embodiments, in the fourth state, robotic arm 420 is driven to move from the third position to a fourth position (e.g., the second position) at tube assembly rack 431 to place the swab back in tube assembly rack 431.

Fig. 4E illustrates a top view of an example system 400 in a fifth state during a specimen collection from a patient. In some embodiments, in the fifth state, robotic arm 420 is driven to move from the fourth position to a fifth position (e.g., the first initial position) . In addition, specimen processing device 430 is configured to retrieve tube assembly rack 431 for further processing the collected specimen.

Fig. 5A illustrates a top view of an example specimen processing device 500 in an initial or final state and Fig. 5B illustrates a top view of an example specimen processing device 500 in a specimen collection state. In some embodiments, specimen processing device 500 may correspond to specimen processing device 430 in Fig. 4A to 4E. In some embodiments, specimen processing device 500 includes, but not limited to, handle 531, front door 532, switch 533, side door 534, tube assembly rack 535 and transition region 536. In some embodiments, tube assembly rack 535 may correspond to tube assembly rack 431 in Fig. 4B to 4D.

In some embodiments, in the initial state, a technician (not illustrated) may use handle 531 to open front door 532 so that tube assembly rack 535 is exposed from specimen processing device 500. The healthcare worker may place, not limited to, a swab, a vial, a tube and a pipette on tube assembly rack 535.

In some embodiments, the healthcare worker may then close front door 532 and push switch 533 to actuate a holder which supports tube assembly rack 535 and open side door 534. In some embodiments, in the specimen collection state, the holder may move tube assembly rack 535 through transition region 536 and expose tube assembly rack 535 from side door 534.

In the specimen collection state, in conjunction with Fig. 4B, robotic arm 420 is driven to hold the swab placed in tube assembly rack 535. In conjunction with Fig. 4D, after robotic arm 420 collects specimens from patient 410 with the swab, robotic arm 420 is driven to place the swab back in tube assembly rack 535. The healthcare worker may push switch 533 to retrieve tube assembly rack 535 back to transition region 536 and close side door 534.

In some embodiments, collected specimens are processed in transition region 536 and will be further described below. After the collected specimens are processed, in the final state, the holder may move tube assembly rack 535 from transition region 536 to an initial position in the initial state as illustrated in Fig. 5A. The healthcare worker may retrieve the tube from front door 532. The retrieved tube will be analyzed by technical feasible approaches, such as polymerase chain reaction, to determine whether a pathogen exists in the specimens.

Fig. 6 illustrates example components in a specimen processing device 600. In some embodiments, these components include, but not limited to, secondary robotic arm 610, tube assembly rack 620, holder 630, sanitizing components 640 and

medical waste collectors

651 and 652. Secondary robotic arm 610 may further include a robotic claw 612. In some embodiments, swab 622, swab holder 623, vial 624, tube 626 and pipette 628 are disposed on tube assembly rack 620. Holder 630 may include a displacement mechanism to support and move tube assembly rack 620. Sanitizing components 640 are configured to sanitize the specimen processing device. In some embodiments, in conjunction with Fig. 5A and 5B, secondary robotic arm 610, tube assembly rack 620, holder 630 and sanitizing components 640 are disposed in transition region 536 of specimen processing device 500.

In some embodiments, in conjunction with Fig. 5B, after tube assembly rack 620/535 is retrieved from side door 534 back to transition region 536, swab 622 includes specimens collected from a patient. Robotic claw 612 is configured to place swab 622 in swab holder 623, open a cover of vial 624 and a cover of tube 626 and place the covers at appropriate places in specimen processing device 600. Robotic claw 612 is then configured to hold swab 622 and place swab 622 into vial 624. Vial 624 may include a fluid. Robotic claw 612 is configured to place swab 622 in vial 624 for a predetermined amount of time so that collected specimens are dispersed in the fluid of vial 624. After the predetermined amount of time, robotic claw 612 may pull swab 622 away from vial 624, place swab 622 in swab holder 623 and place swab 623 in which swab 612 is placed in medical waste collector 651.

Alternatively, in conjunction with Fig. 4D and 5B, in some other embodiments, before tube assembly rack 620/535 is retrieved from side door 534 back to transition region 536, primary robotic arm 420 is configured to place swab 622 into vial 624 for a predetermined amount of time so that collected specimens are dispersed in the fluid of vial 624. After tube assembly rack 620/535 is retrieved from side door 534 back to transition region 536, robotic claw 612 may pull swab 622 away from vial 624, place swab 622 in medical waste collector 651 and place swab holder 623 in medical waste collector 651.

In some embodiments, robotic claw 612 is then configured to hold pipette 628, place pipette 628 in vial 624 and collect an amount of the fluid from vial 624. Robotic claw 612 is then configured to pull pipette 628 away from vial 624 and inject the collected fluid into tube 626. Robotic claw 612 is then configured to place pipette 628 in medical waste collector 651.

In some embodiments, robotic claw 612 is then configured to hold the cover of tube 626, place the cover of tube 626 at the top of tube 626 and seal tube 626 with the cover. Robotic claw 612 is then configured to hold the cover of vial 624, place the cover of vial 624 at the top of vial 624, seal vial 624, hold sealed vial 624 and place sealed vial 624 in medical waste collector 652.

After the operations set forth above, there is only sealed tube 626 disposed on tube assembly rack 620. Swab 622, vial 624 and pipette 628 are disposed in

medical waste collectors

651 and 652. In some embodiments, sanitizing components 640 are activated to sanitize the transition region of specimen processing device 600. Sanitizing components 640 may include, but not limited to, a sanitizer spraying device configured to spray a sanitizer in the transition region of specimen processing device 600 or an ultraviolet light source configured to generate ultraviolet lights in the transition region of specimen processing device 600. Therefore, the transition region and an outer surface of seal tube 626 may be sanitized. Accordingly, in conjunction with Fig. 5A, the healthcare worker may retrieve the sanitized sealed tube 626 from front door 532 without being affected by a pathogen potentially existed in the specimens.

Fig. 7A illustrates an example laboratory 700 including one or more systems to collect specimens from one or more patients. Fig. 7B is a top view of the example laboratory. In some embodiments, laboratory 700 may be a preconfigured module, such as a container, with dividers placed in the module to define the different areas. Alternatively, each of the different areas of laboratory 700 (e.g., 752, 754, and 760) may be a preconfigured module. Therefore, laboratory 700 may be quickly deployed and scaled.

In some embodiments, laboratory 700 defines a first door 712, a second door 714 and a third set of doors 720. First door 712 is a door of first specimen collection area 752 and a first patient may enter or leave first specimen collection area 752 through first door 712. Second door 714 is a door of second specimen collection area 754 and a second patient may enter or leave second specimen collection area 754 through second door 714.

In some embodiments, first specimen collection area 752 may include a first system to collect a specimen from the first patient. In conjunction with Fig. 3, the first system may correspond to system 300. First system may include robotic arm 754 and specimen processing device 756. Robotic arm 754 may correspond to robotic arm 320 and specimen processing device 756 may correspond to specimen processing device 330. Similarly, second specimen collection area 754 may include a second system to collect a specimen from the second patient. The second system may correspond to the first system.

In some embodiments, operation area 760 is adjacent to both first specimen collection area 752 and second specimen collection area 754. With such a configuration, a healthcare worker may access operation area 760 through any of the third set of doors 720, which are doors of operation area 760, and operate either of or both the first system in first specimen collection area 752 and the second system in second specimen collection area 754. In some embodiments, operation area 760 is isolated from first specimen collection area 752 and second specimen collection area 754 so that the healthcare worker may keep a safe distance from the first patient and the second patient.

The foregoing detailed description has set forth various embodiments of the devices and/or processes via the use of block diagrams, flowcharts, and/or examples. Insofar as such block diagrams, flowcharts, and/or examples contain one or more functions and/or operations, it will be understood by those within the art that each function and/or operation within such block diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. In some embodiments, several portions of the subject matter described herein may be implemented via Application Specific Integrated Circuits (ASICs) , Field Programmable Gate Arrays (FPGAs) , digital signal processors (DSPs) , or other integrated formats. However, those skilled in the art will recognize that some aspects of the embodiments disclosed herein, in whole or in part, can be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems) , as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors) , as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of skill in the art in light of this disclosure. In addition, those skilled in the art will appreciate that the mechanisms of the subject matter described herein are capable of being distributed as a program product in a variety of forms, and that an illustrative embodiment of the subject matter described herein applies regardless of the particular type of signal bearing medium used to actually carry out the distribution. Examples of a signal bearing medium include, but are not limited to, the following: a recordable type medium such as a floppy disk, a hard disk drive, a Compact Disc (CD) , a Digital Versatile Disk (DVD) , a digital tape, a computer memory, etc. ; and a transmission type medium such as a digital and/or an analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communications link, a wireless communication link, etc. ) .

From the foregoing, it will be appreciated that various embodiments of the present disclosure have been described herein for purposes of illustration, and that various modifications may be made without departing from the scope and spirit of the present disclosure. Accordingly, the various embodiments disclosed herein are not intended to be limiting.

Claims

A method to determine a pathway in a patient for an apparatus to collect a specimen from the patient, comprising:

obtaining two-dimensional image information and three-dimensional image information associated with the patient with one or more optical devices;

selecting a set of two-dimensional feature points from the two-dimensional image information;

transforming the set of two-dimensional feature points to a set of three-dimensional feature points in the three-dimensional image information;

determining points associated with the pathway based on the set of three-dimensional feature points; and

transforming the points from a first coordinate system associated with the one or more optical devices to a second coordinate system associated with the apparatus.
The method of claim 1, wherein the set of two-dimensional feature points includes a feature point associated with a sampling tool disposed on the patient.
The method of claim 1, wherein the transforming is based on a depth information associated with the patient collected by the one or more optical devices
The method of claim 1, wherein the determining includes determining an entry point, an entry angle and a destination point of the pathway.
The method of claim 4, wherein the destination point is determined based on a distance associated with contours of the patient.
A non-transitory computer-readable storage medium that includes a set of instructions which, in response to execution by a processor, cause the processor to perform any of the method of claims 1 to 5 to determine a pathway in a patient for an apparatus to collect a specimen from the patient.
A system to determine a pathway in a patient for an apparatus to collect a specimen from the patient, comprising:

a processor; and

a non-transitory computer-readable medium having stored thereon program code that, upon being executed by the processor, causes the processor to perform any of the method of claims 1 to 5.
A system to process a specimen from a patient, comprising:

a specimen processing device including a sanitizing component configured to receive the specimen and to sanitize a region of the specimen processing device with the sanitizing component before the specimen is retrieved from the specimen processing device.
The system of claim 8, further comprising a primary robotic arm configured to collect the specimen from the patient, and the specimen processing device is configured to receive the specimen from the primary robotic arm.
The system of claim 9, wherein the specimen processing device is configured to define a front door accessible by a healthcare worker and a side door accessible by the primary robotic arm.
The system of claim 9, wherein the specimen processing device further includes a secondary robotic arm, a swab, a vial, a tube and a pipette.
The system of claim 11, wherein the primary robotic arm is configured to hold the swab, collect the specimen from the patient with the swab and place the swab in a fluid included in the vial.
The system of claim 11, wherein the secondary robotic arm is configured to hold the swab after the primary robotic arm collects the specimen from the patient and place the swab in a fluid included in the vial.
The system of claim 13, wherein the secondary robotic arm is configured to, after placing the swab in the vial, hold the pipette and collect the fluid with the pipette.
The system of claim 14, wherein the secondary robotic arm is configured to, after having collected the fluid with the pipette, inject the fluid in the tube and seal the tube.
The system of claim 15, wherein the sanitizing component is configured to, after sealing the tube, sanitize the region of the specimen processing device and the sealed tube.
The system of claim 11, wherein the secondary robotic arm is configured to place the swab, the vial and the pipette in a medical waste collector.
A laboratory, comprising:

a first specimen collection area having a first door to receive a first patient, wherein the first specimen collection area includes a first system having a first primary robotic arm and a first specimen processing device having a sanitizing component, and the first primary robotic arm is configured to collect a specimen from the first patient, and the first specimen processing device is configured to receive the specimen from the first primary robotic arm and sanitize a region of the first specimen processing device with the sanitizing component before the specimen is retrieved from the first specimen processing device; and

an operation area isolated from the first specimen collection area, wherein the operation area includes a third door to receive a healthcare worker to operate the first system in the first specimen collection area.
The laboratory of claim 18, further comprising:

a second specimen collection area having a second door to receive a second patient,

wherein the second specimen collection area includes a second system having a second primary robotic arm and a second specimen processing device having a sanitizing component, and the second primary robotic arm is configured to collect a specimen from the second patient, and the second specimen processing device is configured to receive the specimen from the second primary robotic arm and sanitize a region of the second specimen processing device with the sanitizing component before the specimen is retrieved from the second specimen processing device, and

wherein the second specimen collection area is isolated from the operation area, allowing the healthcare worker in the operation area to operate the second system in the second specimen collection area.
The laboratory of claim 18, wherein the laboratory includes one or more preconfigured modules.
The laboratory of claim 19, wherein the first specimen collection area and the second specimen collection area are adjacent to the operation area.