GB2585907A - Loading of consumables in autamted analyzer systems - Google Patents

Abstract

A system for controlling the loading and unloading of material to an automated analyser system comprises a device, comprising at least one camera 5 for capturing a video stream, and a data processing unit, comprising artificial intelligence for evaluation of the captured video stream. Preferably, the device comprises a microphone 8, a loudspeaker 7, and means 6 for illuminating the area for capturing the video stream. The system may comprise means for a locking member and/or an opening member for locking a loading area 12 of the automated analyzer system. Preferably the system is configured to recognize at least one from the group of: persons, objects, barcodes, gestures and voices. A control unit 10 may be provided for coordinating image capture, recognition, regulating loading and/or unloading of material, generation of audio and/or visual signals and access or blocking of loading areas. minimizing the number of interactions that are necessary during loading of automated analyser systems with components like samples, reagents and consumables and provides a system for controlling the loading and unloading of material to automated analyzer systems, comprising a device, comprising at least one camera for capturing a video stream; and a data processing unit, comprising artificial intelligence for evaluation of the captured video stream.

Description

LOADING OF CONSUMABLES IN AUTMATED ANALYZER SYSTEMS

DESCRIPTION

Field of the Invention

[0001] The invention relates to a system for minimizing the number of interactions that are necessary during loading of automated analyser systems with components like samples, reagents and consumables.

Brief description of the related art

[0002] Automated analyser systems for use in clinical diagnostics, in vitro diagnostics and life sciences are produced by a number of companies. For example, STRATEC' SE, Birkenfeld, Germany, produces a number of devices for specimen handling and detection for use in automated analyser systems and other laboratory instrumentation.

[0003] Operators or users will have to provide different kinds of materials to automated analyzer systems like patient samples, reagents and other consumables and to remove them once they have been processed or are empty. Such interactions are usually done through dedicated loading areas that may be protected by lockable doors, drawers or racks.

[0004] In order to request access to such an area the user has to know where the material has to be loaded and how to perform the loading or unloading action. This is commonly achieved by interaction with the graphical user interface (GUI) of a software that is installed on the automated analyzer's system or by using provided hardware buttons. Software may be used for unlocking automatically areas when the system does not have to be locked anymore. If the loading area is unlocked, the operator will be able to open it and load or unload material. Using GUI or hardware buttons requires an operator training and provision of manuals because such loading procedures are usually not self-explaining.

[0005] A further disadvantage of solution known in the prior is related to dedicated storage areas for applying material as mentioned above so that the operator has a free hand to operate the GUI, hardware buttons or instrument doors. Such storage areas require space that is needed only temporarily during the loading or unloading procedure.

[0006] Another problem of known solutions is related to trays or carriers that allow for one-handed carrying so that the operator has a free hand to operate the GUI, hardware buttons or instrument doors. Such trays and carriers are not mounted to the instrument, so that they can get lost or need a storage area.

Object of the Invention [0007] It is therefore the object of this invention to provide a solution facilitating loading and unloading of automated analyzer systems.

Summary of the Invention

[0008] The present invention provides a system for controlling the loading and unloading of material to automated analyzer systems, comprising a device, comprising at least one camera for capturing a video stream; and a data processing unit, comprising artificial intelligence for evaluation of the captured video stream.

[0009] In a further aspect, the system's device may further comprise at least one illuminant for illumination of the area for capturing the video stream.

[0010] In a further embodiment of the system, the device may further comprise at least one loudspeaker.

[0011] It is also envisaged that the device may further comprise at least one microphone.

[0012] In another aspect, the device may further comprise a locking and/or opening member for locking a loading are of the automated analyzer system.

[0013] In another embodiment of a system according to the invention, the device may further comprise visual elements for producing signals.

[0014] It is also envisaged that the artificial intelligence may be configured to recognize at least one selected from the group comprising persons, objects, barcodes, gestures and voices.

[0015] The software of a system according to the invention may further comprise a control unit for coordinating image capture, recognition, regulating loading and/or unloading of material, generation of audio and/or visual signals and access or blocking of loading areas.

[0016] Another object of the present invention refers to a method for controlling the loading and unloading of material to automated analyzer systems, comprising the steps of a. Capturing a video stream in an area next to a loading area of the automated analyzer system; b. Analysing the video stream for the presence of a person c. Recognizing material carried by the person; d. Allowing access to the loading are.

[0017] The method may in a further aspect comprise the step of identifying the intended use of recognized material by reading our information that is attached to the material.

[0018] It is also envisaged that the method may comprise the step of requesting information regarding the material and/or its intended use.

[0019] In another embodiment, the method of the present invention may comprise the step of producing audio and/or visual signal for interaction with the person.

[0020] Still other aspects, features, and advantages of the present invention are readily apparent from the following detailed description, simply by illustrating preferable embodiments and implementations. The present invention is also capable of other and different embodiments and its several details can be modified in various obvious respects, all without departing from the spirit and scope of the present invention. Accordingly, the drawings and descriptions are to be regarded as illustrative in nature, and not as restrictive. Additional objects and advantages of the invention will be set forth in part in the description which follows and in part will be obvious from the description, or may be learned by practice of the invention.

Summary of the Figures

[0021] The invention will be described based on figures. It will be understood that the embodiments and aspects of the invention described in the figures are only examples and do not limit the protective scope of the claims in any way. The invention is defined by the claims and their equivalents. It will be understood that features of one aspect or embodiment of the invention can be combined with a feature of a different aspect or aspects of other embodiments of the invention, in which: [0022] FIG. 1 shows an overview of the present invention.

[0023] FIG. 2 shows a diagram summarizing state of the art operator-machine interactions.

[0024] FIG. 3 shows a diagram adding features according to the invention to the diagram shown in FIG. 2.

[0025] FIG. 4 shows a material loading workflow.

[0026] FIG. 5 shows a material unloading work flow.

Detailed Description of the Invention and the Figures [0027] The technical problem is solved by the independent claims. The dependent claims cover further specific embodiments of the invention.

[0001] The present invention combines several known technologies to improve the loading process of analytical instruments like automated analysers. The invention is based on one or more cameras in the loading area (FIG. 1) of the instrument and the application of software (FIG. 5) which are used to detect the intent of the operator when performing the corresponding instrument interactions (FIG. 2). The software comprises artificial intelligence (AI) for the object recognition (sample material, reagents, consumables), for voice recognition and for gesture recognition. A microphone and loudspeakers may further be used for user feedback and confirmation.

[0028] The camera aided intelligent instrument loading (CAIL[DST1) shall minimize the amount of interactions required by the user to load resources that are needed by an analytical instrument like sample material, reagents and consumables and guides the operator through the loading process. Usually, instrument areas which are intended for loading of such materials have to be protected by interlocks to avoid unintentional process interruption, mechanical damage or operator injuries. Consequently, the operator has to manually request an unlock operation using the graphical user interface (GUI) or a hardware button. If a GUI is used, a training of the operator will be a prerequisite for using the software. An operator's qualification becomes more and more an issue as the diversity of instruments in today's laboratories increases. The operator may not have a free hand available for performing required operations to initialize loading or unloading when carrying materials or because of multiple handlings that may be required, in particular when handlings have to be performed in parallel.

[0029] The camera aided intelligent instrument loading is suitable to detect whether an operator arrived or stays in front of an instrument. It may further identify the type of material to be loaded by object recognition or the detection of barcodes which are applied to the materials. The CALL may further guide an operator through the corresponding loading operation and interactively assist the operator. e.g. by unlocking the corresponding loading area, opening of a door or flap, visually indicating the correct loading area or asking the user for confirmation via GUI interaction, gesture or voice control.

[0002] The CALL sub-system extends an analytical instrument system with hardware components (FIG. 1) and software (FIG. 5). The hardware comprises so-called core components like a computer unit optimized for image processing to provide object, gesture and voice recognition, one or more digital cameras for monitoring the loading area, illumination means to guarantee a high quality of images, loudspeaker for an audible user feedback and a microphone for voice control. Such interaction components do also belong to the provided hardware components, comprising an interlock of the loading module, an active or passive opening mechanism and means for visual signaling of the loading area status, e.g. LEDs or other illuminants.

[0003] The CALL related software (FIG. 5) comprises program modules for object recognition, barcode recognition, gesture recognition module, voice recognition, speech generation, intelligent loading and unloading control in terms of guiding and controlling loading and unloading, as well as image capturing.

[0004] With respect to FIG. 2, the operator has to perform two primary interactions with an automated analyzer, which relate to loading and unloading of material. To insert or remove material from the analytical instrument the operator needs to get access to the loading area, which may be locked and closed. The operator has to execute secondary system interactions that are traditionally performed manually in order to unlock and open the loading area. These secondary operations are automatically initialized by the intelligent loading module through deriving the operator's intent to interact with the automated analyzer by using a camera system.

[0005] The system will have to be able to identify the type of material an operator is carrying for recognizing the respective intended use and kind of intended use. Therefore, the system has to detect in a first instance that an operator arrived or is staying in front of the instrument. In a following step, the system has to detect whether the operator is carrying material and if yes, what kind of material the operator is carrying. Standard algorithms may be used for the general person detection and no special training is required. Material type detection has to be trained as part of the particular product development.

[0006] The operator has to approach the instrument with the material to be loaded in a way that it is in the field of view (FOV) of the camera. In case of dimmed environmental light, the system applies additional illumination for a better image recognition. The FOV is continuously scanned for the presence of a person. The Al has to be trained for material type recognition and/or a barcode belonging to an identified object that may be attached to material.

[0007] The information that an operator with material approached the instrument will be transferred to the Instrument Control Unit (ICU) which can derive appropriate actions based on the type of material, identified barcodes and the status of the instrument. In case the type of material could not be identified, or the operator's intent is unclear the intelligent loading module can be used by the instrument control unit as a standard voice command device to get an interactive user feedback. The operator should always be able to carry out the traditional manual interactions because there exists a remaining likelihood of incorrect recognition by the AI. The ICU takes care of pausing a currently performed process for allowing to unlock and open the corresponding loading area if CALL has been able to identify an operator, the type of material and an instrument status allowing loading of material. Finally, the operator can insert the material and close the doors (FIG. 4).

[0008] The FOV is continuously scanned for the presence of a person and if an operator will approach the instrument with material, the operator will be recognized in front of the instrument. Such information will be transferred to the Instrument Control Unit (ICU), which can derive appropriate actions based on the state of the instrument. Again, the operator should always be able to carry out the traditional manual interactions because there is a remaining likelihood of incorrect recognition by the Al. In case CALL was able to identify an operator carrying no material and the instrument status indicates that unloading of material is required, the ICU takes care to pause a currently ongoing process if required and unlocks and opens the corresponding loading area. Finally, the operator may remove material and close the doors again (FIG. 5).

[0009] The CALL shall be able to enhance existing and new analytical instrument models or automated analyzer systems. Therefore, CAIL is intended to be a standalone device composed of a processing unit, a camera, illuminants, loudspeaker and microphones. The processing unit is connected to the hardware components and provides a software interface to the instrument control unit (FIG. 1).

[0010] The following software modules are intended to run on the processing unit (FIG. 1): - Intelligent Loading Control - Image Capture - Object Recognition - Barcode Recognition - Gesture Recognition - Sound Capture - Voice Recognition - Speech Generation [0011] The intelligent loading control module takes care of the orchestration of the sub-modules and provides an interface to the instrument control unit (ICU). The interface allows the following actions and events: - Event: Operator Detected Generating a signal, when a person is recognized in the field of view. Contains information whether the operator carries material or not. Additionally, the certainty of detection is reported.

- Event: Material Detected Generating a signal, when a person carries material known or recognized by the system.

Contains barcode information if barcodes have been identified.

Additionally, the certainty of detection is reported.

- Action: Request user feedback Can be used by the ICU to use CAIL as a voice command device for confirmation or ambiguous situations.

- Event: Operator response detected Contains information whether the response is positive or negative within the meaning of a successful detection.

Contains information whether the response was obtained via gestures or by voice input.

Additionally, the certainty of detection is reported.

[0012] The Image Capture module is connected to the camera and delivers an image video stream to the object-, barcode-and gesture-recognition modules.

[0013] The Object Recognition module is able to recognize material and the type of material in the context of an analytical instrument. Each analyzer model has to be trained for different classes of material types that the instrument may handle.

[0014] The Barcode Recognition module is able to decode barcodes from the image video stream for higher material type detection accuracy. Barcodes have to be large enough to allow reliable detection.

[0015] The Gesture Recognition module is able to recognize standard person's gestures like a person nodding or shaking their head. The output can be used for an operator response detection in case that a feedback was requested.

[0016] The Sound Capture module is connected to the microphone and delivers the audio stream to the Voice Recognition module. It is further able to translate natural human language into text. The output can be used for an operator response detection in case that feedback was requested.

[0017] The Speech Generation module is able to generate natural human language from text. This can be used to request feedback from the operator in case of unclear situations or to get confirmation or further information from the user or operator.

[0018] The advantages of the invention can be summarized as follows: - reducing the amount of user/operator interactions; - reducing operator's training time by intelligent and intuitive system guidance; - increasing operator's efficiency by allowing to use both hands to carry material; - enabling people with inabilities to use the instrument; - reducing the likelihood of operating errors or user confusion; - no need of dedicated areas for intermediate material storage reducing the instrument workplace space requirements in the laboratory.

[0019] The instrument according to the present invention can be connected to existing track systems for sample tube transportation (LAS). The connection solves material loading problems only partially because those systems are dedicated to sample tube transportation. Reagents and other consumables will still have to be loaded manually.

[0020] The instrument can be connected directly to a waste container or the like in order to get rid of liquid waste. This requires an appropriate infrastructure of the lab.

[0021] The instrument may further be connected directly to plumbing for providing DI water. Dl water can be mixed furthermore with reagent concentrates onboard e.g. for obtaining a wash buffer. This requires a respective infrastructure of the lab and the provision of concentrates that will still have to be loaded manually.

[0022] The instrument may further be connected to a track system for a reagent cartridge transport.

[0023] The foregoing description of the preferred embodiment of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention. The embodiment was chosen and described in order to explain the principles of the invention and its practical application to enable one skilled in the art to utilize the invention in various embodiments as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims appended hereto, and their equivalents. The entirety of each of the aforementioned documents is incorporated by reference herein.

Reference Numerals 1 intelligent loading processing unit camera 6 illumination 7 loudspeaker 8 microphone instrument control unit 11 monitor 12 lockable loading area barcode labelled loadable material operator

Claims (12)

1. CLAIMSA system for controlling the loading and unloading of material to automated analyzer systems, comprising a device, comprising at least one camera for capturing a video stream; and a data processing unit, comprising artificial intelligence for evaluation of the captured video stream.
2. The system of claim 1, wherein the device further comprises at least one illuminant for illumination of the area for capturing the video stream.
3. 3. The system of claim 1 or 2, wherein the device further comprises at least one loudspeaker.
4. The system of any one of claims 1 to 3, wherein the device further comprises at least one microphone.
5. The system of any one of claims 1 to 4, wherein the device further comprises a locking and/or opening member for locking a loading are of the automated analyzer system.
6. The system of any one of claims 1 to 5, wherein the device further comprising visual elements for producing signals.
7. The system of any one of claims 1 to 6, wherein the artificial intelligence is configured to recognize at least one selected from the group comprising persons, objects, barcodes, gestures and voices.
8. The system of any one of claims 1 to 7, wherein the software further comprises a control unit for coordinating image capture, recognition, regulating loading and/or unloading of material, generation of audio and/or visual signals and access or blocking of loading areas.
9. A method for controlling the loading and unloading of material to automated analyzer systems, comprising the steps of a. Capturing a video stream in an area next to a loading area of the automated analyzer system; b. Analysing the video stream for the presence of a person c. Recognizing material carried by the person; d. Allowing access to the loading are.
10. 10. The method of claim 9, further comprising the step of identifying the intended use of recognized material by reading our information that is attached to the material.
11. 11 The method of any one of claims 9 or 10, further comprising the step of requesting information regarding the material and/or its intended use.
12. 12. The method of any one of claims 9 to 11, further comprising the step of producing audio and/or visual signal for interaction with the person.