EP4732237A2

EP4732237A2 - System, method, and computer program product for detecting and classifying medical devices for vascular access management

Info

Publication number: EP4732237A2
Application number: EP24826724.7A
Authority: EP
Inventors: Sagar S DEYAGOND; Yashar SEYED VAHEDEIN; Amir Mohammad TAHMASEBI MARAGHOOSH
Original assignee: Becton Dickinson and Co
Current assignee: Becton Dickinson and Co
Priority date: 2023-06-22
Filing date: 2024-06-21
Publication date: 2026-04-29
Also published as: CN121569319A; WO2024263903A2; WO2024263903A3

Abstract

A system, method, and computer program product may process, with a first machine learning model, at least one image to detect a plurality of medical devices therein and to provide a prediction of a device class associated with each medical device detected; identify, based on the prediction of the device class associated with each medical device, one or more medical devices for further classification; and for each medical device identified for further classification: select, based on the prediction of the device class associated with that medical device, a second machine learning model associated with the device class; process, with the selected second machine learning model, a region of interest in the at least one image including that medical device to generate a prediction of a device subclass associated with that medical device; and provide the device class and the device subclass associated with that medical device.

Description

SYSTEM, METHOD, AND COMPUTER PROGRAM PRODUCT FOR DETECTING AND CLASSIFYING MEDICAE DEVICES FOR VASCULAR ACCESS

MANAGEMENT

CROSS-REFERENCE TO RELATED APPLICATION

[0001] The present application claims priority to United States Provisional Application No. 63/509,555 entitled “System, Method, and Computer Program Product for Detecting and Classifying Medical Devices for Vascular Access Management” filed June 22, 2023, the disclosure of which is hereby incorporated by reference in its entirety.

BACKGROUND

[0002] When a patient is admitted to a hospital, a variety of disposable drug delivery devices (e.g., a needleless connector, an IV tubing, an extension set, a catheter, etc.) may be attached to the patient to deliver medication from an infusion pump or syringe to the patient at one or more catheter insertion sites. A single IV line may be constructed from multiple devices. A catheter with multiple lumens may form parts of multiple IV lines, which may be referred to as a “catheter tree”. Nurses may label IV lines with color stickers (or other methods) to track medication infused from each individual pump module of the infusion pump to each IV line.

[0003] Multiple IV lines attached to a patient may have implications on drug compatibility for the patient. Further, during a patient stay, the components of the IV lines may be replaced and/or the line configurations may be changed to accommodate different patient care needs. Nurses may need to keep track of these changes to the IV lines by documenting a dwell time of components and/or connections thereof to ensure line cleanliness and drug compatibility.

SUMMARY

[0004] Accordingly, provided are improved systems, devices, products, apparatus, and/or methods for detecting and classifying medical devices for vascular access management.

[0005] According to some non-limiting embodiments or aspects, provided is a system, including: at least one processor coupled to a memory and configured to: process, with a first machine learning model, at least one image to detect a plurality of medical devices in the at least one image and to provide a prediction of a device class associated with each medical device of the plurality of medical devices, wherein the first machine learning model receives, as input, the at least one image and provides, as output, for each medical device of the plurality of medical devices detected in the at least one image, a region of interest in the at least one image including that medical device and the prediction of the device class associated with that medical device; identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices, one or more medical devices of the plurality of medical devices for further classification; and for each medical device of the one or more medical devices identified for further classification: select, based on the prediction of the device class associated with that medical device, a second machine learning model associated with the device class from a plurality of second machine learning models associated with the plurality of device classes; process, with the selected second machine learning model, the region of interest in the at least one image including that medical device, wherein the selected second machine learning model receives, as input, the region of interest in the at least one image including that medical device and provides, as output, a prediction of a device subclass associated with that medical device; and provide the device class and the device subclass associated with that medical device.

[0006] In some non-limiting embodiments or aspects, the at least one processor is further configured to: identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices, at least one medical device of the plurality of medical devices for no further classification; and for each medical device of the at least one medical device identified for no further classification: provide the device class associated with that medical device.

[0007] In some non-limiting embodiments or aspects, the at least one processor is further configured to: determine, based on the device class and the device subclass associated with each medical device of the one or more medical devices identified for further classification, at least one first medical device of the plurality of medical devices as a subcomponent of a second medical device of the plurality of medical devices.

[0008] In some non-limiting embodiments or aspects, the at least one processor is further configured to: modify, based on at least one predetermined rule associated with the device subclass associated with the second medical device, a first device subclass associated with the at least one first medical device of the plurality of medical devices identified as the subcomponent to a second device subclass different than the first device subclass.

[0009] In some non-limiting embodiments or aspects, the at least one first medical device of the plurality of medical devices identified as the subcomponent includes at least two first medical devices identified as at least two subcomponents of the second medical device, and wherein the at least one processor is further configured to: merge, based on at least one predetermined rule associated with the device subclass associated with the second medical device and a distance between the at least two first medical devices, the at least two first medical devices identified as the at least two subcomponents of the second medical device into a single first medical device identified as a single subcomponent of the second medical device. [0010] In some non-limiting embodiments or aspects, for each second machine learning model of the plurality of second machine learning models, the plurality of device subclasses includes a plurality of first device subclasses associated with the device class and at least one second device subclass associated with an indication that the device class is incorrect.

[0011] In some non-limiting embodiments or aspects, the at least one processor is further configured to: for each medical device of the one or more medical devices identified for further classification, in response to the selected second machine learning model providing, as output, the device subclass associated with that medical device as the at least one second device subclass associated with the indication that the device class is incorrect, process, with the first machine learning model, the region of interest in the at least one image including that medical device, wherein the first machine learning model receives, as input, the region of interest in the at least one image including that medical device and a flag indicating that the medical device is to be classified in a different device class than the device class associated with that medical device.

[0012] According to some non-limiting embodiments or aspects, provided is a method, including: processing, with at least one processor, with a first machine learning model, at least one image to detect a plurality of medical devices in the at least one image and to provide a prediction of a device class associated with each medical device of the plurality of medical devices, wherein the first machine learning model receives, as input, the at least one image and provides, as output, for each medical device of the plurality of medical devices detected in the at least one image, a region of interest in the at least one image including that medical device and the prediction of the device class associated with that medical device; identifying, with the at least one processor, based on the prediction of the device class associated with each medical device of the plurality of medical devices, one or more medical devices of the plurality of medical devices for further classification; and for each medical device of the one or more medical devices identified for further classification: selecting, with the at least one processor, based on the prediction of the device class associated with that medical device, a second machine learning model associated with the device class from a plurality of second machine learning models associated with the plurality of device classes; processing, with the at least one processor, with the selected second machine learning model, the region of interest in the at least one image including that medical device, wherein the selected second machine learning model receives, as input, the region of interest in the at least one image including that medical device and provides, as output, a prediction of a device subclass associated with that medical device; and providing, with the at least one processor, the device class and the device subclass associated with that medical device.

[0013] In some non-limiting embodiments or aspects, the method further includes: identifying, with the at least one processor, based on the prediction of the device class associated with each medical device of the plurality of medical devices, at least one medical device of the plurality of medical devices for no further classification; and for each medical device of the at least one medical device identified for no further classification: providing, with the at least one processor, the device class associated with that medical device.

[0014] In some non-limiting embodiments or aspects, the method further includes: determining, with the at least one processor, based on the device class and the device subclass associated with each medical device of the one or more medical devices identified for further classification, at least one first medical device of the plurality of medical devices as a subcomponent of a second medical device of the plurality of medical devices.

[0015] In some non-limiting embodiments or aspects, the method further includes: modifying, with the at least one processor, based on at least one predetermined rule associated with the device subclass associated with the second medical device, a first device subclass associated with the at least one first medical device of the plurality of medical devices identified as the subcomponent to a second device subclass different than the first device subclass.

[0016] In some non-limiting embodiments or aspects, the at least one first medical device of the plurality of medical devices identified as the subcomponent includes at least two first medical devices identified as at least two subcomponents of the second medical device, and wherein the method further includes: merging, with the at least one processor, based on at least one predetermined rule associated with the device subclass associated with the second medical device and a distance between the at least two first medical devices, the at least two first medical devices identified as the at least two subcomponents of the second medical device into a single first medical device identified as a single subcomponent of the second medical device. [0017] In some non-limiting embodiments or aspects, for each second machine learning model of the plurality of second machine learning models, the plurality of device subclasses includes a plurality of first device subclasses associated with the device class and at least one second device subclass associated with an indication that the device class is incorrect.

[0018] In some non-limiting embodiments or aspects, the method further includes: for each medical device of the one or more medical devices identified for further classification, in response to the selected second machine learning model providing, as output, the device subclass associated with that medical device as the at least one second device subclass associated with the indication that the device class is incorrect, processing, with the at least one processor, with the first machine learning model, the region of interest in the at least one image including that medical device, wherein the first machine learning model receives, as input, the region of interest in the at least one image including that medical device and a flag indicating that the medical device is to be classified in a different device class than the device class associated with that medical device.

[0019] According to some non-limiting embodiments or aspects, provided is a computer program product including at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: process, with a first machine learning model, at least one image to detect a plurality of medical devices in the at least one image and to provide a prediction of a device class associated with each medical device of the plurality of medical devices, wherein the first machine learning model receives, as input, the at least one image and provides, as output, for each medical device of the plurality of medical devices detected in the at least one image, a region of interest in the at least one image including that medical device and the prediction of the device class associated with that medical device; identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices, one or more medical devices of the plurality of medical devices for further classification; and for each medical device of the one or more medical devices identified for further classification: select, based on the prediction of the device class associated with that medical device, a second machine learning model associated with the device class from a plurality of second machine learning models associated with the plurality of device classes; process, with the selected second machine learning model, the region of interest in the at least one image including that medical device, wherein the selected second machine learning model receives, as input, the region of interest in the at least one image including that medical device and provides, as output, a prediction of a device subclass associated with that medical device; and provide the device class and the device subclass associated with that medical device.

[0020] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to: identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices, at least one medical device of the plurality of medical devices for no further classification; and for each medical device of the at least one medical device identified for no further classification: provide the device class associated with that medical device.

[0021] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to: determine, based on the device class and the device subclass associated with each medical device of the one or more medical devices identified for further classification, at least one first medical device of the plurality of medical devices as a subcomponent of a second medical device of the plurality of medical devices.

[0022] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to: modify, based on at least one predetermined rule associated with the device subclass associated with the second medical device, a first device subclass associated with the at least one first medical device of the plurality of medical devices identified as the subcomponent to a second device subclass different than the first device subclass.

[0023] In some non-limiting embodiments or aspects, the at least one first medical device of the plurality of medical devices identified as the subcomponent includes at least two first medical devices identified as at least two subcomponents of the second medical device, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: merge, based on at least one predetermined rule associated with the device subclass associated with the second medical device and a distance between the at least two first medical devices, the at least two first medical devices identified as the at least two subcomponents of the second medical device into a single first medical device identified as a single subcomponent of the second medical device.

[0024] In some non-limiting embodiments or aspects, for each second machine learning model of the plurality of second machine learning models, the plurality of device subclasses includes a plurality of first device subclasses associated with the device class and at least one second device subclass associated with an indication that the device class is incorrect, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each medical device of the one or more medical devices identified for further classification, in response to the selected second machine learning model providing, as output, the device subclass associated with that medical device as the at least one second device subclass s associated with the indication that the device class is incorrect, process, with the first machine learning model, the region of interest in the at least one image including that medical device, wherein the first machine learning model receives, as input, the region of interest in the at least one image including that medical device and a flag indicating that the medical device is to be classified in a different device class than the device class associated with that medical device.

[0025] Further non-limiting embodiments or aspects are set forth in the following numbered clauses:

[0026] Clause 1. A system, comprising: at least one processor coupled to a memory and configured to: process, with a first machine learning model, at least one image to detect a plurality of medical devices in the at least one image and to provide a prediction of a device class associated with each medical device of the plurality of medical devices, wherein the first machine learning model receives, as input, the at least one image and provides, as output, for each medical device of the plurality of medical devices detected in the at least one image, a region of interest in the at least one image including that medical device and the prediction of the device class associated with that medical device; identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices, one or more medical devices of the plurality of medical devices for further classification; and for each medical device of the one or more medical devices identified for further classification: select, based on the prediction of the device class associated with that medical device, a second machine learning model associated with the device class from a plurality of second machine learning models associated with the plurality of device classes; process, with the selected second machine learning model, the region of interest in the at least one image including that medical device, wherein the selected second machine learning model receives, as input, the region of interest in the at least one image including that medical device and provides, as output, a prediction of a device subclass associated with that medical device; and provide the device class and the device subclass associated with that medical device.

[0027] Clause 2. The system of clause 1, wherein the at least one processor is further configured to: identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices, at least one medical device of the plurality of medical devices for no further classification; and for each medical device of the at least one medical device identified for no further classification: provide the device class associated with that medical device.

[0028] Clause 3. The system of any of clause 1 or clause 2, wherein the at least one processor is further configured to: determine, based on the device class and the device subclass associated with each medical device of the one or more medical devices identified for further classification, at least one first medical device of the plurality of medical devices as a subcomponent of a second medical device of the plurality of medical devices.

[0029] Clause 4. The system of any of clauses 1-3, wherein the at least one processor is further configured to: modify, based on at least one predetermined rule associated with the device subclass associated with the second medical device, a first device subclass associated with the at least one first medical device of the plurality of medical devices identified as the subcomponent to a second device subclass different than the first device subclass.

[0030] Clause 5. The system of any of clauses 1-4, wherein the at least one first medical device of the plurality of medical devices identified as the subcomponent includes at least two first medical devices identified as at least two subcomponents of the second medical device, and wherein the at least one processor is further configured to: merge, based on at least one predetermined rule associated with the device subclass associated with the second medical device and a distance between the at least two first medical devices, the at least two first medical devices identified as the at least two subcomponents of the second medical device into a single first medical device identified as a single subcomponent of the second medical device. [0031] Clause 6. The system of any of clauses 1-5, wherein, for each second machine learning model of the plurality of second machine learning models, the plurality of device subclasses includes a plurality of first device subclasses associated with the device class and at least one second device subclass associated with an indication that the device class is incorrect. [0032] Clause 7. The system of any of clauses 1-6, wherein the at least one processor is further configured to: for each medical device of the one or more medical devices identified for further classification, in response to the selected second machine learning model providing, as output, the device subclass associated with that medical device as the at least one second device subclass associated with the indication that the device class is incorrect, process, with the first machine learning model, the region of interest in the at least one image including that medical device, wherein the first machine learning model receives, as input, the region of interest in the at least one image including that medical device and a flag indicating that the medical device is to be classified in a different device class than the device class associated with that medical device.

[0033] Clause 8. A method, comprising: processing, with at least one processor, with a first machine learning model, at least one image to detect a plurality of medical devices in the at least one image and to provide a prediction of a device class associated with each medical device of the plurality of medical devices, wherein the first machine learning model receives, as input, the at least one image and provides, as output, for each medical device of the plurality of medical devices detected in the at least one image, a region of interest in the at least one image including that medical device and the prediction of the device class associated with that medical device; identifying, with the at least one processor, based on the prediction of the device class associated with each medical device of the plurality of medical devices, one or more medical devices of the plurality of medical devices for further classification; and for each medical device of the one or more medical devices identified for further classification: selecting, with the at least one processor, based on the prediction of the device class associated with that medical device, a second machine learning model associated with the device class from a plurality of second machine learning models associated with the plurality of device classes; processing, with the at least one processor, with the selected second machine learning model, the region of interest in the at least one image including that medical device, wherein the selected second machine learning model receives, as input, the region of interest in the at least one image including that medical device and provides, as output, a prediction of a device subclass associated with that medical device; and providing, with the at least one processor, the device class and the device subclass associated with that medical device.

[0034] Clause 9. The method of clause 8, further comprising: Identifying, with the at least one processor, based on the prediction of the device class associated with each medical device of the plurality of medical devices, at least one medical device of the plurality of medical devices for no further classification; and for each medical device of the at least one medical device identified for no further classification: providing, with the at least one processor, the device class associated with that medical device.

[0035] Clause 10. The method of any of clause 8 or clause 9, further comprising: determining, with the at least one processor, based on the device class and the device subclass associated with each medical device of the one or more medical devices identified for further classification, at least one first medical device of the plurality of medical devices as a subcomponent of a second medical device of the plurality of medical devices.

[0036] Clause 11. The method of any of clauses 8-10, further comprising: modifying, with the at least one processor, based on at least one predetermined rule associated with the device subclass associated with the second medical device, a first device subclass associated with the at least one first medical device of the plurality of medical devices identified as the subcomponent to a second device subclass different than the first device subclass.

[0037] Clause 12. The method of any of clauses 8-11, wherein the at least one first medical device of the plurality of medical devices identified as the subcomponent includes at least two first medical devices identified as at least two subcomponents of the second medical device, and wherein the method further comprises: merging, with the at least one processor, based on at least one predetermined rule associated with the device subclass associated with the second medical device and a distance between the at least two first medical devices, the at least two first medical devices identified as the at least two subcomponents of the second medical device into a single first medical device identified as a single subcomponent of the second medical device.

[0038] Clause 13. The method of any of clauses 8-12, wherein, for each second machine learning model of the plurality of second machine learning models, the plurality of device subclasses includes a plurality of first device subclasses associated with the device class and at least one second device subclass associated with an indication that the device class is incorrect. [0039] Clause 14. The method of any of clauses 8-13, further comprising: for each medical device of the one or more medical devices identified for further classification, in response to the selected second machine learning model providing, as output, the device subclass associated with that medical device as the at least one second device subclass associated with the indication that the device class is incorrect, processing, with the at least one processor, with the first machine learning model, the region of interest in the at least one image including that medical device, wherein the first machine learning model receives, as input, the region of interest in the at least one image including that medical device and a flag indicating that the medical device is to be classified in a different device class than the device class associated with that medical device.

[0040] Clause 15. A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: process, with a first machine learning model, at least one image to detect a plurality of medical devices in the at least one image and to provide a prediction of a device class associated with each medical device of the plurality of medical devices, wherein the first machine learning model receives, as input, the at least one image and provides, as output, for each medical device of the plurality of medical devices detected in the at least one image, a region of interest in the at least one image including that medical device and the prediction of the device class associated with that medical device; identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices, one or more medical devices of the plurality of medical devices for further classification; and for each medical device of the one or more medical devices identified for further classification: select, based on the prediction of the device class associated with that medical device, a second machine learning model associated with the device class from a plurality of second machine learning models associated with the plurality of device classes; process, with the selected second machine learning model, the region of interest in the at least one image including that medical device, wherein the selected second machine learning model receives, as input, the region of interest in the at least one image including that medical device and provides, as output, a prediction of a device subclass associated with that medical device; and provide the device class and the device subclass associated with that medical device.

[0041] Clause 16. The computer program product of clause 15, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices, at least one medical device of the plurality of medical devices for no further classification; and for each medical device of the at least one medical device identified for no further classification: provide the device class associated with that medical device.

[0042] Clause 17. The computer program product of any of clause 15 or clause 16, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: determine, based on the device class and the device subclass associated with each medical device of the one or more medical devices identified for further classification, at least one first medical device of the plurality of medical devices as a subcomponent of a second medical device of the plurality of medical devices.

[0043] Clause 18. The computer program product of any of clauses 15-17, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: modify, based on at least one predetermined rule associated with the device subclass associated with the second medical device, a first device subclass associated with the at least one first medical device of the plurality of medical devices identified as the subcomponent to a second device subclass different than the first device subclass.

[0044] Clause 19. The computer program product of any of clauses 15-18, wherein the at least one first medical device of the plurality of medical devices identified as the subcomponent includes at least two first medical devices identified as at least two subcomponents of the second medical device, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: merge, based on at least one predetermined rule associated with the device subclass associated with the second medical device and a distance between the at least two first medical devices, the at least two first medical devices identified as the at least two subcomponents of the second medical device into a single first medical device identified as a single subcomponent of the second medical device. [0045] Clause 20. The computer program product of any of clauses 15-19, wherein, for each second machine learning model of the plurality of second machine learning models, the plurality of device subclasses includes a plurality of first device subclasses associated with the device class and at least one second device subclass associated with an indication that the device class is incorrect, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each medical device of the one or more medical devices identified for further classification, in response to the selected second machine learning model providing, as output, the device subclass associated with that medical device as the at least one second device subclass s associated with the indication that the device class is incorrect, process, with the first machine learning model, the region of interest in the at least one image including that medical device, wherein the first machine learning model receives, as input, the region of interest in the at least one image including that medical device and a flag indicating that the medical device is to be classified in a different device class than the device class associated with that medical device.

BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Additional advantages and details are explained in greater detail below with reference to the exemplary embodiments that are illustrated in the accompanying schematic figures, in which:

[0047] FIG. 1A is a diagram of non-limiting embodiments or aspects of an environment in which systems, devices, products, apparatus, and/or methods, described herein, can be implemented;

[0048] FIG. IB is a diagram of non-limiting embodiments or aspects of an implementation of an environment in which systems, devices, products, apparatus, and/or methods, described herein, can be implemented;

[0049] FIG. 2 is a diagram of non-limiting embodiments or aspects of components of one or more devices and/or one or more systems of FIGS. 1A and IB;

[0050] FIG. 3 is a flow chart of non-limiting embodiments or aspects of a process for vascular access management;

[0051] FIG. 4 is a flow chart of an implementation of non-limiting embodiments or aspects of a process for vascular access management;

[0052] FIG. 5A and 5B are images of an example catheter insertion site;

[0053] FIGS. 6 A and 6B are images of an example catheter;

[0054] FIGS. 7 A and 7B are images of an example catheter; and [0055] FIGS. 8 A and 8B are images of an example catheter insertion site.

DETAILED DESCRIPTION

[0056] It is to be understood that the present disclosure may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary and non-limiting embodiments or aspects. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered as limiting.

[0057] For purposes of the description hereinafter, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and derivatives thereof shall relate to embodiments or aspects as they are oriented in the drawing figures. However, it is to be understood that embodiments or aspects may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply non-limiting exemplary embodiments or aspects. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects of the embodiments or aspects disclosed herein are not to be considered as limiting unless otherwise indicated.

[0058] No aspect, component, element, structure, act, step, function, instruction, and/or the like used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items, and may be used interchangeably with “one or more” and “at least one.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and may be used interchangeably with “one or more” or “at least one.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise.

[0059] As used herein, the terms “communication” and “communicate” may refer to the reception, receipt, transmission, transfer, provision, and/or the like of information (e.g., data, signals, messages, instructions, commands, and/or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and/or the like) to be in communication with another unit means that the one unit is able to directly or indirectly receive information from and/or transmit information to the other unit. This may refer to a direct or indirect connection that is wired and/or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and/or routed between the first and second unit. For example, a first unit may be in communication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit (e.g., a third unit located between the first unit and the second unit) processes information received from the first unit and communicates the processed information to the second unit. In some non-limiting embodiments or aspects, a message may refer to a network packet (e.g., a data packet and/or the like) that includes data. It will be appreciated that numerous other arrangements are possible.

[0060] As used herein, the term “computing device” may refer to one or more electronic devices that are configured to directly or indirectly communicate with or over one or more networks. A computing device may be a mobile or portable computing device, a desktop computer, a server, and/or the like. Furthermore, the term “computer” may refer to any computing device that includes the necessary components to receive, process, and output data, and normally includes a display, a processor, a memory, an input device, and a network interface. A “computing system” may include one or more computing devices or computers. An “application” or “application program interface” (API) refers to computer code or other data sorted on a computer-readable medium that may be executed by a processor to facilitate the interaction between software components, such as a client-side front-end and/or server-side back-end for receiving data from the client. An “interface” refers to a generated display, such as one or more graphical user interfaces (GUIs) with which a user may interact, either directly or indirectly (e.g., through a keyboard, mouse, touchscreen, etc.). Further, multiple computers, e.g., servers, or other computerized devices directly or indirectly communicating in the network environment may constitute a “system” or a “computing system”.

[0061] It will be apparent that systems and/or methods, described herein, can be implemented in different forms of hardware, software, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and/or methods is not limiting of the implementations. Thus, the operation and behavior of the systems and/or methods are described herein without reference to specific software code, it being understood that software and hardware can be designed to implement the systems and/or methods based on the description herein. [0062] Some non-limiting embodiments or aspects are described herein in connection with thresholds. As used herein, satisfying a threshold may refer to a value being greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, fewer than the threshold, lower than the threshold, less than or equal to the threshold, equal to the threshold, etc.

[0063] Existing object detection methods are typically based on a single model architecture with a large training dataset used to make the model’s performance more accurately, which may result in a lot of time and effort wasted on data collection, labeling, and training of the models.

[0064] Non-limiting embodiments or aspects of the present disclosure may provide systems, methods, and computer program products for detecting and classifying medical devices for vascular access management that process, with a first machine learning model, at least one image to detect a plurality of medical devices in the at least one image and to provide a prediction of a device class associated with each medical device of the plurality of medical devices, wherein the first machine learning model receives, as input, the at least one image and provides, as output, for each medical device of the plurality of medical devices detected in the at least one image, a region of interest in the at least one image including that medical device and the prediction of the device class associated with that medical device; identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices, one or more medical devices of the plurality of medical devices for further classification; and for each medical device of the one or more medical devices identified for further classification: select, based on the prediction of the device class associated with that medical device, a second machine learning model associated with the device class from a plurality of second machine learning models associated with the plurality of device classes; process, with the selected second machine learning model, the region of interest in the at least one image including that medical device, wherein the selected second machine learning model receives, as input, the region of interest in the at least one image including that medical device and provides, as output, a prediction of a device subclass associated with that medical device; and provide the device class and the device subclass associated with that medical device.

[0065] In this way, non-limiting embodiments or aspects of the present disclosure may use multiple models in cascade to make performance full-proof with certain predefined information using filtering logics and hierarchical outputs of the object of interest to detect and classify non-tagged medical devices used for vascular access management (e.g., non-tagged catheter components, such as hubs, lumens, needleless connectors, stopcocks, extension sets, and/or the like, etc.). This data augmentation may make the system more dependable for the model performance and accuracy.

[0066] Referring now to FIG. 1A, FIG. 1A is a diagram of an example environment 100 in which devices, systems, methods, and/or products described herein, may be implemented. As shown in FIG. 1A, environment 100 includes user device 102, management system 104, and/or communication network 106. Systems and/or devices of environment 100 can interconnect via wired connections, wireless connections, or a combination of wired and wireless connections. [0067] Referring also to FIG. IB, FIG. IB is a diagram of non-limiting embodiments or aspects of an implementation of environment 100 in which systems, devices, products, apparatus, and/or methods, described herein, can be implemented. For example, as shown in FIG. IB, environment 100 may include a hospital room including a patient, one or more medical devices 108, and/or a caretaker (e.g., a nurse, etc.).

[0068] User device 102 may include one or more devices capable of receiving information and/or data from management system 104 (e.g., via communication network 106, etc.) and/or communicating information and/or data to management system 104 (e.g., via communication network 106, etc.). For example, user device 106 may include one or more computing systems including one or more processors (e.g., one or more computing devices, one or more server computers, one or more mobile computing devices, one or more tablet computers, etc.). In some non-limiting embodiments or aspects, user device 102 may include a tablet computer or mobile computing device, such as an Apple® iPad, an Apple® iPhone, an Android® tablet, an Android® phone, and/or the like.

[0069] User device 102 may include one or more image capture devices (e.g., one or more cameras, one or more sensors, etc.) configured to capture one or more images of an environment (e.g., environment 100, etc.) surrounding the one or more image capture devices. For example, user device 102 may include one or more image capture devices configured to capture one or more images of the one or more medical devices 108 and/or the patient. As an example, an image capture device of user device 102 may include at least one of the following: a plurality of image capture devices, a monocular camera, a stereo camera, a color camera configured to capture and/or detect one or more predetermined wavelengths of light, a camera including a filter configured to filter a predetermined wavelength of light, an infrared (IR) camera (e.g., a sensor configured to detect a changes in temperatures based on which heat and/or movement may be detected, etc.), a thermal sensor configured to capture thermal images of syringe 103 illuminated by infrared wavelengths of a light source, a pan, tilt, and zoom (PTZ) camera including a variable field-of-view (FOV) and an automatic zoom function, a master and slave camera system including a static camera and a dynamic camera, a camera including a filter configured to filter a predetermined wavelength of light, a LiDAR system, a RADAR system, a microwave sensor (a sensor configured to detect movement of objects by emitting microwave pulses and measuring reflected microwaves of a moving object, etc.), an ultrasonic motion sensor, and/or the like, or any combination thereof.

[0070] Management system 104 may include one or more devices capable of receiving information and/or data from user device 102 (e.g., via communication network 106, etc.) and/or communicating information and/or data to user device 102 (e.g., via communication network 110, etc.). For example, management system 104 may include one or more computing systems including one or more processors (e.g., one or more computing devices, one or more server computers, one or more mobile computing devices, etc.). In some non-limiting embodiments or aspects, management system 104 includes and/or is accessible via a nurse station or terminal in a hospital. For example, management system 104 may provide bedside nurse support, nursing station manager support, retrospective reporting for nursing administration, and/or the like.

[0071] Communication network 106 may include one or more wired and/or wireless networks. For example, communication network 110 may include a cellular network (e.g., a long-term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a sixth generation (6G) network a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber optic -based network, a cloud computing network, and/or the like, and/or a combination of these or other types of networks.

[0072] A medical device 108 may include at least one of the following types of medical devices: a peripheral IV catheter (PIVC), a peripherally inserted central catheter (PICC), a midline catheter, a central venous catheter (CVC), a dual port catheter, a single port catheter, a diffusics catheter, a Power midline dual catheter, a Power midline single catheter, a poly midline catheter, a PICC single catheter, a PICC dual catheter, a PICC triple catheter, a PICC solo dual catheter, a PICC solo single catheter, a Teleflex CVC quad catheter, a Teleflex CVC triple catheter, a needleless or needle free connector (NFC), a catheter dressing, a catheter stabilization device, a disinfectant cap, a disinfectant swab or wipe, an IV tubing set, an extension set, a Y connector, a stopcock, an infusion pump, a flush syringe, a medication delivery syringe, an IV fluid bag, a lumen adapter (e.g., a number of lumen adapters associated with a catheter may indicate a number of lumens included in the catheter, etc.), a pentagon tag (e.g., including fiducial markers, etc.) or any combination thereof.

[0073] The number and arrangement of systems and devices shown in FIGS. 1A and IB are provided as an example. There can be additional systems and/or devices, fewer systems and/or devices, different systems and/or devices, or differently arranged systems and/or devices than those shown in FIGS. 1A and IB. Furthermore, two or more systems or devices shown in FIGS. 1A and IB can be implemented within a single system or a single device, or a single system or a single device shown in FIGS. 1A and IB can be implemented as multiple, distributed systems or devices. Additionally, or alternatively, a set of systems or a set of devices (e.g., one or more systems, one or more devices, etc.) of environment 100 can perform one or more functions described as being performed by another set of systems or another set of devices of environment 100.

[0074] Referring now to FIG. 2, FIG. 2 is a diagram of example components of a device 200. Device 200 may correspond to user device 102 (e.g., one or more devices of a system of user device 102, etc.) and/or one or more devices of management system 104. In some non-limiting embodiments or aspects, user device 102 (e.g., one or more devices of a system of user device 102, etc.) and/or one or more devices of management system 104 may include at least one device 200 and/or at least one component of device 200. As shown in FIG. 2, device 200 may include bus 202, processor 204, memory 206, storage component 208, input component 210, output component 212, and communication interface 214.

[0075] Bus 202 may include a component that permits communication among the components of device 200. In some non-limiting embodiments or aspects, processor 204 may be implemented in hardware, software, or a combination of hardware and software. For example, processor 204 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and/or any processing component (e.g., a field-programmable gate array (FPGA), an application- specific integrated circuit (ASIC), etc.) that can be programmed to perform a function. Memory 206 may include random access memory (RAM), read-only memory (ROM), and/or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and/or instructions for use by processor 204.

[0076] Storage component 208 may store information and/or software related to the operation and use of device 200. For example, storage component 208 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid state disk, etc.), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, and/or another type of computer-readable medium, along with a corresponding drive.

[0077] Input component 210 may include a component that permits device 200 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). Additionally or alternatively, input component 210 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). Output component 212 may include a component that provides output information from device 200 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.).

[0078] Communication interface 214 may include a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 214 may permit device 200 to receive information from another device and/or provide information to another device. For example, communication interface 214 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, and/or the like.

[0079] Device 200 may perform one or more processes described herein. Device 200 may perform these processes based on processor 204 executing software instructions stored by a computer-readable medium, such as memory 206 and/or storage component 208. A computer- readable medium (e.g., a non-transitory computer-readable medium) is defined herein as a non- transitory memory device. A memory device includes memory space located inside of a single physical storage device or memory space spread across multiple physical storage devices.

[0080] Software instructions may be read into memory 206 and/or storage component 208 from another computer-readable medium or from another device via communication interface 214. When executed, software instructions stored in memory 206 and/or storage component 208 may cause processor 204 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments or aspects described herein are not limited to any specific combination of hardware circuitry and software. [0081] Memory 206 and/or storage component 208 may include data storage or one or more data structures (e.g., a database, etc.). Device 200 may be capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage or one or more data structures in memory 206 and/or storage component 208.

[0082] The number and arrangement of components shown in FIG. 2 are provided as an example. In some non-limiting embodiments or aspects, device 200 may include additional components, fewer components, different components, or differently arranged components than those shown in FIG. 2. Additionally or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200.

[0083] Referring now to FIG. 3, FIG. 3 is a flowchart of non-limiting embodiments or aspects of a process 300 for detecting and classifying medical devices for vascular access management. In some non-limiting embodiments or aspects, one or more of the steps of process 4300 may be performed (e.g., completely, partially, etc.) by user device 102 (e.g., one or more devices of a system of user device 102, etc.). In some non-limiting embodiments or aspects, one or more of the steps of process 300 may be performed (e.g., completely, partially, etc.) by another device or a group of devices separate from or including user device 102, such as management system 104 (e.g., one or more devices of management system 104, etc.).

[0084] As shown in FIG. 3, at step 302, process 300 includes obtaining at least one image. For example, user device 102 may obtain at least one image (e.g., a single image, a plurality of images, a series of images, etc.). As an example, user device 102 may obtain at least one image including a plurality of medical devices 108. In such an example, the at least one image may be captured by an image capture device, such as an image capture device of user device 102. For example, a nurse may use user device 102 to take one or more images of a catheter site of a patient and/or an infusion pump connected to the catheter site.

[0085] As shown in FIG. 3, at step 304, process 300 includes processing, with a first machine learning model, at least one image to detect a plurality of medical devices in the at least one image and to provide a prediction of a device class associated with each medical device of the plurality of medical devices. For example, user device 102 may process, with a first machine learning model, the at least one image to detect a plurality of medical devices 108 in the at least one image and to provide a prediction of a device class associated with each medical device of the plurality of medical devices 108. As an example, the first machine learning model may receive, as input, the at least one image and provide, as output, for each medical device of the plurality of medical devices 108 detected in the at least one image, a region of interest in the at least one image including that medical device (e.g., bounding box coordinates in a given region of interest in the at least one image, etc.) and the prediction of the device class (e.g., an object class, a class of catheter components that are part of a same or similar family, etc.) associated with that medical device from a plurality of device classes. In such an example, the device class may be selected from a plurality of device classes.

[0086] A first machine learning model may include a detector and/or a classifier model. For example, the plurality of medical devices 108 may be identified by user device 102 processing the at least one image using one or more object detection techniques (e.g., a deep learning technique, an image processing technique, and/or an image segmentation technique, etc.) to identify or detect the plurality of medical devices 108 in the at least one image (and/or position information associated with a 3D position of the identified or detected medical devices 108 relative to an image capture device that captured the at least one image, for example, including x, y, and z coordinates of the medical devices 108, directional vectors for Z, Y, and X axes of the medical devices 108, and/or the 2D positions of the identified or detected medical devices 108 in the at least one image itself). For example, a deep learning technique may include a bounding box technique that generates a region or interest or a box label for objects (e.g., medical devices 108, etc.) of interest in images, an image masking technique (e.g., masked FRCNN (RCNN or CNN) that captures specific shapes of objects (e.g., medical devices 108, etc.) in images, a trained neural network that identifies objects (e.g., medical devices 108, etc.) in images, a classifier that classifies identified objects into classes or types of the objects (e.g., into medical device classes, etc.), and/or the like. As an example, an image processing technique may include a cross correlation image processing technique, an image contrasting technique, a binary or colored filtering technique, and/or the like. As an example, different catheter lumens may include unique colors that can be used by the image processing to identify a type of the catheter.

[0087] User device 102 may generate a detector/classifier machine learning model using machine learning techniques including, for example, supervised and/or unsupervised techniques, such as decision trees (e.g., gradient boosted decision trees, random forests, etc.), logistic regressions, artificial neural networks (e.g., convolutional neural networks, etc.), Bayesian statistics, learning automata, Hidden Markov Modeling, linear classifiers, quadratic classifiers, association rule learning, and/or the like. The detector/classifier machine learning model may be trained to provide an output including, for each medical device detected in the at least one image, a region of interest in the at least one image including that medical device and the prediction of the device class associated with that medical device. In such an example, the prediction may include a probability (e.g., a likelihood, etc.) that the medical device is associated with that device class. User device 102 may generate the detector/classifier machine model based on a plurality of training images (e.g., training data, etc.). In some implementations, the detector/classifier machine learning model is designed to receive, as an input, at least one image and provide, as an output, for each medical device detected in the at least one image, a region of interest in the at least one image including that medical device and a prediction (e.g., a probability, a likelihood, a binary output, a yes-no output, a score, a prediction score, a classification, etc.) as the device class of a plurality of device classes associated with that medical device. In some non-limiting embodiments or aspects, user device 102 stores the detector/classifier machine learning model (e.g., stores the model for later use). In some non-limiting embodiments or aspects, user device 102 stores the detector/classifier machine learning model in a data structure (e.g., a database, a linked list, a tree, etc.). In some non-limiting embodiments, the data structure is located within user device 102 or external (e.g., remote from) user device 102 (e.g., within management system 104, etc.).

[0088] Referring also to FIG. 4, which is a is a flow chart of an implementation 400 of nonlimiting embodiments or aspects of a process for vascular access management, a first machine learning model (e.g., a detector/classifier machine learning model, etc.) may be configured to classify each medical device detected in the at least one image into a device class of a plurality of device classes associated with vascular access management device. For example, the first machine learning model (e.g., the detector/classifier machine learning model, etc.) may be configured to classify each medical device detected in the at least one image into one of the following device classes: a needle free connector (NFC) class; a stopcock class; an extension class; a catheter class; a cap class, a y-site class; a pentagon tag class, a lumen class, and/or the like. As an example, and referring also to FIG. 5A, which is an image of an example catheter insertion site, user device 102 may display the at least one image with the regions of interest in the at least one image including the detected medical devices superimposed thereon (e.g., with the bounding boxes surrounding the medical devices in the at least one image, etc.) and with the prediction of the device classes (e.g., the class of catheter components in which the medical device is classified, etc.) associated with the respective medical devices displayed adjacent thereto.

[0089] In some non-limiting embodiments or aspects, user device 102 may resize the at least one image with padding, for example, to a predetermined resolution, before processing the at least one image with the first machine learning model. In some non-limiting embodiments or aspects, the first machine learning model may use non-maximum suppression (e.g., a computer vision method that selects a single entity out of many overlapping entities, for example bounding boxes in object detection, and discards entities that are below a given probability bound, etc.) to detect the plurality of medical devices in the at least one image, each of which may be assigned respective bound box coordinates in the at least one image identifying the region of interest in the at least one image including that medical device for further processing in a batch inference by selected second machine learning models of the plurality of second machine learning models as described herein in more detail.

[0090] As shown in FIG. 3, at step 306, process 300 includes identifying, based on the prediction of the device class associated with each medical device of the plurality of medical devices, one or more medical devices of the plurality of medical devices for further classification. For example, user device 102 may identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices 108, one or more medical devices of the plurality of medical devices for further classification. As an example, user device 102 may identify for further classification (e.g., for subclass prediction, etc.) one or more medical devices that are associated with the device classes for which a second machine learning model (e.g., a specific device-type classifier, etc.) is available. In such an example, the second machine learning model associated with the device class may be selected from a plurality of second machine learning models associated with the plurality of device classes.

[0091] In some non-limiting embodiments or aspects, user device 102 may identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices, at least one medical device of the plurality of medical devices for no further classification. As an example, user device 102 may identify for no further classification (e.g., for no subclass prediction, etc.) at least one medical device that is associated with the device classes for which a second machine learning model (e.g., a specific device-type classifier, etc.) is not available. However, non-limiting embodiments or aspects are not limited thereto and, in some on-limiting embodiments or aspects, a second machine learning model (e.g., a specific device-type classifier, etc.) may be available for each device class of the plurality of device classes.

[0092] For example, and referring again to FIG. 4, user device 102 may identify medical devices associated with the NFC class, the catheter class, and/or the lumen class for further classification. As an example, the plurality of second machine learning models may include a NFC-type classifier, a catheter-type classifier, and/or a lumen color classifier. For example, user device 102 may identify medical devices associated with the stopcock class, the extension class, the cap class, the y-site class, and/or the pentagon tag class for no further classification. As an example, the plurality of second machine learning models may not include a stopcock- type classifier, an extension-type classifier, a cap-type classifier, a y-site type classifier, and/or a pentagon tag-type classifier.

[0093] As shown in FIG. 3, at step 308, process 300 includes, for each medical device of the one or more medical devices identified for further classification, selecting a second machine learning model and processing the region of interest including that medical device with the second machine learning model to predict a device subclass associated with that medical device. For example, user device 102 may, for each medical device of the one or more medical devices identified for further classification: select, based on the prediction of the device class associated with that medical device, a second machine learning model associated with the device class from a plurality of second machine learning models associated with the plurality of device classes; and process, with the selected second machine learning model, the region of interest in the at least one image including that medical device. As an example, the selected second machine learning model may receive, as input, the region of interest in the at least one image including that medical device and provide, as output, a prediction of a device subclass associated with that medical device (e.g., a classification of the identified bounding box from the first machine learning module into a next granular class level, a further classification into a subclass based on various features, such as color, geometry, texture, and/or the like, to identify a more granular subclass than the device class, etc.).

[0094] A second machine learning model may include a device-type classifier model. For example, a plurality of second machine learning models may include a specific device-type classifier model for one or more device classes of a plurality of device classes (e.g., a NFC- type classifier model for a NFC class, a catheter-type classifier model for a catheter class, etc.). The second machine learning model may use one or more object detection techniques (e.g., a deep learning technique, an image processing technique, and/or an image segmentation technique, etc.) to identify the medical device in the region of interest. As an example, a deep learning technique may include a bounding box technique that generates a region or interest or a box label for objects (e.g., medical devices 108, etc.) of interest in images, an image masking technique (e.g., masked FRCNN (RCNN or CNN) that captures specific shapes of objects (e.g., medical devices 108, etc.) in images, a trained neural network that identifies objects (e.g., medical devices 108, etc.) in images, a classifier that classifies identified objects into subclasses or sub-types of the objects (e.g., into medical device subclasses, etc.), and/or the like. As an example, an image processing technique may include a cross correlation image processing technique, an image contrasting technique, a binary or colored filtering technique, and/or the like. As an example, different catheter lumens may include unique colors that can be used by the image processing to identify a type of the catheter.

[0095] User device 102 may generate a device-type classifier machine learning model using machine learning techniques including, for example, supervised and/or unsupervised techniques, such as decision trees (e.g., gradient boosted decision trees, random forests, etc.), logistic regressions, artificial neural networks (e.g., convolutional neural networks, etc.), Bayesian statistics, learning automata, Hidden Markov Modeling, linear classifiers, quadratic classifiers, association rule learning, and/or the like. The device-type classifier machine learning model may be trained to provide an output including a prediction of a device subclass associated with a medical device associated with a specific device class. In such an example, the prediction may include a probability (e.g., a likelihood, etc.) that the medical device is associated with that device subclass. User device 102 may generate the device-type classifier machine learning model based on a plurality of training images or regions of interest (e.g., training data, etc.) associated with medical devices having a same device class as that of the device-type classifier machine learning model. In some implementations, the device-type classifier machine learning model is designed to receive, as an input, a region of interest in at least one image including a medical device and provides, as output, a prediction e.g., a probability, a likelihood, a binary output, a yes-no output, a score, a prediction score, a classification, etc.) of a device subclass associated with the medical device. In some nonlimiting embodiments or aspects, user device 102 stores the detector/classifier machine learning model (e.g., stores the model for later use). In some non-limiting embodiments or aspects, user device 102 stores the device-type classifier machine learning model in a data structure (e.g., a database, a linked list, a tree, etc.). In some non-limiting embodiments, the data structure is located within user device 102 or external (e.g., remote from) user device 102 (e.g., within management system 104, etc.).

[0096] In this way, the first machine learning model and the selected second machine learning model may be used in combination for a hierarchical and cascaded architecture to detect and classify the objects of interest in a catheter setup for vascular access management. These models may be assisted by filtering logic and/or algorithms that may include predefined information from a user and/or predetermined catheter configurations, and/or which may enable further reducing detection and classification errors during pre/post processing, thereby improving overall model accuracy.

[0097] For example, and referring again to FIG. 4, user device 102 may process a region of interest including a detected medical device assigned to the NFC class with a NFC-type classifier, process a region of interest including a detected medical device assigned to the catheter class with a catheter-type classifier, and/or process a region of interest including a detected medical device assigned to the lumen class with a lumen color classifier. As an example, the plurality of second machine learning models may include a NFC-type classifier, a catheter-type classifier, and/or a lumen color classifier. In such an example, the NFC-type classifier may be configured to classify each medical device associated with or assigned to the NFC class into a device subclass of a plurality of device subclasses associated with the NFC class. For example, the NFC-type classifier may be configured to classify each medical device associated with or assigned to the NFC class into one of the following device subclasses: a tagged subclass, an untagged subclass, a SmartSite subclass, an NFC_others subclass (e.g., a subclass associated with an indication that the device class of NFC for the medical device is incorrect, etc.), and/or the like. In such an example, the catheter-type classifier may be configured to classify each medical device associated with or assigned to the catheter class into a device subclass of a plurality of device subclasses associated with the catheter class. For example, the catheter- type classifier may be configured to classify each medical device associated with or assigned to the catheter class into one of the following device subclasses: a Nexiva dual port subclass, a Nexiva single port subclass, a Nexiva diffusics subclass, a Power midline dual subclass, a Power midline single subclass, a poly midline subclass, a PowerPICC single subclass, a PowerPICC dual subclass, a PowerPICC triple subclass, a PowerPICC solo dual subclass, a PowerPICC solo single subclass, a Teleflex CVC quad subclass, a Teleflex CVC triple subclass, a Cathether_others subclass (e.g., a subclass associated with an indication that the device class of catheter for the medical device is incorrect, etc.), and/or the like. In such an example, the lumen color classifier may be configured to classify each medical device associated with or assigned to the lumen class into a device subclass of a plurality of device subclasses associated with the lumen class. For example, the lumen color classifier may be configured to classify each medical device associated with or assigned to the lumen class into one of the following device subclasses: a red color subclass, a purple color subclass, a green color subclass, a yellow color subclass, a blue color subclass, a gray color subclass, a white color subclass, a pink color subclass, a brown color subclass, a lumen_others subclass (e.g., a subclass associated with an indication that the device class of lumen for the medical device is incorrect, etc.), and/or the like.

[0098] In some non-limiting embodiments or aspects, for each second machine learning model of the plurality of second machine learning models, the plurality of device subclasses includes a plurality of first device subclasses associated with the device class and at least one second device subclass associated with an indication that the device class is incorrect. For example, and referring again to FIG. 4, the NFC-type classifier may include a NFC_others subclass (e.g., a subclass associated with an indication that the device class of NFC for the medical device is incorrect, etc.), the catheter-type classifier may include a Cathether_others subclass (e.g., a subclass associated with an indication that the device class of catheter for the medical device is incorrect, etc.), and/or the lumen color classifier may include a lumen_others subclass (e.g., a subclass associated with an indication that the device class of lumen for the medical device is incorrect, etc.). As an example, non-limiting embodiments or aspects may improve a performance of the first machine learning model (e.g., the detector/classifier machine learning model, etc.) by introducing a mechanism in the plurality of second machine learning models (e.g., in the sub-class classifiers, etc.), because if the first machine learning model makes a mistake in identifying or classifying a medical device in a device class (e.g., confuses a Needle-free Connector for a lumen, etc.), the corresponding second machine learning model for lumen color classification may cannot recover from that mistake. To address this shortcoming, a subclass associated with an indication that the device class for the medical device is incorrect may be provided for each second machine learning model fo the plurality of second machine learning models. The input labeled data for this newly introduced class may be a random combination of super-class classes that are most confused with the desired object.

[0099] As an example, for each medical device of the one or more medical devices identified for further classification, in response to the selected second machine learning model providing, as output, the device subclass associated with that medical device as the at least one second device subclass associated with the indication that the device class is incorrect, user device 102 may process (e.g., re-process, etc.), with the first machine learning model, the region of interest in the at least one image including that medical device, wherein the first machine learning model receives, as input, the region of interest in the at least one image including that medical device and a flag indicating that the medical device is to be classified in a different device class than the device class associated with that medical device. For example, if training a second machine learning model including a subclass classifier for “lumen color”, the classifier may expect to receive a bounding box including a lumen as an input from the first machine learning model. By analyzing the performance of the first machine learning model (e.g., by reviewing the confusion matrix, etc.) it can be determined which classes are most confused with the “lumen” class. For example, assuming “NFC” and “Y-site” are the classes that are most confused with the “Lumen” class, at a time of training the second machine learning model including the subclass classifier for lumen color, a new class may be introduced (e.g., “nonlumen” or “others” class, etc.), and to train this design with N+l classes (where N: number of lumen colors), a stratified or uniform sample of “NFC” and “Ysite” classes may be randomly selected and used as training data for the N+l sub-class classifier. In such an example, the output label for this new class (e.g., a mix of NFC and Ysite bounding box images) may be “non-lumen” or “others”.

[0100] For example, and referring to FIGS. 8 A and 8B, are images of an example catheter insertion site, and still referring to FIG. 4, an NFC is falsely classified as a lumen and inside the subclass classifier for lumen color the NFC is further classified as a blue lumen. However, as described herein, the subclass color classification model can be designed to include a class called “other”, and this “other” class may classify objects with confidence below a threshold confidence in the other class and feedback the region of interest or bounding box to the first machine learning model (e.g., to step 304 of FIG. 3, etc.) to be reclassified as a next probable option or device class (e.g., the NFC in FIG. 8B, etc.), which may enable non-limiting embodiments or aspects to have a second layer of self-correction in case the first machine learning model incorrectly predicts the device class of a medical device.

[0101] As shown in FIG. 3, at step 310, process 300 includes determining at least one first medical device of the plurality of medical devices as a subcomponent of a second medical device of the plurality of medical devices. For example, the first machine learning model may be trained to automatically group at least one first medical device of the plurality of medical devices as a subcomponent of a second medical device of the plurality of medical devices. As an example, user device 102 may, determine, based on the device class and the device subclass associated with each medical device of the one or more medical devices identified for further classification, the device class associated with each medical device of the at least one medical device identified for no further classification, and/or distances between the plurality of medical devices, at least one first medical device of the plurality of medical devices as a subcomponent of a second medical device of the plurality of medical devices. As an example, user device 102 may apply the at least one predetermined rule and/or filter logic to determine subcomponents of detected medical devices, and based on distances between medical devices that are determined to be possible co-components of a same medical device and/or thresholds therefor, determine at least one first medical device of the plurality of medical devices as a subcomponent of a second medical device of the plurality of medical devices.

[0102] As shown in FIG. 3, at step 312, process 300 includes merging and/or modifying the at least one first medical device of the plurality of medical devices identified as a subcomponent according to at least one predetermined rule. For example, user device 102 may merge and/or modify he at least one first medical device of the plurality of medical devices identified as a subcomponent according to at least one predetermined rule.

[0103] In some non-limiting embodiments or aspects, user device 102 may modify, based on at least one predetermined rule associated with the device subclass associated with the second medical device, a first device subclass associated with the at least one first medical device of the plurality of medical devices identified as the subcomponent to a second device subclass different than the first device subclass. For example, and referring also to FIGS. 6A and 6B, which are images of an example catheter, the at least one predetermined rule may indicate that a PICC dual lumen catheter always has a purple lumen and a red lumen as subcomponents. If the selected second machine learning model incorrectly predicts one of the lumens as a falsenegative or a false-positive of another color (e.g., a brown lumen as shown in FIG. 6A, etc.), user device 102 may apply the at least one predetermined rule or filtering logic to correct the color subclass of the lumen (e.g., to a red lumen as shown in FIG. 6B, etc.).

[0104] In some non-limiting embodiments or aspects, the at least one first medical device of the plurality of medical devices identified as the subcomponent includes at least two first medical devices identified as at least two subcomponents of the second medical device. For example, user device 102 may merge, based on at least one predetermined rule associated with the device subclass associated with the second medical device and a distance between the at least two first medical devices, the at least two first medical devices identified as the at least two subcomponents of the second medical device into a single first medical device identified as a single subcomponent of the second medical device. As an example, and referring also to FIGS. 7 A and 7B, which are images of an example catheter, the at least one predetermined rule may indicate that a Poly midline single catheter always includes a single gray lumen as a subcomponent. If the first machine learning model and/or the selected second machine learning model incorrectly predicts a Poly midline single catheter including two gray lumens as shown in FIG. 7A, user device 102 may apply the at least one predetermined rule or filtering logic to understand that there is no chance of two lumens on this type of catheter, determine a Euclidean distance between the two bounding boxes or regions of interest for the two lumens, and determine, based on the determined distance satisfying a distance threshold, to merge the two bounding boxes or regions of interest into a single bounding box or region of interest including a single lumen with the appropriate lumen class name as shown in FIG. 7B. In this way, the predetermined rules or filtering logic can be expanded to combination of catheter components and a probability of the proximity of subcomponents. [0105] As shown in FIG. 3, at step 314, process 300 includes providing a device class and/or a device subclass for each medical device of the plurality of medical devices detected in the at least one image. For example, user device 102 may provide a device class and/or a device subclass for each medical device of the plurality of medical devices detected in the at least one image. As an example, user device 102 may, for each medical device of the one or more medical devices identified for further classification, provide the device class and the device subclass associated with that medical device. As an example, user device 102 may, for each medical device of the at least one medical device identified for no further classification, provide the device class associated with that medical device. In such an example, and referring also to FIG. 5B, which is an image of an example catheter insertion site, user device 102 may display the at least one image with the regions of interest in the at least one image including the detected medical devices superimposed thereon (e.g., with the bounding boxes surrounding the medical devices in the at least one image, etc.) and with the predictions of the device classes and subclasses associated with the respective medical devices displayed adjacent thereto.

[0106] In some non-limiting embodiments or aspects, user device 102 may use the device class and the device subclass associated with each medical device and/or their respective regions of interest or bounding boxes to determine pairs of medical devices that are connected to each other, to generate a representation of one or more IV lines to, issue an alert and/or the infusion pump to stop fluid flow and/or adjust fluid flow, and/or to determine a dwell time of medical devices 108 (e.g., in environment 100, etc.) and/or connections thereof (e.g., an amount or duration of time a medical device is connected to another medical device and/or the patient, etc.), as described in Indian Provisional Patent Application No. 202211042859, filed July 26, 2022, the contents of which is hereby incorporated by reference in its entirety.

[0107] Although embodiments or aspects have been described in detail for the purpose of illustration and description, it is to be understood that such detail is solely for that purpose and that embodiments or aspects are not limited to the disclosed embodiments or aspects, but, on the contrary, are intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect can be combined with one or more features of any other embodiment or aspect. In fact, many of these features can be combined in ways not specifically recited in the claims and/or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

Claims

WHAT IS CLAIMED IS:

1. A system, comprising: at least one processor coupled to a memory and configured to: process, with a first machine learning model, at least one image to detect a plurality of medical devices in the at least one image and to provide a prediction of a device class associated with each medical device of the plurality of medical devices, wherein the first machine learning model receives, as input, the at least one image and provides, as output, for each medical device of the plurality of medical devices detected in the at least one image, a region of interest in the at least one image including that medical device and the prediction of the device class associated with that medical device; identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices, one or more medical devices of the plurality of medical devices for further classification; and for each medical device of the one or more medical devices identified for further classification: select, based on the prediction of the device class associated with that medical device, a second machine learning model associated with the device class from a plurality of second machine learning models associated with the plurality of device classes; process, with the selected second machine learning model, the region of interest in the at least one image including that medical device, wherein the selected second machine learning model receives, as input, the region of interest in the at least one image including that medical device and provides, as output, a prediction of a device subclass associated with that medical device; and provide the device class and the device subclass associated with that medical device.

2. The system of claim 1, wherein the at least one processor is further configured to: identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices, at least one medical device of the plurality of medical devices for no further classification; and for each medical device of the at least one medical device identified for no further classification: provide the device class associated with that medical device.

3. The system of claim 1, wherein the at least one processor is further configured to: determine, based on the device class and the device subclass associated with each medical device of the one or more medical devices identified for further classification, at least one first medical device of the plurality of medical devices as a subcomponent of a second medical device of the plurality of medical devices.

4. The system of claim 3, wherein the at least one processor is further configured to: modify, based on at least one predetermined rule associated with the device subclass associated with the second medical device, a first device subclass associated with the at least one first medical device of the plurality of medical devices identified as the subcomponent to a second device subclass different than the first device subclass.

5. The system of claim 3, wherein the at least one first medical device of the plurality of medical devices identified as the subcomponent includes at least two first medical devices identified as at least two subcomponents of the second medical device, and wherein the at least one processor is further configured to: merge, based on at least one predetermined rule associated with the device subclass associated with the second medical device and a distance between the at least two first medical devices, the at least two first medical devices identified as the at least two subcomponents of the second medical device into a single first medical device identified as a single subcomponent of the second medical device.

6. The system of claim 1 , wherein, for each second machine learning model of the plurality of second machine learning models, the plurality of device subclasses includes a plurality of first device subclasses associated with the device class and at least one second device subclass associated with an indication that the device class is incorrect.

7. The system of claim 6, wherein the at least one processor is further configured to: for each medical device of the one or more medical devices identified for further classification, in response to the selected second machine learning model providing, as output, the device subclass associated with that medical device as the at least one second device subclass associated with the indication that the device class is incorrect, process, with the first machine learning model, the region of interest in the at least one image including that medical device, wherein the first machine learning model receives, as input, the region of interest in the at least one image including that medical device and a flag indicating that the medical device is to be classified in a different device class than the device class associated with that medical device.

8. A method, comprising: processing, with at least one processor, with a first machine learning model, at least one image to detect a plurality of medical devices in the at least one image and to provide a prediction of a device class associated with each medical device of the plurality of medical devices, wherein the first machine learning model receives, as input, the at least one image and provides, as output, for each medical device of the plurality of medical devices detected in the at least one image, a region of interest in the at least one image including that medical device and the prediction of the device class associated with that medical device; identifying, with the at least one processor, based on the prediction of the device class associated with each medical device of the plurality of medical devices, one or more medical devices of the plurality of medical devices for further classification; and for each medical device of the one or more medical devices identified for further classification: selecting, with the at least one processor, based on the prediction of the device class associated with that medical device, a second machine learning model associated with the device class from a plurality of second machine learning models associated with the plurality of device classes; processing, with the at least one processor, with the selected second machine learning model, the region of interest in the at least one image including that medical device, wherein the selected second machine learning model receives, as input, the region of interest in the at least one image including that medical device and provides, as output, a prediction of a device subclass associated with that medical device; and providing, with the at least one processor, the device class and the device subclass associated with that medical device.

9. The method of claim 8, further comprising: identifying, with the at least one processor, based on the prediction of the device class associated with each medical device of the plurality of medical devices, at least one medical device of the plurality of medical devices for no further classification; and for each medical device of the at least one medical device identified for no further classification: providing, with the at least one processor, the device class associated with that medical device.

10. The method of claim 8, further comprising: determining, with the at least one processor, based on the device class and the device subclass associated with each medical device of the one or more medical devices identified for further classification, at least one first medical device of the plurality of medical devices as a subcomponent of a second medical device of the plurality of medical devices.

11. The method of claim 10, further comprising: modifying, with the at least one processor, based on at least one predetermined rule associated with the device subclass associated with the second medical device, a first device subclass associated with the at least one first medical device of the plurality of medical devices identified as the subcomponent to a second device subclass different than the first device subclass.

12. The method of claim 10, wherein the at least one first medical device of the plurality of medical devices identified as the subcomponent includes at least two first medical devices identified as at least two subcomponents of the second medical device, and wherein the method further comprises: merging, with the at least one processor, based on at least one predetermined rule associated with the device subclass associated with the second medical device and a distance between the at least two first medical devices, the at least two first medical devices identified as the at least two subcomponents of the second medical device into a single first medical device identified as a single subcomponent of the second medical device.

13. The method of claim 8, wherein, for each second machine learning model of the plurality of second machine learning models, the plurality of device subclasses includes a plurality of first device subclasses associated with the device class and at least one second device subclass associated with an indication that the device class is incorrect.

14. The method of claim 13, further comprising: for each medical device of the one or more medical devices identified for further classification, in response to the selected second machine learning model providing, as output, the device subclass associated with that medical device as the at least one second device subclass associated with the indication that the device class is incorrect, processing, with the at least one processor, with the first machine learning model, the region of interest in the at least one image including that medical device, wherein the first machine learning model receives, as input, the region of interest in the at least one image including that medical device and a flag indicating that the medical device is to be classified in a different device class than the device class associated with that medical device.

15. A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: process, with a first machine learning model, at least one image to detect a plurality of medical devices in the at least one image and to provide a prediction of a device class associated with each medical device of the plurality of medical devices, wherein the first machine learning model receives, as input, the at least one image and provides, as output, for each medical device of the plurality of medical devices detected in the at least one image, a region of interest in the at least one image including that medical device and the prediction of the device class associated with that medical device; identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices, one or more medical devices of the plurality of medical devices for further classification; and for each medical device of the one or more medical devices identified for further classification: select, based on the prediction of the device class associated with that medical device, a second machine learning model associated with the device class from a plurality of second machine learning models associated with the plurality of device classes; process, with the selected second machine learning model, the region of interest in the at least one image including that medical device, wherein the selected second machine learning model receives, as input, the region of interest in the at least one image including that medical device and provides, as output, a prediction of a device subclass associated with that medical device; and provide the device class and the device subclass associated with that medical device.

16. The computer program product of claim 15, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: identify, based on the prediction of the device class associated with each medical device of the plurality of medical devices, at least one medical device of the plurality of medical devices for no further classification; and for each medical device of the at least one medical device identified for no further classification: provide the device class associated with that medical device.

17. The computer program product of claim 15, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: determine, based on the device class and the device subclass associated with each medical device of the one or more medical devices identified for further classification, at least one first medical device of the plurality of medical devices as a subcomponent of a second medical device of the plurality of medical devices.

18. The computer program product of claim 17, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: modify, based on at least one predetermined rule associated with the device subclass associated with the second medical device, a first device subclass associated with the at least one first medical device of the plurality of medical devices identified as the subcomponent to a second device subclass different than the first device subclass.

19. The computer program product of claim 17, wherein the at least one first medical device of the plurality of medical devices identified as the subcomponent includes at least two first medical devices identified as at least two subcomponents of the second medical device, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: merge, based on at least one predetermined rule associated with the device subclass associated with the second medical device and a distance between the at least two first medical devices, the at least two first medical devices identified as the at least two subcomponents of the second medical device into a single first medical device identified as a single subcomponent of the second medical device.

20. The computer program product of claim 15, wherein, for each second machine learning model of the plurality of second machine learning models, the plurality of device subclasses includes a plurality of first device subclasses associated with the device class and at least one second device subclass associated with an indication that the device class is incorrect, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each medical device of the one or more medical devices identified for further classification, in response to the selected second machine learning model providing, as output, the device subclass associated with that medical device as the at least one second device subclass s associated with the indication that the device class is incorrect, process, with the first machine learning model, the region of interest in the at least one image including that medical device, wherein the first machine learning model receives, as input, the region of interest in the at least one image including that medical device and a flag indicating that the medical device is to be classified in a different device class than the device class associated with that medical device.