EP4732239A2 - System, method, and computer program product for tracking real-time syringe volume - Google Patents

Abstract

A system, method, and computer program product for tracking real-time syringe volume may obtain at least two images of a syringe including a fiducial marker, the at least two images including an initial image and a current image, and the fiducial marker encapsulating syringe data; for each image of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and segment, based on a color threshold for that image, the region of interest in that image to generate a masked image; determine, based on a pixel-to-volume mapping associated with the syringe, the masked image generated from the initial image, and the masked image generated from the current image, a volume of a fluid dispensed from and/or remaining in the syringe; and provide the volume of a fluid dispensed from and/or remaining in the syringe.

Description

SYSTEM, METHOD, AND COMPUTER PROGRAM PRODUCT FOR TRACKING REAU-TIME SYRINGE VOEUME

CROSS-REFERENCE TO RELATED APPLICATION

[0001] The present application claims priority to United States Provisional Application No. 63/509,607 entitled “System, Method, and Computer Program Product for Tracking Real-Time Syringe Volume” filed June 22, 2023, the disclosure of which is hereby incorporated by reference in its entirety.

BACKGROUND

[0002] Wrong dose and wrong administration rate are two common intravenous (IV) push medication errors that can have significant impacts on patient safety and recovery.

SUMMARY

[0003] Accordingly, provided are improved systems, devices, products, apparatuses, and/or methods for tracking real-time syringe volume.

[0004] According to some non-limiting embodiments or aspects, provided is a system, including: at least one processor coupled to a memory and configured to: obtain at least two images of a syringe including a fiducial marker, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the fiducial marker encapsulates syringe data associated with the syringe; determine, based on the fiducial marker in at least one image of the syringe of the at least two images of the syringe, a pixel-to-volume mapping associated with the syringe; for each image of the syringe of the at least two images of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and segment, based on a color threshold for that image, the region of interest in that image to generate a masked image; determine, based on the pixel-to-volume mapping associated with the syringe, the masked image generated from the initial image of the syringe, and the masked image generated from the current image of the syringe, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and provide the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof. [0005] In some non-limiting embodiments or aspects, the syringe includes at least one color calibration marker including at least one color, wherein a moveable component of the syringe includes the at least one color, and wherein the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: generate, based on the at least one color of the at least one color calibration marker in that image, the color threshold for that image.

[0006] In some non-limiting embodiments or aspects, the at least one color calibration marker includes at least two color calibration markers including a first color calibration marker including a first color and a second color calibration marker including a second color different than the first color, wherein a first portion of the moveable component of the syringe includes the first color, wherein a second portion of the moveable component of the syringe different than the first portion includes the second color, and wherein the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: generate, based on the first color of the first color calibration marker in that image and the second color of the second color calibration marker in that image, the color threshold for that image.

[0007] In some non-limiting embodiments or aspects, the at least two images of the syringe include a first color space, and wherein the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: convert that image from including the first color space to including a second color space different than the first color space, wherein the color threshold for that image is generated based on the at least one color of the at least one color calibration marker in that converted image, and wherein the region of interest in that converted image is segmented based on the color threshold for that image to generate the masked image.

[0008] In some non-limiting embodiments or aspects, the at least one processor is further configured to determine the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof by: for each image of the syringe of the at least two images of the syringe: determining at least one centroid pixel of at least one plurality of pixels in the masked image that satisfied the color threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to- volume mapping associated with the syringe, the at least one centroid pixel of the masked image generated from the initial image of the syringe, and the at least one centroid pixel of the masked image generated from the current image of the syringe.

[0009] In some non-limiting embodiments or aspects, the at least one processor is further configured to determine, for each image of the syringe of the at least two images of the syringe, at least two centroid pixels of at least two pluralities of pixels in the masked image that satisfied the color threshold for that image including a first centroid pixel of a first plurality of pixels that satisfied a first color range threshold of the color threshold for that image and a second centroid pixel of a second plurality of pixels that satisfied a second color range threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to- volume mapping associated with the syringe, the first centroid pixel of the masked image generated from the initial image of the syringe, the second centroid pixel of the masked image generated from the initial image of the syringe, the first centroid pixel of the masked image generated from the current image of the syringe, and the second centroid pixel of the masked image generated from the current image of the syringe.

[0010] In some non-limiting embodiments or aspects, the pixel-to- volume mapping includes a hard-coded or predetermined pixel-to-volume mapping.

[0011] In some non-limiting embodiments or aspects, the pixel-to-volume mapping includes a machine learning model configured to generate the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof based on the masked image generated from the initial image of the syringe and the masked image generated from the current image of the syringe.

[0012] In some non-limiting embodiments or aspects, the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: control a display to display that image of the syringe with the masked image generated from that image overlaid on that image in the display.

[0013] In some non-limiting embodiments or aspects, the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: determine, based on the fiducial marker in that image, position information associated with a three- dimensional (3D) position of the syringe relative to an image capture device that captured the image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

[0014] According to some non-limiting embodiments or aspects, provided is a method, including: obtaining, with at least one processor, at least two images of a syringe including a fiducial marker, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the fiducial marker encapsulates syringe data associated with the syringe; determining, with the at least one processor, based on the fiducial marker in at least one image of the syringe of the at least two images of the syringe, a pixel-to-volume mapping associated with the syringe; for each image of the syringe of the at least two images of the syringe: determining, with the at least one processor, based on the fiducial marker in that image, a region of interest in that image; and segmenting, with the at least one processor, based on a color threshold for that image, the region of interest in that image to generate a masked image; determining, with the at least one processor, based on the pixel-to-volume mapping associated with the syringe, the masked image generated from the initial image of the syringe, and the masked image generated from the current image of the syringe, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and providing, with the at least one processor, the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

[0015] In some non-limiting embodiments or aspects, the syringe includes at least one color calibration marker including at least one color, wherein a moveable component of the syringe includes the at least one color, and wherein the method further comprises: for each image of the syringe of the at least two images of the syringe: generating, with the at least one processor, based on the at least one color of the at least one color calibration marker in that image, the color threshold for that image.

[0016] In some non-limiting embodiments or aspects, the at least one color calibration marker includes at least two color calibration markers including a first color calibration marker including a first color and a second color calibration marker including a second color different than the first color, wherein a first portion of the moveable component of the syringe includes the first color, wherein a second portion of the moveable component of the syringe different than the first portion includes the second color, and wherein the method further comprises: for each image of the syringe of the at least two images of the syringe: generating, with the at least one processor, based on the first color of the first color calibration marker in that image and the second color of the second color calibration marker in that image, the color threshold for that image.

[0017] In some non-limiting embodiments or aspects, the at least two images of the syringe include a first color space, and wherein the method further comprises: for each image of the syringe of the at least two images of the syringe: converting, with the at least one processor, that image from including the first color space to including a second color space different than the first color space, wherein the color threshold for that image is generated based on the at least one color of the at least one color calibration marker in that converted image, and wherein the region of interest in that converted image is segmented based on the color threshold for that image to generate the masked image.

[0018] In some non-limiting embodiments or aspects, determining the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof further includes: for each image of the syringe of the at least two images of the syringe: determining, with the at least one processor, at least one centroid pixel of at least one plurality of pixels in the masked image that satisfied the color threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to-volume mapping associated with the syringe, the at least one centroid pixel of the masked image generated from the initial image of the syringe, and the at least one centroid pixel of the masked image generated from the current image of the syringe. [0019] In some non-limiting embodiments or aspects, determining the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof further includes: for each image of the syringe of the at least two images of the syringe: determining, with the at least one processor, at least two centroid pixels of at least two pluralities of pixels in the masked image that satisfied the color threshold for that image including a first centroid pixel of a first plurality of pixels that satisfied a first color range threshold of the color threshold for that image and a second centroid pixel of a second plurality of pixels that satisfied a second color range threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel- to-volume mapping associated with the syringe, the first centroid pixel of the masked image generated from the initial image of the syringe, the second centroid pixel of the masked image generated from the initial image of the syringe, the first centroid pixel of the masked image generated from the current image of the syringe, and the second centroid pixel of the masked image generated from the current image of the syringe.

[0020] In some non-limiting embodiments or aspects, the pixel-to-volume mapping includes a hard-coded or predetermined pixel-to-volume mapping.

[0021] In some non-limiting embodiments or aspects, the pixel-to-volume mapping includes a machine learning model configured to generate the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof based on the masked image generated from the initial image of the syringe and the masked image generated from the current image of the syringe. [0022] In some non-limiting embodiments or aspects, the method further includes: for each image of the syringe of the at least two images of the syringe: controlling, with the at least one processor, a display to display that image of the syringe with the masked image generated from that image overlaid on that image in the display.

[0023] In some non-limiting embodiments or aspects, the method further includes: for each image of the syringe of the at least two images of the syringe: determining, with the at least one processor, based on the fiducial marker in that image, position information associated with a three-dimensional (3D) position of the syringe relative to an image capture device that captured the image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

[0024] According to some non-limiting embodiments or aspects, provided is a computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: obtain at least two images of a syringe including a fiducial marker, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the fiducial marker encapsulates syringe data associated with the syringe; determine, based on the fiducial marker in at least one image of the syringe of the at least two images of the syringe, a pixel-to-volume mapping associated with the syringe; for each image of the syringe of the at least two images of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and segment, based on a color threshold for that image, the region of interest in that image to generate a masked image; determine, based on the pixel-to-volume mapping associated with the syringe, the masked image generated from the initial image of the syringe, and the masked image generated from the current image of the syringe, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and provide the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

[0025] In some non-limiting embodiments or aspects, the syringe includes at least one color calibration marker including at least one color, wherein a moveable component of the syringe includes the at least one color, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: generate, based on the at least one color of the at least one color calibration marker in that image, the color threshold for that image.

[0026] In some non-limiting embodiments or aspects, the at least one color calibration marker includes at least two color calibration markers including a first color calibration marker including a first color and a second color calibration marker including a second color different than the first color, wherein a first portion of the moveable component of the syringe includes the first color, wherein a second portion of the moveable component of the syringe different than the first portion includes the second color, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: generate, based on the first color of the first color calibration marker in that image and the second color of the second color calibration marker in that image, the color threshold for that image.

[0027] In some non-limiting embodiments or aspects, the at least two images of the syringe include a first color space, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: convert that image from including the first color space to including a second color space different than the first color space, wherein the color threshold for that image is generated based on the at least one color of the at least one color calibration marker in that converted image, and wherein the region of interest in that converted image is segmented based on the color threshold for that image to generate the masked image.

[0028] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to determine the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof by: for each image of the syringe of the at least two images of the syringe: determining at least one centroid pixel of at least one plurality of pixels in the masked image that satisfied the color threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to- volume mapping associated with the syringe, the at least one centroid pixel of the masked image generated from the initial image of the syringe, and the at least one centroid pixel of the masked image generated from the current image of the syringe.

[0029] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to determine, for each image of the syringe of the at least two images of the syringe, at least two centroid pixels of at least two pluralities of pixels in the masked image that satisfied the color threshold for that image including a first centroid pixel of a first plurality of pixels that satisfied a first color range threshold of the color threshold for that image and a second centroid pixel of a second plurality of pixels that satisfied a second color range threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to- volume mapping associated with the syringe, the first centroid pixel of the masked image generated from the initial image of the syringe, the second centroid pixel of the masked image generated from the initial image of the syringe, the first centroid pixel of the masked image generated from the current image of the syringe, and the second centroid pixel of the masked image generated from the current image of the syringe.

[0030] In some non-limiting embodiments or aspects, the pixel-to- volume mapping includes a hard-coded or predetermined pixel-to-volume mapping.

[0031] In some non-limiting embodiments or aspects, the pixel-to-volume mapping includes a machine learning model configured to generate the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof based on the masked image generated from the initial image of the syringe and the masked image generated from the current image of the syringe.

[0032] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: control a display to display that image of the syringe with the masked image generated from that image overlaid on that image in the display.

[0033] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: determine, based on the fiducial marker in that image, position information associated with a three-dimensional (3D) position of the syringe relative to an image capture device that captured the image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

[0034] Further non-limiting embodiments or aspects are set forth in the following numbered clauses: [0035] Clause 1. A system, comprising: at least one processor coupled to a memory and configured to: obtain at least two images of a syringe including a fiducial marker, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the fiducial marker encapsulates syringe data associated with the syringe; determine, based on the fiducial marker in at least one image of the syringe of the at least two images of the syringe, a pixel-to-volume mapping associated with the syringe; for each image of the syringe of the at least two images of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and segment, based on a color threshold for that image, the region of interest in that image to generate a masked image; determine, based on the pixel-to-volume mapping associated with the syringe, the masked image generated from the initial image of the syringe, and the masked image generated from the current image of the syringe, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and provide the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

[0036] Clause 2. The system of clause 1, wherein the syringe includes at least one color calibration marker including at least one color, wherein a moveable component of the syringe includes the at least one color, and wherein the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: generate, based on the at least one color of the at least one color calibration marker in that image, the color threshold for that image.

[0037] Clause 3. The system of any of clauses 1 or 2, wherein the at least one color calibration marker includes at least two color calibration markers including a first color calibration marker including a first color and a second color calibration marker including a second color different than the first color, wherein a first portion of the moveable component of the syringe includes the first color, wherein a second portion of the moveable component of the syringe different than the first portion includes the second color, and wherein the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: generate, based on the first color of the first color calibration marker in that image and the second color of the second color calibration marker in that image, the color threshold for that image.

[0038] Clause 4. The system of any of clauses 1-3, wherein the at least two images of the syringe include a first color space, and wherein the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: convert that image from including the first color space to including a second color space different than the first color space, wherein the color threshold for that image is generated based on the at least one color of the at least one color calibration marker in that converted image, and wherein the region of interest in that converted image is segmented based on the color threshold for that image to generate the masked image.

[0039] Clause 5. The system of any of clauses 1-4, wherein the at least one processor is further configured to determine the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof by: for each image of the syringe of the at least two images of the syringe: determining at least one centroid pixel of at least one plurality of pixels in the masked image that satisfied the color threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to-volume mapping associated with the syringe, the at least one centroid pixel of the masked image generated from the initial image of the syringe, and the at least one centroid pixel of the masked image generated from the current image of the syringe. [0040] Clause 6. The system of any of clauses 1-5, wherein the at least one processor is further configured to determine, for each image of the syringe of the at least two images of the syringe, at least two centroid pixels of at least two pluralities of pixels in the masked image that satisfied the color threshold for that image including a first centroid pixel of a first plurality of pixels that satisfied a first color range threshold of the color threshold for that image and a second centroid pixel of a second plurality of pixels that satisfied a second color range threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to-volume mapping associated with the syringe, the first centroid pixel of the masked image generated from the initial image of the syringe, the second centroid pixel of the masked image generated from the initial image of the syringe, the first centroid pixel of the masked image generated from the current image of the syringe, and the second centroid pixel of the masked image generated from the current image of the syringe.

[0041] Clause 7. The system of any of clauses 1-6, wherein the pixel-to-volume mapping includes a hard-coded or predetermined pixel-to-volume mapping.

[0042] Clause 8. The system of any of clauses 1-7, wherein the pixel-to-volume mapping includes a machine learning model configured to generate the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof based on the masked image generated from the initial image of the syringe and the masked image generated from the current image of the syringe. [0043] Clause 9. The system of any of clauses 1-8, wherein the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: control a display to display that image of the syringe with the masked image generated from that image overlaid on that image in the display.

[0044] Clause 10. The system of any of clauses 1-9, wherein the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: determine, based on the fiducial marker in that image, position information associated with a three-dimensional (3D) position of the syringe relative to an image capture device that captured the image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

[0045] Clause 11. A method, comprising: obtaining, with at least one processor, at least two images of a syringe including a fiducial marker, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the fiducial marker encapsulates syringe data associated with the syringe; determining, with the at least one processor, based on the fiducial marker in at least one image of the syringe of the at least two images of the syringe, a pixel-to-volume mapping associated with the syringe; for each image of the syringe of the at least two images of the syringe: determining, with the at least one processor, based on the fiducial marker in that image, a region of interest in that image; and segmenting, with the at least one processor, based on a color threshold for that image, the region of interest in that image to generate a masked image; determining, with the at least one processor, based on the pixel-to-volume mapping associated with the syringe, the masked image generated from the initial image of the syringe, and the masked image generated from the current image of the syringe, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and providing, with the at least one processor, the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof. [0046] Clause 12. The method of clause 11, wherein the syringe includes at least one color calibration marker including at least one color, wherein a moveable component of the syringe includes the at least one color, and wherein the method further comprises: for each image of the syringe of the at least two images of the syringe: generating, with the at least one processor, based on the at least one color of the at least one color calibration marker in that image, the color threshold for that image. [0047] Clause 13. The method of any of clauses 11 or 12, wherein the at least one color calibration marker includes at least two color calibration markers including a first color calibration marker including a first color and a second color calibration marker including a second color different than the first color, wherein a first portion of the moveable component of the syringe includes the first color, wherein a second portion of the moveable component of the syringe different than the first portion includes the second color, and wherein the method further comprises: for each image of the syringe of the at least two images of the syringe: generating, with the at least one processor, based on the first color of the first color calibration marker in that image and the second color of the second color calibration marker in that image, the color threshold for that image.

[0048] Clause 14. The method of any of clauses 11-13, wherein the at least two images of the syringe include a first color space, and wherein the method further comprises: for each image of the syringe of the at least two images of the syringe: converting, with the at least one processor, that image from including the first color space to including a second color space different than the first color space, wherein the color threshold for that image is generated based on the at least one color of the at least one color calibration marker in that converted image, and wherein the region of interest in that converted image is segmented based on the color threshold for that image to generate the masked image.

[0049] Clause 15. The method of any of clauses 11-14, wherein determining the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof further includes: for each image of the syringe of the at least two images of the syringe: determining, with the at least one processor, at least one centroid pixel of at least one plurality of pixels in the masked image that satisfied the color threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to-volume mapping associated with the syringe, the at least one centroid pixel of the masked image generated from the initial image of the syringe, and the at least one centroid pixel of the masked image generated from the current image of the syringe. [0050] Clause 16. The method of any of clauses 11-15, wherein determining the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof further includes: for each image of the syringe of the at least two images of the syringe: determining, with the at least one processor, at least two centroid pixels of at least two pluralities of pixels in the masked image that satisfied the color threshold for that image including a first centroid pixel of a first plurality of pixels that satisfied a first color range threshold of the color threshold for that image and a second centroid pixel of a second plurality of pixels that satisfied a second color range threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel- to-volume mapping associated with the syringe, the first centroid pixel of the masked image generated from the initial image of the syringe, the second centroid pixel of the masked image generated from the initial image of the syringe, the first centroid pixel of the masked image generated from the current image of the syringe, and the second centroid pixel of the masked image generated from the current image of the syringe.

[0051] Clause 17. The method of any of clauses 11-16, wherein the pixel-to-volume mapping includes a hard-coded or predetermined pixel-to-volume mapping.

[0052] Clause 18. The method of any of clauses 11-17, wherein the pixel-to-volume mapping includes a machine learning model configured to generate the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof based on the masked image generated from the initial image of the syringe and the masked image generated from the current image of the syringe.

[0053] Clause 19. The method of any of clauses 11-18, further comprising: for each image of the syringe of the at least two images of the syringe: controlling, with the at least one processor, a display to display that image of the syringe with the masked image generated from that image overlaid on that image in the display.

[0054] Clause 20. The method of any of clauses 11-19, further comprising: for each image of the syringe of the at least two images of the syringe: determining, with the at least one processor, based on the fiducial marker in that image, position information associated with a three-dimensional (3D) position of the syringe relative to an image capture device that captured the image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

[0055] Clause 21. A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: obtain at least two images of a syringe including a fiducial marker, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the fiducial marker encapsulates syringe data associated with the syringe; determine, based on the fiducial marker in at least one image of the syringe of the at least two images of the syringe, a pixel-to-volume mapping associated with the syringe; for each image of the syringe of the at least two images of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and segment, based on a color threshold for that image, the region of interest in that image to generate a masked image; determine, based on the pixel-to-volume mapping associated with the syringe, the masked image generated from the initial image of the syringe, and the masked image generated from the current image of the syringe, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and provide the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

[0056] Clause 22. The computer program product of clause 21, wherein the syringe includes at least one color calibration marker including at least one color, wherein a moveable component of the syringe includes the at least one color, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: generate, based on the at least one color of the at least one color calibration marker in that image, the color threshold for that image.

[0057] Clause 23. The computer program product of any of clauses 21 or 22, wherein the at least one color calibration marker includes at least two color calibration markers including a first color calibration marker including a first color and a second color calibration marker including a second color different than the first color, wherein a first portion of the moveable component of the syringe includes the first color, wherein a second portion of the moveable component of the syringe different than the first portion includes the second color, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: generate, based on the first color of the first color calibration marker in that image and the second color of the second color calibration marker in that image, the color threshold for that image.

[0058] Clause 24. The computer program product of any of clauses 21-23, wherein the at least two images of the syringe include a first color space, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: convert that image from including the first color space to including a second color space different than the first color space, wherein the color threshold for that image is generated based on the at least one color of the at least one color calibration marker in that converted image, and wherein the region of interest in that converted image is segmented based on the color threshold for that image to generate the masked image.

[0059] Clause 25. The computer program product of any of clauses 21-24, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to determine the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof by: for each image of the syringe of the at least two images of the syringe: determining at least one centroid pixel of at least one plurality of pixels in the masked image that satisfied the color threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to-volume mapping associated with the syringe, the at least one centroid pixel of the masked image generated from the initial image of the syringe, and the at least one centroid pixel of the masked image generated from the current image of the syringe. [0060] Clause 26. The computer program product of any of clauses 21-25, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to determine, for each image of the syringe of the at least two images of the syringe, at least two centroid pixels of at least two pluralities of pixels in the masked image that satisfied the color threshold for that image including a first centroid pixel of a first plurality of pixels that satisfied a first color range threshold of the color threshold for that image and a second centroid pixel of a second plurality of pixels that satisfied a second color range threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to-volume mapping associated with the syringe, the first centroid pixel of the masked image generated from the initial image of the syringe, the second centroid pixel of the masked image generated from the initial image of the syringe, the first centroid pixel of the masked image generated from the current image of the syringe, and the second centroid pixel of the masked image generated from the current image of the syringe.

[0061] Clause 27. The computer program product of any of clauses 21-26, wherein the pixel- to-volume mapping includes a hard-coded or predetermined pixel-to-volume mapping.

[0062] Clause 28. The computer program product of any of clauses 21-27, wherein the pixel- to-volume mapping includes a machine learning model configured to generate the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof based on the masked image generated from the initial image of the syringe and the masked image generated from the current image of the syringe. [0063] Clause 29. The computer program product of any of clauses 21-28, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: control a display to display that image of the syringe with the masked image generated from that image overlaid on that image in the display.

[0064] Clause 30. The computer program product of any of clauses 21-29, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: determine, based on the fiducial marker in that image, position information associated with a three-dimensional (3D) position of the syringe relative to an image capture device that captured the image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Additional advantages and details are explained in greater detail below with reference to the exemplary embodiments that are illustrated in the accompanying schematic figures, in which:

[0066] FIG. 1A is a diagram of non-limiting embodiments or aspects of an environment in which systems, devices, products, apparatuses, and/or methods, described herein, can be implemented;

[0067] FIGS. IB and 1C are perspective views of implementations of non-limiting embodiments or aspects of a syringe;

[0068] FIG. 2 is a diagram of non-limiting embodiments or aspects of components of one or more devices and/or one or more systems of FIG. 1 A;

[0069] FIG. 3 is a flow chart of non-limiting embodiments or aspects of a process for tracking real-time syringe volume; and

[0070] FIGS. 4-6 illustrate implementations of non-limiting embodiments or aspects of images of syringes overlaid with corresponding masked images. DETAILED DESCRIPTION

[0071] It is to be understood that the present disclosure may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary and non-limiting embodiments or aspects. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered as limiting.

[0072] For purposes of the description hereinafter, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and derivatives thereof shall relate to embodiments or aspects as they are oriented in the drawing figures. However, it is to be understood that embodiments or aspects may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply non-limiting exemplary embodiments or aspects. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects of the embodiments or aspects disclosed herein are not to be considered as limiting unless otherwise indicated.

[0073] No aspect, component, element, structure, act, step, function, instruction, and/or the like used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items, and may be used interchangeably with “one or more” and “at least one.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and may be used interchangeably with “one or more” or “at least one.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise.

[0074] As used herein, the terms “communication” and “communicate” may refer to the reception, receipt, transmission, transfer, provision, and/or the like of information (e.g., data, signals, messages, instructions, commands, and/or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and/or the like) to be in communication with another unit means that the one unit is able to directly or indirectly receive information from and/or transmit information to the other unit. This may refer to a direct or indirect connection that is wired and/or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and/or routed between the first and second unit. For example, a first unit may be in communication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit (e.g., a third unit located between the first unit and the second unit) processes information received from the first unit and communicates the processed information to the second unit. In some non-limiting embodiments or aspects, a message may refer to a network packet (e.g., a data packet and/or the like) that includes data. It will be appreciated that numerous other arrangements are possible.

[0075] As used herein, the term “computing device” may refer to one or more electronic devices that are configured to directly or indirectly communicate with or over one or more networks. A computing device may be a mobile or portable computing device, a desktop computer, a server, and/or the like. Furthermore, the term “computer” may refer to any computing device that includes the necessary components to receive, process, and output data, and normally includes a display, a processor, a memory, an input device, and a network interface. A “computing system” may include one or more computing devices or computers. An “application” or “application program interface” (API) refers to computer code or other data sorted on a computer-readable medium that may be executed by a processor to facilitate the interaction between software components, such as a client-side front-end and/or server-side back-end for receiving data from the client. An “interface” refers to a generated display, such as one or more graphical user interfaces (GUIs) with which a user may interact, either directly or indirectly (e.g., through a keyboard, mouse, touchscreen, etc.). Further, multiple computers, e.g., servers, or other computerized devices directly or indirectly communicating in the network environment may constitute a “system” or a “computing system”.

[0076] It will be apparent that systems and/or methods, described herein, can be implemented in different forms of hardware, software, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and/or methods is not limiting of the implementations. Thus, the operation and behavior of the systems and/or methods are described herein without reference to specific software code, it being understood that software and hardware can be designed to implement the systems and/or methods based on the description herein.

[0077] Some non-limiting embodiments or aspects are described herein in connection with thresholds. As used herein, satisfying a threshold may refer to a value being greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, fewer than the threshold, lower than the threshold, less than or equal to the threshold, equal to the threshold, etc.

[0078] Referring now to FIG. 1A, FIG. 1A is a diagram of an example environment 100 in which devices, systems, methods, apparatuses, and/or products described herein, may be implemented. As shown in FIG. 1A, environment 100 includes user device 102, syringe 103, management system 104, and/or communication network 106. Systems and/or devices of environment 100 can interconnect via wired connections, wireless connections, or a combination of wired and wireless connections.

[0079] User device 102 may include one or more devices capable of receiving information and/or data from management system 104 (e.g., via communication network 106, etc.) and/or communicating information and/or data to management system 104 (e.g., via communication network 106, etc.). For example, user device 102 may include one or more computing systems including one or more processors (e.g., one or more computing devices, one or more server computers, one or more mobile computing devices, one or more tablet computers, etc.). In some non-limiting embodiments or aspects, user device 102 may include a tablet computer or a mobile computing device, such as an Apple® iPad, an Apple® iPhone, an Android® tablet, an Android® phone, and/or the like.

[0080] User device 102 may include one or more image capture devices (e.g., one or more cameras, one or more sensors, etc.) configured to capture one or more images of an environment (e.g., environment 100, etc.) surrounding the one or more image capture devices. For example, user device 102 may include one or more image capture devices configured to capture one or more images of syringe 103. As an example, an image capture device of user device 102 may include at least one of the following: a plurality of image capture devices, a monocular camera, a stereo camera, a color camera configured to capture and/or detect one or more predetermined wavelengths of light, a camera including a filter configured to filter a predetermined wavelength of light, an infrared (IR) camera, a thermal sensor configured to capture thermal images of syringe 103 illuminated by infrared wavelengths of a light source, a pan, tilt, and zoom (PTZ) camera including a variable field-of-view (FOV) and an automatic zoom function, a master-slave camera system including a static camera and a dynamic camera, a LiDAR system, a RADAR system, and/or the like, or any combination thereof.

[0081] In some non-limiting embodiments or aspects, an image capture device of user device 102 includes a stereo camera. For example 3D position information associated with a 3D position of syringe 103 and/or components thereof (e.g., of plunger 134, of stopper 136, of fiducial marker 138, etc.) relative to the image capture device (and/or user device 102) may be determined using a Structure from Motion (SfM) algorithm. As an example, user device 102 may include a stereo camera setup, which is available in many mobiles devices, such Apple® iPads, Apple® iPhones, Android® tablets, Android® phones, and/or the like. User device 102 may process images from the stereo camera using SfM algorithms to extract 3D information which may enhance object feature recognition of fiducial marker 138 and/or of the components of syringe 103. As an example, the SfM processing may improve extraction of the 3D features from the components of syringe 103, which may improve chances of image feature accumulation, e.g., by using a burst image capture/video mode which captures images in a proper direction/registration (e.g., pan/tilt, etc.) based on the setup or location of syringe 103. In such examples, user device 102 may generate a 3D depth or point cloud of syringe 103 using the stereo camera and detect and classify components thereof based on the depth information of the 3D point cloud and structural feature mapping of the syringe components. For example, user device 102 may use depth information to detect and track the distance between plunger 134 and the image capture device (and/or user device 102) and/or to map the real-time volume of the medication inside the syringe, which may enhance an accuracy of the tracking of the plunger depth and movement to provide better pixel-to-volume mapping for tracking the volume of the medication in the syringe.

[0082] In some non-limiting embodiments or aspects, an image capture device of user device 102 includes a LiDAR system and/or a RADAR system. For example, an image may include a LiDAR point cloud and/or a RADAR point cloud. As an example, user device 102 may include a mini-LiDAR system, which is available in many mobiles devices, such Apple® iPads, Apple® iPhones, Android® tablets, Android® phones, and/or the like. In such an example, use of LiDAR images may improve accuracy for 3D feature detection because LiDAR images directly provide 3D world information as point clouds, which may accelerate the 3D data collection with reduced or minimum protocols when compared to stereo setup. For example, user device 102 may use existing image registration and/or transformation techniques to overlap 2D object information from camera images, such as color, texture and/or the like, with the 3D LiDAR point cloud to detect 3D features to detect and track the distance between the syringe components and the image capture device (and/or user device 102) for mapping the real-time volume of the medication inside syringe 103, which may enhance an accuracy of the tracking of the plunger depth and movement to provide better pixel-to-volume mapping for tracking the volume of the medication in the syringe. [0083] Referring also to FIGS. IB and 1C, which are a perspective views of implementations of non-limiting embodiments or aspects of syringe 103, syringe 103 may include syringe barrel 132, plunger rod 134, and/or stopper 136. Syringe barrel 132 may extend between a proximal end including a proximal opening configured to receive stopper 136 and/or plunger rod 134, and a distal end including a distal opening (e.g., a luer lock opening, etc.) configured to be connected to an IV line or catheter (e.g., via a needleless connector, etc.) and/or a needle or cannula and through which a fluid (e.g. a medication, etc.) can be expelled from syringe 103.

[0084] Syringe 103 may include fiducial marker 138 (e.g., a tag, a label, a code, etc.). Fiducial marker 138 may be associated with (e.g., removably attached to, permanently attached to, integrated with, implemented on, etc.) syringe 103. For example, as shown in FIGS. IB and 1C, fiducial marker 138 may be located on an exterior surface of syringe barrel 132 of syringe 103 such that fiducial marker 138 is included in an image of syringe 103 captured by an image capture device. In some non-limiting embodiments or aspects, syringe 103 may include a plurality of fiducial markers 138 arranged about an axis of syringe 103, which may enable at least one fiducial marker 138 being presented to a FOV of an image capture device regardless of an orientation of the syringe 103.

[0085] Fiducial marker 138 may encapsulate syringe data associated with syringe 103. For example, syringe data may include at least one of the following parameters associated with syringe 103: a volume of syringe 103 (e.g., an initial volume of fluid contained in syringe 103, etc.), a size of syringe 103, a type of syringe 103 (e.g., a syringe type identifier, etc.), a unique identifier of syringe 103 (e.g., that uniquely identifies syringe 103 from other syringes, etc.), a type of medication contained in syringe 103, a concentration of medication contained in syringe 103, a prescribed rate of delivery of the medication contained in syringe 103, and/or the like, or any combination thereof.

[0086] Fiducial marker 138 may encapsulate pose information associated with a 3D position of the fiducial marker 138. For example, fiducial marker 138 may include markings that, when captured in an image, enable computing of a precise 3D position of fiducial marker 138 with respect to an image capture device that captured the image (e.g., an x, y, z coordinate position of the fiducial marker 138, etc.) and/or a precise 2D position of fiducial marker 138 in the image itself (e.g., an x, y coordinate positon of fiducial marker 138 in the image, etc.).

[0087] In some non-limiting embodiments or aspects, fiducial marker 138 includes an AprilTag. For example, fiducial marker 138 may include an AprilTag V3 of type custom Tag 48hl2, which enables using AprilTag V3 detection to determine a unique ID, which may indicate a type of syringe 103 associated with fiducial marker 138 (e.g., in leading digits, etc.) and/or a unique serial number for that specific syringe 103 (e.g., in the trailing digits, etc.), and/or a location (e.g., x, y, and z coordinates, directional vectors for Z, Y, and X axes, etc.) of fiducial marker 138 in a field-of-view (FOV) of an image capture device. However, nonlimiting embodiments or aspects are not limited thereto, and fiducial marker 138 may include a QR code, a barcode (e.g., a ID barcode, a 2D barcode, etc.), an Aztec code, a Data Matrix code, an ArUco marker, a colored pattern, a reflective pattern, a fluorescent pattern, a predetermined shape and/or color (e.g., a red pentagon, a blue hexagon, etc.), an LED pattern, a hologram, and/or the like that encapsulates syringe data associated with syringe 103 and/or encapsulates pose information associated with a 2D and/or 3D position of fiducial marker 138. [0088] Referring now to FIG. IB, syringe 103 may include color calibration marker 139 (e.g., a tag, a label, etc.). Color calibration marker 139 may be associated with (e.g., removably attached to, permanently attached to, integrated with, implemented on, etc.) syringe 103. For example, as shown in FIGS. IB and 1C, color calibration marker 139 may be located on an exterior surface of syringe barrel 132 of syringe 103 such that color calibration marker 139 is included in an image of syringe 103 captured by an image capture device. In some non-limiting embodiments or aspects, syringe 103 may include a plurality of color calibration markers 139 arranged about an axis of syringe 103, which may enable at least one color calibration marker 139 being presented to a FOV of an image capture device regardless of an orientation of the syringe 103.

[0089] Color calibration marker 139 may include a color corresponding to a color of a moveable component of syringe 103 (e.g., plunger 134, stopper 136, etc.). For example, color calibration marker 139 and stopper 136 (e.g., a portion of stopper 136, a color marker ring on stopper 136, an entirety of stopper 136, etc.) may include a same color (e.g., blue, red, green, fluorescent, a color or surface that absorbs or reflects light in a near infrared band, etc.), which may be used to calibrate color for detecting stopper 136 in a varying or wider range of lighting conditions. Color calibration marker 139 may be positioned adjacent to fiducial marker 138. For example, color calibration marker 139 may be located on syringe 103 relative to fiducial marker 138 such that color calibration marker 139 is included in a region of interest defined or identified by fiducial marker 138 in an image of syringe 103 including fiducial marker 138. As an example, fiducial marker 138 may define or identify a region of interest in an image that has a predetermined width relative to fiducial marker 138 in the image, a predetermined length relative to fiducial marker 138 in the image, and/or a predetermined location relative to fiducial marker 138 in the image. [0090] Referring now to FIG. 1C, color calibration marker 139 may include at least two color calibration markers including a first color calibration marker 139a including a first color and a second color calibration marker 139b including a second color different than the first color, and a first portion 136a of stopper 136 (and/or plunger 134) of syringe 103 may include the first color, and a second portion 136b of stopper 136 (and/or plunger 134) of syringe 103 different than the first portion 136a may include the second color. For example, the use of two or more color calibration markers corresponding to two or more identically or similarly colored portions of stopper 136 (and/or plunger 134) may improve a noise residence when processing images thereof as compared to use of a single color calibration marker.

[0091] In some non-limiting embodiments or aspects, syringe 103 may be maintained at a same position and/or orientation relative to an image capture device of user device 102 to provide a consistent region of interest and a fixed field of view. For example, syringe 103 may be maintained at the same position and/or orientation relative to the image capture device of user device 102 (e.g., via a hardware configuration or fixture, etc.) as the image capture devices captures a series of images of syringe 103 over a period of time for tracking a real-time volume of fluid dispensed from and/or remaining in syringe 103. However, non-limiting embodiments or aspects of the present disclosure are not limited thereto, and in some non-limiting embodiments or aspects a position and/or orientation of syringe 103 relative to an image capture device of user device 102 may vary (e.g., due to being held in a hand of a user, etc.) as the image capture device captures a series of images of syringe 103 over a period of time for tracking a real-time volume of fluid dispensed from and/or remaining in syringe 103.

[0092] Management system 104 may include one or more devices capable of receiving information and/or data from user device 102 (e.g., via communication network 106, etc.) and/or communicating information and/or data to user device 102 (e.g., via communication network 106, etc.). For example, management system 104 may include one or more computing systems including one or more processors (e.g., one or more computing devices, one or more server computers, one or more mobile computing devices, etc.). In some non-limiting embodiments or aspects, management system 104 includes and/or is accessible via a nurse station or terminal in a hospital. For example, management system 104 may provide bedside nurse support, nursing station manager support, retrospective reporting for nursing administration, and/or the like.

[0093] Communication network 106 may include one or more wired and/or wireless networks. For example, communication network 106 may include a cellular network (e.g., a long-term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a sixth generation (6G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber optic -based network, a cloud computing network, and/or the like, and/or a combination of these or other types of networks.

[0094] The number and arrangement of systems and devices shown in FIGS. 1A-1C are provided as an example. There can be additional systems and/or devices, fewer systems and/or devices, different systems and/or devices, or differently arranged systems and/or devices than those shown in FIGS. 1A-1C. Furthermore, two or more systems or devices shown in FIGS. 1A-1C can be implemented within a single system or a single device, or a single system or a single device shown in FIGS. 1A-1C can be implemented as multiple, distributed systems or devices. Additionally, or alternatively, a set of systems or a set of devices (e.g., one or more systems, one or more devices, etc.) of environment 100 can perform one or more functions described as being performed by another set of systems or another set of devices of environment 100.

[0095] Referring now to FIG. 2, FIG. 2 is a diagram of example components of a device 200. Device 200 may correspond to user device 102 (e.g., one or more devices of a system of user device 102, etc.) and/or one or more devices of management system 104. In some non-limiting embodiments or aspects, user device 102 (e.g., one or more devices of a system of user device 102, etc.) and/or one or more devices of management system 104 may include at least one device 200 and/or at least one component of device 200. As shown in FIG. 2, device 200 may include bus 202, processor 204, memory 206, storage component 208, input component 210, output component 212, and communication interface 214.

[0096] Bus 202 may include a component that permits communication among the components of device 200. In some non-limiting embodiments or aspects, processor 204 may be implemented in hardware, software, or a combination of hardware and software. For example, processor 204 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and/or any processing component (e.g., a field-programmable gate array (FPGA), an application- specific integrated circuit (ASIC), etc.) that can be programmed to perform a function. Memory 206 may include random access memory (RAM), read-only memory (ROM), and/or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and/or instructions for use by processor 204.

[0097] Storage component 208 may store information and/or software related to the operation and use of device 200. For example, storage component 208 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid state disk, etc.), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, and/or another type of computer-readable medium, along with a corresponding drive.

[0098] Input component 210 may include a component that permits device 200 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). Additionally or alternatively, input component 210 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, an image capture device, etc.). Output component 212 may include a component that provides output information from device 200 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.).

[0099] Communication interface 214 may include a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 214 may permit device 200 to receive information from another device and/or provide information to another device. For example, communication interface 214 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, and/or the like.

[0100] Device 200 may perform one or more processes described herein. Device 200 may perform these processes based on processor 204 executing software instructions stored by a computer-readable medium, such as memory 206 and/or storage component 208. A computer- readable medium (e.g., a non-transitory computer-readable medium) is defined herein as a non- transitory memory device. A memory device includes memory space located inside of a single physical storage device or memory space spread across multiple physical storage devices.

[0101] Software instructions may be read into memory 206 and/or storage component 208 from another computer-readable medium or from another device via communication interface 214. When executed, software instructions stored in memory 206 and/or storage component 208 may cause processor 204 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments or aspects described herein are not limited to any specific combination of hardware circuitry and software. [0102] Memory 206 and/or storage component 208 may include data storage or one or more data structures (e.g., a database, etc.). Device 200 may be capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage or one or more data structures in memory 206 and/or storage component 208.

[0103] The number and arrangement of components shown in FIG. 2 are provided as an example. In some non-limiting embodiments or aspects, device 200 may include additional components, fewer components, different components, or differently arranged components than those shown in FIG. 2. Additionally or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200.

[0104] Referring now to FIG. 3, FIG. 3 is a flowchart of non-limiting embodiments or aspects of a process 300 for tracking real-time syringe volume. In some non-limiting embodiments or aspects, one or more of the steps of process 300 may be performed (e.g., completely, partially, etc.) by user device 102 (e.g., one or more devices of a system of user device 102, etc.). In some non-limiting embodiments or aspects, one or more of the steps of process 300 may be performed (e.g., completely, partially, etc.) by another device or a group of devices separate from or including user device 102, such as management system 104 (e.g., one or more devices of management system 104, etc.).

[0105] As shown in FIG. 3, at step 302, process 300 includes obtaining at least two images of a syringe including a fiducial marker. For example, user device 102 (and/or management system 104) may obtain at least two images of syringe 103 including fiducial marker 138. The at least two images of syringe 103 may include an initial image of syringe 103 and a current image of syringe 103. For example, the initial image of syringe 103 may include an image of syringe 103 captured before the current image of syringe 103 in a time series of images including the at least two images captured over a period of time, such as an earlier image and/or a first image of syringe 103 in the time series of images of syringe 103, and/or the like. In such examples, fiducial marker 138 may encapsulate syringe data associated with syringe 103.

[0106] As shown in FIG. 3, at step 304, process 300 includes determining, based on the fiducial marker in at least one image of the syringe, a pixel-to-volume mapping associated with the syringe. For example, user device 102 (and/or management system 104) may determine, based on fiducial marker 138 in at least one image of syringe 103 of the at least two images of syringe 103 (e.g., in the initial image of syringe 103, in the current image of syringe 103, etc.), a pixel-to-volume mapping associated with syringe 103. As an example, user device 102 (and/or management system 104) may process the at least one image of syringe 103 to detect fiducial marker 138 therein, read, from fiducial marker 138 in the at least one image of syringe 103, the syringe data associated with syringe 103 to classify a type and/or an initial or full volume of syringe 103, and/or retrieve the pixel-to-volume mapping associated with syringe 103 based on the type and/or initial or full volume of syringe 103 (or provide the type and/or initial or full volume of syringe 103 with other input to a machine learning model to generate the pixel-to-volume mapping). Further details regarding non-limiting embodiments or aspects of a pixel-to-volume mapping are provided below with respect to step 312 of FIG. 3.

[0107] As shown in FIG. 3, at step 306, process 300 includes, for each image of the syringe, determining, based on the fiducial marker in that image, a region of interest in that image. For example, user device 102 (and/or management system 104) may, for each image of syringe 103 of the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), determine, based on fiducial marker 138 in that image, a region of interest in that image. As an example, fiducial marker 138 may define or identify a region of interest in an image that has a predetermined width relative to fiducial marker 138 in the image, a predetermined length relative to fiducial marker 138 in the image, and/or a predetermined location relative to fiducial marker 138 in the image. In such an example, as shown in FIGS. 4-6, the region of interest in the image may include color calibration marker 139 (or first and second color calibration markers 139a and 139b) and/or a movement path of stopper 136 and/or plunger 134 from an initial or full syringe volume position thereof to a final or fully dispensed syringe volume position thereof. For example, fiducial marker 138 may identify or define as a region of interest only color calibration marker 139 and/or a middle section of syringe body 132 that only this exact region of interest of each image is processed instead of processing the full image, thereby reducing a processing load associated with processing the images and/or increasing an accuracy when detecting stopper 136 and/or plunger 134 in the images.

[0108] In some non-limiting embodiments or aspects, user device 102 (and/or management system 104) may, for each image of syringe 103 of the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), determine, based on fiducial marker 138 in that image, position information associated with a 3D position of fiducial marker 138 (and/or syringe 103) relative to the image capture device and/or a 2D position of fiducial marker 138 (and/or syringe 103) in the image itself. For example, user device 102 (and/or management system 104) may determine the position information associated with the 3D position of fiducial marker 138 (and/or syringe 103) relative to the image capture device and/or the 2D position of the fiducial marker 138 (and/or syringe 103) in the image itself by determining or identifying, based on the image, fiducial marker 138 and the pose information associated with the 3D positions and/or 2D positions of fiducial marker 138. For example, fiducial marker 138 may include an AprilTag, and, user device 102 (and/or management system 104) may process the image using AprilTag detection software to determine syringe data encapsulated by fiducial marker 138 and/or a unique serial number for syringe 103 and to compute a precise 3D position, orientation, and/or identity of fiducial marker 138 (and/or syringe 103) relative to the image capture device that captured the image and/or a precise 2D position of fiducial marker 138 (and/or syringe 103) in the image itself. As an example, the position information associated with the 3D position of the fiducial marker 138 (and/or syringe 103) relative to the image capture device may be determined as the 3D position of the fiducial marker 138 associated with syringe 103 relative to the image capture device and/or the position information associated with the 2D positon of syringe 103 in the image itself may be determined as the 2D position of fiducial marker 138 associated with syringe 103. In such an example, the 3D position of fiducial marker 138 (and/or syringe 103) may include x, y, and z coordinates of fiducial marker 138 and/or directional vectors for Z, Y, and X axes of fiducial marker 138. In such an example, the 2D position of fiducial marker 138 in the image itself may include x, y coordinates of fiducial marker 138 in the image and/or direction vectors for Y and X axes of fiducial marker 138.

[0109] In some non-limiting embodiments or aspects, user device 102 (and/or management system 104) may, for each image of syringe 103 of the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), determine position information associated with a 3D position fiducial marker 138 (and/or syringe 103) relative to the image capture device using a stereo camera, a LiDAR system, a RADAR system, and/or the like, generate a 3D depth or point cloud of syringe 103.

[0110] As shown in FIG. 3, at step 308, process 300 includes, for each image of the syringe, generating a color threshold for that image. For example, user device 102 (and/or management system 104) may, for each image of syringe 103 of the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), generate a color threshold for that image. As an example, a color threshold may include one or more colors (e.g., a range of colors, a plurality of ranges of colors, one or more color values, etc.) of a color space (e.g., a RGB color space, an HSV color space, an HSL color space, etc.). [0111] In some non-limiting embodiments or aspects, user device 102 (and/or management system 104) may, for each image of syringe 103 of the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), generate the color threshold for that image based on a predetermined color threshold associated with syringe 103. For example, user device 102 (and/or management system 104) may determine, based on fiducial marker 138 in at least one image of syringe 103 of the at least two images of syringe 103, the predetermined color threshold associated with syringe 103 from a plurality of predetermined color thresholds (e.g., from a plurality of predetermined color thresholds associated with a plurality of syringe types and/or volumes stored in a database, etc.).

[0112] In some non-limiting embodiments or aspects, syringe 103 includes color calibration marker 139, including at least one color, and a moveable component of syringe 103 (e.g., plunger 134, stopper 136, etc.) includes the at least one color. For example, user device 102 (and/or management system 104) may, for each image of syringe 103 of the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), generate, based on the at least one color of color calibration marker 139 in that image, the color threshold for that image. As an example, user device 102 (and/or management system 104) may, for each image of syringe 103 of the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), generate the color threshold for that image as a range of colors surrounding a color of color calibration marker 139 in that image.

[0113] In some non-limiting embodiments or aspects, color calibration marker 139 includes at least two color calibration markers including first color calibration marker 139a including a first color and second color calibration marker 139b including a second color different than the first color, a first portion of the moveable component of the syringe (e.g., a first portion of plunger 134, a first portion 136a of stopper 136, etc.) includes the first color, and a second portion of the moveable component of the syringe (e.g., a second portion of plunger 134, a second portion 136b of stopper 136, etc.) different than the first portion includes the second color. For example, user device 102 (and/or management system 104) may, for each image of syringe 103 of the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), generate, based on the first color of the first color calibration marker in that image and the second color of the second color calibration marker in that image, the color threshold for that image. As an example, user device 102 (and/or management system 104) may, for each image of syringe 103 of the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), generate, the color threshold for that image as a first range of colors surrounding a color of first color calibration marker 139a in that image and a second range of colors surrounding a color of second color calibration marker 139b in that image.

[0114] In some non-limiting embodiments or aspects, the at least two images of the syringe (e.g., the initial image of syringe 103, the current image of syringe 103, etc.) include a first color space (e.g., a RGB color space, etc.), and user device 102 (and/or management system 104) may, for each image of syringe 103 of the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), convert (e.g., using OpenCV, etc.) that image from including the first color space (e.g., the RGB color space, etc.) to including a second color space (e.g., an HSV color space, etc.) different than the first color space. For example, the color threshold for that image may be generated based on a color of color calibration marker 139 in that converted image (e.g., based on a color of first color calibration marker 139a in that image and a color of second color calibration marker 139b in that image, etc.), and the region of interest in that converted image may be segmented based on the color threshold for that image to generate the masked image as described herein in more detail. As an example, user device 102 (and/or management system 104) may convert input image frames from the RGB color space to the HSV color space, which may enable the system to understand the colors or values thereof more precisely. In such an example, user device 102 (and/or management system 104) may generate a HSV color threshold for each image to detect and segment the color of stopper 136 in the converted images as described herein in more detail.

[0115] In this way, non-limiting embodiments or aspects of the present disclosure may calibrate the color threshold per each image or frame continuously over a series of images captured over a period of time for improving detection of the color of the stopper and/or plunger and tracking the real-time volume of the syringe in lighting conditions that may vary over the period of time.

[0116] As shown in FIG. 3, at step 310, process 300 includes, for each image of the syringe, segmenting the region of interest in that image based on the color threshold to generate a masked image. For example, user device 102 (and/or management system 104) may, for each image of syringe 103 of the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), segment, based on the color threshold for that image, the region of interest in that image to generate a masked image. As an example, user device 102 (and/or management system 104) may, for each image of syringe 103 of the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), generate the masked image as a binary mask (and/or dual binary masks for a syringe including two color calibration markers) that maps or defines a region of stopper 136 and/or plunger 134 within the color threshold for that image. In such an example, user device 102 (and/or management system 104) may, for each image of syringe 103 of the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), apply one or more morphological operations (e.g., using OpenCV, etc.) to the binary mask to generate contours and/or connect groups or blobs of pixels in the masked image.

[0117] In some non-limiting embodiments or aspects, user device 102 (and/or management system 104) may, for each image of syringe 103 of the at least two images of syringe 103 (e.g., for the initial image of syringe 103, for the current image of syringe 103, etc.), control a display to display that image of the syringe with the masked image generated from that image overlaid on that image in the display. For example, as shown in FIGS. 4-6, which illustrate implementations 400, 500, 600 of non-limiting embodiments or aspects of images of syringes overlaid with corresponding masked images, a masked image generated from segmenting a region of interest of an image of syringe 103 may be overlaid on that image of syringe 103 to visualize tracking of stopper 136 and/or plunger 134 in real-time as stopper 136 and/or plunger 134 move relative to syringe barrel 132 over time in a series of images of syringe 103 as fluid is dispensed from syringe 103.

[0118] As shown in FIG. 3, at step 312, process 300 includes determining, based on the pixel- to-volume mapping associated with the syringe and the masked image generated from each image of the syringe, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof. For example, user device 102 (and/or management system 104) may, determine, based on the pixel-to -volume mapping associated with syringe 103, the masked image generated from the initial image of syringe 103, and the masked image generated from the current image of syringe 103, at least one of a volume of a fluid dispensed from syringe 103, a volume of a fluid remaining in syringe 103, or any combination thereof.

[0119] As an example, user device 102 (and/or management system 104) may determine the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof by, for each image of syringe 103 of the at least two images of syringe 103, determining at least one centroid pixel of at least one plurality of pixels in the masked image that satisfied the color threshold for that image (e.g., a centroid pixel of a region of stopper 136 and/or plunger 134 within the color threshold for that image, a centroid pixel of a blob in the masked image, etc.). In such an example, user device 102 (and/or management system 104) may determine the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof based on the pixel-to-volume mapping associated with the syringe, the at least one centroid pixel of the masked image generated from the initial image of the syringe, and the at least one centroid pixel of the masked image generated from the current image of the syringe.

[0120] For example, user device 102 (and/or management system 104) may provide, as input to the pixel-to-volume mapping, the at least one centroid pixel of the masked image and, receive, as output from the pixel-to-volume mapping, the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof. As an example, a centroid pixel associated with the initial image of syringe 103 of the at least two images of syringe 103 (e.g., an earlier image and/or a first image of syringe 103 in a time series of images of syringe 103, etc.) may be assigned to or determined to correspond to an initial or full syringe volume (e.g., an initial or full syringe volume position of stopper 136 and/or plunger 134 in a movement path thereof, etc.) by the pixel-to-volume mapping, and subsequent centroid pixels associated with subsequent images of syringe 103 of the at least two images of syringe 103 (e.g., the current image of syringe 103 in the time series of images of syringe 103, etc.) input to the pixel-to-volume mapping may provide a change in the syringe volume from the initial or full syringe volume or another earlier or previous volume determined from another earlier or previous image in the time series of images (e.g., a change in a position of stopper 136 and/or plunger 134 in the movement path thereof from the initial or full volume position or from another earlier or previous position of stopper 136 and/or plunger 134 in the movement path thereof, etc.).

[0121] In some non-limiting embodiments or aspects, user device 102 (and/or management system 104) determine, for each image of syringe 103 of the at least two images of syringe 103, at least two centroid pixels of at least two pluralities of pixels in the masked image that satisfied the color threshold for that image including a first centroid pixel of a first plurality of pixels that satisfied a first color range threshold of the color threshold for that image (e.g., a first centroid pixel of a first region of stopper 136 and/or plunger 134 within the first color threshold for that image, a first centroid pixel of a first blob in the masked image, etc.) and a second centroid pixel of a second plurality of pixels that satisfied a second color range threshold for that image (e.g., a second centroid pixel of a second region of stopper 136 and/or plunger 134 within the second color threshold for that image, a second centroid pixel of a second blob in the masked image, etc.). For example, syringe 103 may include at least two color calibration markers including first color calibration marker 139a including a first color and second color calibration marker 139b including a second color different than the first color. As an example, user device 102 (and/or management system 104) may determine the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof based on the pixel-to-volume mapping associated with syringe 103, the first centroid pixel of the masked image generated from the initial image of syringe 103, the second centroid pixel of the masked image generated from the initial image of syringe 103, the first centroid pixel of the masked image generated from the current image of syringe 103, and the second centroid pixel of the masked image generated from the current image of syringe 103.

[0122] In this way, a continuous mapping of the centroid pixel or pixels of each image or frame to syringe volume may enable real-time tracking of the volume of fluid dispensed from and/or remaining in the syringe.

[0123] In some non-limiting embodiments or aspects, a pixel-to-volume mapping includes a hard coded or predetermined range of centroid pixels mapped to volume levels of syringe 103. For example, a range of centroid pixels may be generated from an experimental setup in which syringe 103 is held at a static position and/or orientation relative to an image capture device that captures a series of images of syringe 103 as stopper 136 and/or plunger 134 moves from an initial or full syringe volume position thereof to a final or fully dispensed syringe volume position and corresponding volumes of syringe 103 are measured (e.g., via a flow sensor, etc.). As an example, the range of centroid pixels of masked images for each image of the series of images may be mapped to the corresponding volume of syringe 103 measured at the time that image in the series of images was captured to generate a hard-coded or predetermined pixel- to-volume mapping. In such an example, the hard-coded or predetermined pixel-to-volume mapping may be stored (e.g., in a database, etc.) in association with syringe 103 (e.g., in association with a syringe type and/or volume of syringe 103, etc.) and/or the position and/or orientation of syringe 103 relative to the image capture device. Accordingly, user device 102 (and/or management system 104) may retrieve the stored, hard-coded, or predetermined pixel- to-volume mapping associated with syringe 103 from a plurality of stored, hard-coded, or predetermined pixel-to-volume mappings based on the syringe data determined from fiducial marker 138 associated with syringe 103 and/or a 2D or 3D position and/or orientation of syringe 103 determined based on one or more images of syringe 103 of the at least two images of syringe 103. [0124] In some non-limiting embodiments or aspects, a pixel-to- volume mapping includes and/or is generated by a machine learning model configured to generate the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof from an input including the masked image generated from the initial image of the syringe and the masked image generated from the current image of the syringe (e.g., the centroid pixel or pixels associated with the initial image of syringe 103 and the centroid pixel or pixels associated with the current image of syringe 103, etc.). For example, user device 102 (and/or management system 104) may process the images of syringe 103, the region of interest or bounding boxes thereof, the syringe data, and/or the position and/or orientation of syringe 103 determined for each image with a machine learning model to generate a predicted volume dispensed from and/or remaining in syringe 103. As an example, user device 102 (and/or management system 104) may generate a prediction model (e.g., an estimator, a classifier, a prediction model, a detector model, etc.) using machine learning techniques including, for example, supervised and/or unsupervised techniques, such as decision trees (e.g., gradient boosted decision trees, random forests, etc.), logistic regressions, artificial neural networks (e.g., convolutional neural networks, etc.), Bayesian statistics, learning automata, Hidden Markov Modeling, linear classifiers, quadratic classifiers, association rule learning, and/or the like. The prediction machine learning model may be trained to provide an output including a prediction of at least one of a volume of a fluid dispensed from syringe 103, a volume of a fluid remaining in syringe 103, or any combination thereof (e.g., a pixel-to-volume mapping at the pixel level where each pixel may correspond to a different amount of change in syringe volume, etc.). In such an example, the prediction may include a probability (e.g., a likelihood, etc.) of the at least one of a volume of a fluid dispensed from syringe 103, a volume of a fluid remaining in syringe 103, or any combination thereof.

[0125] User device 102 (and/or management system 104) may generate the prediction model based on training data determined from an experimental setup in which syringe 103, which may be held at a static and/or a varying position and/or orientation relative to an image capture device that captures a series of images of syringe 103 as stopper 136 and/or plunger 134 moves from an initial or full syringe volume position thereof to a final or fully dispensed syringe volume position and corresponding volumes of syringe 103 are measured (e.g., via a flow sensor, etc.). In some implementations, the prediction model is designed to receive, as an input, the masked image generated from the initial image of the syringe (e.g., the centroid pixel or pixels associated with the initial image of syringe 103, etc.), the masked image generated from the current image of the syringe (e.g., the centroid pixel or pixels associated with the current image of syringe 103, etc.), the syringe data (e.g., the syringe type and/or initial or full volume of syringe 103, etc.), and/or a positon and/or orientation of syringe associated with each image (e.g., relative to the image capture device and/or user device 102, etc.) and provide, as an output, a prediction (e.g., a probability, a likelihood, a binary output, a yes-no output, a score, a prediction score, a classification, etc.) of the at least one of a volume of a fluid dispensed from syringe 103, a volume of a fluid remaining in syringe 103, or any combination thereof. In such an implementation, the measured volumes of syringe 103 may be used as labels for the corresponding input. In some non-limiting embodiments or aspects, user device 102 stores the prediction model (e.g., stores the model for later use). In some non-limiting embodiments or aspects, user device 102 (and/or management system 104) stores the initial prediction model in a data structure (e.g., a database, a linked list, a tree, etc.). In some non-limiting embodiments, the data structure is located within user device 102 (and/or management system 104) or external (e.g., remote from) user device 102 (and/or management system 104) (e.g., within a database system, etc.).

[0126] In some non-limiting embodiments or aspects, a prediction model associated with syringe 103 may be stored in association with syringe 103 (e.g., in association with a syringe type and/or volume of syringe 103, etc.) and/or the position and/or orientation of syringe 103 relative to the image capture device. Accordingly, user device 102 (and/or management system 104) may retrieve the stored, prediction model associated with syringe 103 from a plurality of stored, prediction models based on the syringe data determined from fiducial marker 138 associated with syringe 103 and/or a 2D or 3D position and/or orientation of syringe 103 determined based on one or more images of syringe 103 of the at least two images of syringe 103.

[0127] In some non-limiting embodiments or aspects, a prediction model may be associated with a plurality of different syringe types and/or volumes and/or syringe positions and/or orientations relative to an image capture device. For example, the prediction model may be trained based on training data determined from a plurality of experimental setups in which a plurality of syringes of different syringe types and/or volumes, which may be held at static and/or varying positions and/or orientations relative to a plurality of image capture devices that capture a plurality of series of images of the plurality of syringes as their stoppers 136 and/or plungers 134 move from initial or full syringe volume positions thereof to final or fully dispensed syringe volume positions and corresponding volumes of the plurality of syringes are measured (e.g., via flow sensors, etc.). [0128] Accordingly, a pixel-to-volume mapping may be configured to track a volume level in a syringe by calculating pixel information using hard coded or predetermined ranges and/or machine learning techniques to generate automatic pixel-to-volume mapping models for tracking real-time syringe volume.

[0129] As shown in FIG. 3, at step 314, process 300 includes providing the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof. For example, user device 102 (and/or management system 104) may provide the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof. As an example, user device 102 (and/or management system 104) may control a display (e.g., output component 212, etc.) to display, in real-time, the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof. For example, as shown in FIGS. 4-6, the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof may be displayed in real-time (e.g., concurrently with a masked image generated from segmenting a region of interest of an image of syringe 103 overlaid on that image of syringe 103 to visualize tracking of stopper 136 and/or plunger 134 in real-time, etc.) as stopper 136 and/or plunger 134 move relative to syringe barrel 132 over time in a series of images of syringe 103 as fluid is dispensed from syringe 103.

[0130] In some non-limiting embodiments or aspects, user device 102 (and/or management system 104) may compare the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof (and/or a rate of fluid delivery determined based thereon) to one or more thresholds and, if the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof (and/or the rate of fluid delivery determined based thereon) violates the one or more thresholds, provide an alert associated therewith (e.g., via user device 102, etc.).

[0131] In some non-limiting embodiments or aspects, user device 102 (and/or management system 104) may aggregate the at least one of the volume of the fluid dispensed from syringe 103, the volume of the fluid remaining in syringe 103, or any combination thereof (and/or a rate of fluid delivery determined based thereon) from a plurality of injections or fluid delivery procedures to determine one or more metrics associated with a syringe type and/or volume of syringe 103 and/or a user that administered the plurality of injections. [0132] Although embodiments or aspects have been described in detail for the purpose of illustration and description, it is to be understood that such detail is solely for that purpose and that embodiments or aspects are not limited to the disclosed embodiments or aspects, but, on the contrary, are intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect can be combined with one or more features of any other embodiment or aspect. In fact, many of these features can be combined in ways not specifically recited in the claims and/or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

Claims

WHAT IS CLAIMED IS:

1. A system, comprising: at least one processor coupled to a memory and configured to: obtain at least two images of a syringe including a fiducial marker, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the fiducial marker encapsulates syringe data associated with the syringe; determine, based on the fiducial marker in at least one image of the syringe of the at least two images of the syringe, a pixel-to-volume mapping associated with the syringe; for each image of the syringe of the at least two images of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and segment, based on a color threshold for that image, the region of interest in that image to generate a masked image; determine, based on the pixel-to-volume mapping associated with the syringe, the masked image generated from the initial image of the syringe, and the masked image generated from the current image of the syringe, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and provide the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

2. The system of claim 1, wherein the syringe includes at least one color calibration marker including at least one color, wherein a moveable component of the syringe includes the at least one color, and wherein the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: generate, based on the at least one color of the at least one color calibration marker in that image, the color threshold for that image.

3. The system of claim 2, wherein the at least one color calibration marker includes at least two color calibration markers including a first color calibration marker including a first color and a second color calibration marker including a second color different than the first color, wherein a first portion of the moveable component of the syringe includes the first color, wherein a second portion of the moveable component of the syringe different than the first portion includes the second color, and wherein the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: generate, based on the first color of the first color calibration marker in that image and the second color of the second color calibration marker in that image, the color threshold for that image.

4. The system of claim 2, wherein the at least two images of the syringe include a first color space, and wherein the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: convert that image from including the first color space to including a second color space different than the first color space, wherein the color threshold for that image is generated based on the at least one color of the at least one color calibration marker in that converted image, and wherein the region of interest in that converted image is segmented based on the color threshold for that image to generate the masked image.

5. The system of claim 1, wherein the at least one processor is further configured to determine the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof by: for each image of the syringe of the at least two images of the syringe: determining at least one centroid pixel of at least one plurality of pixels in the masked image that satisfied the color threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to-volume mapping associated with the syringe, the at least one centroid pixel of the masked image generated from the initial image of the syringe, and the at least one centroid pixel of the masked image generated from the current image of the syringe.

6. The system of claim 5, wherein the at least one processor is further configured to determine, for each image of the syringe of the at least two images of the syringe, at least two centroid pixels of at least two pluralities of pixels in the masked image that satisfied the color threshold for that image including a first centroid pixel of a first plurality of pixels that satisfied a first color range threshold of the color threshold for that image and a second centroid pixel of a second plurality of pixels that satisfied a second color range threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to- volume mapping associated with the syringe, the first centroid pixel of the masked image generated from the initial image of the syringe, the second centroid pixel of the masked image generated from the initial image of the syringe, the first centroid pixel of the masked image generated from the current image of the syringe, and the second centroid pixel of the masked image generated from the current image of the syringe.

7. The system of claim 1, wherein the pixel-to-volume mapping includes a hard-coded or predetermined pixel-to-volume mapping.

8. The system of claim 1, wherein the pixel-to-volume mapping includes a machine learning model configured to generate the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof based on the masked image generated from the initial image of the syringe and the masked image generated from the current image of the syringe.

9. The system of claim 1, wherein the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: control a display to display that image of the syringe with the masked image generated from that image overlaid on that image in the display.

10. The system of claim 1, wherein the at least one processor is further configured to: for each image of the syringe of the at least two images of the syringe: determine, based on the fiducial marker in that image, position information associated with a three-dimensional (3D) position of the syringe relative to an image capture device that captured the image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

11. A method, comprising: obtaining, with at least one processor, at least two images of a syringe including a fiducial marker, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the fiducial marker encapsulates syringe data associated with the syringe; determining, with the at least one processor, based on the fiducial marker in at least one image of the syringe of the at least two images of the syringe, a pixel-to-volume mapping associated with the syringe; for each image of the syringe of the at least two images of the syringe: determining, with the at least one processor, based on the fiducial marker in that image, a region of interest in that image; and segmenting, with the at least one processor, based on a color threshold for that image, the region of interest in that image to generate a masked image; determining, with the at least one processor, based on the pixel-to-volume mapping associated with the syringe, the masked image generated from the initial image of the syringe, and the masked image generated from the current image of the syringe, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and providing, with the at least one processor, the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

12. The method of claim 11, wherein the syringe includes at least one color calibration marker including at least one color, wherein a moveable component of the syringe includes the at least one color, and wherein the method further comprises: for each image of the syringe of the at least two images of the syringe: generating, with the at least one processor, based on the at least one color of the at least one color calibration marker in that image, the color threshold for that image.

13. The method of claim 12, wherein the at least one color calibration marker includes at least two color calibration markers including a first color calibration marker including a first color and a second color calibration marker including a second color different than the first color, wherein a first portion of the moveable component of the syringe includes the first color, wherein a second portion of the moveable component of the syringe different than the first portion includes the second color, and wherein the method further comprises: for each image of the syringe of the at least two images of the syringe: generating, with the at least one processor, based on the first color of the first color calibration marker in that image and the second color of the second color calibration marker in that image, the color threshold for that image.

14. The method of claim 12, wherein the at least two images of the syringe include a first color space, and wherein the method further comprises: for each image of the syringe of the at least two images of the syringe: converting, with the at least one processor, that image from including the first color space to including a second color space different than the first color space, wherein the color threshold for that image is generated based on the at least one color of the at least one color calibration marker in that converted image, and wherein the region of interest in that converted image is segmented based on the color threshold for that image to generate the masked image.

15. The method of claim 11, wherein determining the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof further includes: for each image of the syringe of the at least two images of the syringe: determining, with the at least one processor, at least one centroid pixel of at least one plurality of pixels in the masked image that satisfied the color threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to-volume mapping associated with the syringe, the at least one centroid pixel of the masked image generated from the initial image of the syringe, and the at least one centroid pixel of the masked image generated from the current image of the syringe.

16. The method of claim 15, wherein determining the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof further includes: for each image of the syringe of the at least two images of the syringe: determining, with the at least one processor, at least two centroid pixels of at least two pluralities of pixels in the masked image that satisfied the color threshold for that image including a first centroid pixel of a first plurality of pixels that satisfied a first color range threshold of the color threshold for that image and a second centroid pixel of a second plurality of pixels that satisfied a second color range threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to- volume mapping associated with the syringe, the first centroid pixel of the masked image generated from the initial image of the syringe, the second centroid pixel of the masked image generated from the initial image of the syringe, the first centroid pixel of the masked image generated from the current image of the syringe, and the second centroid pixel of the masked image generated from the current image of the syringe.

17. The method of claim 11, wherein the pixel-to- volume mapping includes a hard-coded or predetermined pixel-to-volume mapping.

18. The method of claim 11, wherein the pixel-to-volume mapping includes a machine learning model configured to generate the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof based on the masked image generated from the initial image of the syringe and the masked image generated from the current image of the syringe.

19. The method of claim 11, further comprising: for each image of the syringe of the at least two images of the syringe: controlling, with the at least one processor, a display to display that image of the syringe with the masked image generated from that image overlaid on that image in the display.

20. The method of claim 11, further comprising: for each image of the syringe of the at least two images of the syringe: determining, with the at least one processor, based on the fiducial marker in that image, position information associated with a three-dimensional (3D) position of the syringe relative to an image capture device that captured the image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.

21. A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: obtain at least two images of a syringe including a fiducial marker, wherein the at least two images of the syringe include an initial image of the syringe and a current image of the syringe, and wherein the fiducial marker encapsulates syringe data associated with the syringe; determine, based on the fiducial marker in at least one image of the syringe of the at least two images of the syringe, a pixel-to-volume mapping associated with the syringe; for each image of the syringe of the at least two images of the syringe: determine, based on the fiducial marker in that image, a region of interest in that image; and segment, based on a color threshold for that image, the region of interest in that image to generate a masked image; determine, based on the pixel-to-volume mapping associated with the syringe, the masked image generated from the initial image of the syringe, and the masked image generated from the current image of the syringe, at least one of a volume of a fluid dispensed from the syringe, a volume of a fluid remaining in the syringe, or any combination thereof; and provide the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof.

22. The computer program product of claim 21, wherein the syringe includes at least one color calibration marker including at least one color, wherein a moveable component of the syringe includes the at least one color, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: generate, based on the at least one color of the at least one color calibration marker in that image, the color threshold for that image.

23. The computer program product of claim 22, wherein the at least one color calibration marker includes at least two color calibration markers including a first color calibration marker including a first color and a second color calibration marker including a second color different than the first color, wherein a first portion of the moveable component of the syringe includes the first color, wherein a second portion of the moveable component of the syringe different than the first portion includes the second color, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: generate, based on the first color of the first color calibration marker in that image and the second color of the second color calibration marker in that image, the color threshold for that image.

24. The computer program product of claim 22, wherein the at least two images of the syringe include a first color space, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: convert that image from including the first color space to including a second color space different than the first color space, wherein the color threshold for that image is generated based on the at least one color of the at least one color calibration marker in that converted image, and wherein the region of interest in that converted image is segmented based on the color threshold for that image to generate the masked image.

25. The computer program product of claim 21, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to determine the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof by: for each image of the syringe of the at least two images of the syringe: determining at least one centroid pixel of at least one plurality of pixels in the masked image that satisfied the color threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to-volume mapping associated with the syringe, the at least one centroid pixel of the masked image generated from the initial image of the syringe, and the at least one centroid pixel of the masked image generated from the current image of the syringe.

26. The computer program product of claim 25, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to determine, for each image of the syringe of the at least two images of the syringe, at least two centroid pixels of at least two pluralities of pixels in the masked image that satisfied the color threshold for that image including a first centroid pixel of a first plurality of pixels that satisfied a first color range threshold of the color threshold for that image and a second centroid pixel of a second plurality of pixels that satisfied a second color range threshold for that image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the pixel-to-volume mapping associated with the syringe, the first centroid pixel of the masked image generated from the initial image of the syringe, the second centroid pixel of the masked image generated from the initial image of the syringe, the first centroid pixel of the masked image generated from the current image of the syringe, and the second centroid pixel of the masked image generated from the current image of the syringe.

27. The computer program product of claim 21, wherein the pixel-to- volume mapping includes a hard-coded or predetermined pixel-to-volume mapping.

28. The computer program product of claim 21, wherein the pixel-to- volume mapping includes a machine learning model configured to generate the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof based on the masked image generated from the initial image of the syringe and the masked image generated from the current image of the syringe.

29. The computer program product of claim 21, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: control a display to display that image of the syringe with the masked image generated from that image overlaid on that image in the display.

30. The computer program product of claim 21, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for each image of the syringe of the at least two images of the syringe: determine, based on the fiducial marker in that image, position information associated with a three-dimensional (3D) position of the syringe relative to an image capture device that captured the image, wherein the at least one of the volume of the fluid dispensed from the syringe, the volume of the fluid remaining in the syringe, or any combination thereof is determined based on the 3D position of the syringe in the initial image and the 3D position of the syringe in the current image.