US11896425B2 - Guiding instrument insertion - Google Patents

Guiding instrument insertion Download PDF

Info

Publication number
US11896425B2
US11896425B2 US17/239,323 US202117239323A US11896425B2 US 11896425 B2 US11896425 B2 US 11896425B2 US 202117239323 A US202117239323 A US 202117239323A US 11896425 B2 US11896425 B2 US 11896425B2
Authority
US
United States
Prior art keywords
ultrasound
blood vessel
blood vessels
image
blood
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US17/239,323
Other languages
English (en)
Other versions
US20220338833A1 (en
Inventor
Davinder S. Dhatt
Christopher Aleksandr White
Adam Benjamin Pely
Thomas Michael Duffy
Paul Tomotaro DANSET
Diku Pranav Mandavia
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujifilm Sonosite Inc
Original Assignee
Fujifilm Sonosite Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujifilm Sonosite Inc filed Critical Fujifilm Sonosite Inc
Priority to US17/239,323 priority Critical patent/US11896425B2/en
Assigned to FUJIFILM SONOSITE, INC. reassignment FUJIFILM SONOSITE, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: DHATT, Davinder S., MANDAVIA, DIKU PRANAV, WHITE, CHRISTOPHER ALEKSANDR, DANSET, Paul Tomotaro, PELY, ADAM BENJAMIN, DUFFY, THOMAS MICHAEL
Priority to EP22792637.5A priority patent/EP4326160A1/fr
Priority to PCT/US2022/026122 priority patent/WO2022226398A1/fr
Priority to JP2023565274A priority patent/JP2024514957A/ja
Priority to CN202280030439.1A priority patent/CN117222367A/zh
Publication of US20220338833A1 publication Critical patent/US20220338833A1/en
Priority to US18/397,826 priority patent/US20240156429A1/en
Publication of US11896425B2 publication Critical patent/US11896425B2/en
Application granted granted Critical
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/08—Detecting organic movements or changes, e.g. tumours, cysts, swellings
    • A61B8/0833—Detecting organic movements or changes, e.g. tumours, cysts, swellings involving detecting or locating foreign bodies or organic structures
    • A61B8/085—Detecting organic movements or changes, e.g. tumours, cysts, swellings involving detecting or locating foreign bodies or organic structures for locating body or organic structures, e.g. tumours, calculi, blood vessels, nodules
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/08—Detecting organic movements or changes, e.g. tumours, cysts, swellings
    • A61B8/0833—Detecting organic movements or changes, e.g. tumours, cysts, swellings involving detecting or locating foreign bodies or organic structures
    • A61B8/0841—Detecting organic movements or changes, e.g. tumours, cysts, swellings involving detecting or locating foreign bodies or organic structures for locating instruments
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/08—Detecting organic movements or changes, e.g. tumours, cysts, swellings
    • A61B8/0891—Detecting organic movements or changes, e.g. tumours, cysts, swellings for diagnosis of blood vessels
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/12—Diagnosis using ultrasonic, sonic or infrasonic waves in body cavities or body tracts, e.g. by using catheters
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/44—Constructional features of the ultrasonic, sonic or infrasonic diagnostic device
    • A61B8/4427—Device being portable or laptop-like
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/44—Constructional features of the ultrasonic, sonic or infrasonic diagnostic device
    • A61B8/4444—Constructional features of the ultrasonic, sonic or infrasonic diagnostic device related to the probe
    • A61B8/445—Details of catheter construction
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/44—Constructional features of the ultrasonic, sonic or infrasonic diagnostic device
    • A61B8/4444—Constructional features of the ultrasonic, sonic or infrasonic diagnostic device related to the probe
    • A61B8/4472—Wireless probes
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/46—Ultrasonic, sonic or infrasonic diagnostic devices with special arrangements for interfacing with the operator or the patient
    • A61B8/461—Displaying means of special interest
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/46—Ultrasonic, sonic or infrasonic diagnostic devices with special arrangements for interfacing with the operator or the patient
    • A61B8/467—Ultrasonic, sonic or infrasonic diagnostic devices with special arrangements for interfacing with the operator or the patient characterised by special input means
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/52—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/5207—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of raw data to produce diagnostic data, e.g. for generating an image
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/52—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/5215—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data
    • A61B8/5223—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data for extracting a diagnostic or physiological parameter from medical diagnostic data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Definitions

  • One or more exemplary embodiments relate to an ultrasound machine and a method of operating the same, and more particularly, to an ultrasound machine that identifies blood vessels in ultrasound images and displays the blood vessels in an enhanced manner by including information about the blood vessels useful to an operator of the ultrasound machine.
  • Ultrasound systems radiate an ultrasonic signal generated from an ultrasound probe into an object, such as a patient, and receive echo signals reflected from an internal part of the object. An image of the internal part of the object is generated using the received echo signals. More specifically, ultrasound diagnostic machines generate an ultrasound image by using ultrasonic image data acquired from an ultrasound probe and display the generated ultrasound image on a screen to provide the ultrasound image to a user.
  • the ultrasound machine can include a control panel for controlling the ultrasound machine and setting various functions, such as a gain or frequency setting.
  • Procedures for which ultrasound machines are often used include ultrasound-guided insertion of an interventional instrument, such as peripheral intravenous (PIV) catheterization.
  • PIV catheterization a clinician may initially perform a survey scan of a body using the ultrasound machine to look for appropriate veins for cannulation.
  • Suitable veins have the following characteristics: generally greater than 0.3 mm diameter; generally not located near an artery that could be accidentally damaged during cannulation; they are sufficiently large such that the catheter-to-vein diameter ratio is greater than 0.45 (although other rules of thumb exist such as 0.33); the vein is relatively straight and not tortuous; and the vein does not contain valves that would be hit by the catheter.
  • the clinician can either eyeball (e.g., estimate) the measurement, which is prone to error, or they can turn on the calipers and measure to get a more precise diameter and depth.
  • eyeball e.g., estimate
  • clinicians rarely use the calipers because of the time it takes and the desire to not touch the ultrasound machine while they are performing this procedure (both for sterility and workflow).
  • the clinician can easily see and identify a blood vessel suitable for inserting a catheter, a needle, etc.
  • a method is implemented by a computing device for guiding instrument insertion, such as an ultrasound machine or a tablet coupled to the ultrasound machine, and includes receiving an ultrasound image that includes one or more blood vessels, and determining, with a neural network implemented at least partially in hardware of the computing device, diameters of the one or more blood vessels in the ultrasound image. The method also includes receiving a user selection of an instrument size, and indicating, in the ultrasound image, at least one blood vessel of the one or more blood vessels based on the instrument size and the diameters of the one or more blood vessels.
  • a computing device for guiding instrument insertion such as an ultrasound machine or a tablet coupled to the ultrasound machine, comprises a memory to maintain one or more ultrasound images, and a neural network implemented at least partially in hardware of the computing device to identify blood vessels in the one or more ultrasound images and determine diameters of the blood vessels.
  • the computing device can also include a processor to determine, for one or more of the blood vessels, an instrument size based on the diameters of the one or more of the blood vessels.
  • the processor can select a color based on the instrument size, and indicate, in the one or more ultrasound images, the one or more of the blood vessels with the color.
  • an ultrasound system for identifying blood vessels includes an image module to generate one or more ultrasound images based on ultrasound echo signals, and a neural network module to identify the blood vessels in the one or more ultrasound images and assign one of a vein classification or an artery classification to each blood vessel of the blood vessels.
  • the ultrasound system can also include a processor to determine, for each blood vessel, a confidence level for the vein classification or the artery classification, and cause display in the one or more ultrasound images of an outline of at least one blood vessel of the blood vessels with an opacity based on the confidence level determined for the at least one blood vessel.
  • a method is implemented by a computing device for identifying blood vessels, such as an ultrasound machine or a tablet coupled to the ultrasound machine, and includes receiving an ultrasound image that includes the blood vessels, and determining, with a neural network implemented at least partially in hardware of the computing device, locations and depths of the blood vessels in the ultrasound image. The method also includes determining, based on the locations and the depths, one of the blood vessels as being more centered in the ultrasound image than other blood vessels of the blood vessels and unobstructed by the other blood vessels, and indicating, in the ultrasound image, the one of the blood vessels.
  • FIGS. 1 A and 1 B illustrate an example of detecting two blood vessels and identifying them with boxes.
  • FIGS. 2 A and 2 B illustrate detecting an outline of a vein.
  • FIG. 3 illustrates an example of an oval with intercrossing diameters.
  • FIGS. 4 A and 4 B represent one image without probe pressure (compression) applied and one with probe pressure (compression) applied.
  • FIGS. 5 A and 5 B illustrate enhancement on a detected vein.
  • FIG. 6 illustrates an example of vein diameters used for calculation.
  • FIG. 7 illustrates a flow diagram of one embodiment of a process for displaying veins on an ultrasound image.
  • FIGS. 8 A- 8 D illustrate an example of a process for displaying blood vessels.
  • FIG. 9 illustrates a flow diagram of one embodiment of a process for manipulating a display of blood vessels (or other body structure) on a monitor.
  • FIGS. 10 A- 10 C illustrate one embodiment of a process of displaying an image of a previous detection result in response to the sudden disappearance of the detection result.
  • FIGS. 11 A- 11 D illustrate an example of an enhanced display of an ultrasound image produced when a probe is in motion.
  • FIG. 12 illustrates a flow diagram of one embodiment of a process for displaying one or more blood vessels (or other body structure) as detection results on a monitor with at least one hypothetical detection.
  • FIGS. 13 A- 13 D illustrate a period during which the pressure by the probe decreases from the certain level to the released state and the shape of a blood vessel changes from another shape back to the normal shape.
  • FIG. 14 A- 14 D illustrate the period (ii) during which the compression by the probe decreases from the certain level to the released state and the shape of a blood vessel changes from the additional shape back to the normal shape.
  • FIG. 15 illustrates a flow diagram of one embodiment of a process for using detection results of one period for creation of detection results for another period.
  • FIG. 16 illustrates a flow diagram of one embodiment of a process for using detection results of one period for creation of detection results for another period.
  • FIG. 17 illustrates an example of an image with a best image of a detected blood vessel superimposed on the image.
  • FIGS. 18 A and 18 B show an example of guide information, together with information regarding a best probe position.
  • FIG. 19 A is a flow diagram of another embodiment of a process for identifying blood vessels.
  • FIG. 19 B is a flow diagram of one embodiment of a process for identifying blood vessels in which a misclassification of a blood vessel occurs.
  • FIG. 19 C is a flow diagram of another embodiment of a process for identifying blood vessels.
  • FIG. 19 D is a flow diagram of yet another embodiment of a process for identifying blood vessels.
  • FIG. 20 A is a flow diagram of yet another embodiment of a process for identifying blood vessels for uses such as a medical procedure.
  • FIG. 20 B is a flow diagram of another embodiment of a process for guiding instrument insertion with an ultrasound machine.
  • FIG. 21 is a flow diagram of one embodiment of a process for guiding instrument insertion.
  • FIG. 22 illustrates a block diagram of one embodiment of an ultrasound machine.
  • FIG. 23 illustrates an example of a hand-held ultrasound machine.
  • FIG. 24 illustrates a data flow diagram of a blood vessel identification and display subsystem.
  • ultrasound machine For purposes herein the term “ultrasound machine”, “ultrasound system”, and “ultrasound imaging system” may be used interchangeably.
  • the ultrasound machine executes detection software to perform blood vessel detection on areas of an ultrasound image, and the results of that detection are displayed on a monitor or display of the ultrasound machine, such as, for example, a clinical display or a tablet coupled to the ultrasound machine.
  • the execution can be performed by one or more processors or execution engines.
  • the ultrasound machine performs blood vessel detection using template matching, artificial intelligence (AI) or machine-learning (e.g., adaptive boosting (adaboost), deep-learning, supervised learning models, support vector machine (SVM), sequence models including recurrent neural networks (RNN), Gated Recurrent Unit (GRU), convolutional GRID (ConvGRU), long short-term memory (LSTM), etc., to process frame information in sequence, etc.), and/or another suitable detection method.
  • AI artificial intelligence
  • SVM support vector machine
  • sequence models including recurrent neural networks (RNN), Gated Recurrent Unit (GRU), convolutional GRID (ConvGRU), long short-term memory (LSTM), etc.
  • LSTM long short-term memory
  • the ultrasound machine can execute an AI algorithm and/or use a neural network to identify veins and arteries and locate them in an ultrasound image.
  • the ultrasound machine displays the blood vessels on a monitor or display of the ultrasound system.
  • the ultrasound machine displays the detected blood vessels in an enhanced manner to provide information to the operator, e.g., a user of the ultrasound machine.
  • the ultrasound machine can draw outlines or other forms of blood vessel indicia (e.g., identifiers) around or in the proximity of the blood vessels.
  • an outline of a vein can be changed to match the color coding of the largest catheter that could fit within that vein to the extent of catheter-to-vein diameter ratio (e.g., 0.45, or 0.33).
  • catheter-to-vein diameter ratio e.g. 0.45, or 0.33
  • an operator may select a catheter size based on the blood vessels and their indicia being displayed.
  • the ultrasound machine can identify all veins in an ultrasound image that are appropriate for a particular catheter size.
  • an operator may select a catheter size on a user interface of the ultrasound machine, and in response, the ultrasound machine can designate the veins in the ultrasound image suitable for the catheter size (e.g., based on ratios of the catheter size to diameters of the veins), such as by changing a color of an indicator of the veins suitable for the catheter size, removing indicators of veins that are not suitable for the catheter size, combinations thereof, and the like.
  • an operator is able to touch a vein in the ultrasound image and have the ultrasound machine display the diameter and depth for that blood vessel.
  • the ultrasound machine can automatically identify the most central and shallowest vein, and automatically provide the diameter and depth for that vein on a display of the ultrasound machine.
  • a triad is a vein-artery-vein collection where a central artery is closely bounded on either side by a vein.
  • the AI can distinctly detect and classify the veins and arteries in the triad as individual components. However, because of the unique configuration of the triad structure, this grouping can be detected as an additional unique classification alongside veins and arteries. By doing so, the overall accuracy of the detection can increase as the triad always occurs in vein-artery-vein configuration.
  • a group of three closely detected vessels for example, vein, vein, artery, where the central vein is bounded by a vein and an artery, is likely a misclassification or misdetection.
  • this group is likely a misclassification or misdetection.
  • the ultrasound machine calculates a likelihood value for each detected blood vessel as an indication of a confidence level associated with the detection results.
  • the likelihood value can include a value between zero and one that indicates a confidence level of a classification of a blood vessel as an artery or vein.
  • the ultrasound machine generates indicators (e.g., outlines, circles, ellipse, boxes, etc.) of blood vessels, and adjusts an opacity of the indicators based on the likelihood value that represents a confidence of the prediction for that blood vessel. This adjustment of opacity results in improvements over conventional ultrasound systems that do not adjust opacity of an indicator based on a confidence level for the indicator.
  • the clinician or operator of the ultrasound machine is exposed to additional information that is useful for selecting a blood vessel for a medical procedure, such as catheterization.
  • the opacity adjustment makes the display more visually appealing and less confusing, since based on the changing opacity, blood vessels fade in and out gradually over time, rather than suddenly appearing and disappearing.
  • the ultrasound machine tracks blood vessels over multiple frames (e.g., a frame can represent one ultrasound image in a series of ultrasound images).
  • the ultrasound system can determine blood vessels in one ultrasound image, and track the blood vessels in subsequent ultrasound images in a series of ultrasound images. The tracking can be based on properties of the blood vessels, such as locations of the blood vessels, diameters of the blood vessels, classifications as veins or arteries, combinations thereof, and the like.
  • the ultrasound machine can determine that a blood vessel in a first ultrasound image is the same blood vessel in a second ultrasound image based on one or more of the properties of the blood vessel in the first and second ultrasound images.
  • the ultrasound machine determines when blood vessel detection fails and produces hypothetical detection results to display a blood vessel on an ultrasound image. For example, the ultrasound machine can detect a blood vessel in multiple ultrasound images, and then fail to detect the blood vessel in a subsequent ultrasound image (e.g., an ultrasound image that follows the multiple ultrasound images in a video sequence). The ultrasound machine can compare the detection results for the ultrasound images in the video sequence, and declare a detection failure in the subsequent ultrasound image based on the comparison. Based on the detection failure, the ultrasound system can generate a hypothetical detection result, such as, for example, a bounding box, and display the hypothetical detection result in the subsequent ultrasound image to designate the location of the blood vessel, despite that the blood vessel was not detected in the subsequent ultrasound image.
  • a hypothetical detection result such as, for example, a bounding box
  • the ultrasound machine determines a desired entry point for an interventional instrument.
  • the ultrasound machine can then calculate the distance from a transducer face (e.g., an edge of a probe) to the desired entry point, and display an indicator of the distance on a display of the ultrasound machine.
  • the ultrasound machine can display a message with text that includes the distance, and/or an arrow that indicates a direction to move the probe in accordance with the distance.
  • the techniques disclosed herein provide for identifying blood vessels in ultrasound images and displaying the blood vessels in a number of ways to convey useful information to an operator. Accordingly, the ultrasound systems disclosed herein can be suitable for medical procedures in which conventional ultrasound systems are not suitable, since the conventional ultrasound systems can result in undesired consequences for a patient, including discomfort, loss of blood from multiple punctures, risk of infection, and the like.
  • the techniques and systems disclosed herein are not limited to blood vessels and can be used with other body structures, such as nerves, muscles, skeletal parts, and the like.
  • the discussion herein focuses on peripheral intravenous catheterization, the techniques and systems disclosed herein are not limited to catheters, and can be used with any suitable interventional instrument, such as a needle, stint, clamp, guide, etc.
  • an ultrasound system determines a diameter of a blood vessel based on one or more previous ultrasound images. For instance, the ultrasound system can determine the diameter of a blood vessel in each of multiple previous frames (e.g., previous 2, 3, 4 or more frames), and generate a diameter for the blood vessel in a current frame based on the diameters of the blood vessel from the multiple previous frames.
  • multiple previous frames e.g., previous 2, 3, 4 or more frames
  • the ultrasound system can prevent the undesirable fast changing of an indicator of the blood vessel (e.g., a color of a bounding box of the blood vessel that can correspond to a catheter size), which can allow an operator to understand the true size of the blood vessel and take a suitable action, such as selecting the appropriate size of a catheter.
  • an indicator of the blood vessel e.g., a color of a bounding box of the blood vessel that can correspond to a catheter size
  • the ultrasound system generates ultrasound images that include one or more blood vessels, and determines diameters of the blood vessels in the ultrasound images.
  • the diameters can include a respective diameter for each blood vessel in each ultrasound image of the ultrasound images (e.g., an ultrasound video stream).
  • the ultrasound system can include a neural network that determines the sizes and locations of the blood vessels.
  • the neural network can be implemented at least partially in hardware of a computing device (e.g., the ultrasound machine, a computing device coupled to the ultrasound machine, such as a tablet, combinations thereof, and the like).
  • the ultrasound system can calculate a blood vessel diameter based on diameters of a blood vessel.
  • the ultrasound system can generate a blood vessel diameter for use in displaying the blood vessel.
  • the ultrasound machine can select a color based on the blood vessel diameter, and then generate an indicator based on the blood vessel diameter and the selected color to provide an indication in at least one ultrasound image, such that the blood vessel is displayed with the indicator having the color.
  • FIGS. 1 A- 6 illustrate examples of this process.
  • the process starts with the processor of the ultrasound machine executing an image analysis algorithm to perform image analysis on ultrasound images, resulting in the detection of a blood vessel in the ultrasound images.
  • the processor can execute two algorithms, one to detect the blood vessel and another to determine the type of the detected blood vessel (e.g., whether the blood vessel is a vein or an artery).
  • the algorithms can include one or more neural networks, such as a first neural network trained to detect a location of a blood vessel, and a second neural network trained to classify the blood vessel as a vein or artery based on the detection results of the first neural network.
  • these two neural networks can be integrated into a single algorithm.
  • a single neural network comprising a first head of the detection and a second head of the classification is included in a single algorithm, such that the processor includes and can execute the single algorithm.
  • FIG. 1 A illustrates an example of detecting two blood vessels and identifying them with boxes 101 and 102 , respectively, that have a property, such as a color, line width, line shape (e.g., solid or dashed), etc.
  • the boxes 101 and 102 are examples of bounding containers for the blood vessels in FIG. 1 A , and can be generated with a processor in the ultrasound system using information from the neural network.
  • the ultrasound system can then classify the blood vessels as veins or arteries, such as with an additional neural network. After classifying the blood vessels as veins or arteries, the ultrasound system can change a property of one or both of the boxes 101 and 102 to act as an indicator of the classification.
  • FIG. 1 B after the ultrasound system classifies both blood vessels as veins, the ultrasound machine changes the color of the boxes 101 and 102 in FIG. 1 B (represented with hashing) compared to the color of the boxes 101 and 102 in FIG. 1 A .
  • the ultrasound system determines an outline of a detected blood vessel, and approximates the determined outline as an oval. For instance, the ultrasound system can match an oval to one or both of the boxes 101 and 102 . Additionally or alternatively, the ultrasound system can include an implementation of binarizing technology, contour detection technology, or segmentation technology to determine an outline for a blood vessel.
  • FIGS. 2 A and 2 B illustrate detecting an outline of a blood vessel (e.g., a vein).
  • a blood vessel e.g., a vein
  • FIG. 2 A two blood vessels 201 and 202 are shown. Contouring by binarization/edge-detection can be performed inside a box-shaped area (not shown) selected in the image of FIG. 2 A .
  • the ultrasound system can match inner ovals 211 and 212 to boxes, as shown in FIG. 2 B . In these examples, the ultrasound system produces ovals as bounding containers for the blood vessels (e.g., veins).
  • the ultrasound system can determine diameters of blood vessels using the ovals produced as bounding containers.
  • FIG. 3 depicts a major axis and a minor axis (together major/minor axes 301 and 302 for two blood vessels, respectively).
  • the ultrasound system can determine a diameter of one of the blood vessels from the major/minor axes 301 , and a diameter of the other of the blood vessels from the major/minor axes 302 .
  • the ultrasound system can average the major and minor axes of major/minor axes 301 to determine, as a blood vessel diameter, a diameter of the blood vessel on the left side of FIG.
  • the ultrasound system can generate a blood vessel diameter, a diameter of a blood vessel from the weighted average of major and minor axes of an oval (e.g., an ellipse) that represents a bounding container for the blood vessel. For instance, the ultrasound system can weigh one of the axes more heavily than the other of the axes when forming an average.
  • an oval e.g., an ellipse
  • the ultrasound system determines, as a blood vessel diameter, a diameter of a blood vessel from the major and minor axes of an ellipse by matching a property of the ellipse to a property of a circle. For instance, the ultrasound system can determine the diameter of the blood vessel as a diameter of a circle that has the same area as the ellipse that represents the bounding container of the blood vessel. Let the major and minor axes of the ellipse be d 1 and d 2 , respectively. The ultrasound system can determine the diameter of the blood vessel, d, as the diameter of a circle with a same area as the ellipse, or
  • ⁇ ⁇ ( d 2 ) 2 ⁇ ⁇ d 1 2 ⁇ d 2 2 .
  • the ultrasound system can determine, as a blood vessel diameter, the diameter of a blood vessel from a diameter of a circle that has the same circumference as the ellipse that represents the bounding container of the blood vessel.
  • the ultrasound system can determine the diameter of the blood vessel from
  • the ultrasound system can set the diameter of the blood vessel according to
  • the ultrasound system detects an artery, and displays the artery in an enhanced way with an indicator that is different from an indicator used to identify a vein.
  • the indicator used for an artery can be a red circle with a red line across the circle to notify the operator not to select the artery for a procedure in which a vein is desired.
  • the operator of the ultrasound machine may turn off a calculation of the artery diameter, and hence may turn off a calculation of blood vessel diameter of the artery.
  • the operator may individually (e.g., one after another in succession) enable or disable enhancements of a vein or artery, such as by first disabling a designator of a vein or artery classification, followed by disabling a displayed diameter of the vein or artery.
  • the ultrasound system applies tracking technology for a detected blood vessel to determine if the same blood vessel appears in a plurality of ultrasound images.
  • the tracking technology can use the diameters and the locations of the detected blood vessels.
  • the blood vessel diameters are calculated in advance to the tracking.
  • the tracking of detected blood vessels comes before calculating the blood vessel diameters. Note that the blood vessel diameter can be calculated before or after tracking because the blood vessel diameter is calculated based on two intercrossing diameters, and the tracking only requires at least calculating these two diameters and does not necessarily require blood vessel diameter.
  • the use of tracking technology also enables calculating a current diameter value of a blood vessel from previously-determined diameters for the blood vessel.
  • the ultrasound system can set the current blood vessel diameter of a blood vessel to a maximum value of previously-determined blood vessel diameters for the blood vessel as it appeared in multiple previous ultrasound images, such as from the previous three to five images.
  • the tracking technology enables determination of the same blood vessels that persist through a series of ultrasound images (frames) (e.g., an ultrasound), in some embodiments, the maximum value is calculated for the same blood vessel.
  • FIGS. 4 A and 4 B represent two images during the pressure of a probe in a time series. There are two veins in FIG. 4 A , vein 401 and vein 402 .
  • the ultrasound system can determine a current value of the blood vessel diameter of vein 402 in FIG. 4 B from one or more values of the blood vessel diameter of the identified vein 402 in a predetermined number of previous images including FIG. 4 A . For instance, if the blood vessel diameter of the vein 402 in FIG. 4 B is 1.5 mm and the maximum blood vessel diameter of the vein 402 in a previous image (e.g., FIG. 4 A ) is 2.0 mm, then the ultrasound system can set the current diameter of the vein 402 in FIG. 4 B as 2.0 mm.
  • a new vein 403 is also shown in FIG. 4 B and does not appear in FIG. 4 A .
  • the ultrasound system can determine a blood vessel diameter of the vein 403 from FIG. 4 B , without using blood vessel diameters of the vein 403 in previous images.
  • the tracking uses at least one or more types of information related to: a point, a line, or an area of a blood vessel region.
  • a point of a blood vessel is tracked between frames, and the ultrasound system measures movement of the point.
  • the information related to the point may include a center, a top, a bottom, a left, a right, or a centroid point of a blood vessel region, and the difference in location of the same point between two ultrasound images is evaluated.
  • Information other than point information may be used in tracking.
  • Examples of other information that may be used for tracking include information related to a line (e.g., diameter, or a length of an outline of a blood vessel region, etc.) and information related to an area (e.g., a shape, an area, or illumination information (texture information) of a blood vessel region, etc.). If the movement is within a threshold distance (e.g., less than 2 mm), then the ultrasound system regards the blood vessels of the frames as the same blood vessel.
  • a threshold distance e.g., less than 2 mm
  • the threshold distance is equal to a percentage (e.g., one-half) of the diameter of the blood vessel in the immediately preceding frame (or another previously-generated frame), or a blood vessel diameter (e.g., an average of the diameters) of the blood vessel in a set of two or more previously-generated and consecutive frames.
  • the ultrasound system displays a detected blood vessel superimposed with an enhancement, such as a color-enhancement.
  • the ultrasound system can determine the color for the color enhancement according to the diameter value determined by the ultrasound system for the blood vessel.
  • FIGS. 5 A and 5 B illustrate the same ultrasound image with different color enhancement on a detected vein on a display of the ultrasound system. The image was obtained under the pressure by a probe, for one example. Specifically, FIG. 5 A illustrates a display of the detected vein when taking into consideration only the blood vessel diameter of the current image, while FIG. 5 B illustrates the detected vein with color enhancement according to calculated blood vessel diameter according to some embodiments.
  • the configuration of the enhancement rule in the ultrasound system is such that, if a current value of a blood vessel diameter is equal to or above 2.0 mm, a vein is circled with a color (e.g., pink), and if that value is below 2.0 mm, the vein is circled with another color (e.g., orange).
  • a current value of a blood vessel diameter is equal to or above 2.0 mm
  • a vein is circled with a color (e.g., pink)
  • the vein is circled with another color (e.g., orange).
  • the vein 501 A is enhanced with the color orange (represented by a solid white ellipse)
  • the vein 501 B is enhanced with color pink (represented by hatching).
  • the ultrasound system is configured to calculate a blood vessel diameter from the previous images that include the maximum value of the blood vessel diameter, 2.0 mm.
  • the shape of the same vein as of 501 A and 501 B would be more circular, for example. Therefore, the enhancement display of FIG. 5 B corresponds to a change of the shape of the blood vessel better than that of FIG. 5 A .
  • FIG. 5 B is effective, for example, in a case where the probe releases and/or compresses the object under ultrasonic scan such that the shape of the vessel changes gradually or abruptly.
  • the ultrasound system uses past images (e.g., previously generated and displayed ultrasound images) to determine a property of a blood vessel, such as a vein size.
  • a blood vessel is in a normal state (e.g., not compressed by a probe).
  • the ultrasound system can determine an image that contains a blood vessel having its maximum diameter from multiple ultrasound images, and use the maximum diameter to determine the blood vessel diameter, thus enabling selection of the blood vessel at its correct size when determining its blood vessel diameter.
  • at least three past frames or at least one second worth of frames can be used to determine when the blood vessel is at its correct size. Note that more or less than these numbers of frames can be used.
  • the ultrasound system can determine the real (or true) value of a blood vessel diameter in its normal state, and hence select the correct color (e.g., pink, according to the diameter) to designate the blood vessel. In this manner, it is possible to mitigate the effect of pressure by the probe.
  • the correct color e.g., pink, according to the diameter
  • the blood vessel diameters are calculated for the subsequent images (frames).
  • the ultrasound system can detect veins and perform the calculation of the blood vessel diameters only of the veins for the subsequent images (frames), and then display enhancements only for the veins, thus encouraging the understanding the properties of the blood vessels in ultrasound images.
  • FIG. 6 illustrates an example of blood vessel diameters used for calculation. Referring to FIG. 6 , the ultrasound system identifies veins and calculates diameters for vein 402 for one frame (image) using the axes that are shown. Artery 403 has a diameter, and the blood vessel diameters of vein 402 and artery 403 are not calculated for subsequent frames.
  • FIG. 7 illustrates a flow diagram of one embodiment of a process for displaying veins on an ultrasound image.
  • the process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as is run on a general-purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or combinations thereof.
  • the process is performed by a processor of an ultrasound machine.
  • FIG. 7 illustrates displaying veins on an ultrasound image, the process can additionally or alternatively be used for displaying arteries.
  • the process begins by processing logic obtaining an image frame, such as an ultrasound image, (processing block 701 ) and performing vein (or blood vessel) detection on the image frame (processing block 702 ). Using the results of vein detection, the processing logic extracts the boundaries of veins (processing block 703 ) and calculates the blood vessel diameters of detected veins (processing block 704 ).
  • an image frame such as an ultrasound image
  • vein block 702 performing vein (or blood vessel) detection on the image frame
  • the processing logic extracts the boundaries of veins (processing block 703 ) and calculates the blood vessel diameters of detected veins (processing block 704 ).
  • processing logic After calculating blood vessel diameters, processing logic performs tracking (processing block 705 ). During tracking, processing logic determines whether each individual detected vein exists in a past frame (processing block 706 ). If not, the process transitions to processing block 708 . If so, processing logic calculates the maximum blood vessel diameter of that vein based on its blood vessel diameters in previous frames (processing block 707 ) as a current value of the blood vessel diameter and transitions to processing block 708 .
  • processing logic determines whether all the veins have been tracked and whether the maximum blood vessel diameters have been calculated for those veins being tracked. If not, the process transitions back to processing block 705 where the process continues. If so, the process transitions to processing block 709 where processing logic displays each of the veins in an ultrasound image with a color or other indicator selected based on its maximum diameter.
  • the current value of the blood vessel diameter is calculated in one of a number of different ways.
  • the current value of the blood vessel diameter of a detected blood vessel can be determined from the maximum, minimum, average, median, mode, standard deviation, or a max-min value from the blood vessel diameters of the same vein in the past several frames.
  • the ultrasound machine determines the calculation algorithm in advance of an examination, or before imaging the blood vessel or using any enhancement of the blood vessel in the image (e.g., indicator, color, etc.). This determination can be part of a configuration or boot-up process of the ultrasound machine.
  • the calculation algorithm e.g., maximum, minimum, average, etc.
  • the operator of the ultrasound machine is selected by the operator of the ultrasound machine.
  • One benefit of choosing standard deviation or a max-min value is that a normal vein can compress and recover easily during the pressure by the probe, and if a vein is not in a healthy condition, it is not so elastic.
  • One benefit of choosing the average, median, or mode value is that it results in the exclusion of an extraordinary value that unintentionally or erroneously appears during one or more frames.
  • Another benefit of choosing the average, median, or mode value is a compromise between the two points: (i) a smaller gauge catheter would cause less damage to the blood vessel, but (ii) a smaller gauge catheter would also transport less liquid.
  • a medical instrument/tool such as, for example, but not limited to, needle or catheter, is smaller than the diameter of the blood vessel.
  • the processor is configured to determine the calculation algorithm automatically (e.g., without explicit user selection of the calculation algorithm). In some embodiments, the selection is based on whether the probe is pressing into a subject being examined (as determined, for example, by a pressure sensor in the probe), thereby reducing the size of the blood vessel. In one such embodiment, (i) if the probe is pressing on the same part of the object, the processor chooses the maximum calculation algorithm, and (ii) if the probe is moving along the blood vessel (e.g., the longitudinal direction), the processor chooses the average calculation algorithm. In some embodiments, to distinguish between (i) and (ii), the ultrasound machine performs image analysis to determine whether the periphery of the detected blood vessels has changed or not. Additionally or alternatively, a location sensor (e.g., magnetic sensor, an accelerometer, a gyro sensor, etc.) inside the probe can be used to distinguish between (i) and (ii).
  • a location sensor e.g., magnetic sensor, an accelerometer, a
  • the ultrasound machine marks the displayed blood vessels with a colored indicator (e.g., a circle, an ellipse, etc.).
  • the color can be based on whether the detected blood vessel is an artery or a vein.
  • the ultrasound machine can mark an artery with a red indicator (e.g., a red circle, or a crossed-out circle icon).
  • the ultrasound system can mark a blood vessel with an indicator having a color based on which size (e.g., gauge) of needle or catheter could be inserted into the blood vessel.
  • the color can be in accordance with a standard color chart that assigns unique colors to instrument gauges.
  • blood vessels having different ranges of diameters are enhanced by using different colors.
  • only veins having different ranges of diameters are enhanced by using colors that are different from the color of enhancement used for arteries, since in one configuration of the ultrasound system blood vessel diameters are calculated only for veins.
  • the ultrasound machine detects and displays some enhancement on blood vessels irrespective of which blood vessels are of interest to an operator of the ultrasound machine.
  • the display of enhancements on blood vessels can include one or more enhanced blood vessels that an operator has a desire to see along with one or more other enhanced blood vessels in which the operator has expressed no interest in viewing.
  • the operator can input a desire to view or not view a particular blood vessel via a user interface of the ultrasound machine.
  • the operator is able to remove enhancements on unwanted blood vessels from the display.
  • the operator may select a detected blood vessel to cause the enhancements on the detected blood vessel to switch between an ON-state in which the blood vessel is displayed with enhancements (e.g., a bounding container, a color, a color based on type (e.g., vein or artery), a color based on the size of a needle that is insertable into the blood vessel, text indicating a diameter, an icon, such as a stop sign, thumbs up, or thumbs down, combinations thereof, etc.) and an OFF-state in which the detected blood vessel is displayed without any enhancements.
  • enhancements e.g., a bounding container, a color, a color based on type (e.g., vein or artery), a color based on the size of a needle that is insertable into the blood vessel, text indicating a diameter, an icon, such as a stop sign, thumbs up, or thumbs down, combinations thereof, etc.
  • FIGS. 8 A- 8 D illustrate an example of a process for displaying enhancements on blood vessels.
  • blood vessels that include vein 811 , artery 812 , and vein 813 are shown and enhanced on monitor 800 .
  • monitor 800 includes a touch-sensitive screen type display or touch surface display device.
  • the three blood vessels 811 - 813 are detected by artificial intelligence (AI). Additionally or alternatively, the detected blood vessels can be marked with a colored indicator (e.g., circle or ellipse) by AI.
  • the operator touches monitor 800 at the location of one of the three blood vessels, such as vein 813 , in FIG. 8 A the colored circle disappears as shown in FIG. 8 B .
  • FIG. 8 C if the operator touches monitor 800 at the location of the one blood vessel that was previously touched to make the circle disappear, the circle appears on the blood vessel again as shown in FIG. 8 D .
  • FIG. 9 illustrates a flow diagram of one embodiment of a process for manipulating an enhanced display of blood vessels (or other body structure) on a monitor.
  • the process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as is run on a general-purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or combinations thereof.
  • the process is performed by a processor of an ultrasound machine.
  • the process begins by processing logic displaying enhancements on one or more blood vessels (or other body structure) on a monitor of an ultrasound machine with colored circle (or with another indication of other enhanced manner) (processing block 901 ).
  • the monitor is a touch-sensitive screen type monitor or touch surface display device.
  • processing logic receives an indication that a user has touched on the monitor (processing block 902 ). Additionally or alternatively, the user could make a selection on the monitor by moving a cursor with a cursor control device to a location on the monitor and pressing a selection button.
  • processing logic determines the location of the touch and whether a blood vessel was being enhanced at that location on the monitor when the touch occurred (processing block 903 ). If so, processing logic causes the colored circle being displayed with the blood vessel (or any other indicator/enhancement being used in conjunction with the display of the blood vessel) to be removed from the display (processing block 904 ).
  • processing logic determines if a blood vessel is being detected at the location of the touch but its enhancement is being suppressed from the display (processing block 905 ), and if so, causes the colored circle (or other indicia/enhancement) to reappear on the monitor (processing block 906 ).
  • the process of FIG. 9 can repeat as blood vessels are displayed on the monitor and the operator touches the monitor. For instance, as an operator moves the ultrasound probe, new blood vessels can appear and previous blood vessels can disappear from view.
  • the ultrasound machine can repeat the process of FIG. 9 for the newly-displayed blood vessels when the operator touches the monitor.
  • the ultrasound system generates ultrasound images continuously and continues the detection process of blood vessels and the tracking technology (and additionally the calculation of the blood vessel diameter) on the generated images regardless of the ON-state or OFF-state of the enhancement, such that if an operator touches a blood vessel that was not previously enhanced on the monitor, a circle with an appropriate color is displayed at the blood vessel, wherein the color is additionally based on the calculated blood vessel diameter and/or the standard color corresponding to a catheter size.
  • the enhancement to the ultrasound images comes in the form of indicating a blood vessel that was not detected.
  • some abrupt change in the examination circumstance can cause the detection results (e.g., a colored circle on a detected blood vessel of an ultrasound image) to disappear suddenly.
  • the detection results e.g., a colored circle on a detected blood vessel of an ultrasound image
  • the blood vessel of an ultrasound image can suddenly collapse, which can cause no detection result to be displayed on the monitor.
  • This lack of detection display can be troubling to a user, because the user may still see the actual location of the collapsed blood vessel in a B-mode image.
  • the processing logic of the ultrasound machine fails to detect a blood vessel in a current image, the detection results of one or several previous ultrasound images are searched.
  • the processing logic can determine a hypothetical detection result based on the detection results of one or more of those previous ultrasound images, and the hypothetical detection result can be superimposed on the current image and displayed by the ultrasound machine.
  • the ultrasound machine only displays the hypothetical detection results for two to three frames to prevent flickering.
  • the ultrasound machine determines diameters and locations of the blood vessels in a subset of the ultrasound images. As discussed above, this determination can be made using a neural network or other detection mechanism implemented at least partially in hardware of a computing device, such as the ultrasound machine itself, a computing device coupled to the ultrasound machine, or combinations thereof. Alternatively or additionally, the determination can be made using hardware (e.g., circuitry, dedicated logic, etc.), software (such as is run on a general-purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or combinations thereof.
  • hardware e.g., circuitry, dedicated logic, etc.
  • software such as is run on a general-purpose computer system or a dedicated machine
  • firmware e.g., software programmed into a read-only memory
  • the ultrasound machine Based on the diameters and locations of blood vessels that are found in each of the images, the ultrasound machine is able to determine whether a detection failure for one of the blood vessels in one of the ultrasound images occurred because the blood vessel was absent from those blood vessels identified by the neural network (or other detection mechanism). In response to that determination, the ultrasound machine can attempt to track the previously detected blood vessel by alternative means and display an indicator of the diameter and the location of the one of the blood vessels in one of the ultrasound images. In some embodiments, the indicator is based on the diameters and the locations determined by the neural network for the subset of the ultrasound images.
  • FIGS. 10 A- 10 C illustrate one embodiment of the process of displaying an image of a previous detection result in response to a sudden disappearance of a detection result. This situation may occur when the probe is compressing gradually, for instance. As a blood vessel is compressed further, the shape of the blood vessel becomes collapsed. In such a case, the detection of such a collapsed blood vessel only from the current image becomes much more difficult.
  • FIG. 10 A blood vessels 1001 and 1002 are displayed on monitor 800 as a result of the detection process determining their presence.
  • FIG. 10 B while detection continues, blood vessels 1001 and 1002 continue to be displayed on monitor 800 , but blood vessel 1002 is shown with an elliptical shape (as opposed to the circular shape of blood vessel 1002 in FIG. 10 A ).
  • FIG. 10 C illustrates that detection only shows blood vessel 1001 being displayed, as blood vessel 1002 is not being displayed.
  • FIG. 10 C reflects the result of the last successful detection of the blood vessel 1002 .
  • the ultrasound machine can fail to detect the blood vessel 1002 in FIG. 10 C .
  • the system displays blood vessel 1002 as a dashed-line ellipse, which represents a hypothetical detection result, even though the detection process did not produce a result for blood vessel 1002 .
  • An advantage of displaying the hypothetical result in the same shape as, but with different types of outline from, the successful result is to prevent an operator from feeling uncomfortable.
  • the color of solid-line (actual) and dashed-line (hypothetical) ellipse can be selected from the calculated result of blood vessel diameters in previous images. Since a current value of the blood vessel diameter is calculated based on those of previous images, the fluctuation of color can be suppressed and only the line of the shape is changed, which makes it easier for an operator to identify the same blood vessel (vein).
  • the hypothetical detection result is displayed in a manner different from that of the actual detection result.
  • the hypothetical detection result is displayed in a dashed-line ellipse, while the actual detection result would be displayed in a solid-line bounding container, such as is shown for blood vessel 1002 in FIG. 10 A and FIG. 10 B .
  • different colors, different types of filling for the bounding container, colors, labels, icons, text, or other types of distinguishing features may be used to distinguish actual detection results from hypothetical detection results. Note also that since blood vessel 1001 is an artery, its shape does not change so much as a vein's shape changes.
  • FIGS. 11 A- 11 D illustrate an example of an enhanced display of an ultrasound image produced when a probe is in motion.
  • blood vessels 1101 - 1103 are shown on the ultrasound image.
  • Blood vessel 1101 is depicted as a vein with a first colored circle
  • blood vessel 1102 is depicted as an artery
  • blood vessel 1103 is not identified as either a vein or an artery (e.g., identified with a dotted circle).
  • FIG. 11 B shows another ultrasound image of the same location where the shape of blood vessel 1101 is represented in a more oval shape with a different enhancement (e.g., a different color) than that of FIG. 11 A .
  • the shape of blood vessel 1101 may be due to the operator pressing the probe more into the body at the location.
  • FIG. 11 C shows another ultrasound image of the same location where the shape of blood vessel 1101 is represented in a smaller oval shape with a different enhancement (e.g., a different color) than that of FIG. 11 B .
  • FIG. 11 D shows another ultrasound image of the same location where the shape of blood vessel 1101 is represented in an even smaller oval shape with a different enhancement (e.g., a different color) than that of FIG. 11 C and with dotted lines. Accordingly, FIGS. 11 A- 11 D illustrate how enhancements for blood vessels can change due to the motion and/or pressure of a probe.
  • FIG. 12 illustrates a flow diagram of one embodiment of a process for displaying one or more blood vessels (or other body structure) as detection results on a monitor with at least one hypothetical detection result.
  • the process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as is run on a general-purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or combinations thereof.
  • the process is performed by a processor of an ultrasound machine.
  • the process begins by processing logic detecting one or more blood vessels from an ultrasound image (processing block 1201 ).
  • the processing logic detects blood vessels using a neural network, template matching, machine-learning (e.g., adaboost, deep-learning, SVM, sequence models including RNN, GRU, ConvGRU, LSTM, etc., to process frame information in sequence, etc.), and others.
  • processing logic can calculate a likelihood value for each blood vessel indicative of a level of confidence that the detection result is a blood vessel (processing block 1202 ).
  • the likelihood value can be between zero and one, with one representing high confidence in the detection result, and zero representing low confidence in the detection result.
  • Processing logic also tracks the detected blood vessels between frames and identifies the same detected blood vessel in different images (processing block 1203 ). In some embodiments, the processing logic uses location information and/or the type of blood vessel for the tracking and/or identification process.
  • Processing logic compares the likelihood value to a threshold, such as 0.7 (processing block 1204 ). If processing logic determines that the likelihood value for the same blood vessel becomes less than the threshold, the processing logic determines that the detection of the blood vessel is failed (processing block 1205 ) and produces a hypothetical detection result for display (processing block 1206 ). In some embodiments, to produce the hypothetical detection result, processing logic searches for the result of one or plural previous ultrasound images and uses that result as the basis for the display of the hypothetical detection result. The hypothetical detection result can be calculated using a calculation method (algorithm) performed in a number of ways. In some embodiments, processing logic obtains the most recent, successful detection result and uses that for the hypothetical detection result.
  • a threshold such as 0.7
  • the processing logic can select a detection result with the highest likelihood value from a predetermined number of previous frames as the hypothetical detection result, such as from the previous two, three, four, or more frames.
  • the number of frames may depend at least partially on the acquisition frame rate; a higher frame rate may enable more frames to be selected.
  • processing logic estimates the shape and/or the location of the blood vessel based on the result of one or more previous frames generated during a predetermined time length in the past, such as from the previous 500 msec of image frames. In an example, this estimation is done by employing motion-vector technology. Additionally or alternatively, this estimation can be done by employing optical-flow technology. Thereafter, processing logic displays the hypothetical detection result in the ultrasound image with any other detection results, actual or hypothetical (processing block 1207 ).
  • the probe of the ultrasound machine has a sensor to detect its motion. If the movement of the probe is above a certain velocity, the processing logic does not display the hypothetical detection result on the monitor. This suppression of the display of the hypothetical detection result is because, when the probe is moving, it is harder for the processing logic to estimate the hypothetical detection result, because it cannot estimate, for example, the symmetric change of shape and/or location of the blood vessel when the probe is moving.
  • FIGS. 13 A- 13 D illustrate period (i) during which the pressure by the probe increases to a certain level and the shape of a blood vessel changes (e.g., collapses) from a normal shape (usually a circular oval) to an additional shape (e.g., a collapsed oval).
  • FIGS. 14 A- 14 D illustrate the period (ii) during which the compression by the probe decreases from the certain level to the released state and the shape of a blood vessel changes from the additional shape back to the normal shape.
  • ultrasound images are collected and blood vessels are detected.
  • the detection of blood vessels is performed using AI (e.g., AI software which implements a neural network).
  • the detection results of ultrasound images that are acquired during the period (i) are analyzed, and results are created and used for the detection/analysis of the ultrasound images during the period (ii).
  • the ultrasound system can use the detection results from period (i) as conditional inputs to a neural network to generate additional detection results for period (ii). Additionally or alternatively, the ultrasound system can use the location and the size of each of the detected blood vessels during the period (i) to predict (e.g., limit) the area for searching the blood vessels during the period (ii), thereby contributing to the fast and correct detection of the same blood vessel during the time period (ii).
  • the ultrasound system can generate a region of interest based on the location and the size of each of the detected blood vessels during the period (i), and provide this region of interest as a conditional input to a neural network, which can then generate inferences for locations and sizes of blood vessels in accordance with the region of interest for the period (ii).
  • the ultrasound machine adjusts for a difference in the collapsing speed and the recovery speed of a blood vessel when determining the detection results of the period (ii) based on the detection results of the period (i). For instance, in an ultrasound examination using the probe, the probe gradually presses the examined part during the period (i), and the operator releases the probe during the period (ii). Therefore, the collapsing speed is generally dependent on the pressure caused by the probe, which is based on the pressing action by the operator.
  • the recovery speed is mainly dependent on the property of the blood vessel itself (i.e., almost a constant speed). As has been observed by inventors, the recovery speed of the blood vessel itself is usually lower than the collapsing speed.
  • FIG. 15 includes a graph illustrating the relationship between the vessel diameter and time.
  • a delay 1501 represents the gap between the actual speed of recovery and that of the predicted speed that is assumed to be as the same as the collapsing speed, and has to be filled.
  • the ultrasound system calculates the collapsing speed by employing the detection of blood vessels, the determination of diameters, and the tracking technology, and then detects the timing of transition from the period (i) and period (ii), and then calculates the (actual) recovery speed by using first some frames of period (ii), and then a coefficient between the actual and the predicted speed is generated. Therefore, using the coefficient value, results of the detection during the period (i) is appropriately reflected in the detection process in the period (ii). Details are also explained using the flowchart of FIG. 16 below. In one example, such a coefficient can be updated by reiterently reflecting the detection results of the period (ii) as the time proceeds because the actual results (i.e.
  • the blood vessel diameter can be used instead of the diameters in FIGS. 15 and 16 .
  • the technology explained with reference to FIGS. 15 and 16 can be based on a case where the period (i) refers to the collapsing speed and the period (ii) refers to the recovery speed, the technology can be also applied to a case where the period (i) refers to the recovery speed and the period (ii) refers to the collapsing speed.
  • FIG. 16 illustrates a flow diagram of one embodiment of a process for using detection results of one period for creation of detection results for another period, for example, as previously described with respect to FIGS. 13 A- 15 .
  • the process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as is run on a general-purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or combinations thereof.
  • the process is performed by a processor of an ultrasound machine.
  • the process starts by processing logic detecting a blood vessel from an ultrasound image (processing block 1601 ).
  • processing logic detecting a blood vessel from an ultrasound image (processing block 1601 ).
  • many different algorithms can be used to detect blood vessels, including, but not limited to, template-matching, machine-learning (e.g., adaboost, deep-learning, SVM, sequence models including RNN, GRU, ConvGRU, LSTM, etc., to process frame information in sequence, etc.), and the like.
  • Processing logic also tracks the detected blood vessel between frames (images) to identify the same blood vessel in these frames (processing block 1602 ).
  • processing logic analyzes the change of the shape, the position, the size, combinations thereof, etc. of the same blood vessel frame by frame (processing block 1603 ).
  • processing logic determines the first period (i) in which the shape transitions from the normal oval (no pressure) to the most collapsed oval (maximum pressure) (processing block 1604 ).
  • processing logic calculates the time of the first period and the collapsing speed of the blood vessel (processing block 1605 ).
  • processing logic obtains information of the blood vessel during period (i).
  • the processor determines the frame which is the starting frame of the recovery of the blood vessel (processing block 1606 ). For instance, the processing logic can equate the recovery speed of the blood vessel with the collapsing speed of the blood vessel. Then, the processor can determine the frame which is the starting frame of the recovery of the blood vessel, e.g., from this starting frame, it enters into the period (ii). The processor can then estimate the position, diameter, or other information of the blood vessel in the current frame, based on the detection results during the period (i) and the time interval between the current time and the time of the starting frame. In other words, at processing block 1606 , some frames of period (ii) are also acquired, such that the time of change from period (i) to period (ii) is determined.
  • the processor can calculate the recovery speed by using the same frames of the period (ii) since these frames are already obtained, thereby a coefficient (corresponding to the delay 1501 of FIG. 15 ) to be applied to detection results during the period (i) is calculated based on the some frames of period (ii).
  • a new, current image (belonging to period (ii)) is obtained. Since the time interval between the current time and the time of change from period (i) to period (ii) and the coefficient are known, the information about the blood vessel at the appropriate time of period (i) is obtained for use in creating the new image. More specifically, the processor can account for differences between the recovery speed and the collapsing speed of the blood vessel. For example, in some embodiments, the processor modifies the detection results of the period (i), such as by multiplying an appropriate coefficient to the detection results of the period (i) to delay the recovery feature, to match the actual recovery speed of the blood vessel (processing block 1607 ).
  • the detection results at an appropriate time in the period (i) are input to the processor (e.g., AI algorithm) to enhance the detection of the blood vessels in the new, current image.
  • the locations and/or the diameters of the blood vessels at an appropriate time in the period (i) are input to the processor (e.g., AI algorithm), thereby the region specified for search can be limited, and/or the size of blood vessel for search can be limited, thereby reducing the burden for the processor.
  • a user may want to search for the best location of a blood vessel into which a needle or other interventional instrument can be inserted.
  • a blood vessel can extend from an elbow to a wrist, from an upper arm to an elbow, etc., and an operator may want to be able to search for a particular location, or position, of the blood vessel for needle insertion.
  • an operator moves a probe from one position to another, thereby obtaining a number of ultrasound images continuously that may be checked. In such a case, the operator may miss the best position for the testing and/or intervention as the probe is moving.
  • ultrasound images that include a blood vessel are obtained.
  • the ultrasound machine can generate the ultrasound images based on ultrasound echo signals received by the ultrasound probe.
  • the ultrasound machine determines at least one ultrasound image of the ultrasound images based on a location of the ultrasound probe with respect to the blood vessel when the ultrasound image is received, and determines one or more features of the blood vessel in the ultrasound image.
  • the ultrasound machine displays an indicator of each feature of the blood vessel in an additional ultrasound image (e.g. in a current ultrasound image).
  • the ultrasound machine selects one image from the past images, based on the analysis (hereafter the selected one image is also referred as ‘best image’).
  • the image selected is the ultrasound image obtained when the probe was located at a desired part of the blood vessel for insertion.
  • the determination of which location is best or desired can be based on predefined criteria.
  • predefined criteria i can include the size of the diameter of the blood vessels, location of the predetermined blood vessels are shallower than a predetermined depth, etc. Examples of such predefined criteria are explained in greater detail below.
  • FIG. 17 illustrates an example of a current (live) image with such a superposition.
  • a vein 1701 is enhanced with an ellipse together with the addition of an L-shaped depth indicator.
  • the L-shaped depth indicator can be additionally displayed if the location of a detected blood vessel is shallower than a predetermined depth.
  • another blood vessel 1702 is shown superimposed on the current image as the information regarding the blood vessel of the best image of a blood vessel.
  • Blood vessel 1703 is shown on the image as well and represents an artery. Hence, blood vessel 1703 is indicated with a strike-through, as opposed to a circle or ellipse lacking a strike-through, to emphasize blood vessel 1703 as an artery rather than a vein.
  • the ultrasound machine can include a probe having a location sensor to provide location information that the processor analyzes.
  • the location sensor includes a gyro sensor or an accelerometer.
  • the images are stored with the location information of the probe in a memory.
  • the ultrasound machine also includes a processor(s) that can detect the blood vessel from an ultrasound image and select an image among a plurality of images as the best image, based on at least one of the following:
  • the information related to the best image can be superimposed on the monitor that is currently displaying live B-mode images.
  • the information includes one or more of the following:
  • FIGS. 18 A and 18 B show an example of the guide information, together with the information regarding the best position.
  • the solid-line oval represents the detected vein in the current image
  • the dashed-line oval represents the detected (same) vein in the past best image.
  • the arrows 1801 and 1802 represent the guide information.
  • arrow 1801 goes into the dashed-line oval, with its distal portion becoming narrower. This arrow configuration indicates that the probe should move to the back side with respect to the monitor.
  • arrow 1802 goes out from the dashed-line oval, with its distal portion becoming wider. This arrow configuration indicates that the probe should move to the front side with respect to the monitor.
  • the guide information is displayed together with the information regarding the detected blood vessel of the best image.
  • the ultrasound machine can display the guide information together with a diameter of the detected blood vessel and/or an indication of the classification of the blood vessel as a vein or artery as an annotation.
  • the ultrasound machine can display the guide information together with a dashed circle of a color that can be selected based at least on the blood vessel diameter and/or the standard color corresponding to a catheter size.
  • the ultrasound machine can provide audible information and/or tactile information to the operator of the ultrasound machine.
  • audible information the frequency and/or the volume is changed according to the relative distance between the position of the best image and the current position.
  • the audible information changes in one or more ways (e.g., becomes louder) to alert the user they are getting closer, or one or more other ways (e.g., becomes quieter) to alert the user they are getting farther away.
  • the power of vibration and/or the pattern of vibration by the main unit (e.g., smartphone or tablet) of the probe is changed to indicate a proximity of the probe to the position of the best image.
  • the movement of the probe can be tracked to determine whether the probe is moved in accordance with the guide information or not.
  • the ultrasound machine can track the movement of the probe to determine whether the probe is moved in accordance with the audible information or not.
  • the ultrasound system can provide an indicator of the wrong direction, such as by displaying a stop-sign icon, caution icon, wrong-direction icon, and the like, and/or playing an audio signal that is not pleasing to the operator, such as the words “stop”, “wrong direction”, etc., or a tone with an unpleasant beep.
  • the ultrasound system can provide an indicator of the correct direction, such as by displaying a proceed icon, traffic light icon with green light illuminated, and the like, and/or playing an audio signal that is pleasing to the operator, such as the words “proceed”, “correct direction”, etc., applause, cheers, or a series of tones with a pleasant harmony.
  • the ultrasound system identifies blood vessels and guides instrument insertion.
  • the ultrasound system includes an image module that generates one or more ultrasound images based on ultrasound echo signals, a neural network module and a processor(s) that operate together to perform blood vessel identification and/or generate and display guidance information for instrument insertion, and a map module that generates a map image indicating locations of blood vessels.
  • These modules can be implemented as any type of module or component in software (e.g., as software instructions that are executable with a processing system), hardware, or combinations thereof, as a standalone application or as a module or component of another device application, and in any type of computing device.
  • the instrument can be a catheter, needle or another medical tool.
  • the neural network module identifies blood vessels in one or more ultrasound images and assigns to each blood vessel a classification as a vein or an artery.
  • the neural network module implements one or more artificial intelligence (AI) algorithms (e.g., neural networks) to identify blood vessels in an image and classify them as veins or arteries.
  • AI artificial intelligence
  • the neural network module operates with a map module of the ultrasound system to identify the blood vessels in the ultrasound images.
  • the map module can be executed by a processor of the ultrasound system and generates a map image that indicates locations of blood vessels in an image identified by the neural network module. That is, the neural network module identifies the blood vessels in an image and then the map module generates a map image that indicates their location.
  • the neural network module is configured to generate a feature map from an image that represents the content of the image in a feature space.
  • the neural network module can then concatenate the feature map and a map image from the map module to identify the blood vessels in the image.
  • the map image can correspond to a previous image in a sequence of ultrasound images in which the neural network module has identified the blood vessels.
  • the neural network module can use the map image from one ultrasound image to identify blood vessels in another ultrasound image.
  • the map module is configured to generate the map image to indicate the locations of blood vessels using bounding containers that can include outlines of the blood vessels.
  • the map image can include just the bounding containers to designate the locations of blood vessels and no ultrasound image data.
  • the map module can be configured to generate the map image as a mask that indicates the locations of the blood vessels with a first color, or pixel value, (e.g., white), and additional locations in the map image that do not correspond to the blood vessels with another color, or second pixel value, (e.g., black).
  • the neural network module adapts over time.
  • the AI algorithm(s) performed by the neural network module can be modified such as by changing one or more neural network coefficients. These coefficients can be adjusted based on a function (e.g., a loss function, L2 norm, etc.), and the coefficient adjustments can be based on results of identifying and/or determining the locations of the blood vessels in one image or multiple images (e.g., images in the same video, same examination, same procedure, etc.).
  • a function e.g., a loss function, L2 norm, etc.
  • FIGS. 19 A- 21 are examples of processes for blood vessel identification and/or generating and displaying guidance information.
  • FIG. 19 A is a flow diagram of another embodiment of a process for identifying blood vessels.
  • the process may be related to the technique described in conjunction with FIGS. 1 A and 1 B .
  • the process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as is run on a general-purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or combinations thereof.
  • the process is performed by a neural network and/or processor of an ultrasound machine.
  • the process begins by processing logic determining bounding containers of blood vessels in ultrasound images (processing block 1901 ). Boxes 101 and 102 in FIG. 1 A are examples of bounding containers determined by the processing logic in processing block 1901 .
  • the processing logic determines the bounding containers of blood vessels in the ultrasound images using a neural network of the neural network module.
  • the neural network is implemented in at least partially hardware of a computing device. However, the neural network can be implemented at least partially in software, hardware, and/or firmware.
  • processing logic determines amounts that blood vessels have been compressed based on the bounding containers (processing block 1902 ).
  • the processing logic determines the compression amounts of the blood vessels by determining first lengths of the bounding containers (e.g., a horizontal length across the bounding containers), second lengths of the bounding containers (e.g., a vertical length across the bounding containers), and then ratios of the first lengths to the second lengths.
  • the processing logic can determine the compression amount of a blood vessel as a ratio of a major axis to a minor axis of an ellipse (e.g., a bounding container) that denotes the blood vessel.
  • processing logic assigns a vein classification or an artery classification to classify each blood vessel in the ultrasound images (processing block 1903 ). In one embodiment, processing logic assigns the vein or artery classification using a neural network. In one embodiment, processing logic determines the amount that blood vessels have become compressed and assigns the vein/artery classifications for blood vessels in one ultrasound image. The determination and assignment can be based on the compression amounts and assigned vein/artery classifications for blood vessels in one or more additional ultrasound images (e.g., images in the same sequence or video of ultrasound images in the one ultrasound image).
  • processing logic determines labels for the blood vessels based on their compression amounts and their vein/artery classifications (processing block 1904 ).
  • the labels indicate, for each of the blood vessels, whether they are classified as a vein or an artery.
  • processing logic determines labels for the blood vessels by comparing the compression amounts to a compression threshold. If the compression amount is greater than or equal to the compression threshold, the processing logic can assign a vein classification to the blood vessel. Otherwise, if the compression amount for the blood vessel is less than the compression threshold, the processing logic can assign an artery classification to the blood vessel.
  • processing logic displays the labels for the blood vessels in at least one ultrasound image (processing block 1905 ).
  • the processing logic can display the labels as part of another designator, such as by color-coding a bounding container, with the color based on the vein/artery classification. Additionally or alternatively, the processing logic can display the labels separate from bounding containers, such as with an ‘A’ for artery and ‘V’ for vein.
  • the ultrasound machine uses template matching, centroid tracking, or other object tracking methods to monitor a detection (e.g., blood vessel detection) that is made throughout a video sequence of ultrasound images.
  • a detection e.g., blood vessel detection
  • the ultrasound machine can track a detection from frame to frame.
  • such a tracking technology can be performed between the block 1901 and the block 1902 , or can be embedded in the block 1901 or 1902 by the processing logic.
  • the ultrasound machine determines a detection for a blood vessel was a vein for 10 frames and then the blood vessel is suddenly classified as an artery, it is likely that the artery classification is a misclassification which should be overturned in favor of a classification as a vein. Applying this method over a few frames prevents flickering between classes for a continuous vessel.
  • the neural network can make another determination that classifies a blood vessel, previously classified as a vein, as an artery (or vice versa), which is a misclassification.
  • the ultrasound system is able to correct the misclassification for purposes of displaying the vein/artery classification in the ultrasound image in the display.
  • FIG. 19 B is a flow diagram of one embodiment of process for identifying blood vessels in which a misclassification of a blood vessel occurs.
  • the process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as is run on a general-purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or combinations thereof.
  • the process is performed by a neural network and/or processor of an ultrasound machine.
  • the process begins with processing logic determining bounding containers (processing block 1908 ), performing tracking to identify the same blood (processing block 1908 ), and determining compression amounts (processing block 1908 ). Note that these operations are not performed in all embodiments and thus may not performed prior to the remaining operations in FIG. 19 B .
  • processing logic assigning a classification as a vein or classification as an artery to one or more blood vessels is based on assigning the classification as a vein or an artery in at least one additional ultrasound image.
  • assigning the blood vessel a classification as a vein or an artery in one ultrasound image is based on the ultrasound image itself and at least one set of additional ultrasound images.
  • these additional ultrasound images are gray scale images and assigning a vein/artery classification to one blood vessel in the one ultrasound image includes receiving gray scale images on separate color channels by the neural network and having the neural network use those images to make vein/artery classifications.
  • processing logic determines a misclassification for one blood vessel of the one or more blood vessels (processing block 1912 ).
  • the misclassification denotes the neural network assigning the one of the vein classification and the artery classification to the one blood vessel in one ultrasound image of the ultrasound images and the other of the vein classification and the artery classification to the one blood vessel in additional ones of the ultrasound images.
  • the misclassification denotes that the neural network has assigned a blood vessel classification as a vein or an artery in one ultrasound image and the classification should have been the opposite based on the classification that occurred in one or more additional images.
  • the determination that a misclassification occurred may be the result of processing logic detecting a blood vessel in multiple ultrasound images and then failing to detect the blood vessel in a subsequent ultrasound image (e.g., an ultrasound image that follows the multiple ultrasound images in a video sequence).
  • the failure to detect the blood vessel in a subsequent image can be determined based on the likelihood value associated with the classification of the blood vessel falling below a threshold that needed to be achieved in order to be classified as it was in the previous images.
  • processing logic can use a hysteresis method by which a vessel is very confidently detected at some point and then its probability threshold falls to a point that must be reached to no longer be considered a valid detection.
  • the determination of the misclassification can be based on the condition that the likelihood value for the one of an artery or a vein has been higher than that for the other (and optionally higher than the threshold) in multiple images and suddenly becomes lower (and optionally lower than the threshold) in a subsequent ultrasound image.
  • the one ultrasound image and the additional ultrasound image can represent frames in an ultrasound video.
  • the one ultrasound image represents a later frame in the ultrasound video than the additional ultrasound images that represent previous frames in the ultrasound video.
  • the previous frames represented by the additional ultrasound images are consecutive frames in the ultrasound video.
  • the process includes receiving a user selection that indicates a time interval over which the blood vessel determination should be made.
  • the neural network can use the one ultrasound image and the additional ultrasound images representing frames in an ultrasound video in determining the misclassification based on the number of frames corresponding to or covered by the time interval.
  • processing logic displays, in the one ultrasound image, an indication of the other of the vein classification and the artery classification for the one blood vessel (processing block 1913 ).
  • processing logic displays the correct classification for the misclassified blood vessel as a vein or artery in the displayed image.
  • the processing logic designates the correct classification for a misclassified blood vessel in a manner to indicate that the classification has been corrected for the misclassified blood vessel, such as with shaded, hashed, dotted, etc. indicators of the classification.
  • such an indication of the corrected classification can be in a different from than that of the confident classification.
  • traditional tracking could help compensate for frames in which there was a missed detection, as the tracking can identify the same blood vessel in consecutive frames. For example, if a vessel is centered on the screen and appears to not be compressed for 10 frames, and then in the next frame it is no longer detected, then it is possible to estimate its position using template matching, information from other detected vessels, or traditional tracking, and persist the detection over several frames, giving time for the detector to start detecting the vessel again. To prevent persistence of vessels that shouldn't be persisted, certain rules such as not persisting frames with compressed aspect ratios or those very near the edges could be implemented. As another example, the tracking can help search for likelihood values of the same blood vessel.
  • the processor can track a blood vessel over a duration of a video based on detections of the blood vessel, the processor can ensure that a blood vessel that was previously compressed will continue to be classified as a vein, adding an element of information from previous frames about the nature of the blood vessel to the detection in subsequent frames.
  • FIG. 19 C is a flow diagram of another embodiment of a process for identifying blood vessels.
  • the process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as is run on a general-purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or combinations thereof.
  • the process or portions thereof are performed by a neural network and/or processor of an ultrasound machine.
  • the process begins by processing logic generating one or more ultrasound images based on ultrasound echo signals (processing block 1921 ).
  • processing logic generating ultrasound images based on ultrasound echo signals is performed by an image module.
  • Processing logic identifies the blood vessels in the one or more ultrasound images (processing block 1922 ) and assigns one of a vein classification or an artery classification to each blood vessel of the blood vessels (processing block 1923 ). In some embodiments, these operations are performed with a neural network module.
  • processing logic determines, for each blood vessel, a confidence level for the vein classification or the artery classification (processing block 1924 ). In some embodiments, determining the confidence level for the vein classification or the artery classification is performed with a processor. Additionally or alternatively, a neural network can determine a confidence level for the classification, such as by generating a probability of correct classification, and determining the confidence level based on the probability of correct classification.
  • Processing logic causes the display in the one or more ultrasound images of an outline of at least one blood vessel of the blood vessels with an opacity based on the confidence level determined for the at least one blood vessel (processing block 1925 ). In some embodiments, causing the display in the one or more ultrasound images of an outline of at least one blood vessel of the blood vessels with an opacity based on the confidence level is performed with a processor.
  • the process includes processing logic determining diameters and depths of the blood vessels (processing block 1926 ) and determining, for a blood vessel of the blood vessels, a threshold insertion length of a catheter to be inserted into the blood vessel based on a diameter of the blood vessel (processing block 1927 ).
  • the threshold insertion length indicates an amount of the catheter inside the blood vessel.
  • determining diameters and depths of the blood vessels is performed with a neural network module, while a processor determines the threshold insertion length of the catheter.
  • the process also includes processing logic determining a length of the catheter based on the threshold insertion length and a depth of the blood vessel (processing block 1928 ) and causing display in the one or more ultrasound images of an indication of the length of the catheter to be inserted into the blood vessel (processing block 1929 ). In some embodiments, these operations are performed with a processor.
  • the process also includes processing logic determining an insertion angle for the catheter (processing block 1930 ), and determining the length of the catheter based on the insertion angle (processing block 1931 ). For instance, the processor can determine the sum of the threshold insertion length and the depth of the blood vessel, and weight this sum by a coefficient that accounts for the insertion angle, such as by multiplying the sum by the inverse of the cosine of the insertion angle. The processor can then determine the length of the catheter to be at least as long as the weighted result.
  • the process also includes determining an insertion point for a needle of the catheter based on the depth of the blood vessel and the insertion angle for the catheter (processing block 1932 ), and indicating the insertion point as a distance from the transducer (optional) (processing block 1933 ). In one embodiment, these operations are performed with a processor. In some embodiments, the processor determines the insertion angle for the catheter using information other than the diameter or the depth of the blood vessel. For example, the processor can read the insertion angle from a default setting, such as 45 degrees set in a default register. In some embodiments, the process further includes determining the distance from a face of the transducer.
  • FIG. 19 D is a flow diagram of yet another embodiment of a process for identifying blood vessels.
  • the process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as is run on a general-purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or combinations thereof.
  • the process or portions thereof are performed by a neural network and/or processor of an ultrasound machine.
  • the process begins by processing logic generating ultrasound images based on ultrasound echo signals (processing block 1941 ).
  • processing logic generating ultrasound images based on ultrasound echo signals is performed by an image module.
  • processing logic After generating the ultrasound images, processing logic identifies the blood vessels in a first image of the ultrasound images (processing block 1942 ). In some embodiments, identifying the blood vessels in a first image of the ultrasound images is performed with a neural network module.
  • Processing logic also generates a map image that indicates locations of the blood vessels in the first image identified by the neural network module (processing block 1943 ).
  • generating a map image is performed with a map module.
  • the map module can be configured to generate the map image to indicate the locations of the blood vessels in the first image with bounding containers.
  • the bounding containers can include outlines of the blood vessels.
  • the map module is configured to generate the map image as a mask that indicates the locations of the blood vessels with a first pixel value and additional locations in the map image that do not correspond to the blood vessels with a second pixel value.
  • processing logic identifies the blood vessels in a second image of the ultrasound images based on the map image and the second image (processing block 1944 ).
  • identifying the blood vessels in a second image of the ultrasound images based on the map image and the second image is performed with the neural network module.
  • the neural network module is configured to adjust at least one neural network coefficient based on a loss function of the locations of the blood vessels in the first image and additional locations of the blood vessels in the second image.
  • the process further comprises generating a feature map from the second image that represents content of the second image in a feature space, concatenating the feature map and the map image, and identifying the blood vessels in the second image based on results of concatenation. In some embodiments, these operations are performed by a neural network module.
  • a neural network of the neural network module identifies blood vessels and their locations for use in a medical procedure.
  • the neural network performs the identification of blood vessels in conjunction with other determinations such as the determination of blood vessel depth and size (e.g., diameter, radius, area, etc.), appropriate or suitable implements for use in the medical procedure, as well as guidance information to help a user during the medical procedure.
  • FIG. 20 A is a flow diagram of yet another embodiment of a process for identifying blood vessels for uses such as a medical procedure.
  • the process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as is run on a general-purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or combinations thereof.
  • the process is performed by a neural network and/or processor of an ultrasound machine.
  • the process begins with the processing logic receiving an ultrasound image that includes blood vessels (processing block 2001 ).
  • processing logic determines the locations and depths of the blood vessels in the ultrasound image (processing block 2002 ).
  • the ultrasound system determines the locations and depths of the blood vessels in the ultrasound image using a neural network.
  • the neural network can be implemented at least partially in hardware. Additionally or alternatively, the neural network can be implemented at least partially in software and/or firmware.
  • determining the locations and the depths of the blood vessels is based on additional ultrasound images. For example, the ultrasound images in these additional ultrasound images can represent frames of an ultrasound video.
  • determining the locations and depths of the blood vessels is based on additional ultrasound images where at least one of the additional ultrasound images indicates a previously determined location and depth of one of the blood vessels determined by the neural network.
  • processing logic determines one of the blood vessels as being more centered in the ultrasound image than other blood vessels of the blood vessels that have been determined (processing block 2003 ).
  • determining one of the blood vessels is more centered in the ultrasound image is made by determining that the blood vessel is centrally located between both left and right sides of the ultrasound image.
  • the ultrasound system can determine that one of the blood vessels is more centered than the other blood vessels in the image by determining that the one of the blood vessels is aligned, based on an alignment threshold, with a center line of the ultrasound image.
  • the ultrasound system draws a center line between the left and right sides of an image down from the top and determines if the one blood vessel crosses that line.
  • the determination of whether the blood vessel crosses the line can be based on whether a bounding box used to indicate the location of the blood vessel in the ultrasound image crosses the line, which can be done by the ultrasound system based on a comparison of coordinates of a bounding container (e.g., x,y endpoints of abounding box) and the points of the line.
  • a bounding container e.g., x,y endpoints of abounding box
  • the process can determine whether the one blood vessel is unobstructed by the other blood vessels that are located at the same horizontal position as the one blood vessel in an ultrasound image. In one embodiment, determining if one of the blood vessels is unobstructed by the other blood vessels is performed by the processor by determining that the one blood vessel has a shallower depth than the depths of the other blood vessels.
  • processing logic After determining one of the blood vessels is more centered in the ultrasound image than other blood vessels, processing logic indicates the blood vessel in the ultrasound image (processing block 2004 ). In some embodiments, processing logic indicates the one blood vessel by providing an indication of the blood vessel in the ultrasound image, as discussed with reference to FIGS. 11 A- 11 D and FIG. 17 .
  • the process of FIG. 20 A optionally includes processing logic determining, with a neural network, a diameter of the one blood vessel in the ultrasound image (processing block 2005 ), determining an instrument size based on the determined diameter (processing block 2006 ), and providing an indication as to the instrument size in the ultrasound image (processing block 2007 ).
  • processing logic determines the instrument size based on determining the largest appropriate instrument for a blood vessel having that sized diameter. For example, processing logic can determine an appropriately sized instrument based on the results of comparing the ratios of the instrument size to the diameters of the blood vessels to a threshold ratio size. These ratios can be determined by a processor of the ultrasound machine.
  • the processor can determine that an appropriately sized instrument for a particular blood vessel is limited in gauge to a percentage of a diameter for the blood vessel, such as no greater than 50% of the diameter of the blood vessel.
  • the process may optionally include processing logic for calculating a blood vessel diameter, such that the determination of an instrument size is based on the blood vessel diameter. In some embodiments, this calculation can be performed between processing blocks 2005 and 2006 .
  • indicating the instrument size includes displaying the bounding container of a blood vessel in a color corresponding to the instrument size.
  • the bounding container includes a circle, an ellipse, a highlight, etc., as an outline of the blood vessel.
  • a processor in the ultrasound system selects a color based on the instrument size and indicates one or more of the blood vessels with the color in one or more ultrasound images. For example, the processor selects the color based on instrument size and then displays the blood vessels with that color in one or more ultrasound images.
  • the neural network determines depths of the blood vessels in ultrasound images, and the processor system determines one of the blood vessels having the shallowest depth.
  • the processor system can then cause display of a depth and diameter of the one blood vessel, as determined by the neural network, in the ultrasound images automatically and without user intervention.
  • the process also includes receiving a user selection of at least one of the blood vessels. For example, a user may touch a touchscreen display at the location of the blood vessel to indicate the blood vessel, enable a selection of a pull-down tab, and the like.
  • the processor can cause the display in one or more ultrasound images of a diameter of that selected blood vessel.
  • FIG. 20 B is a flow diagram of another embodiment of a process for guiding instrument insertion with an ultrasound machine.
  • the process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as is run on a general-purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or combinations thereof.
  • the process or portions thereof are performed by a neural network and/or processor of the ultrasound machine.
  • the process begins by processing logic maintaining one or more ultrasound images (processing block 2011 ). Using these ultrasound images, processing logic identifies blood vessels in the one or more ultrasound images and determines diameters of the blood vessels (processing block 2012 ). In some embodiments, identifying the blood vessels in a first image of the ultrasound images is performed with a neural network module.
  • processing logic determines, for one or more of the blood vessels, an instrument size based on the diameters of the one or more of the blood vessels (processing block 2013 ) and selects a color based on the instrument size (processing block 2014 ). In some embodiments, these operations are performed by the processor system.
  • the process also optionally includes processing logic indicating, in the one or more ultrasound images, the one or more of the blood vessels with the color (processing block 2015 ).
  • the process can also include receiving a user selection of at least one of the blood vessels (processing block 2016 ), and causing display in the one or more ultrasound images of a diameter of the at least one of the blood vessels or a depth of the at least one of the blood vessels (processing block 2017 ).
  • these operations are performed by the processor system.
  • a parameter associated with a blood vessel diameter can be used instead of diameter in FIG. 20 B .
  • the processing logic can determine a blood vessel diameter in the block 2012 and can determine, for one or more of the blood vessels, an instrument size based on the blood vessel diameters of the one or more of the blood vessels in the block 2013 .
  • the process also optionally includes processing logic determining depths of the blood vessels in the one or more ultrasound images (processing block 2018 ), determining one of the blood vessels as having a shallowest depth (processing block 2019 ), and causing display in the one or more ultrasound images, automatically and without user intervention, of a depth and a diameter of the one of the blood vessels (processing block 2020 ).
  • determining depths of the blood vessels in the one or more ultrasound images is performed by a neural network, while the other operations are performed by the processor system.
  • FIG. 21 is a flow diagram of one embodiment of a process for guiding instrument insertion.
  • the process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as is run on a general-purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or combinations thereof.
  • the process is performed by a neural network and/or processor of an ultrasound machine.
  • processing logic begins by processing logic receiving the ultrasound image that includes one or more blood vessels (processing block 2101 ).
  • processing logic also determines the locations of the blood vessels in the ultrasound image and classifies blood vessels as veins or arteries.
  • a neural network determines these locations and performs the classification.
  • the neural network also determines a confidence level associated with the assigned vein/artery classification.
  • the ultrasound machine can display an outline of each blood vessel (e.g., a bounding container) with an opacity based on the confidence level associated with the vein/artery classification determined for the blood vessel.
  • processing logic determines the distance of at least one blood vessel to an edge of the ultrasound image, and then displays the outline for the at least one blood vessel by displaying the outline with an opacity based on that distance.
  • processing logic determines the diameter of one (or more) of the blood vessels in the ultrasound image (processing block 2102 ).
  • the techniques disclosed herein are not limited to determining a size of a blood vessel as the diameter of the blood vessel.
  • processing logic determines and uses other size measurements such as a radius, an area, a circumference, combinations thereof, such as ratios, etc.
  • processing logic determines a blood vessel diameter discussed above as the diameter of the blood vessel.
  • a neural network determines a size measurement of the blood vessels based on the ultrasound image and one or more additional ultrasound images that include the same blood vessels. The additional images may be part of the same ultrasound video or part of the same ultrasound examination/procedure.
  • the ultrasound system can generate the additional ultrasound images based on an imaging parameter that is set to a different value than a value of the imaging parameter used to generate the ultrasound image being processed. For example, the ultrasound system can generate one or more additional ultrasound images based on a beamform angle that is different than an additional beamform angle used to generate the ultrasound signal used for generating the ultrasound image being processed. The ultrasound system can generate these additional images with information indicating locations corresponding to one or more blood vessels using, for example, one or more boundary containers.
  • processing logic receives a user selection of instrument size (processing block 2103 ).
  • the instrument size corresponds to an instrument that is going to be inserted into the blood vessel, such as, for example, a catheter, a needle, etc.
  • processing logic After receiving the user selection of instrument size, processing logic indicates, in the ultrasound image, at least one blood vessel of the one or more blood vessels based on the instrument size and the diameters of the one or more blood vessels (processing block 2104 ).
  • processing logic indicates the one blood vessel by providing an indication with at least one blood vessel in the ultrasound image indicating the appropriate instrument size for each of the one or more blood vessels (e.g., for blood vessels of that diameter, blood vessels of that radius, blood vessels of that area).
  • the processing logic can determine the appropriate instrument size based on instruments that are used for a medical procedure to be performed by the ultrasound machine operator.
  • processing logic provides an indication of the blood vessel(s) by displaying an outline of the identified blood vessels in an ultrasound image.
  • the processing logic can display the outline of at least one of the blood vessels by displaying an outline in a particular color that is associated with the instrument size.
  • processing logic can provide an indication regarding at least one blood vessel and the appropriately sized instrument to use based on the results of a comparison between ratios to a threshold ratio size, where the ratios correspond to ratios of the instrument size to the diameters of the blood vessels.
  • processing logic can determine the ratios of the instrument size to the diameters of the blood vessels and compare those ratios to the threshold ratio size. Note that the comparisons may be performed using measurements (e.g., radius, area, etc.) other than diameters.
  • a neural network is configured to make inferences based on multiple frames.
  • Such multi-frame interferences can be used to help improve overall interframe model consistency of the neural network models and monitor a vein through compressions.
  • the multi-frame inferences are performed by feeding inferences from previous frames into the model as additional inputs (e.g., conditional inputs) to provide hints to the model where blood vessels might be found as well as the size of the blood vessels.
  • additional inputs includes a map image generated by the map module, as discussed above.
  • the map image generated by the map image module for a previous image frame can be provided as a conditional input to the neural network for generating inferences for a current, or subsequent, image frame.
  • the procedures described herein constitute improvements over procedures that do not display blood vessels in ultrasound images with enhancements. Rather, the procedures described herein generate enhancements that include useful, and sometimes critical, information, and display the information to an operator of an ultrasound system so that the operator can apply the information to the ultrasound procedure in real-time.
  • the procedures described herein can result in significant improvements to a patient's experience, including reducing, or even minimizing, the number of times an interventional instrument is inserted into a patient, resulting in pain and infection risk for the patient, and as well as preventing an undesired insertion of the instrument, e.g., into an artery rather than a vein.
  • the patient does not perceive unnecessary pain and discomfort, and the risk of one or more of infection, thrombus due to insertion of too large of a catheter, extravasation, and damage to a vessel wall is reduced compared to conventional ultrasound procedures that do not display blood vessels with enhancements. Therefore, the procedures described herein can be suitable to medical procedures and examinations for which conventional ultrasound procedures are not suitable.
  • FIG. 22 is a block diagram of one embodiment of an ultrasound machine.
  • any type of ultrasound machine can be used to perform the blood vessel identification and enhanced display techniques described herein, including a stationary-type, a portable-type, a handheld type, and combinations thereof.
  • the block diagram is one example of an ultrasound machine that can be constructed using the blocks in FIG. 22 .
  • signals can be redefined, and blocks can be modified, combined, divided, added, or removed to form a modified system, without altering the functionality of the block diagram. Accordingly, such modified systems are considered to be within the scope of the disclosure.
  • the blocks, modules and units of an ultrasound machine such as the ultrasound machine illustrated in FIG.
  • 22 can be implemented as any type of module or component in software (e.g., as software instructions that are executable with a processing system), hardware, or combinations thereof, as a standalone application or as a module or component of another device application, and in any type of computing device.
  • the ultrasound machine includes an ultrasound transducer probe 2201 coupled to an ultrasound device 2202 .
  • Ultrasound transducer probe 2201 includes an ultrasonic transducer 2201 A electrically coupled to a receiver 2201 B and a transmitter 2201 C.
  • Ultrasonic transducer 2201 A has one or more transducer elements and in operation, transmits ultrasound energy from the one or more transducer elements toward a subject in response to transmitter signals from transmitter 2201 C and receives ultrasound echoes from the subject using receiver 2201 B.
  • Receiver 2210 B and transmitter 2210 C can include any type of circuit, or any types of other form such as a processor, as described below.
  • Receiver 2210 B and transmitter 2210 C can be formed separately or integrated into a single form.
  • Receiver 2210 B and transmitter 2210 C can be integrated into a processor that has other functions, such as processor 2202 C described below.
  • the ultrasound echoes are converted into electrical signals by receiver 2201 B and electrically coupled to electronics (e.g., analog/digital (A/D) converter 2202 A, one or more processor(s) 2202 C, memory modules 2202 B, beamformers, FPGAs, etc.) in ultrasound device 2202 configured to process the electrical signals and form one or more ultrasound images.
  • electronics e.g., analog/digital (A/D) converter 2202 A, one or more processor(s) 2202 C, memory modules 2202 B, beamformers, FPGAs, etc.
  • ultrasound probe 2201 includes an A/D converter (not shown in FIG. 22 ) that can be configured to digitize the electrical signals generated by receiver 2201 B that are based on the ultrasound echoes. Furthermore, ultrasound probe 2201 can also include a partial or full beamformer (not shown in FIG. 22 ) configured to sum the electrical signals in a phase-corrected manner. Hence, the ultrasound probe 2201 can be implemented to couple digital signals to ultrasound device 2202 , such as over a wired or wireless communication link. For instance, ultrasound probe 2201 and ultrasound device 2202 can each include transceivers (not shown in FIG. 22 ) that communicate using communication signals that have been digitally modulated based on the digital signals.
  • Processor 2202 C can include any type or any number of an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an integrated circuit (IC), logic, etc., partially or wholly.
  • processor 2202 C includes a beamformer, and includes processing to form an ultrasound image. Note also that the processing described herein can be performed at least partially in the cloud, on an attached or networked processing device, on a discrete or external GPU, or custom specific attached hardware such as a USB attached device.
  • Capturing ultrasound data from a subject using an ultrasound probe 2201 generally includes generating ultrasound, transmitting ultrasound into the subject, and receiving ultrasound reflected by the subject.
  • a wide range of frequencies of ultrasound may be used to capture ultrasound data, such as, for example, low frequency ultrasound (e.g., less than 15 MHz) and/or high frequency ultrasound (e.g., greater than or equal to 15 MHz) can be used.
  • low frequency ultrasound e.g., less than 15 MHz
  • high frequency ultrasound e.g., greater than or equal to 15 MHz
  • Those of ordinary skill in the art can readily determine which frequency range to use based on factors such as, for example, but not limited to, depth of imaging and/or desired resolution.
  • ultrasound device 2202 includes one or more processor(s) 2202 C that cause electrical currents to be sent to the ultrasound probe 2201 to emit sound waves and also receives the electrical pulses from probe 2201 that were created from the returning echoes.
  • the A/D converter 2202 A receives analog signals from receiver 2201 B and converts them into digital data that is stored in memory 2202 B.
  • Memory 2202 B also stores software and other data (e.g., templates, user preferences for enhancements, etc.) related to detecting blood vessels (e.g., veins, arteries, and capillaries), and analytical results along with ultrasound images and other data associated therewith.
  • Monitor 2203 A processes the raw data associated with the received electrical pulses and forms an image that is sent to display device 2203 , which displays the image on monitor 2203 A.
  • monitor 2203 A can display ultrasound images from the ultrasound data processed by processor 2202 C of ultrasound device 2202 .
  • Monitor 2203 A can also display blood vessel detection results, as well as guide information of the probe as discussed herein.
  • monitor 2203 A includes a clinical display device. Additionally or alternatively, monitor 2203 A can include a display of a computing device coupled to the ultrasound machine, such as a tablet.
  • processor 2202 C performs blood vessel detection using template matching, artificial intelligence or machine-learning (e.g., adaptive boosting (adaboost), deep-learning, support vector machine (SVM), sequence models including RNN, GRU, ConvGRU, LSTM, etc., to process frame information in sequence, etc.), and/or another detection method.
  • processor 2202 C executes an AI algorithm and/or uses a neural network to identify veins and arteries and locate them in an ultrasound image.
  • processor 2202 C displays the blood vessels on a monitor or display of the ultrasound system.
  • the detected blood vessels are displayed by processor 2202 C in an enhanced manner to provide information to the operator, or user of the ultrasound machine.
  • processor 2202 C draws outlines or other forms of blood vessel indicia (e.g., identifiers) around or in the proximity of the blood vessels.
  • an outline of the vein is changed to match the color coding of the largest catheter that could fit within that vein.
  • the color coding may be an industry standard color coding for catheters.
  • an operator is able to quickly know the catheter that may be used with a particular blood vessel.
  • an operator selects a catheter size based on the blood vessels being displayed.
  • an operator could select a catheter size and have the ultrasound system identify all blood vessels (e.g., veins) in the image that are appropriate for that catheter size.
  • processor 2202 C could identify all veins in the image that are appropriate for a particular catheter size.
  • an operator is able to touch a vein in the image and have processor 2202 C display the diameter and depth for that blood vessel.
  • processor 2202 C automatically identifies the most central and shallowest vein and automatically provides its diameter and depth for that vein on the display of the ultrasound machine.
  • processor 2202 C calculates a likelihood value for each detected blood vessel as an indication of a confidence level associated with the detection results.
  • processor 2202 C generates outlines of vein/artery that are made more or less opaque to indicate the confidence of the prediction for that vessel.
  • processor 2202 C adjusts the opacity with fading that can be done by alpha blending the outline (e.g., an ellipse outline) with the b-mode image.
  • the amount of fading used may not be linear to the confidence, but rather an empirically derived curve based on what is determined to be pleasing to the user.
  • processor 2202 C performs fading based on vessel size.
  • the ultrasound machine can fade out vessels as their detected size gets close to the minimum size.
  • the accuracy of the model decreases for vessels detected towards the bottom and the sides. This is partially due to the ambiguity of determining when a vessel is too far out of the image to annotate or detect. In such a case, processor 2202 C can perform fading based on proximity to the sides and bottom of the image.
  • processor 2202 C tracks the detected blood vessels across frames and identifies the same detected blood vessel in multiple images.
  • the processor 2202 C can use blood vessel location information or the type of blood vessel for the tracking and/or identification.
  • processor 2202 C uses the blood vessel identification and vein/artery classification information output from the neural network to generate and display information for PIV or other medical procedures.
  • the information can include suggested catheter gauges, needle entry point information, guidance information (e.g., calculated distances from the transducer face or edge to a desired entry point of a PIV needle, etc.), a threshold insertion length of a catheter to be inserted into the blood vessel, catheter or other medical instrument length, combinations thereof, and the like.
  • this information is determined by processor 2202 C based on diameters and depths of the blood vessels, threshold exertion length, insertion angle for the catheter determined by the neural network or processor 2202 C.
  • processor 2202 C can determine the insertion angle for the catheter based on something other than the diameter or the depth of the blood vessel. For instance, processor 2202 C can determine the insertion angle as a default insertion angle, such as 45 degrees, that processor 2202 C reads from memory 2202 b and does not depend on the diameter or depth of a blood vessel determined by the ultrasound machine. After generating this information, the processor can display any portion or all of the information to the user in an ultrasound image or in another portion of the user interface of the ultrasound machine.
  • a default insertion angle such as 45 degrees
  • the ultrasound system has one or more user input devices 2204 (e.g., a keyboard, a cursor control device, etc.) that input data from at least one operation unit 2204 A.
  • the ultrasound system also has a storage device 2205 that includes storage 2205 A (e.g., hard, floppy, compact disks (CD), digital video discs (DVDs)) for storing the acquired images (e.g., past images, current images, etc.), which may be used for calculating a blood vessel diameter, selecting the best image for the insertion, other another operation described herein.
  • the ultrasound system further includes a data communication device 2206 with a data communication interface 2206 A for providing communication between ultrasound device 2202 and an external server 2207 .
  • the probe of the ultrasound machine includes an additional sensor, such as an accelerometer or six degrees-of-freedom (6DOF) inertial measurement unit (IMU).
  • the additional sensor can be used by the operator to perform functions such as, for example, estimating probe pose and providing user guidance (e.g., for directing the user back to a best vein for cannulation) and for providing a better understanding of the artery and vein structures along a limb. For example, when a vein with appropriate depth and diameter is found, the ultrasound system can track the probe location to ensure that the vein is not tortuous and does not have valves over a length that is appropriate for the length of the catheter.
  • 6DOF degrees-of-freedom
  • the ultrasound machine includes a hand-held ultrasound apparatus having a probe and a tablet or smartphone connected to each other.
  • the connection may be wireless or wired.
  • FIG. 23 illustrates an example of such an ultrasound machine.
  • a wireless ultrasound probe 2300 wirelessly communicates with smart phone 2302 .
  • probe 2300 includes hardware and software that function to perform transmission/reception by an array of transducers, A/D conversion; beamforming; quadrature detection; and wireless transmission.
  • smart phone 2302 includes hardware and software that function to produce an ultrasound image (e.g., a B-mode image) and display the image on its display.
  • the hardware and software of the smart phone 2302 can detect a vein and calculate a blood vessel diameter; calculate a value (e.g., maximum value) using a plurality of blood vessel diameters; and generate and apply color-enhancement of the detected vein on the monitor.
  • a value e.g., maximum value
  • FIG. 24 illustrates a data flow diagram of a blood vessel identification and display subsystem.
  • the subsystem is part of an ultrasound machine, such as that described previously in FIG. 22 . Additionally or alternatively, the subsystem can operate in conjunction with an ultrasound machine by providing information to the ultrasound machine for use and/or display, and receiving information from the ultrasound machine.
  • memory 2401 stores ultrasound images 2402 .
  • Neural network (or other blood vessel detection mechanism) 2403 receives or otherwise obtains the ultrasound images 2402 .
  • neural network 2403 determines blood vessel locations and diameters in ultrasound images 2402 using AI or other machine learning algorithms and outputs this information as blood vessel location/diameter information 2404 to display generator 2406 .
  • neural network 2403 also determines when there has been a blood vessel detection failure using AI or other machine learning algorithms and outputs this information as blood vessel detection failure information 2405 to display generator 2406 .
  • neural network 2403 includes one or more processors.
  • display generator 2406 In response to blood vessel location/diameter information 2404 and/or blood vessel detection failure information 2405 , display generator 2406 generates blood vessel graphics or other indicia for superimposition onto one or more ultrasound images.
  • neural network 2403 is a convolutional neural network (CNN), such as, for example, but not limited to, RetinaNet.
  • CNN convolutional neural network
  • the neural network can also contain memory units which pass state information from one instance of the network to another thus allowing the network to learn from past, and future inputs.
  • the neural network uses “intersection over min area” (IOMA) instead of “intersection over union” (IOU), which is used in Non-Maximal Suppression (NMS).
  • IOMA intersection over min area
  • IOU intersection over union
  • IOMA intersection over union
  • NMS Non-Maximal Suppression
  • the ultrasound system uses rectangular boxes around what are best approximated as ellipses that represent blood vessels. As vessels themselves cannot overlap of physically enclose other vessels, an algorithm that better suppresses detections that are fully or mostly surrounded by other detections, some traditional neural networks use NMS to compute an “intersection over union” (IOU) to determine an amount of overlap between rectangular boxes.
  • the neural network computes the areas of the ellipses contained within the boxes, instead of computing the area of the boxes. Using the computed ellipse area in the IOMA, the neural network can use a very small threshold for allowing overlap of the vessels and therefore removes those false positives. To reduce the computation load of computing the actual ellipse area, the neural network can use an approximation of the ellipse area.
  • the neural network computes the radius of the two ellipses (on the line that intersects their centers) and compares the sum of the radii to the length of the segment connecting the two centers. If the sum is less than the segment length, then the neural network determines that the vessels are overlapping. Using this method, the neural network does not need to compute the area of either ellipse/box, and this lack of area computations could speed up the processing.
  • the ultrasound machine acquires additional echo frames that are acquired with different look angles.
  • the additional look angles are done at a fairly steep angle to provide the AI model with more information about the sides of the blood vessel walls.
  • the ultrasound machine uses this additional information about the sides of the blood vessel walls to help the model (e.g., neural network) better determine locations of the blood vessels and aiding in the discrimination between arteries and veins.
  • these additional frames are of lower quality and resolution compared to ultrasound image frames displayed to the user, but instead the additional frames can be fed into the model along with the image frames corresponding to the echo data and that are displayed to a user. The lower resolution allows for less impact on the frame rate.
  • the ultrasound machine obtains color frames along with the echo frames and provides them to the model of the neural network to inform the model of vessel blood flow.
  • the color frames are low resolution, low quality images that may not be shown to the user, but instead can be fed into the model along with image frames corresponding to the echo data and that are displayed to a user. Adding the additional color flow information to the frame artery/vein detection model provides the model another dimension to the data from which to learn. The lower resolution would allow for less impact on the frame rate.
  • the color information is coupled with the post inference detection information to reinforce or modify the detection.
  • the systems described herein constitute improvements over systems that do not display blood vessels in ultrasound images with enhancements. Rather, the systems described herein generate enhancements that include useful, and sometimes critical, information, and display the information to an operator of an ultrasound system so that the operator can apply the information to the ultrasound procedure in real real-time.
  • the systems described herein can result in significant improvements to a patient's experience, including reducing, or even minimizing, the number of times an interventional instrument is inserted into a patient, resulting in pain and infection risk for the patient, and as well as preventing an undesired insertion of the instrument, e.g., into an artery rather than a vein.
  • the patient does not perceive unnecessary pain and discomfort, and the risk of one or more of infection, thrombus due to insertion of too large of a catheter, extravasation, and damage to a vessel wall is reduced compared to conventional ultrasound systems that do not display blood vessels with enhancements. Therefore, the systems described herein can be suitable to medical procedures and examinations for which conventional ultrasound systems are not suitable.
  • Example 1 is a method implemented by a computing device for aiding a clinician in selecting an appropriate vein for catheterization, where the method comprises: receiving an ultrasound image that includes one or more blood vessels; determining, with a neural network implemented at least partially in hardware of the computing device, diameters of the one or more blood vessels in the ultrasound image; receiving a user selection of an instrument size; and indicating, in the ultrasound image, at least one blood vessel of the one or more blood vessels based on the instrument size and the diameters of the one or more blood vessels.
  • Example 2 is the method of example 1 that may optionally include determining ratios of the instrument size to the diameters of the one or more blood vessels; and comparing the ratios to a threshold ratio, wherein the indicating the at least one blood vessel is based on the comparing.
  • Example 3 is the method of example 1 that may optionally include that the indicating includes displaying an outline of the at least one blood vessel.
  • Example 4 is the method of example 3 that may optionally include that the displaying the outline of the at least one blood vessel includes displaying the outline in a color associated with the instrument size.
  • Example 5 is the method of example 3 that may optionally include assigning, with the neural network, one of a vein classification or an artery classification to the one or more blood vessels; and determining a confidence level for the vein classification or the artery classification assigned to the one or more blood vessels, wherein the displaying the outline of the at least one blood vessel includes displaying the outline with an opacity based on the confidence level determined for the at least one blood vessel.
  • Example 6 is the method of example 3 that may optionally include that the displaying the outline of the at least one blood vessel includes displaying the outline with an opacity based on a distance of the at least one blood vessel to an edge of the ultrasound image.
  • Example 7 is the method of example 1 that may optionally include that the determining the diameters with the neural network is based on the ultrasound image and one or more additional ultrasound images that include the one or more blood vessels.
  • Example 8 is the method of example 7 that may optionally include generating the one or more additional ultrasound images based on an imaging parameter that is set to a different value than a value of the imaging parameter used to generate the ultrasound image.
  • Example 9 is the method of example 7 that may optionally include that generating at least one of the one or more additional ultrasound images based on a beamform angle that is different than an additional beamform angle used to generate the ultrasound image.
  • Example 10 is the method of example 1 that may optionally include that generating an additional image that indicates locations corresponding to the one or more blood vessels, wherein the determining the diameters with the neural network is based on the ultrasound image and the additional image.
  • Example 11 is the method of example 10 that may optionally include that the additional image indicates the locations corresponding to the one or more blood vessels with one or more bounding containers.
  • Example 12 is a computing device to implement an ultrasound system for guiding instrument insertion, where the computing device comprises: a memory to maintain one or more ultrasound images; a neural network implemented at least partially in hardware of the computing device to identify blood vessels in the one or more ultrasound images and determine diameters of the blood vessels; and a processor system to: determine, for one or more of the blood vessels, an instrument size based on the diameters of the one or more of the blood vessels; select a color based on the instrument size; and indicate, in the one or more ultrasound images, the one or more of the blood vessels with the color.
  • Example 13 is the computing device of example 12 that may optionally include that the processor system is implemented to determine the instrument size as a largest catheter size for the one or more of the blood vessels.
  • Example 14 is the computing device of example 12 that may optionally include that the processor system is implemented to: receive a user selection of at least one of the blood vessels; and cause display in the one or more ultrasound images of a diameter of the at least one of the blood vessels.
  • Example 15 is the computing device of example 12 that may optionally include the neural network is implemented to determine depths of the blood vessels in the one or more ultrasound images; and the processor system is implemented to: receive a user selection of at least one of the blood vessels; and cause display in the one or more ultrasound images of a depth of the at least one of the blood vessels determined by the neural network.
  • Example 16 is the computing device of example 12 that may optionally include the neural network is implemented to determine depths of the blood vessels in the one or more ultrasound images; and the processor system is implemented to: determine one of the blood vessels as having a shallowest depth; and cause display in the one or more ultrasound images, automatically and without user intervention, of a depth and a diameter of the one of the blood vessels determined by the neural network.
  • the neural network is implemented to determine depths of the blood vessels in the one or more ultrasound images
  • the processor system is implemented to: determine one of the blood vessels as having a shallowest depth; and cause display in the one or more ultrasound images, automatically and without user intervention, of a depth and a diameter of the one of the blood vessels determined by the neural network.
  • Example 17 is an ultrasound system for identifying blood vessels, where the ultrasound system comprises: an image module to generate one or more ultrasound images based on ultrasound echo signals; a neural network module to: identify the blood vessels in the one or more ultrasound images; and assign one of a vein classification or an artery classification to each blood vessel of the blood vessels; and a processor to: determine, for said each blood vessel, a confidence level for the vein classification or the artery classification; and cause display in the one or more ultrasound images of an outline of at least one blood vessel of the blood vessels with an opacity based on the confidence level determined for the at least one blood vessel.
  • Example 18 is the ultrasound system of example 17 that may optionally include that the neural network module is implemented to determine diameters and depths of the blood vessels; and the processor is implemented to: determine, for a blood vessel of the blood vessels, a threshold insertion length of a catheter to be inserted into the blood vessel based on a diameter of the blood vessel determined by the neural network module, the threshold insertion length indicating an amount of the catheter inside the blood vessel; determine a length of the catheter based on the threshold insertion length and a depth of the blood vessel determined by the neural network module; and cause display in the one or more ultrasound images of an indication of the length of the catheter to be inserted into the blood vessel.
  • the neural network module is implemented to determine diameters and depths of the blood vessels
  • the processor is implemented to: determine, for a blood vessel of the blood vessels, a threshold insertion length of a catheter to be inserted into the blood vessel based on a diameter of the blood vessel determined by the neural network module, the threshold insertion length indicating an amount of the catheter inside the blood vessel; determine
  • Example 19 is the ultrasound system of example 18 that may optionally include that the processor is implemented to: determine an insertion angle for the catheter; and determine the length of the catheter based on the insertion angle.
  • Example 20 is the ultrasound system of example 19 that may optionally include a transducer to generate ultrasound signals and receive the ultrasound echo signals based on the ultrasound signals, wherein the processor is implemented to: determine an insertion point for a needle of the catheter based on the depth of the blood vessel and the insertion angle for the catheter; and indicate the insertion point as a distance from the transducer.
  • a transducer to generate ultrasound signals and receive the ultrasound echo signals based on the ultrasound signals
  • the processor is implemented to: determine an insertion point for a needle of the catheter based on the depth of the blood vessel and the insertion angle for the catheter; and indicate the insertion point as a distance from the transducer.
  • Example 21 is the ultrasound system of example 20 that may optionally include that the processor is implemented to determine the distance from a face of the transducer.
  • Example 22 is the ultrasound system of example 19 that may optionally include that the processor is implemented to determine the insertion angle for the catheter not based on the diameter or the depth of the blood vessel.
  • Example 23 is a method implemented by a computing device for identifying blood vessels, where the method comprises: receiving an ultrasound image that includes the blood vessels; determining, with a neural network implemented at least partially in hardware of the computing device, locations and depths of the blood vessels in the ultrasound image; determining, based on the locations and the depths, one of the blood vessels as being more centered in the ultrasound image than other blood vessels of the blood vessels and unobstructed by the other blood vessels; and indicating, in the ultrasound image, the one of the blood vessels.
  • Example 24 is the method of example 23 that may optionally include determining, with the neural network, a diameter of the one of the blood vessels in the ultrasound image; determining an instrument size based on the diameter; and indicating, in the ultrasound image, the instrument size.
  • Example 25 is the method of example 24 that may optionally include that the indicating the instrument size includes displaying a bounding container of the one of the blood vessels in a color corresponding to the instrument size.
  • Example 26 is the method of example 25 that may optionally include that the bounding container includes a circle or an ellipse as an outline of the one of the blood vessels.
  • Example 27 is the method of example 23 that may optionally include that the determining the one of the blood vessels as being more centered than the other blood vessels includes determining that the one of the blood vessels is aligned, within an alignment threshold, with a center line of the ultrasound image.
  • Example 28 is the method of example 23 that may optionally include that the determining the one of the blood vessels as being unobstructed by the other blood vessels includes determining that the one of the blood vessels has a shallower depth than the depths of the other blood vessels.
  • Example 29 is the method of example 23 that may optionally include that the determining the locations and the depths of the blood vessels is based on additional ultrasound images, the ultrasound image and the additional ultrasound images representing frames of an ultrasound video.
  • Example 30 is the method of example 23 that may optionally include that the determining the locations and the depths of the blood vessels is based on additional ultrasound images, at least one of the additional ultrasound images indicating a previous location and a previous depth of the one of the blood vessels determined by the neural network prior to the determining the locations and the depths.
  • the present invention also relates to apparatus for performing the operations herein.
  • This apparatus may be specially constructed for the required purposes, or it may include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer.
  • a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.
  • a machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer).
  • a machine-readable medium includes read only memory (“ROM”); random access memory (“RAM”); magnetic disk storage media; optical storage media; flash memory devices; electrical, optical, acoustical or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.); etc.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Public Health (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Pathology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Veterinary Medicine (AREA)
  • Radiology & Medical Imaging (AREA)
  • Vascular Medicine (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Physiology (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Ultra Sonic Daignosis Equipment (AREA)
US17/239,323 2021-04-23 2021-04-23 Guiding instrument insertion Active US11896425B2 (en)

Priority Applications (6)

Application Number Priority Date Filing Date Title
US17/239,323 US11896425B2 (en) 2021-04-23 2021-04-23 Guiding instrument insertion
CN202280030439.1A CN117222367A (zh) 2021-04-23 2022-04-25 引导器械插入
PCT/US2022/026122 WO2022226398A1 (fr) 2021-04-23 2022-04-25 Insertion d'instruments de guidage
JP2023565274A JP2024514957A (ja) 2021-04-23 2022-04-25 器具挿入案内
EP22792637.5A EP4326160A1 (fr) 2021-04-23 2022-04-25 Insertion d'instruments de guidage
US18/397,826 US20240156429A1 (en) 2021-04-23 2023-12-27 Guiding instrument insertion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US17/239,323 US11896425B2 (en) 2021-04-23 2021-04-23 Guiding instrument insertion

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/397,826 Division US20240156429A1 (en) 2021-04-23 2023-12-27 Guiding instrument insertion

Publications (2)

Publication Number Publication Date
US20220338833A1 US20220338833A1 (en) 2022-10-27
US11896425B2 true US11896425B2 (en) 2024-02-13

Family

ID=83694774

Family Applications (2)

Application Number Title Priority Date Filing Date
US17/239,323 Active US11896425B2 (en) 2021-04-23 2021-04-23 Guiding instrument insertion
US18/397,826 Pending US20240156429A1 (en) 2021-04-23 2023-12-27 Guiding instrument insertion

Family Applications After (1)

Application Number Title Priority Date Filing Date
US18/397,826 Pending US20240156429A1 (en) 2021-04-23 2023-12-27 Guiding instrument insertion

Country Status (5)

Country Link
US (2) US11896425B2 (fr)
EP (1) EP4326160A1 (fr)
JP (1) JP2024514957A (fr)
CN (1) CN117222367A (fr)
WO (1) WO2022226398A1 (fr)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11710306B1 (en) * 2022-06-24 2023-07-25 Blackshark.Ai Gmbh Machine learning inference user interface

Citations (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6245018B1 (en) 1997-12-15 2001-06-12 Medison Co., Ltd. Ultrasonic color doppler imaging system capable of discriminating artery and vein
JP2007222291A (ja) 2006-02-22 2007-09-06 Yunekusu:Kk 動脈血管判定方法および装置
US20080226017A1 (en) * 2005-09-06 2008-09-18 Koninklijke Philips Electronics N. V. Data Handling and Analysis in Computed Tomography With Multiple Energy Windows
US20080269605A1 (en) 2007-04-25 2008-10-30 Shigemitsu Nakaya Ultrasonic diagnostic apparatus and image display method thereof
US20090103794A1 (en) 2007-10-19 2009-04-23 Boston Scientific Scimed, Inc. Display of classifier output and confidence measure in an image
US20100161023A1 (en) 2007-03-08 2010-06-24 Sync-Rx, Ltd. Automatic tracking of a tool upon a vascular roadmap
JP2010269018A (ja) 2009-05-22 2010-12-02 Shimadzu Corp 超音波診断装置
US20130102889A1 (en) * 2011-10-21 2013-04-25 C. R. Bard, Inc. Systems and Methods for Ultrasound-Based Medical Device Assessment
US20130150710A1 (en) * 2011-12-01 2013-06-13 John Zentgraf Surgical navigation for repair of heart valve leaflets
US20130197367A1 (en) 2012-03-14 2013-08-01 Jeffrey Smok Method and apparatus for locating and distinguishing blood vessel
US20140031690A1 (en) 2012-01-10 2014-01-30 Panasonic Corporation Ultrasound diagnostic apparatus and method for identifying blood vessel
US20140188133A1 (en) * 2007-11-26 2014-07-03 C. R. Bard, Inc. Iconic Representations for Guidance of an Indwelling Medical Device
US20140249404A1 (en) 2013-03-01 2014-09-04 Pacesetter, Inc. Vascular branch characterization
US20140343431A1 (en) 2011-12-16 2014-11-20 Koninklijke Philips N.V. Automatic blood vessel identification by name
US20150005630A1 (en) 2013-07-01 2015-01-01 Samsung Electronics Co., Ltd. Method of sharing information in ultrasound imaging
US20150126865A1 (en) 2013-11-05 2015-05-07 Seiko Epson Corporation Ultrasonic probe and ultrasonic measuring device
US20150209113A1 (en) 2014-01-29 2015-07-30 Becton, Dickinson And Company Wearable Electronic Device for Enhancing Visualization During Insertion of an Invasive Device
US20150238165A1 (en) 2014-02-21 2015-08-27 Seiko Epson Corporation Ultrasonic measurement apparatus and ultrasonic measurement method
US20160117814A1 (en) 2013-04-10 2016-04-28 The Asan Foundation Method for distinguishing pulmonary artery and pulmonary vein, and method for quantifying blood vessels using same
US9731066B2 (en) * 2011-09-30 2017-08-15 General Electric Company Device, system and method of automatic vessel access based on real time volumetric ultrasound
US20180015256A1 (en) * 2016-07-14 2018-01-18 C. R. Bard, Inc. Automated Catheter-To-Vessel Size Comparison Tool And Related Methods
US20180014810A1 (en) 2015-04-03 2018-01-18 Koninklijke Philips N.V. Ultrasound system and method of vessel identification
JP2018015356A (ja) 2016-07-29 2018-02-01 セイコーエプソン株式会社 画像処理装置および画像処理方法
US20180137244A1 (en) * 2016-11-17 2018-05-17 Terarecon, Inc. Medical image identification and interpretation
US20180243033A1 (en) 2017-02-24 2018-08-30 Heartflow, Inc. Systems and Methods for Identifying Anatomically Relevant Blood Flow Characteristics in a Patient
US20180247154A1 (en) 2017-02-27 2018-08-30 Fujifilm Corporation Image classification apparatus, method, and program
JP2018171177A (ja) 2017-03-31 2018-11-08 大日本印刷株式会社 眼底画像処理装置
US20180374233A1 (en) 2017-06-27 2018-12-27 Qualcomm Incorporated Using object re-identification in video surveillance
WO2019171986A1 (fr) 2018-03-05 2019-09-12 キヤノン株式会社 Dispositif et procédé de traitement d'image, et programme
US20190365350A1 (en) 2016-11-16 2019-12-05 Teratech Corporation Portable ultrasound system
WO2020002810A1 (fr) 2018-06-25 2020-01-02 Universite D'orleans Détection de nerfs dans une série d'images échographiques par la détermination de régions d'intérêts définies par la détection d'artères
WO2020044769A1 (fr) 2018-08-27 2020-03-05 富士フイルム株式会社 Dispositif de diagnostic à ultrasons et procédé de commande de dispositif de diagnostic à ultrasons
US20200394789A1 (en) 2019-06-12 2020-12-17 Carl Zeiss Meditec Inc Oct-based retinal artery/vein classification
US10902585B2 (en) 2018-03-19 2021-01-26 General Electric Company System and method for automated angiography utilizing a neural network
WO2021014767A1 (fr) 2019-07-23 2021-01-28 富士フイルム株式会社 Dispositif de diagnostic ultrasonore et procédé de commande de dispositif de diagnostic ultrasonore
US20210045716A1 (en) 2019-08-13 2021-02-18 GE Precision Healthcare LLC Method and system for providing interaction with a visual artificial intelligence ultrasound image segmentation module
US20210045711A1 (en) 2019-08-16 2021-02-18 Massachusetts Institute Of Technology Systems and methods for portable ultrasound guided cannulation
WO2021033446A1 (fr) 2019-08-19 2021-02-25 富士フイルム株式会社 Appareil de diagnostic ultrasonore et procédé de commande d'un appareil de diagnostic ultrasonore
WO2021033491A1 (fr) 2019-08-16 2021-02-25 富士フイルム株式会社 Appareil de diagnostic à ultrasons et procédé de commande d'un appareil de diagnostic à ultrasons
JP2021037239A (ja) 2019-09-05 2021-03-11 キヤノン株式会社 領域分類方法
US20210113194A1 (en) * 2019-10-17 2021-04-22 Verathon Inc. Systems and methods for ultrasound scanning
US20220230363A1 (en) 2021-01-21 2022-07-21 Canon Medical Systems Corporation Medical information processing apparatus

Patent Citations (44)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6245018B1 (en) 1997-12-15 2001-06-12 Medison Co., Ltd. Ultrasonic color doppler imaging system capable of discriminating artery and vein
US20080226017A1 (en) * 2005-09-06 2008-09-18 Koninklijke Philips Electronics N. V. Data Handling and Analysis in Computed Tomography With Multiple Energy Windows
JP2007222291A (ja) 2006-02-22 2007-09-06 Yunekusu:Kk 動脈血管判定方法および装置
US20100161023A1 (en) 2007-03-08 2010-06-24 Sync-Rx, Ltd. Automatic tracking of a tool upon a vascular roadmap
US20080269605A1 (en) 2007-04-25 2008-10-30 Shigemitsu Nakaya Ultrasonic diagnostic apparatus and image display method thereof
US20090103794A1 (en) 2007-10-19 2009-04-23 Boston Scientific Scimed, Inc. Display of classifier output and confidence measure in an image
US20140188133A1 (en) * 2007-11-26 2014-07-03 C. R. Bard, Inc. Iconic Representations for Guidance of an Indwelling Medical Device
JP2010269018A (ja) 2009-05-22 2010-12-02 Shimadzu Corp 超音波診断装置
US9731066B2 (en) * 2011-09-30 2017-08-15 General Electric Company Device, system and method of automatic vessel access based on real time volumetric ultrasound
US20130102889A1 (en) * 2011-10-21 2013-04-25 C. R. Bard, Inc. Systems and Methods for Ultrasound-Based Medical Device Assessment
US20130150710A1 (en) * 2011-12-01 2013-06-13 John Zentgraf Surgical navigation for repair of heart valve leaflets
US20140343431A1 (en) 2011-12-16 2014-11-20 Koninklijke Philips N.V. Automatic blood vessel identification by name
US20140031690A1 (en) 2012-01-10 2014-01-30 Panasonic Corporation Ultrasound diagnostic apparatus and method for identifying blood vessel
US20130197367A1 (en) 2012-03-14 2013-08-01 Jeffrey Smok Method and apparatus for locating and distinguishing blood vessel
US20140249404A1 (en) 2013-03-01 2014-09-04 Pacesetter, Inc. Vascular branch characterization
US20160117814A1 (en) 2013-04-10 2016-04-28 The Asan Foundation Method for distinguishing pulmonary artery and pulmonary vein, and method for quantifying blood vessels using same
US20150005630A1 (en) 2013-07-01 2015-01-01 Samsung Electronics Co., Ltd. Method of sharing information in ultrasound imaging
US20150126865A1 (en) 2013-11-05 2015-05-07 Seiko Epson Corporation Ultrasonic probe and ultrasonic measuring device
US20150209113A1 (en) 2014-01-29 2015-07-30 Becton, Dickinson And Company Wearable Electronic Device for Enhancing Visualization During Insertion of an Invasive Device
US20150238165A1 (en) 2014-02-21 2015-08-27 Seiko Epson Corporation Ultrasonic measurement apparatus and ultrasonic measurement method
US20180014810A1 (en) 2015-04-03 2018-01-18 Koninklijke Philips N.V. Ultrasound system and method of vessel identification
US20180015256A1 (en) * 2016-07-14 2018-01-18 C. R. Bard, Inc. Automated Catheter-To-Vessel Size Comparison Tool And Related Methods
JP2018015356A (ja) 2016-07-29 2018-02-01 セイコーエプソン株式会社 画像処理装置および画像処理方法
US20190365350A1 (en) 2016-11-16 2019-12-05 Teratech Corporation Portable ultrasound system
US20180137244A1 (en) * 2016-11-17 2018-05-17 Terarecon, Inc. Medical image identification and interpretation
US20180243033A1 (en) 2017-02-24 2018-08-30 Heartflow, Inc. Systems and Methods for Identifying Anatomically Relevant Blood Flow Characteristics in a Patient
US20180247154A1 (en) 2017-02-27 2018-08-30 Fujifilm Corporation Image classification apparatus, method, and program
JP2018139693A (ja) 2017-02-27 2018-09-13 富士フイルム株式会社 画像分類装置、方法およびプログラム
JP2018171177A (ja) 2017-03-31 2018-11-08 大日本印刷株式会社 眼底画像処理装置
US20180374233A1 (en) 2017-06-27 2018-12-27 Qualcomm Incorporated Using object re-identification in video surveillance
WO2019171986A1 (fr) 2018-03-05 2019-09-12 キヤノン株式会社 Dispositif et procédé de traitement d'image, et programme
JP2019150345A (ja) 2018-03-05 2019-09-12 キヤノン株式会社 画像処理装置、画像処理方法及びプログラム
US10902585B2 (en) 2018-03-19 2021-01-26 General Electric Company System and method for automated angiography utilizing a neural network
WO2020002810A1 (fr) 2018-06-25 2020-01-02 Universite D'orleans Détection de nerfs dans une série d'images échographiques par la détermination de régions d'intérêts définies par la détection d'artères
WO2020044769A1 (fr) 2018-08-27 2020-03-05 富士フイルム株式会社 Dispositif de diagnostic à ultrasons et procédé de commande de dispositif de diagnostic à ultrasons
US20200394789A1 (en) 2019-06-12 2020-12-17 Carl Zeiss Meditec Inc Oct-based retinal artery/vein classification
WO2021014767A1 (fr) 2019-07-23 2021-01-28 富士フイルム株式会社 Dispositif de diagnostic ultrasonore et procédé de commande de dispositif de diagnostic ultrasonore
US20210045716A1 (en) 2019-08-13 2021-02-18 GE Precision Healthcare LLC Method and system for providing interaction with a visual artificial intelligence ultrasound image segmentation module
US20210045711A1 (en) 2019-08-16 2021-02-18 Massachusetts Institute Of Technology Systems and methods for portable ultrasound guided cannulation
WO2021033491A1 (fr) 2019-08-16 2021-02-25 富士フイルム株式会社 Appareil de diagnostic à ultrasons et procédé de commande d'un appareil de diagnostic à ultrasons
WO2021033446A1 (fr) 2019-08-19 2021-02-25 富士フイルム株式会社 Appareil de diagnostic ultrasonore et procédé de commande d'un appareil de diagnostic ultrasonore
JP2021037239A (ja) 2019-09-05 2021-03-11 キヤノン株式会社 領域分類方法
US20210113194A1 (en) * 2019-10-17 2021-04-22 Verathon Inc. Systems and methods for ultrasound scanning
US20220230363A1 (en) 2021-01-21 2022-07-21 Canon Medical Systems Corporation Medical information processing apparatus

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2022/026087, dated Nov. 2, 2023, 9 pages.
International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2022/026089, dated Nov. 2, 2023, 7 pages.
International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2022/026122, dated Nov. 2, 2023, 8 pages.
International Search Report and Written Opinion on the Patentability of Application No. PCT/US2022/026122 (dated Aug. 9, 2022, 12 pages.
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US22/26087, dated Aug. 11, 2022, 11 pages.
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US22/26089, dated Aug. 11, 2022, 9 pages.
Mishra et al., "Ultrasound Image Segmentation: a Deeply Supervised Network With Attention to Boundaries", IEEE Transactions on Biomedical Engineering, vol. 66, No. 6, Jun. 2019, pp. 1637-1648.

Also Published As

Publication number Publication date
JP2024514957A (ja) 2024-04-03
WO2022226398A1 (fr) 2022-10-27
EP4326160A1 (fr) 2024-02-28
US20220338833A1 (en) 2022-10-27
US20240156429A1 (en) 2024-05-16
CN117222367A (zh) 2023-12-12

Similar Documents

Publication Publication Date Title
US20220361840A1 (en) Displaying blood vessels in ultrasound images
US11331076B2 (en) Method and system for displaying ultrasonic elastic measurement
EP3432803B1 (fr) Système ultrasonore et procédé permettant de détecter un glissement du poumon
CN108733857B (zh) 超声弹性成像装置及弹性成像结果评价方法
KR102188176B1 (ko) 침 시술 가이드 기능을 가진 초음파 영상 기기
JP7022217B2 (ja) 超音波システムのためのエコー窓のアーチファクト分類及び視覚的インジケータ
US20240156429A1 (en) Guiding instrument insertion
JP2020018694A (ja) 超音波診断装置及び超音波画像処理方法
US20180042578A1 (en) Automated ultrasound image measurement system and method
JP2020103883A (ja) 超音波撮像システムおよび対象物体品質レベルを表示するための方法
CN116194048A (zh) 膈肌的超声测量方法及系统
US11900593B2 (en) Identifying blood vessels in ultrasound images
CN112515747A (zh) 用于分析超声场景以提供针引导和警告的方法和系统
US20200229795A1 (en) Method and systems for color flow imaging of arteries and veins
JP2021007512A (ja) 超音波診断装置及び解析装置
JP7299100B2 (ja) 超音波診断装置及び超音波画像処理方法
US12023196B2 (en) Ultrasonic diagnostic apparatus and storage medium
JP7488789B2 (ja) 超音波診断装置及び診断支援方法
US20220202395A1 (en) Ultrasonic imaging system and ultrasonic imaging method
US20210307720A1 (en) Ultrasonic diagnostic apparatus and storage medium
US11413019B2 (en) Method and apparatus for displaying ultrasound image of target object
JP2022038088A (ja) 超音波診断装置
JP2023051175A (ja) コンピュータプログラム、情報処理方法、及び情報処理装置
CN116919466A (zh) 蠕动波检测方法和超声成像装置
JP2020022550A (ja) 超音波画像処理装置およびプログラム

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

AS Assignment

Owner name: FUJIFILM SONOSITE, INC., WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:DHATT, DAVINDER S.;WHITE, CHRISTOPHER ALEKSANDR;PELY, ADAM BENJAMIN;AND OTHERS;SIGNING DATES FROM 20210505 TO 20210517;REEL/FRAME:056313/0377

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE