US11393085B2 - Image analysis using machine learning and human computation - Google Patents
Image analysis using machine learning and human computation Download PDFInfo
- Publication number
- US11393085B2 US11393085B2 US16/538,662 US201916538662A US11393085B2 US 11393085 B2 US11393085 B2 US 11393085B2 US 201916538662 A US201916538662 A US 201916538662A US 11393085 B2 US11393085 B2 US 11393085B2
- Authority
- US
- United States
- Prior art keywords
- image
- machine learning
- user inputs
- learning algorithm
- filter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
- G06T7/0014—Biomedical image inspection using an image reference approach
- G06T7/0016—Biomedical image inspection using an image reference approach involving temporal comparison
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10101—Optical tomography; Optical coherence tomography [OCT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20092—Interactive image processing based on input by user
- G06T2207/20104—Interactive definition of region of interest [ROI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30041—Eye; Retina; Ophthalmic
Definitions
- This disclosure relates generally to image analysis. More specifically, this disclosure relates to the use of machine learning and human computation in image analysis.
- Machine learning or artificial intelligence is being applied in the area of medical research, particularly in areas of research where large amounts of data need to be collected and analyzed.
- machine learning is able to analysis thousands to millions of individual data sets and detect patterns within these large data sets that normally would be impossible or impractical to analyze given the scope.
- Age-related macular degeneration is the leading cause of visual impairment for people over age 50 in the developed world.
- AMD Age-related macular degeneration
- the macula is where the eye processes detailed vision.
- RPE retinal pigment epithelium
- Bruch's membrane As part of the complex anatomic features of AMD, deposits known as drusen form between the retinal pigment epithelium (RPE) and Bruch's membrane.
- RPE retinal pigment epithelium
- Bruch's membrane Bruch's membrane
- OCT optical coherence tomography
- OCT is a key tool in the diagnosis and treatment evaluation of AMD.
- OCT is a non-invasive imaging method utilizing light to provide high-resolution cross-sections of the neurosensory retina.
- OCT measures light waves reflected and scattered by the retinal tissue. Analysis of the cross-sections allows researchers to identify drusen and other anatomic features of AMD and would potentially allow researchers to monitor the progression of AMD in response to treatment.
- FIG. 1 illustrates an example networked system in which various embodiments of the present disclosure may be implemented
- FIG. 2 illustrates an example of a computer system in which various embodiments of the present disclosure may be implemented
- FIG. 3 illustrates an example of a user computing device in which various embodiments of the present disclosure may be implemented
- FIG. 4 illustrates an example OCT image of a normal retina and includes labels indicating the various structures within the retina tissue which may be analyzed in accordance with embodiments of the present disclosure
- FIGS. 5A and 5B illustrate example OCT images of a retina with AMD which may be analyzed in accordance with embodiments of the present disclosure
- FIG. 6 illustrates another example OCT image showing subretinal drusen material having variability in amount and appearance but consistency in anatomic location which may be analyzed in accordance with embodiments of the present disclosure
- FIG. 7 illustrates a user interface for a gaming application that is used to generate human computational data in accordance with various embodiments of the present disclosure
- FIGS. 8A and 8B are graphs illustrating example mean error results for a set of players playing their first and seventh images, respectively in accordance with various embodiments of the present disclosure.
- FIG. 9 illustrates an example flowchart of a process for analyzing an image in accordance with various embodiments of the present disclosure.
- Embodiments of the present disclosure provide for analyzing an image using machine learning and human computation.
- a method for analyzing an image includes providing, via multiple instances of an interactive application for analysis of the image, multiple instances, respectively, of the image and receiving, via the interactive application, data from results of analyses of the image including multiple sets of user inputs from the analyses of the multiple instances of the image, respectively.
- the multiple sets of user inputs are input via the multiple instances of the interactive application, respectively; the multiple sets of user inputs are from multiple users, respectively; and the multiple users are associated with the multiple instances of the interactive application, respectively.
- the method further includes processing the received data to identify one or more areas of interest present within the image based on the multiple sets of user inputs and analyzing the image using a machine learning algorithm to identify one or more structures present in the image based on the identified one or more areas of interest present within the image.
- a system for analyzing an image includes a communication interface and a processor operably connected to the communication interface.
- the communication interface is configured to provide, via multiple instances of an interactive application for analysis of the image, multiple instances, respectively, of the image and receive, via the interactive application, data from results of analyses of the image including multiple sets of user inputs from the analyses of the multiple instances of the image, respectively.
- the multiple sets of user inputs are input via the multiple instances of the interactive application, respectively; the multiple sets of user inputs are from multiple users, respectively; and the multiple users are associated with the multiple instances of the interactive application, respectively.
- the processor is configured to process the received data to identify one or more areas of interest present within the image based on the multiple sets of user inputs and analyze the image using a machine learning algorithm to identify one or more structures present in the image based on the identified one or more areas of interest present within the image.
- a non-transitory, computer-readable medium for analyzing an image comprises program code that, when executed by a processor of a system, causes the system to provide, via multiple instances of an interactive application for analysis of the image, multiple instances, respectively, of the image and receive, via the interactive application, data from results of analyses of the image including multiple sets of user inputs from the analyses of the multiple instances of the image, respectively.
- the multiple sets of user inputs are input via the multiple instances of the interactive application, respectively; the multiple sets of user inputs are from multiple users, respectively; and the multiple users are associated with the multiple instances of the interactive application, respectively.
- the computer-readable medium further comprises program code that, when executed by a processor of a system, causes the system to process the received data to identify one or more areas of interest present within the image based on the multiple sets of user inputs and analyze the image using a machine learning algorithm to identify one or more structures present in the image based on the identified one or more areas of interest present within the image.
- FIGS. 1 through 9 discussed below, and the various embodiments used to describe the principles of the present disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably arranged system or device.
- Embodiments of the present disclosure recognize that automated image evaluation procedures exist and continue to improve but have yet to provide a comprehensive solution, and often require verification or modification by a trained evaluator.
- Embodiments of the present disclosure further recognize the technical problem that automated solutions frequently misidentify anatomic boundaries, particularly with deformations such as, for example, those created by drusen as well as anatomic features that represent degenerative features of the retina.
- trained OCT evaluators are often employed to manually evaluate or correct the analysis.
- medical imaging such as OCT
- Embodiments of the present disclosure recognize that a limited number of trained image evaluators exist and demand for new evaluation approaches is increasing.
- Human computing has proven to be an effective way to crowdsource a variety of scientific problems, as well as leverage human pattern-recognition ability.
- Human computing can be used to collect data (e.g., gathering or generating samples), perform human calculation problems (such as solving challenges as a replacement for all or part of an algorithm), or provide human analysis (e.g., classification, data verification).
- video games allow users to interact with the scientific data while also leveraging the elements game developers require to maintain engagement.
- various embodiments of the present disclosure provide image analysis solutions that utilize video games to gain human intuition and leverage that human intuition in connection with machine learning to correctly identify structures (or the absence thereof) in an image.
- FIG. 1 illustrates an example networked system 100 in which various embodiments of the present disclosure may be implemented.
- the embodiment of the networked system 100 shown in FIG. 1 is for illustration only. Other embodiments of the networked system 100 could be used without departing from the scope of this disclosure.
- the system 100 includes a network 101 , which facilitates communication between various components in the system 100 .
- the network 101 may communicate Internet Protocol (IP) packets or other information between network addresses.
- IP Internet Protocol
- the network 101 may include one or more local area networks (LANs); metropolitan area networks (MANs); wide area networks (WANs); a virtual private network (VPN); all or a portion of a global network, such as the Internet; or any other communication system or systems at one or more locations.
- LANs local area networks
- MANs metropolitan area networks
- WANs wide area networks
- VPN virtual private network
- all or a portion of a global network, such as the Internet or any other communication system or systems at one or more locations.
- the network 101 facilitates communications among various computer systems 102 - 104 and various user computing devices 106 - 112 .
- Each of the computer systems 102 - 104 may be any suitable electronic computing or processing device(s) that can provide computing services including software for one or more user computing devices 106 - 112 .
- Each of the computer systems 102 - 104 could, for example, include one or more processing devices, one or more memories storing instructions and data, and one or more network interfaces facilitating communication over the network 101 .
- computer system 102 may be a computer system for a researcher or data processor that is used to run machine learning algorithms that leverage human intuition to correctly identify structures (or the absence thereof) in an image.
- computer system 104 may be associated with a medical provider or researcher to supply images for analysis and receive the results thereof.
- Each user computing device 106 - 112 represents any suitable electronic computing or processing device that interacts with at least one computer system or other computing device(s) over the network 101 .
- the user computing devices 106 - 112 include a desktop computer 106 , a mobile telephone or smartphone 108 , a laptop computer 110 , a video game console 112 ; a set-top box and/or television, etc.
- any other or additional user computing devices could be used in the networked system 100 .
- any Internet or network connectable device or Internet of Things (IoT) device e.g., Smart TVs, refrigerators, Raspberry PIs, etc.
- IoT Internet of Things
- user computing devices 106 - 112 may be employed to provide a video game that drives user engagement to provide human intuition to help identify structures in images.
- FIG. 1 illustrates one example of a networked system 100
- the system 100 could include any number of each component in any suitable arrangement and each of computer systems 102 - 104 and user computing devices 106 - 112 may be representative of any number of computer systems and/or user computing devices that are part of system 100 .
- computing and communication systems come in a wide variety of configurations, and FIG. 1 does not limit the scope of this disclosure to any particular configuration. While FIG. 1 illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.
- FIGS. 2 and 3 illustrate example computing devices in a networked system according to various embodiments of the present disclosure.
- FIG. 2 illustrates an example computer system 200
- FIG. 3 illustrates an example user computing device 300 .
- the computer system 200 represents any one of the computer systems 102 - 104 in FIG. 1
- the user computing device 300 could represent one or more of the user computing devices 106 - 112 in FIG. 1 .
- the computer system 200 includes a bus system 205 , which supports communication between processor(s) 210 , storage devices 215 , communication interface (or circuit) 220 , and input/output (I/O) unit 225 .
- the processor(s) 210 executes instructions that may be loaded into a memory 230 .
- the processor(s) 210 may include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement.
- Example types of processor(s) 210 include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuitry.
- the memory 230 and a persistent storage 235 are examples of storage devices 215 , which represent any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, and/or other suitable information on a temporary or permanent basis).
- the memory 230 may represent a random-access memory or any other suitable volatile or non-volatile storage device(s).
- the persistent storage 235 may contain one or more components or devices supporting longer-term storage of data, such as a read-only memory, hard drive, Flash memory, or optical disc.
- persistent storage 235 may store images for analysis, human computational data 240 (such as training data representing human intuition inputs), and/or a machine learning algorithm 245 to process the images.
- the communication interface 220 supports communications with other systems or devices.
- the communication interface 220 could include a network interface card or a wireless transceiver facilitating communications over the network 101 .
- the communication interface 220 may support communications through any suitable physical or wireless communication link(s).
- the I/O unit 225 allows for input and output of data.
- the I/O unit 225 may provide a connection for user input through a keyboard, mouse, keypad, touchscreen, or other suitable input devices.
- the I/O unit 225 may also send output to a display, printer, or other suitable output devices.
- FIG. 2 illustrates one example of a computer system 200
- various changes may be made to FIG. 2 .
- various components in FIG. 2 could be combined, further subdivided, or omitted and additional components could be added according to particular needs.
- the computer system 200 may include multiple computer system systems that may be remotely located.
- FIG. 3 illustrates an example user computing device 300 according to embodiments of the present disclosure.
- the embodiment of the user computing device 300 illustrated in FIG. 3 is for illustration only, and the user computing devices 106 - 112 of FIG. 1 could have the same or similar configuration.
- user computing devices come in a wide variety of configurations, and FIG. 3 does not limit the scope of this disclosure to any particular implementation of an electronic device.
- the user computing device 300 includes a communication interface (or circuit) 305 , processor(s) 310 , an input/output (I/O) interface 315 , an input 325 , a display 320 , and a memory 330 .
- the memory 330 includes an operating system (OS) 332 and one or more gaming applications 334 .
- OS operating system
- the communication interface or circuit 305 supports communications with other systems or devices.
- the communication interface 305 could include a network interface card or a wireless transceiver facilitating communications over the network 101 .
- the communication interface 305 may support communications through any suitable physical or wireless communication link(s).
- the communication interface 305 may receive an incoming RF signal via one or more antennas using a variety of wireless communication protocols, (e.g., Bluetooth, Wi-Fi, cellular, LTE communication protocols etc.).
- the processor(s) 310 can include one or more processors or other processing devices and execute the OS 332 stored in the memory 330 in order to control the overall operation of the user computing device 300 .
- the processor(s) 310 is also capable of executing gaming application(s) 334 resident in the memory 330 , such as, program code for one or more gaming applications for performing human computing tasks, such as, for example, games that incentivize engagement in image analysis as discussed in greater detail below.
- the gaming application may be downloaded to and run by the user computing device 300 or may be web-based and played by accessing a server, for example, computer system 200 .
- the processor(s) 310 , communication interface 305 and/or memory 330 constitute the computing resources of the user computing device 300 .
- the processor(s) 310 may include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement.
- Example types of processor(s) 310 include microprocessors, microcontrollers, graphical processing units (GPUs), digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuitry.
- the processor(s) 310 can move data into or out of the memory 330 as required by an executing process.
- the processor(s) 310 is also coupled to the I/O interface 315 , which provides the user computing device 300 with the ability to connect to other devices, such as laptop computers and handheld computers.
- the I/O interface 315 provides a communication path between accessories and the processor(s) 310 .
- the processor(s) 310 is also coupled to the input 325 and the display 320 .
- the operator of the user computing device 300 can use the input 325 to enter data and inputs into the user computing device 300 .
- the input 325 may be a touchscreen, button, keyboard, trackball, mouse, stylus, electronic pen, video game controller, etc.
- the display 320 may be a liquid crystal display, light emitting diode display, or other display capable of rendering text and/or at least limited graphics, such as from websites.
- the memory 330 is coupled to the processor(s) 310 . Part of the memory 330 could include a random-access memory (RAM), and another part of the memory 330 could include a Flash memory or other read-only memory (ROM).
- RAM random-access memory
- ROM read-only memory
- FIG. 3 illustrates one example of user computing device 300
- various changes may be made to FIG. 3 .
- various components in FIG. 3 could be combined, further subdivided, or omitted and additional components could be added according to particular needs.
- the processor(s) 310 could be divided into multiple processors, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs).
- the display 320 may be externally connected to or not a part of the user computing device 300 , such as for example, with a video game console or desktop computer.
- FIG. 4 illustrates an example OCT image 400 of a normal retina and includes labels indicating the various structures within the retina tissue which may be analyzed in accordance with embodiments of the present disclosure.
- OCT images are generated by an OCT unit that can, noninvasively and without risk, capture two-dimensional images of the retina and be reconstructed to provide processed three-dimensional information as well.
- OCT image 400 details of the retinal architecture can be captured and analyzed. For example, laser light reflected into the OCT machine to reveal surfaces and layers within the object being imaged and the retina can have a large number of cell types and many layers of material and anatomic interfaces.
- OCT image analysis can help in the diagnosis and management of patients with degenerative AMD.
- challenges that researchers face in fully utilizing OCT to manage degenerative AMD patients are that 1) at the initial stages of the disease the OCT changes can be very subtle; 2) the progression of changes seen on OCT in patients with degenerative AMD can be difficult to detect and 3) the extent of possible changes seen on the OCT in patients with degenerative AMD can be very complex.
- various embodiments of the present disclosure provide image analysis solutions that utilize interactive applications (such as video games or other applications requiring or using human input) to gain human intuition and leverage that human intuition in connection with machine learning to correctly identify structures (or the absence thereof) in an image.
- FIGS. 5A and 5B are example OCT images 500 of a retina with AMD which may be analyzed in accordance with embodiments of the present disclosure.
- FIG. 5A is annotated with has asterisks ( 501 and 502 ) identifying bumps which represent drusen material from a degenerative AMD subject.
- FIG. 5B is the same image but showing upper and lower lines ( 505 and 510 , respectively) generated as a result of line segmentation having performed for use in measuring the volume of the drusen material in the image.
- the amount of the material may be medically significant (relevant to progression or regression of the disease).
- each OCT image is generated at 5 microns or smaller in thickness and approximately 90 such OCT images, or more, are generated per eye. While some of the images can be processed by machine run algorithms, these image analysis techniques can be highly inaccurate. Moreover, often times the image may be out of focus, for example, if the patient is moving or low-resolution images (e.g., 750 pixels) that do not contain significant enough amounts of information for machine run algorithms to generate accurate results. In some instances, the layers to be quantitated may be missing or very small (e.g., due to atrophy).
- the machine run algorithms may not provide enough useful information, for example, by not identifying exactly where the segmentation between the layers is.
- the lines 505 and 510 are inserted above or below where the actual segment between the layers is present in the image.
- technicians manually fix the placement of the lines 505 and 510 which is very labor intensive particularly given the scope and quantity of the images.
- FIG. 6 illustrates another example OCT image showing subretinal drusen material (as denoted by the asterisks labeled 601 and 602 ) having variability in amount and appearance which may be analyzed in accordance with embodiments of the present disclosure.
- FIG. 7 illustrates a user interface (UI) 700 for a gaming application (such as gaming application 334 ) that is used to generate the human computational data 240 in accordance with various embodiments of the present disclosure.
- UI user interface
- relevant portions i.e., retinal tissue layers
- FIGS. 5A and 5B the inverted color scans of relevant portions (i.e., retinal tissue layers) of the OCT image, such as illustrated in FIGS. 5A and 5B , are displayed on the UI 700 as roads or trails that enemy robots will travel (e.g., as denoted by upper and lower lines).
- the user engages in the macular degeneration research by encouraging the creation of the upper and lower lines 705 and 710 defining the drusen volume within a given scan.
- This provided a constrained playfield, modeled after tower defense games where players would defend predefined paths by placing towers.
- the users are able to place points as “towers”, for defense.
- the enemy robots travel the path and the closer the player was to the correct lines, the higher score the player receives.
- Additional decoration such as trees and buildings are added in the upper and lower regions of the intended playfield to provide more of a gaming experience.
- the enemy robots follow the optimal routes defined by the manual corrected OCT analysis.
- Players unaware of the optimal analysis, place nodes to define an upper and lower line to mirror the “roads” the enemy robots will follow.
- the players submit the information which is sent to an online database storing the image solutions.
- the players watch the robots walk along their respective paths, leaving a trail behind to visually train players on the types of features (e.g., segments and layers in the retina) to follow in future levels representing new OCT scans.
- targeting lasers damage the robots depending on the overall accuracy of the line. The closer players place their nodes to the optimal path, the more damage the lasers do, with perfect placement eliminating all enemies.
- the players After completing an image, the players receive a final scoring report detailing their accuracy. Accuracy extends from keeping their defined lines within a given number of units, which translated to the user as direct hits, solid hits, partial hits, and misses. After receiving their score, players proceed to the next stage where they interact with a new image with this process repeating for several images.
- FIGS. 8A and 8B are graphs illustrating example mean error results for a set of players playing their first and seventh images, respectively in accordance with various embodiments of the present disclosure.
- the results of the gameplay are processed, for example, by the computing system to determine the accuracy of results and/or improve machine learning image analysis.
- the same images are played in sequential order.
- the average results from the set of all (e.g., 500 players) for each image is calculated.
- the error between the known solution and this average user position is calculated.
- the players are fairly poor at identifying the correct locations of the layers in the image as illustrated in FIG. 8A .
- the average player position for the top and bottom lines ( 815 and 820 , respectively) is quite different than the position for the top and bottom lines ( 805 and 810 , respectively) for the known solution for the first image.
- the shaded portions between the lines represent the error or deviation.
- this error decreases overtime, for example, by the seventh image the results are reasonably accurate, for example, about a pixel difference, which is consistent with the range of accuracy in professional image analyzers.
- the average player position for the top and bottom lines ( 865 and 870 , respectively) much more closely tracks the position for the top and bottom lines ( 855 and 860 , respectively) for the known solution for the seventh image. While illustrated as a line, in other embodiments, the user inputs in the image may form any other pattern or shape (e.g., circle, rectangle, etc.) for other types of objects or aspects thereof in the image being analyzed.
- the data from subsequent image analyses can be used as a reliable potential solution for the image that does not have an accepted or professionally determined solution.
- the results of several analyses are averaged to determine the final solution.
- the results data set may be limited to relevant results by, for example, discarding proposed solutions that fall at least a predetermined or weighted threshold above or below from the mean results for each point along the lines (i.e., obviously incorrect solutions) and/or removing all results for a user ID associated with the unreliable results.
- a subset of the images that have known solutions are reserved and not used for generating the training data.
- the subset known solutions can be compared against the human-generated solutions for the same images to determine whether the overall set of generated training data is reliable and/or ascertain a rate of error or standard deviation that can be extrapolated to the overall training dataset.
- these results are used by themselves as the proposed solution and/or form human computational data, such as training data, that can be used as inputs to improve a machine learning algorithm that performs additional image analysis.
- the gamification provides a mechanism to insert human intuition or human computation into the image analysis. This gamification not only trains users to perform image analysis but also the processing of the result data increases the accuracy of the ultimate proposed solution. Going one step further, in some embodiments, gamification is used as a training methodology to train technicians (i.e., professional image analyzers) on how to analyze the images. In other words, these embodiments use the game as a tool to teach technicians in reading centers how to analyze the images.
- the lower line is relatively straight and receives fewer inputs to denote the curve of the line.
- several additional points are placed on the upper line to approximate the curve around the bump.
- the number of points entered to approximate positions on the curve can be used to identify potential areas of interest in the image.
- additional image analyses can be run just on that area of the image, for example, using the game-based analysis approach, flagged for review or fine-tuning by medical professionals, or used as an input for machine learning. In so doing, embodiments of the present disclosure generate meta-level data for the image analysis.
- machine learning algorithms process the image data, for example, passing a filter, such as convolution filters or color filters, over the image data for image analysis, with little ability to identify the relevance of areas in the image.
- a filter such as convolution filters or color filters
- embodiments of the present discourse generate a heat-map image showing the density of dots or a derivative of points on average and provide this meta-level data to a machine learning algorithm.
- the machine learning algorithm can identify that particular areas of the image are more relevant and perform additional processing (e.g., iterative processing) thereon, rather than wasting processing resources on less relevant areas such as normal tissue layer interfaces, noise, or background pixels.
- Various embodiments of the present disclosure utilize several filter steps in real-time image analysis. For example, steps of edge detection, grayscale, color, density filters, derivative filters, etc. can be performed by the machine learning algorithm in any order. Additionally, for these embodiments, the present disclosure includes a “human intuition” filter as one or more of the steps in the image analysis. Providing these human intuition inputs to the algorithm can provide suggestions or mutations that allow the solutions provided machine learning algorithm to jump up a level in accuracy. Accordingly, in addition to or instead of assisting in identifying areas of interest/removing areas of non-interest, the generated meta-level data provides an additional input into the machine learning algorithm that indicates where the crowdsourced or human intuition data shows the characteristics of a given line.
- the average position data for points placed by user input is viewable as a heat-map image. This generates a new data set that provides another layer of information captured by user input.
- the meta-level data can not only provide indicators of the relevant areas of the image, but also can providing insight about any given pixel. This, for example, assists the machine learning algorithm in identifying how to change the slope of the line when placing a next point.
- some embodiments provide a methodology to train an algorithm to identify the human intuition input or filter. For example, in deep neural networks, each layer of the deep neural network adds additional complexity or robustness. Using the data about where the user inputs occur in their image analysis (e.g., as a set of training data), embodiments of the present disclosure generate an algorithm that can identify points or curves in a manner similar to the way the human would identify the points or curves. Then in these embodiments, this “human layer” generation is used to improve the overall machine learning algorithm to remove the need for human input in the image analysis.
- FIG. 9 illustrates an example of a process for analyzing an image in accordance with various embodiments of the present disclosure.
- the process depicted in FIG. 9 is described as implemented by the server 200 in FIG. 2 ; the process may also be implemented by any of the server systems 102 - 104 in FIG. 1 , and different servers systems may perform any number or portion of the steps of the process, collectively referred to herein as the system.
- the process begins with the system providing multiple instances of an image for analysis via an interactive application (operation 905 ).
- the system may provide the respective instances of the image via respective instances of an interactive application designed for analysis of the image.
- the interactive application is a gaming application (e.g., such as gaming application 334 ) and includes a user interface representative of the image such as user interface 700 to receive multiple sets of user inputs per image analyzed via the interactive application.
- the system receives data from results of analyses of the image including multiple sets of user inputs (operation 910 ).
- the multiple sets of user inputs are from the analyses of the multiple instances of the image, respectively; the multiple sets of user inputs are input via the multiple instances of the interactive application, respectively; the multiple sets of user inputs are from the multiple users, respectively; and the multiple users associated with the multiple instances of the interactive application, respectively.
- the system may receive these results from each of multiple of client devices 300 from each of multiple users in a collaborative computing or citizen science manner.
- the system may repeat the providing of the multiple instances of the image and the receiving of the data from results of analyses of the image for each of multiple different images.
- the system may provide sequential images of a same or similar type of structure to be analyzed, for example, in successive levels of a game, as discussed, for example, in connection with FIG. 7 above.
- the system determines whether the results of analyses for images are within a predetermined accuracy range (operation 915 ). For example, as discussed above, a first set or a subset of the different images provided for analysis may have a known solution and a second set or the remainder do not have a known solution.
- the system may use the successive providing of the images to train and or provide feedback (e.g., via the interactive application, for example, as a score in a game or as a direct results comparison) to players or technicians regarding those images which have known solutions. Further, as discussed above, the system may perform processing on the results data to filter out or weight user results that deviate from mean results (e.g., for images with or without a known solution) beyond the predetermined accuracy range.
- the system may only use the received data from the results of analyses for images that are within the predetermined accuracy range or deviation for later processing.
- the system may apply a lower weight to results data outside of the predetermined accuracy range, which may, for example, be proportionate in some manner to the amount of deviation from the known solution.
- the system may use additional techniques to determine accuracy. For example, the system may average results and discard or apply lower weight to those solutions outside of a standard or desired deviation therefrom.
- system processes the received data to identify area(s) of interest in the image(s) (operation 920 ).
- the system identifies the area(s) of interest based on the multiple sets of user inputs.
- the system may generate a filter for the image based on both the pattern of each of the user inputs input on the image and a quantity of the user inputs for the pattern in an area of the image.
- both the pattern and quantity of inputs can be indicative of the area interest.
- multiple inputs within a small area can be indicative of some aspect or feature of the image that is significant for processing of the image.
- This filter may be a first filter for a first layer of an aspect of the structure(s) present in the image and multiple such filters can be generated and combined for use in analyzing the image.
- the system may generate a second filter for another aspect of the structures in the image based on both of another pattern of each of the user inputs input on the image and quantity of the user inputs for the other pattern in a second area of the image.
- the structures could be any structures that are medically relevant such as layers of tissue in an organ, such as, for example, the human eye.
- the aspects associated with each of the filters could be, for example, lines representative of an amount of a substance (e.g., drusen) present or could be different aspects, such as different aspects of the image to be analyzed simultaneously, for example.
- the system analyzes the image(s) using a machine learning algorithm to identify structure(s) in the image(s) (operation 925 ).
- the system identifies the structure(s) based on the identified area(s) of interest present within the image using the processed user input results data.
- the system may use the filter(s) for the image as an input for the machine learning algorithm to focus the machine learning algorithm on the areas of interest in the image.
- the system may encode the filter(s) into the image as a data point or channel in the image itself similar to, for example, the red green blue (RGB) channels of an image.
- RGB red green blue
- the system may segment the image using the filter to focus processing of the machine learning algorithm to the one or more areas of interest present within the image, for example, by removing areas outside of the areas of interest using the filter. This may be advantageous in embodiments where reduced image processing time or reduced processing resources (e.g., computing cycles or memory) are desirable. In other embodiments, the system may process the entire image via the machine learning algorithm and use the filter(s) to focus the machine learning algorithm for additional processing inputs on the areas of interest. This may be advantageous in embodiments where precision and accuracy are more important that reduced image processing time or processing resources.
- the system may use the processed user input results data with the machine learning algorithm to improve the machine learning algorithm.
- the system may use the processed user input results data as training data to train the machine learning algorithm, and, with enough training data, train the machine learning algorithm to generate similar results data to use in various aspects of image analysis, for example, without requiring further the user image analysis.
- the system may generate a machine learning model for identifying probable user inputs in the images using the processed user input results data as training data and applying the machine learning model to identify probable user inputs in other images to train the machine learning algorithm to process the other images without requiring the multiple sets of user inputs.
- the system may, additionally or alternatively, use the processed user input results data in other portions of the machine learning algorithm to modify the training or development of and/or improve machine learning algorithm to perform the image processing.
- the system may use the processed user input results data (e.g., the user identified solutions for structure(s) or area(s) of interest in the image) to weight or error correct solutions proposed by the machine learning algorithm.
- the system may modify the training of the machine learning algorithm itself via the processed user input results data.
- the system may analyze images over time to generate important statistical data or results.
- the image may be of a portion of a bodily organ and this portion of a bodily organ may be imaged over time.
- the system may analyze these images provided over time to track a status of the structure(s) in the image over time and generate information using the tracked status. For example, this information can advantageously be used to monitor treatment efficacy for a disease of the bodily organ over the time and track progression of a disease of the bodily organ over the time as discussed in greater detail below.
- FIG. 9 illustrates an example of process for analyzing an image
- various changes could be made to FIG. 9 .
- steps in each figure could overlap, occur in parallel, occur in a different order, or occur multiple times.
- steps may be omitted or replaced by other steps.
- various embodiments of the present disclosure provide an algorithm for generating a solution for individual image analysis and others provide a manner for generating inputs to provide a portion of the solution.
- the algorithm is trained to provide the solution in one step, and in others, a meta-level data set is generated first (e.g., by using human intuition computation or by a machine learning algorithm using training data) and use the algorithm to provide the solution using the meta-level data set.
- OCT images to for AMD analysis
- AMD analysis are example implementations, applications, or use cases and not intended as a limitation on the many different number and type of implementations, applications, or use cases that may be utilized in accordance with the embodiments of the present disclosure.
- Any type of images may be analyzed for any medical or other structural identification reasons in accordance with the principles of the present disclosure.
- any type of OCT image analysis such as retina, cornea, lens, heart, lungs, other organs, can be used etc.
- Other medical diagnoses or purposes such as diabetic retinopathy, other retinal diseases, or OCT angiography (e.g., using consecutive images to track blood flow in blood vessels and generate images of blood flow to identify abnormalities in blood flow) may be utilized in the embodiments of the present disclosure.
- other types of images may be utilized in the embodiments of the present disclosure including but not limited to retinal fundus images, fluorescein angiographic images, indocyanine green angiographic images and ultrasound images.
- other images related to medicine may be utilized in the embodiments of the present disclosure including computerized tomography scans, magnetic resonance imaging, positive emission tomography, or x-rays may be amenable to the described analysis.
- a method for analyzing an image includes providing, via multiple instances of an interactive application for analysis of the image, multiple instances, respectively, of the image and receiving, via the interactive application, data from results of analyses of the image including multiple sets of user inputs from the analyses of the multiple instances of the image, respectively.
- the multiple sets of user inputs are input via the multiple instances of the interactive application, respectively; the multiple sets of user inputs are from multiple users, respectively; and the multiple users are associated with the multiple instances of the interactive application, respectively.
- the method further includes processing the received data to identify one or more areas of interest present within the image based on the multiple sets of user inputs and analyzing the image using a machine learning algorithm to identify one or more structures present in the image based on the identified one or more areas of interest present within the image.
- a system for analyzing an image includes a communication interface and a processor operably connected to the communication interface.
- the communication interface is configured to provide, via multiple instances of an interactive application for analysis of the image, multiple instances, respectively, of the image and receive, via the interactive application, data from results of analyses of the image including multiple sets of user inputs from the analyses of the multiple instances of the image, respectively.
- the multiple sets of user inputs are input via the multiple instances of the interactive application, respectively; the multiple sets of user inputs are from multiple users, respectively; and the multiple users are associated with the multiple instances of the interactive application, respectively.
- the processor is configured to process the received data to identify one or more areas of interest present within the image based on the multiple sets of user inputs and analyze the image using a machine learning algorithm to identify one or more structures present in the image based on the identified one or more areas of interest present within the image.
- a non-transitory, computer-readable medium for analyzing an image comprises program code that, when executed by a processor of a system, causes the system to provide, via multiple instances of an interactive application for analysis of the image, multiple instances, respectively, of the image and receive, via the interactive application, data from results of analyses of the image including multiple sets of user inputs from the analyses of the multiple instances of the image, respectively.
- the multiple sets of user inputs are input via the multiple instances of the interactive application, respectively; the multiple sets of user inputs are from multiple users, respectively; and the multiple users are associated with the multiple instances of the interactive application, respectively.
- the computer-readable medium further comprises program code that, when executed by a processor of a system, causes the system to process the received data to identify one or more areas of interest present within the image based on the multiple sets of user inputs and analyze the image using a machine learning algorithm to identify one or more structures present in the image based on the identified one or more areas of interest present within the image.
- the method further includes repeating the providing of the multiple instances of the image and the receiving of the data from results of analyses of the image for each of multiple different images including the one or more structures, wherein a first set of the multiple different images have a known solution and a second set do not have a known solution; determining, per user in the multiple users, whether the results of analyses for images from the first set are within a predetermined accuracy range; and generating feedback regarding whether the results of analyses for images from the first set are within the predetermined accuracy range.
- the processing of the received data to identify one or more areas of interest comprises generating a filter for the image based on both a pattern of the user inputs input on the image and quantity of the user inputs for the pattern in an area of the image; and the analyzing of the image using the machine learning algorithm comprises using the filter for the image as an input for the machine learning algorithm.
- the filter is a first filter for a first layer of an aspect of the one or more structures present in the image
- processing the received data to identify one or more areas of interest comprises generating a second filter for the image based on both a second pattern of the user inputs input on the image and quantity of the user inputs for the second pattern in a second area of the image, the second filter being for a second layer of the aspect of the one or more structures present in the image
- analyzing the image using the machine learning algorithm further comprises combining the first and second filters to form a multiple layer filter and using the multiple layer filter for the image as an input for the machine learning algorithm.
- using the filter for the image as an input for the machine learning algorithm comprises segmenting the image using the filter to focus processing of the machine learning algorithm to the one or more areas of interest present within the image.
- the method further includes repeating the providing of the multiple instances of the image and the receiving of the data from results of analyses of the image for each of multiple different images including the one or more structures; the analyzing of the image using the machine learning algorithm includes generating a machine learning model for identifying probable user inputs in the multiple different images using the results from the multiple different images as training data; and applying the machine learning model to identify probable user inputs in other images to train the machine learning algorithm to process the other images without requiring the multiple sets of user inputs.
- the analyzing of the image using the machine learning algorithm includes training the machine learning algorithm based on the identified one or more areas of interest present within the image; and analyzing the image using the trained machine learning algorithm to identify the one or more structures present in the image.
- the method further includes the interactive application is a gaming application and includes a user interface representative of the image and the multiple sets of user inputs are input into the user interface representative of the image.
- the image is of a portion of a bodily organ
- the analyzing of the image using a machine learning algorithm to identify one or more structures present in the image includes tracking a status of the one or more structures over time by processing additional images of the one or more structures using the machine learning algorithm; and generating information using the tracked status for at least one of (i) monitoring treatment efficacy for a disease of the bodily organ over the time and (ii) tracking progression of a disease of the bodily organ over the time.
- Couple and its derivatives refer to any direct or indirect communication between two or more elements, whether or not those elements are in physical contact with one another.
- transmit and “communicate,” as well as derivatives thereof, encompass both direct and indirect communication.
- the term “or” is inclusive, meaning and/or.
- phrases “associated with,” as well as derivatives thereof, means to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, have a relationship to or with, or the like.
- the phrase “such as,” when used among terms, means that the latter recited term(s) is(are) example(s) and not limitation(s) of the earlier recited term.
- the phrase “at least one of,” when used with a list of items, means that different combinations of one or more of the listed items may be used, and only one item in the list may be needed. For example, “at least one of: A, B, and C” includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A and B and C.
- various functions described herein can be implemented or supported by one or more computer programs, each of which is formed from computer readable program code and embodied in a computer-readable medium.
- application and “program” refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof adapted for implementation in a suitable computer readable program code.
- computer-readable program code includes any type of computer code, including source code, object code, and executable code.
- computer-readable medium includes any type of medium capable of being accessed by a computer, such as read-only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory.
- ROM read-only memory
- RAM random access memory
- CD compact disc
- DVD digital video disc
- a “non-transitory” computer-readable medium excludes wired, wireless, optical, or other communication links that transport transitory electrical or other signals.
- a non-transitory, computer-readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- General Health & Medical Sciences (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
Description
Claims (24)
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CA3107154A CA3107154A1 (en) | 2018-08-10 | 2019-08-12 | Image analysis using machine learning and human computation |
US16/538,662 US11393085B2 (en) | 2018-08-10 | 2019-08-12 | Image analysis using machine learning and human computation |
PCT/US2019/046233 WO2020033975A1 (en) | 2018-08-10 | 2019-08-12 | Image analysis using machine learning and human computation |
AU2019319298A AU2019319298B2 (en) | 2018-08-10 | 2019-08-12 | Image analysis using machine learning and human computation |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201862717681P | 2018-08-10 | 2018-08-10 | |
US16/538,662 US11393085B2 (en) | 2018-08-10 | 2019-08-12 | Image analysis using machine learning and human computation |
Publications (2)
Publication Number | Publication Date |
---|---|
US20200051241A1 US20200051241A1 (en) | 2020-02-13 |
US11393085B2 true US11393085B2 (en) | 2022-07-19 |
Family
ID=69406075
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/538,662 Active 2039-09-28 US11393085B2 (en) | 2018-08-10 | 2019-08-12 | Image analysis using machine learning and human computation |
Country Status (5)
Country | Link |
---|---|
US (1) | US11393085B2 (en) |
EP (1) | EP3815096A4 (en) |
AU (1) | AU2019319298B2 (en) |
CA (1) | CA3107154A1 (en) |
WO (1) | WO2020033975A1 (en) |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11379697B2 (en) | 2020-05-20 | 2022-07-05 | Bank Of America Corporation | Field programmable gate array architecture for image analysis |
US11295430B2 (en) | 2020-05-20 | 2022-04-05 | Bank Of America Corporation | Image analysis architecture employing logical operations |
US11562467B2 (en) * | 2020-08-31 | 2023-01-24 | Servicenow Canada Inc. | Method and system for designing an optical filter |
EP4288916A4 (en) * | 2021-02-04 | 2024-04-03 | Telefonaktiebolaget LM Ericsson (publ) | Building an explainable machine learning model |
Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110301447A1 (en) * | 2010-06-07 | 2011-12-08 | Sti Medical Systems, Llc | Versatile video interpretation, visualization, and management system |
US20170046833A1 (en) * | 2015-08-10 | 2017-02-16 | The Board Of Trustees Of The Leland Stanford Junior University | 3D Reconstruction and Registration of Endoscopic Data |
WO2017165566A1 (en) | 2016-03-25 | 2017-09-28 | The Regents Of The University Of California | High definition, color images, animations, and videos for diagnostic and personal imaging applications |
US20180060512A1 (en) | 2016-08-29 | 2018-03-01 | Jeffrey Sorenson | System and method for medical imaging informatics peer review system |
US20180064335A1 (en) * | 2014-11-18 | 2018-03-08 | Elwha Llc | Retinal imager device and system with edge processing |
US20180137244A1 (en) * | 2016-11-17 | 2018-05-17 | Terarecon, Inc. | Medical image identification and interpretation |
US20180144214A1 (en) * | 2016-11-23 | 2018-05-24 | General Electric Company | Deep learning medical systems and methods for image reconstruction and quality evaluation |
US20180144466A1 (en) * | 2016-11-23 | 2018-05-24 | General Electric Company | Deep learning medical systems and methods for image acquisition |
US20190029623A1 (en) * | 2017-07-26 | 2019-01-31 | Canon U.S.A., Inc. | Method for evaluating cardiac motion using an angiography image |
US20190392942A1 (en) * | 2016-10-12 | 2019-12-26 | Terarecon, Inc. | System and method for medical image interpretation |
US20210137384A1 (en) * | 2017-12-13 | 2021-05-13 | Washington University | System and method for determining segments for ablation |
-
2019
- 2019-08-12 CA CA3107154A patent/CA3107154A1/en active Pending
- 2019-08-12 EP EP19847194.8A patent/EP3815096A4/en active Pending
- 2019-08-12 WO PCT/US2019/046233 patent/WO2020033975A1/en unknown
- 2019-08-12 US US16/538,662 patent/US11393085B2/en active Active
- 2019-08-12 AU AU2019319298A patent/AU2019319298B2/en active Active
Patent Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110301447A1 (en) * | 2010-06-07 | 2011-12-08 | Sti Medical Systems, Llc | Versatile video interpretation, visualization, and management system |
US20180064335A1 (en) * | 2014-11-18 | 2018-03-08 | Elwha Llc | Retinal imager device and system with edge processing |
US20170046833A1 (en) * | 2015-08-10 | 2017-02-16 | The Board Of Trustees Of The Leland Stanford Junior University | 3D Reconstruction and Registration of Endoscopic Data |
WO2017165566A1 (en) | 2016-03-25 | 2017-09-28 | The Regents Of The University Of California | High definition, color images, animations, and videos for diagnostic and personal imaging applications |
US20180060512A1 (en) | 2016-08-29 | 2018-03-01 | Jeffrey Sorenson | System and method for medical imaging informatics peer review system |
US20190392942A1 (en) * | 2016-10-12 | 2019-12-26 | Terarecon, Inc. | System and method for medical image interpretation |
US20180137244A1 (en) * | 2016-11-17 | 2018-05-17 | Terarecon, Inc. | Medical image identification and interpretation |
US20180144214A1 (en) * | 2016-11-23 | 2018-05-24 | General Electric Company | Deep learning medical systems and methods for image reconstruction and quality evaluation |
US20180144466A1 (en) * | 2016-11-23 | 2018-05-24 | General Electric Company | Deep learning medical systems and methods for image acquisition |
US20190029623A1 (en) * | 2017-07-26 | 2019-01-31 | Canon U.S.A., Inc. | Method for evaluating cardiac motion using an angiography image |
US20210137384A1 (en) * | 2017-12-13 | 2021-05-13 | Washington University | System and method for determining segments for ablation |
Non-Patent Citations (5)
Title |
---|
Bogunović et al., "Machine Learning of the Progression of Intermediate Age-Related Macular Degeneration Based on OCT Imaging", Investigative Ophthalmology & Visual Science, vol. 58, No. 6, Jun. 2017, pp. BIO141-BIO150. |
Extended European Search Report dated Mar. 10, 2022 regarding Application No. 19847194.8, 11 pages. |
Fujii et al., "Histopathological validation of optical coherence tomography findings of the coronary arteries", Journal of Cardiology, vol. 72, No. 3, Sep. 2018, pp. 179-185. |
ISA/US, International Search Report and Written Opinion of the International Searching Authority, International Application No. PCT/US2019/046233, dated Nov. 13, 2019, 7 pages. |
Lee et al., "Use of Mechanical Turk as a MapReduce Framework for Macular OCT Segmentation", Journal of Opthalmology, vol. 2016, May 2016, 6 pages. |
Also Published As
Publication number | Publication date |
---|---|
AU2019319298A1 (en) | 2021-02-18 |
WO2020033975A1 (en) | 2020-02-13 |
CA3107154A1 (en) | 2020-02-13 |
US20200051241A1 (en) | 2020-02-13 |
AU2019319298B2 (en) | 2024-06-20 |
EP3815096A4 (en) | 2022-04-06 |
EP3815096A1 (en) | 2021-05-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11393085B2 (en) | Image analysis using machine learning and human computation | |
US20220165418A1 (en) | Image-based detection of ophthalmic and systemic diseases | |
Abràmoff et al. | Retinal imaging and image analysis | |
US20220323855A1 (en) | System for generating simulated animal data and models | |
AU2018347610A1 (en) | Deep learning-based diagnosis and referral of ophthalmic diseases and disorders | |
KR20220051369A (en) | Information processing apparatus, information processing method, information processing system and program | |
CN112868068B (en) | Processing fundus camera images using machine learning models trained with other modes | |
Kauppi | Eye fundus image analysis for automatic detection of diabetic retinopathy | |
Lin et al. | Retinal image quality assessment for diabetic retinopathy screening: A survey | |
CN109464120A (en) | A kind of screening for diabetic retinopathy method, apparatus and storage medium | |
Goyal et al. | A refined deep learning architecture for diabetic foot ulcers detection | |
CN112869697A (en) | Judgment method for simultaneously identifying stage and pathological change characteristics of diabetic retinopathy | |
Cazañas-Gordón et al. | Ensemble learning approach to retinal thickness assessment in optical coherence tomography | |
Abràmoff | Image processing | |
JP7332463B2 (en) | Control device, optical coherence tomography device, control method for optical coherence tomography device, and program | |
Shi et al. | Artifact-tolerant clustering-guided contrastive embedding learning for ophthalmic images in glaucoma | |
US20240203101A1 (en) | Hierarchical workflow for generating annotated training data for machine learning enabled image segmentation | |
Camara et al. | A comprehensive review of methods and equipment for aiding automatic glaucoma tracking | |
US11967077B1 (en) | System and method for predicting a need for total hip arthroplasty | |
JP7439990B2 (en) | Medical image processing device, medical image processing program, and medical image processing method | |
Sridhar et al. | Artificial intelligence in medicine: diabetes as a model | |
US20230144621A1 (en) | Capturing diagnosable video content using a client device | |
Pavani et al. | Robust semantic segmentation of retinal fluids from SD-OCT images using FAM-U-Net | |
Gullón | Retinal lesions segmentation using CNNs and adversarial training | |
Shylaja | Algorithmic approach for prediction and early detection of diseases using retinal images |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SOUTHERN METHODIST UNIVERSITY, TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CLARK, COREY;CSAKY, KARL;REEL/FRAME:050031/0626 Effective date: 20190812 Owner name: RETINA FOUNDATION OF THE SOUTHWEST, TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CLARK, COREY;CSAKY, KARL;REEL/FRAME:050031/0626 Effective date: 20190812 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: AWAITING TC RESP., ISSUE FEE NOT PAID |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |