Classifications

• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
  • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
  • G06T7/0012—Biomedical image inspection
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
  • G06T7/0012—Biomedical image inspection
  • G06T7/0014—Biomedical image inspection using an image reference approach
  • G06T7/0016—Biomedical image inspection using an image reference approach involving temporal comparison
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
  • G06V40/161—Detection; Localisation; Normalisation
  • G06V40/165—Detection; Localisation; Normalisation using facial parts and geometric relationships
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
  • G06V40/161—Detection; Localisation; Normalisation
  • G06V40/167—Detection; Localisation; Normalisation using comparisons between temporally consecutive images
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/18—Eye characteristics, e.g. of the iris
  • G06V40/193—Preprocessing; Feature extraction
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H30/00—ICT specially adapted for the handling or processing of medical images
  • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
  • G06V40/161—Detection; Localisation; Normalisation
  • G06V40/162—Detection; Localisation; Normalisation using pixel segmentation or colour matching
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H30/00—ICT specially adapted for the handling or processing of medical images
  • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS

Definitions

• the described embodiments relate to diagnoses of disease based on images collected over time.
• facial dimensions are considered at all in a day-to-day medical practice, typically, facial dimensions are examined visually by a physician at the time of a patient visit. Visible facial distortions may be employed by a doctor as one of many clues in a medical diagnosis.
• an image based medical diagnosis system is communicatively coupled to one or more image collection devices available to a human user.
• One or more image collection devices capture images of the human body. The collected images are stored and analyzed.
• images of the body of the human user are collected periodically over a significant period of time (e.g., at least once per month over a period of years) .
• the image based medical diagnosis system precisely measures facial dimensions and changes in facial dimensions over time and identifies whether any measured dimensions, changes in facial dimensions, or both, have exceeded one or more threshold values indicative of the onset or recurrence of a medically significant condition (e.g., onset or recurrence of disease, tumor, etc.) .
• the one or more dimensions are estimated based at least in part on a known value of a reference dimension identifiable in each of the plurality of images.
• an image based medical diagnosis tool determines the number of pixels between the pupils (i.e., inter-pupillary distance) located in each image. Based on the known distance between the pupils and the determined number of pixels, the IBMDT estimates a scaling factor that relates actual distance per pixel in each image. In addition, the IBMDT estimates the actual distance between selected facial features based on the number of pixels between the selected facial features located in each image and the scaling factor associated with each image.
• the distance between any selected facial features in an image is normalized with respect to the measured inter-pupillary distance (IPD) associated with that image. Since the actual IPD is known, the distance between any selected facial features is also determined .
• IPD inter-pupillary distance
• the image based medical diagnosis system communicates an alert (e.g., to the human user, qualified medical personnel, etc.) indicating abnormal body dimensions, changes in body dimensions, or both.
• an alert e.g., to the human user, qualified medical personnel, etc.
• the image based medical diagnosis system prompts medical professionals of the need to further investigate the medical condition of the human user.
• FIG. 1 is a diagram illustrative of an embodiment of an image based medical diagnosis system ( IBMDS ) 100 including one or more imaging devices 102 and an image based diagnosis tool (IBMDT) 101 in at least one aspect.
• IBMDS image based medical diagnosis system
• IBMDT image based diagnosis tool
• FIG. 2 is a diagram illustrative of an exemplary facial image 160 and dimensions of interest measured by an IBMDS.
• FIG. 3 is a diagram illustrative of a memory 131 storing facial records 151-156, including facial images collected over several years.
• FIG. 4 is a flowchart illustrative of a method 200 implementing health monitoring and communications gateway functionality as described herein.
• an image based medical diagnosis system is communicatively coupled to one or more image collection devices available to a human user.
• one or more image collection devices capture images of the face. The collected images are
• FIG. 1 is a diagram illustrative of an embodiment of an image based medical diagnosis system ( IBMDS ) 100 including one or more imaging devices 102 and an image based diagnosis tool (IBMDT) 101.
• imaging device 102 collects images of the face of a human user 107.
• the images 103 are communicated to IBMDT 101.
• IBMDT 101 includes a processor 120, a memory 130, a bus 140, and a wireless communication transceiver 150.
• processor 120, memory 130, and wireless communication transceiver 150 are configured to communicate over bus 140.
• IBMDT 101 is configured to receive images 103 from imaging device 102.
• Memory 130 also includes an amount of memory 132 that stores program code that, when executed by processor 120, causes processor 120 to implement image based medical diagnosis functionality as described herein.
• Memory 130 also includes an amount of memory 131 that stores measurement data extracted from images 103 by IBMDT 101.
• images of the face of the human user are collected periodically over a significant period of time (e.g., at least once per month over a period of years) .
• the image based medical diagnosis system precisely measures facial dimensions and changes in facial dimensions over time and identifies whether any measured facial dimensions, changes in facial dimensions, or both, have exceeded one or more threshold values indicative of the onset or recurrence of a medically significant condition (e.g., onset or recurrence of disease, tumor, etc.) .
• the one or more dimensions are estimated based at least in part on a known value of a reference dimension identifiable in each of the plurality of images .
• IBMDT 101 is communicatively coupled to imaging device 102 by a wired communication link.
• IBMDT 101 receives images 103 and extracts facial dimensions from the collected images.
• FIG. 2 depicts an exemplary image 160.
• IBMDT 101 extracts distances associated with the inter pupillary distance (IPD) , lower facial height (LFH) , and ear to ear (EE) distance.
• IPD inter pupillary distance
• LFH lower facial height
• EE ear to ear
• IBMDT 101 determines the number of pixels between the pupils (i.e., inter-pupillary
• IBMDT 101 estimates a scaling factor that relates actual distance per pixel in each image 103.
• IBMDT 101 estimates the actual distance between selected facial features based on the number of pixels between the selected facial features located in each image 103 and the scaling factor associated with each image 103.
• the distance between any selected facial features in an image is normalized with respect to the measured inter-pupillary distance (IPD) associated with that image. Since the actual IPD is known, the distance between any selected facial features is also determined .
• IPD inter-pupillary distance
• the IPD is described hereinbefore as a preferred reference dimension in facial images, in general, any stable facial dimension, or set of markings, captured in an image of the human face may be
• any suitable dimension may be estimated by IBMDT 101, and any suitable stable body dimension may be employed as a reference dimension.
• the image based medical diagnosis systems described herein may operate based on images of any portion of the human body to track any suitable body dimension provided a reference dimension having a known value is identifiable within each image under consideration.
• IBMDT 101 is communicatively coupled to one or more imaging devices 102 by any suitable communication link (e.g., wired or wireless communication link) .
• IBMDT 101 and an imaging device 102 are communicatively coupled by a wireless
• IBMDT 101 and an imaging device 102 are communicatively coupled with a wired communication link operating in compliance with any suitable wired
• communications protocol e.g., serial data link
• IBMDT 101 is implemented as part of a portable computing device such as a smart phone, tablet computer system, etc.
• the portable computing device integrates an imaging device 102 with a computing system implementing the functionality of IBMDT 101.
• a portable computing device and collected images are uploaded to an external computing system (e.g., external server, cloud based computing system, etc.) via signals 106 communicated from IBMDT 101.
• signals 106 are communicated over a wireless communications link (e.g., via wireless transceiver 150 depicted in FIG. 1) or via a wired communications link (e.g., through a wired connection between the portable computing device and a communication network (e.g., the Internet , etc . ) .
• imaging device 102 includes any suitable imaging device a user 107 may employ to collect facial images.
• imaging device 102 collects two dimensional images of the face of user 107.
• imaging device 102 includes a two dimensional image sensor (e.g., charge coupled device, complementary metal oxide on silicon device, etc.)
• IBMDT 101 generates a three dimensional model of the human face based on a sequence of images collected by imaging device 102 from different angles with respect to the human face.
• Distances between selected facial features are determined by IBMDT 101 based on the three dimensional model. As described hereinbefore, the distance between any selected points in the three dimensional model is normalized with respect to a reference distance (e.g., the measured IPD) associated with that three dimensional model. Since the actual IPD is known, the distance between any selected facial features is also determined.
• a reference distance e.g., the measured IPD
• imaging device 102 is a three dimensional imaging device, such as a scanning laser based imaging device.
• imaging device 102 is a "structure sensor" manufactured by
• imaging device 102 is a LIDAR based sensor. In these embodiments, the three dimensional imaging device 102 generates a three dimensional point cloud of image data 103 that is communicated to IBMDT 101. In other embodiments, imaging device 102 is a three dimensional camera system, such as the TrueDepth ® camera system manufactured by Apple, Inc., Cupertino, California (USA) that is integrated with portable computing devices manufactured by Apple Inc., such as the iPhoneX ® .
• a portable computing device such as a smartphone, tablet computer, etc., implements IBMDT functionality as described with respect to FIG. 1 based on three dimensional images 103 collected from a three dimensional imaging device.
• IBMD system 100 collects facial images and estimates facial dimensions at multiple points in time over a substantially long period.
• facial images are collected and facial dimensions are estimated one or more times per day, week, or month, over a period of years.
• FIG. 3 depicts an illustration of memory 131.
• memory 131 stores facial records 151-156 including facial images collected over several years, the time of image acquisition, and the estimated facial dimensions.
• the image based medical diagnosis system communicates an alert (e.g., to the human user, qualified medical personnel, etc.) indicating abnormal facial dimensions, changes in facial dimensions, or both.
• an alert e.g., to the human user, qualified medical personnel, etc.
• the image based medical diagnosis system prompts medical professionals of the need to further investigate the medical condition of the human user.
• processor 120 is
• an alert is
• processor 120 is configured to process the digital images generated by imaging device 102 to detect anomalies that may indicate an imminent, or on-going medical health emergency. In the event such an anomaly is detected, processor 120 communicates an alarm message 105 (e.g., audibly via a speaker of the portable device
• processor 120 receives a digital image 103 and estimates an abnormal change in EE distance indicative of a brain tumor.
• processor 120 generates an alarm message 105 that is communicated to a medical doctor associated with the medical care of the human user 107.
• IBMDT 101 may send an e-mail, text, or phone message to a physician, a close relative, etc. indicating the nature of the medical emergency.
• IBMDS 100 identifies changes in dimensions of features of a human user (e.g., facial dimensions) that are indicative of tumor onset or recurrence after treatment.
• hormonally active pituitary gland tumors e.g., adenoma secreting growth hormone
• evoke characteristic changes in the shape of the human face e.g., Cushing's moon face or
• these tumors are very small (e.g., 3-5 mm in diameter), and reside in a pituitary gland. For these reasons, these tumors are often difficult to detect and resect completely.
• patients treated by surgery see an initial reversal of facial changes.
• symptoms slowly recur. This may occur over a period of several years.
• IBMDS 100 may identify early symptoms of tumor recurrence based on changes of facial dimensions measured in accordance with the methods and systems described herein. In this manner, patient anxiety evoked by frequent follow-up visits and tests and time and cost associated with these visits are avoided.
• IBMDS 100 identifies changes in dimensions of features of a human user (e.g., facial dimensions) that are indicative of heart failure.
• features measured by IBMDS 100 indicative of enhanced risk of heart failure include the dimension of diagonal creases on the earlobes, i.e., Frank's sign, xanthomas appearing on eyelids, a grey ring around the outside of the iris, a.k.a., arcus senilis.
• IBMDT 101 receives a query 104 from a concerned entity (e.g., human user 107, a medical professional associated with the medical care of human user 107, etc.) .
• a concerned entity e.g., human user 107, a medical professional associated with the medical care of human user 107, etc.
• IBMDT 101 communicates an indication 105 of the health status of the human body to the requesting entity including estimated facial dimensions, their changes over time, or both.
• IBMDS 100 is implemented as part of a portable computing device such as a smart phone, tablet computer system, etc.
• IBMDS 100 is configured to capture images of the human user 107 automatically when the desired portion of the body of human user 107 (e.g., the face of human user 107) is within the field of view of the imaging device 102 of the portable computing system. For example, when a human user 107 visually engages with the portable computing device for any purpose (e.g., interact with a software application, dial a phone number, compose or read a text message, etc.), the IBMDS 100 is
• IBMDS 100 prompts the human user 107 to: 1) locate the portable computing device such the desired portion of the body of the human user 107 is within the field of view of an imaging device of the portable computing system, and 2) instruct the portable computing device to collect one or more images of the human user 107.
• IBMDS 100 may prompt a human user to collect one or more images on a periodic basis (e.g., every hour, every day, etc.) .
• IBMDS 100 may prompt a human user to collect one or more images in response to a request from a query 104 from a medical professional.
• IBMDS 100 is implemented as part of portable computing device integrating three dimensional (3D) display capabilities such as virtual reality goggles, etc.
• an imaging device integrated with the three dimensional display captures images of the face of the human user 107 and estimates facial images such as IPD, pupil size, etc.
• the images captured by the 3D display system are analyzed by IBMDT 101 to determine whether a human user wearing the 3D display is undergoing an epileptic seizure based on measured pupil size.
• FIG. 4 illustrates a flowchart of a method 200 implementing health monitoring and communications gateway functionality as described herein.
• IBMDS 100 is operable in accordance with method 200 illustrated in FIG. 4.
• the method 200 illustrated in FIG. 4 is operable in accordance with method 200 illustrated in FIG. 4.
• execution of method 200 is not limited to the embodiments of IBMDS 100 described with reference to FIG. 1. This illustration and corresponding explanation are provided by way of example as many other embodiments and
• a plurality of facial images of a human is captured by an imaging device.
• one or more dimensions of the human associated with each of the plurality of images are estimated.
• the one or more dimensions are estimated based at least in part on a known value of a reference dimension identifiable in each of the plurality of images .
• an alert message is communicated to an external entity if the one or more dimensions, or changes in the one or more dimensions, exceeds the predetermined threshold value.
• Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another.
• a storage media may be any available media that can be accessed by a general purpose or special purpose computer.
• such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor.
• any connection is properly' termed a computer-readable medium . For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital
• Disk and disc includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically ⁇ ', while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

Landscapes

• Engineering & Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Physics & Mathematics (AREA)
• General Health & Medical Sciences (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Oral & Maxillofacial Surgery (AREA)
• Human Computer Interaction (AREA)
• Multimedia (AREA)
• Medical Informatics (AREA)
• Radiology & Medical Imaging (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Public Health (AREA)
• Quality & Reliability (AREA)
• Biomedical Technology (AREA)
• Epidemiology (AREA)
• Primary Health Care (AREA)
• Geometry (AREA)
• Pathology (AREA)
• Databases & Information Systems (AREA)
• Data Mining & Analysis (AREA)
• Ophthalmology & Optometry (AREA)
• Measuring And Recording Apparatus For Diagnosis (AREA)
• Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
• Image Analysis (AREA)

Priority Applications (3)

Applications Claiming Priority (4)

Publications (1)

Family

ID=67617955

Family Applications (1)

Country Status (5)

Cited By (1)

Families Citing this family (3)

Citations (6)

Family Cites Families (3)

• 2019
  • 2019-02-16 US US16/278,071 patent/US10621414B2/en active Active - Reinstated
  • 2019-02-16 EP EP19754083.4A patent/EP3756132A4/en active Pending
  • 2019-02-16 JP JP2021518423A patent/JP2021520982A/ja active Pending
  • 2019-02-16 AU AU2019221774A patent/AU2019221774B2/en active Active
  • 2019-02-16 WO PCT/US2019/018372 patent/WO2019161316A1/en not_active Ceased

Patent Citations (6)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events