EP2987106A1 - Acquisition et analyse de données physiologiques - Google Patents

Acquisition et analyse de données physiologiques

Info

Publication number
EP2987106A1
EP2987106A1 EP14785516.7A EP14785516A EP2987106A1 EP 2987106 A1 EP2987106 A1 EP 2987106A1 EP 14785516 A EP14785516 A EP 14785516A EP 2987106 A1 EP2987106 A1 EP 2987106A1
Authority
EP
European Patent Office
Prior art keywords
data
information
image
skin
imagery
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
EP14785516.7A
Other languages
German (de)
English (en)
Other versions
EP2987106A4 (fr
Inventor
Bruce L DAVIS
Tony F. Rodriguez
Geoffrey B RHOADS
John Stach
Shankar THAGADUR SHIVAPPA
Alastair M. Reed
Ravi K SHARMA
Richard F GIBSON
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Digimarc Corp
Original Assignee
Digimarc Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US14/206,109 external-priority patent/US20140316235A1/en
Application filed by Digimarc Corp filed Critical Digimarc Corp
Publication of EP2987106A1 publication Critical patent/EP2987106A1/fr
Publication of EP2987106A4 publication Critical patent/EP2987106A4/fr
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G06T7/0014—Biomedical image inspection using an image reference approach
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H70/00—ICT specially adapted for the handling or processing of medical references
    • G16H70/60—ICT specially adapted for the handling or processing of medical references relating to pathologies
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30088—Skin; Dermal
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Definitions

  • Medical diagnosis is an uncertain art, which depends largely on the skill and experience of the practitioner. For example, dermatological diagnosis tends to be based on very casual techniques, like observation by doctor, or on very invasive techniques, like biopsies. Skin condition degrades with age. It is difficult for people to differentiate the effects of normal aging from disease. This leads to lots of worry and unnecessary doctor visits. More rigorous diagnostic techniques can be applied to educate the public, assist medical professionals, and lower health care costs.
  • An example is diagnosis of diseases evidenced by skin rashes and other dermatological symptoms.
  • a skilled dermatologist may be able to accurately identify dozens of obscure conditions by their appearance, whereas a general practitioner may find even some common rashes to be confounding.
  • highly skilled practitioners are sometimes puzzled, e.g., when a rash appears on a traveler recently returned from the tropics, and the practitioner has no experience with tropical medicine.
  • differential diagnosis in dermatology includes location on body, color, texture, shape, and distribution. Other relevant factors include age, race, sex, family tree, and geography of person; and environmental factors including diet, medications, exposure to sun, and occupation. Many skin conditions have topologies and geographies that can be mapped in various dimensions, including depth, color and texture.
  • the prior art includes smartphone apps that are said to be useful in diagnosing skin cancer. Some rely on computerized image analysis. Others refer smartphone snapshots to a nurse or physician for review. The former have been found to perform very poorly. See, e.g., Wolf et al, Diagnostic Inaccuracy of Smartphone Applications for Melanoma Detection, JAMA Dermatology, Vol. 149, No. 4, April 2013 (attached to application 61/872,494).
  • dermatological conditions and other enrollment information is compiled in a crowd- sourced database, together with associated diagnosis information.
  • This reference information may be contributed by physicians and other medical personnel, but can also be provided by the lay public (e.g., relaying a diagnosis provided by a doctor).
  • a user submits a query image to the system (typically with anonymous
  • Image-based derivatives are determined (e.g., color histograms, FFT-based metrics, etc.) for the query image, and are compared against similar derivatives for the reference imagery. In one arrangement, those reference images whose derivatives most closely correspond to the query image are determined, and their associated diagnoses are identified. This information is presented to the user in a ranked listing of possible pathologies. In some embodiments, when the user's submitted query image and associated information is analyzed by the system and several likely diagnoses identified, the system may provide specific questions (guided by the results of the current analysis) to the user, or requests for additional images, to help distinguish among the candidate diagnoses.
  • the analysis identifies diseases that are not consistent with the query image and associated information. Again, this information is reported to the user
  • the imagery is supplemented with 3D information about the surface topology of the skin, and this information is used in the matching process.
  • 3D information can be derived from the imagery, or may be separately sensed.
  • the knowledge base includes profile information about the subjects whose skin conditions are depicted.
  • This profile information can include, e.g., drugs they are taking, places they have visited in the days leading up to onset of symptoms, medical history, lifestyle habits, etc.
  • the system can also report statistically- significant co-occurrence information derived from the profile information. For example, the system may report that 27% of people having a skin condition like that depicted in the user's query image report taking vitamin A supplements.
  • the co-occurrence information is broken down by candidate diagnoses.
  • the system may report that the top candidate diagnosis is miliaria X (42% chance). 35% of people with this diagnosis report having been in the tropics in the 30 days prior to onset of symptoms, and 25% report occasional use of hot tubs or saunas. The next top candidate diagnosis is tinea Y (28% chance). 60% of people with this diagnosis report having chicken pox as a child.
  • Such co-occurrence information can help in making a differential diagnosis from among the offered alternatives.
  • some embodiments of the technology do not attempt to identify, or rule-out, particular diagnoses. Instead, they simply seek to identify correlated factors from the knowledge base created from information from users, image analysis, and crowd-sourced data, so that possibly causative factors might be addressed (e.g., by suspending intake of supplemental vitamin A, in the example given above).
  • the user-submitted information is added to the knowledge base, and forms part of the reference information against which future submissions are analyzed.
  • Audio signals include heart sounds and other cardiovascular sounds (including murmurs, bruits, and other blood flow noises), lung and other respiratory sounds (including crackles, rales, rhonchi, wheezes, coughs, snoring and other air flow noises), bowel and digestive sounds, joint noises (e.g., pops and creaks), as well as speech and other vocalizations.
  • cardiovascular sounds including murmurs, bruits, and other blood flow noises
  • lung and other respiratory sounds including crackles, rales, rhonchi, wheezes, coughs, snoring and other air flow noises
  • bowel and digestive sounds including joint noises (e.g., pops and creaks), as well as speech and other vocalizations.
  • Fig. 1 illustrates components of one implementation of the technology, including plural remote terminals (e.g., smartphones), and one or more central systems.
  • Fig. 2 illustrates conceptual organization of an exemplary diagnostic system using technology disclosed herein.
  • Fig. 3A shows a banknote
  • Fig. 3B shows an excerpt from the banknote.
  • Fig. 4 shows normalized reflectance plots for the Fig. 3B banknote excerpt, and for a white envelope.
  • Fig. 5 is a schematic sectional view of a full-body imaging booth.
  • Figs. 6A and 6B are views depicting features of alternate imaging booths.
  • Figs. 7, 8 and 9 detail sequences of smartphone screen displays that provide illumination of different spectral characteristics, from different positions relative to the smartphone camera.
  • Fig. 10 details how light from different parts of a smartphone screen display illuminates a feature on a skin from different angles.
  • Fig. 1 shows a hardware overview of one embodiment employing principles of the present technology. Included are one or more user terminals (e.g., smartphones), and a central system.
  • user terminals e.g., smartphones
  • central system e.g., a central system.
  • each smartphone includes various functional modules - shown in rectangles. These include one or more processors, a memory, a camera, and a flash. These latter two elements are controlled by the processor in accordance with operating system software and application software stored in the memory.
  • the central system similarly includes one or more processors, a memory, and other conventional components. Particularly shown in Fig. 1 is a knowledge base - a database data structure that facilitates storage and retrieval of data used in the present methods.
  • One aspect of the present technology includes the central system receiving first imagery depicting a part of a human body that evidences a symptom of a pathological condition (e.g., skin rash or bumps). This imagery (and its image metadata) can be uploaded to the central system from one of the user terminals using commonly available image submission means and enrollment. The image then is processed to derive one or more image parameter(s). A data structure containing reference information is then searched, for reference image data that is parametrically similar to the first imagery. Based on results of this search, one or more particular pathological conditions that are not the pathological condition evidenced by the depicted part of the human body are identified. Resulting information is then communicated to the originating user terminal.
  • a pathological condition e.g., skin rash or bumps
  • a machine learning approach will be suitable for determining candidate diagnoses. Many features, whether part of the raw user images, or processed versions thereof, are presented to the machine learning algorithm. Additional features, representing age, gender, race, height, weight, etc., are also likely to be input to the algorithm. The machine learning algorithm can output a set of candidate diagnoses which best match the user's images and additional information. If representative images are to be presented to the user as representative of their diagnoses, the images should be chosen from the database not just based upon visual similarity, but also based upon how well the database image's associated additional features (age, gender, etc.) match the user.
  • representations of a scene and also to encompass other information optically captured from a subject. This can include, for instance, 3D microtopology. Such terms also encompass such information represented in non-spatial domains, e.g., FFT data, which represents the information is a spectral domain.
  • the derived image parameter(s) can be of various types, with some types being more discriminative for some pathologies, while other types are more discriminative for others.
  • One sample derived image parameter is a color histogram. This histogram may be normalized by reference to a "normal" skin color, e.g., as sampled from a periphery of the area exhibiting the symptom.
  • One such histogram is a 3D histogram, in which the first and second histogram dimensions are quantized hues (e.g., red-green, and blue-yellow), and the third histogram dimension is a quantized second derivative of luminance.
  • the first and second histogram dimensions are quantized hues (e.g., red-green, and blue-yellow)
  • the third histogram dimension is a quantized second derivative of luminance.
  • the imagery is spectrally accurate, so that hue-based image derivatives are diagnostically useful.
  • One low cost approach to acquiring such imagery is by gathering multiple frames of imagery under different, spectrally tuned illumination conditions, and processing same, as detailed in co-pending applications 13/840,451, filed March 15, 2013 (now published as 20130308045), and 14/201,852, filed March 8, 2014.
  • Another type of derived image parameter is a transformation of the imagery into a spatial frequency domain representation (e.g., FFT data).
  • FFT data e.g., FFT data
  • Such representation decomposes the image into components of different frequencies, angular orientations, phases and magnitudes
  • the decomposition of the image into such spatial frequency components can be conducted separately in different channels, e.g., yielding two-, three- or more- binned representations of different image chrominance and luminance planes. (More than the usual tri-color image representations can be used. For example, the image may be represented with 4-20 different color channels.)
  • Still another image derivative is wavelet transform data.
  • Such information is again a decomposition of the image information into a collection of orthonormal basis functions - in this case wavelets.
  • region growing A particular method, practiced in the pixel domain, involves selecting a seed pixel, and adding to a blob all of the contiguous pixels whose values are within a threshold value range of the seed pixel, e.g., plus or minus three digital numbers in luminance, on a 0-255 scale.
  • This process can be repeated for seed pixels throughout the image.
  • the seed pixels can be selected based on color or other parameter (e.g., a local maxima in image redness or contrast), or may be chosen randomly. What results is a pattern of 2D regions whose shape and scale parameters are useful as diagnostic indicia.
  • a particular image metric derived from blob analysis is a histogram identifying frequency of occurrence of different shapes. Shapes may be classified in various fashions. A simple two- class division, for example, may distinguish shapes that have exclusively convex boundaries (e.g., circles and ovoids) from shapes that have a concave aspect to part of their peripheries (e.g., blobs that have one or more inwardly-directed dimples). Much more sophisticated techniques are commonly used in blob analysis; an example is a histogram of oriented gradients. (See, e.g., Dalai, et al, Histograms of Oriented Gradients for Human Detection, IEEE Conference on Computer Vision and Pattern Recognition, pp. 886-893, 2005.)
  • luminance was used in the foregoing example, the technique can also be practiced in a particular color channel, or in Boolean logical combinations of color channels (e.g., add to the blob region those pixels whose value in a 500 nm spectral band is within 3 digital numbers of the seed value, OR whose value in a 530 nm spectral band is within 5 digital numbers of the seed value).
  • Boolean logical combinations of color channels e.g., add to the blob region those pixels whose value in a 500 nm spectral band is within 3 digital numbers of the seed value, OR whose value in a 530 nm spectral band is within 5 digital numbers of the seed value).
  • metrics can be computed on different scales.
  • One scale is across the totality of an image.
  • Another is to divide the image into hundreds of portions, and compute the metrics for each such portion.
  • the same image can be re-divided into tens of thousands of portions, with the metrics again recomputed.
  • These portions may be of any shape; rectangular is often computationally efficient, but others can be used.
  • the portions may be disjoint, tiled, or overlap. If computational constraints require, the finer scale metrics can be computed on a subset of all such regions, such as on a random selection of 1% of 100,000 regions.
  • the image derivatives can be computed on different color channels.
  • an image can be captured and accurately decomposed into five or ten or more different spectral bands - each of which may have diagnostic utility.
  • Such spectral- based analysis is not limited to the visible spectrum; infrared and ultraviolet data is also useful.
  • UV Ultraviolet light is absorbed by melanin.
  • illumination with UV can reveal irregular pigment distribution, which can aid, e.g., in defining the borders of melanoma.
  • CMOS and CCD sensors used in conventional digital cameras are typically responsive well into the infrared, provided there is no IR filtering.
  • the image, and image derivatives can also be based on polarized light photography.
  • Bag-of-features techniques can be applied to the image derivatives, e.g., as detailed in Csurka, et al, Visual Categorization with Bags of Keypoints, ECCV, Workshop on Statistical Learning in Computer Vision, 2004.
  • Another image derivative is feature size. Dimensions (e.g., diameter) of lesions and other visually-distinguishable skin features can be assessed from imagery, and this data included with the derivative image data. (The diagnostic profile of a feature is often dependent on its size.)
  • Fig. 2 is an excerpt of a conceptual view of a reference database. It includes a variety of records (rows), each comprising a set of data relating to a reference subject.
  • the first column contains an image (or a set of images) depicting a dermatological condition of the subject.
  • An image can comprise, e.g., a 10 megabyte color TIF file.
  • the second column shows some of the image derivatives computed from the image.
  • the naming convention gives semantic information about the type of data, e.g., indicating whether it is histogram or FFT data, and the coordinate of a tiled sub-region of the image from which the data was derived.
  • the third column shows the location on the subject's body from which the image was captured.
  • the fourth column shows, if available, a diagnosis of the reference subject's affliction. For some entries, no diagnosis is provided.
  • the fifth column shows additional user metadata.
  • Examples include demographic information (e.g., age, gender, weight, height, race, residence location by zip code), and other profile data about the subject. This can include drugs taken in the past thirty days, any on-going medical conditions, foods introduced into the subject's diet in the past thirty days, travel within the past sixty days, lifestyle activities, environmental exposures, family medical history, etc. It will be seen that information in the fourth and fifth columns is tagged using XML-style descriptors, to provide for extensibility and to facilitate text parsing.
  • a query image submitted by the user can similarly be accompanied by the body location and other user metadata information shown in Fig. 2.
  • a server system determines similarity scores between a query image and each of many reference images.
  • One component of such a score can be based on the reciprocal of a Euclidean distance between an image derivative from the query image and a corresponding image derivative for a reference image, in the image derivative feature space. Since each image may have thousands of derivatives (e.g., based on different regions and color channels), there can be many thousands of such components (e.g., comparing a histogram of region 1 of the query image with histograms of regions 1-1,000 of a reference image, and likewise for region 2 of the query image, etc.). Typically, such feature similarity metrics that fall below a statistically significant threshold are ignored.
  • some image derivatives are weighted more heavily than others.
  • the weight given to a particular correspondence between a pair of image derivatives can depend on the scale of the portions between which similarity was found. The larger the feature, the more weight is typically given (e.g., in linear or exponential proportion to feature size).
  • some indicia are more diagnostically relevant than others. Spectral data at 500 nm may be more
  • Weightings can be calculated recursively, accounting for feedback from users of the system about correlations.
  • a sampling, or all, of the reference images in the database are thus scored relative to the query image.
  • the reference images that are scored in the top 5%, or 0.5%, of the universe of evaluated reference images are thereby identified.
  • Associated user metadata for this set of reference images is then analyzed.
  • analysis of the top-scoring set of reference images may find that 40% are associated with diagnostic tags indicating that they depict the condition known as tinea versicolor, and 23% may be similarly tagged as depicting pityriasis rosea. 25% of the top- scoring reference images may be associated with diagnostic tags indicating that the reference subject was taking the blood pressure medicine Atenolol.
  • a statistical breakdown of such correlations is typically provided to the user - in one or more rank-ordered sets.
  • the user may be presented with a rank-ordered listing of the top five or ten possible diagnoses - each including a stated probability based on frequency of occurrence from the top-matching reference image set. Similar listings may be presented for demographic information and other profile data (e.g., drug correlations, diet correlations, lifestyle correlations, etc.).
  • the absence of apparent correlation can additionally, or alternatively, be reported to the user. If less than 0.03% of the reference images in the top-scoring set are associated with tinea versicolor, whereas this condition has a much greater frequency of occurrence in the full reference image set (e.g., 1.5%), then the user can be informed that the skin condition is most likely not tinea versicolor. Likewise with drugs, diet, lifestyle, etc. (The particular threshold used in such evaluation can be determined empirically.)
  • the information presented to the user can also include samples of closely-matching reference imagery - and the diagnosis (if any) associated with each.
  • Another method makes use of changes in the user's depicted symptoms over time.
  • the user submits two images to the system - an initial one, and a second one taken at a later time.
  • the system determines data about a change in the depicted skin symptom between these two times based on the submitted imagery. This determined data is then used in further refining diagnostic information.
  • Expert medical practitioners have the opportunity to "seed" such databases with known imagery examples of a variety of afflictions, paying a great deal of attention to ensuring a wide range of angles, lighting conditions, parts of the body, camera models, etc. This can involve the submission of hundreds, thousands or even more images with clinically derived examples of the major and less major categories of affliction.
  • the capturing of data from skin can employ known and forthcoming imaging
  • a simple one is a smartphone camera. Accessory optics may be employed to provide better close-up capabilities. Other digital cameras - including those on headworn devices - can also be used.
  • Exemplary smartphones include the Apple iPhone 5; smartphones following Google's Android specification (e.g., the Galaxy S5 phone, manufactured by Samsung, and the Google Moto X phone, made by Motorola), and Windows 8 mobile phones (e.g., the Nokia Lumia 1020, which features a 41 megapixel camera).
  • Google's Android specification e.g., the Galaxy S5 phone, manufactured by Samsung, and the Google Moto X phone, made by Motorola
  • Windows 8 mobile phones e.g., the Nokia Lumia 1020, which features a 41 megapixel camera.
  • Some embodiments employ modular mobile device technology, such as Google's Project
  • a mobile device is comprised of detachable components, which can be added, changed or upgraded as needs dictate.
  • a device can be assembled that includes sensors of the sorts detailed in this disclosure, especially adapted for physiologic data capture.
  • Imagery employed in the present technology may be in JPEG format, but preferably is in a higher quality form - such as RAW or TIF.
  • the smartphone or other user device can compute some or all of the derivative information from the sensed data before sending data to the remote database, or the central system can perform such calculations, based on provided sensor data. Or these tasks can be distributed - part performed on one platform, and part on another.
  • image capture can employ purpose-built hardware. Examples are disclosed in patent publication 20110301441. Commercial products include the Dermograph imager by MySkin, Inc., and the Handyscope by FotoFinder Systems. The latter is an accessory for the Apple iPhone 5 device and includes built-in illumination - optionally cross- polarized. It is capable of capturing both contact images (with the device touching the skin), and non-contact images. A variety of other dermatoscopy (aka epiluminescence microscopy) hardware systems are known.
  • a physical fixture can be provided on the imaging device to help establish a consistent imaging distance to the skin.
  • a rigid black, white or clear plastic cowl can extend from the camera lens (and optionally flash) at one end, to an opening that is placed over the skin, for controlled-distance imaging.
  • Software on the smartphone may employ known auto-focus technology to set an initial image focus, and can warn the user if the camera is unable to achieve proper focus.
  • some auto-focus algorithms are easily fooled into focusing on dark hair that may rise above the skin surface. Accordingly, it is preferable to capture several still image exposures - one at the nominal auto-focus setting, and others that are varied under software control from that position, e.g., at focal planes plus and minus two and four millimeters from the auto-focus setting.
  • the software can employ exposure-bracketing, since some features may more easily be distinguished in exposures taken one or two f-stops above, or below, an autoexposure setting.
  • Known high dynamic range methods can be employed to composite such images into an enhanced image frame.
  • a camera's frame capture is triggered based on stability.
  • a stability metric can be based on data from a smartphone sensor (e.g., an accelerometer). Or it can be based on analysis of the viewfinder image data.
  • the Apple iPhone device includes motion estimation hardware, which is most commonly employed for MPEG video compression, but which also can track features in an image frame to assess image stability.
  • imagery captured by mobile cameras is a focus of this disclosure, it will be recognized that imagery captured by whole body scanning systems can likewise be employed.
  • Canfield Scientific is among the commercial providers of whole body scanners.
  • Such apparatus (which may be, e.g., a stand-alone kiosk, or integrated into a weight scale in a doctor's office - capturing frontal face and neck imagery each time a patient is weighed) can be more sophisticated than that found in most smartphones, e.g., providing controlled spectral illumination (e.g., as in applications 13/840,451 (now published as 20130308045) and 14/201,852), thermal imaging, etc. It may provide the user with a hardcopy printout of the results.
  • Such an apparatus may be available for free use, or a nominal charge may be collected (e.g., by coin, dollar, or credit card).
  • photosensitizers e.g., aminolevulinic acid
  • various photosensitizers can be applied to the skin, to highlight certain tumors, etc., such as by changing their absorbance and fluorescence spectra.
  • the user moves a smartphone over a body area, while the camera captures imagery multiple frames of imagery.
  • 3D information about the skin's surface relief (topology) is discerned, e.g., using familiar stereoscopy techniques.
  • Google's patent publication 20130201301 details one such arrangement for creating 3D imagery from smartphone images captured at different viewpoints.
  • SLAM Simultaneous Localization and Mapping
  • SFM Structure from Motion
  • Such a 3D data representation can be virtually flattened, using cartographic techniques, for analysis and rendering to the user.
  • Patent application 13/842,282 filed March 15, 2013, details how the sensor in a moving device can be mounted on a MEMS-actuated pedestal, and moved in a cyclical fashion synchronized with the frame captures, to counteract motion blur.
  • the multiple frames of imagery collected in such a capture arrangement can be combined to yield an enhanced resolution image (e.g., as is taught in Digimarc' s published patent application 20080036886 and in patents 6,570,613 and 5,767,987).
  • patent application 13/842,282 details a particularly advantageous 3D camera sensor, employing photosites that are spectrally tuned - typically providing spectral responses at many more different wavelengths (e.g., at eight different wavelengths - some of which may be outside the visible range) than typical tri-stimulus (red/green/blue color-filter array) sensors of the previous art.
  • Skin topology measured using such skin print techniques is believed to have a higher sensitivity and specificity for machine-based identification of certain skin conditions, as compared with 2D color imagery.
  • skin truth skin topographies, which associate particular topographies with particular expert physician evaluations, are not yet available, these are expected to be forthcoming, when the utility of such measurements becomes widely known.
  • another aspect of the present technology includes aggregating skin prints for a variety of medical conditions in a reference database - at least some of which also include expert diagnoses associated therewith.
  • a related aspect involves deriving features from such reference prints, and then using such features in judging statistical similarities between a query skin print submitted by a user and the reference skin prints, to identify candidate diagnoses and other correlated information - as described earlier.
  • Skin surface minutiae can also be sensed otherwise, such as by systems for capturing human fingerprints. Examples are known from the published patent applications of AuthenTec (subsequently acquired by Apple), including applications 20120085822 and 20110309482. Such sensors are already included in many laptop computers, and will doubtless soon appear in smartphones and the like.
  • Another image data collection technique comprises a flexible sheet with organic transistor circuits.
  • the circuits can comprise photodetectors, as detailed, e.g., in Fuketa, et al, Large- Area and Flexible Sensors with Organic Transistors, 5th IEEE Int'l Workshop on
  • Such media can also include integrated OLED photodetectors - providing controlled illumination.
  • polarized light photography can also be useful with the present technology.
  • This can be implemented with polarized illumination, or with one or more polarizers on the camera or image sensor.
  • polarized illumination or with one or more polarizers on the camera or image sensor.
  • the filters have four different orientations, offset by 45 degrees.
  • image polarizations are sensed.
  • imagery at different polarizations different image features can be revealed and different image effects can be achieved (e.g., increased contrast).
  • Some research also indicates that polarized light, when reflected, has two orthogonal components - one due to the skin surface morphology, and the other "back-scattered" from within the tissue.
  • a close-up e.g., where the lesion spans 25% or more of the image width
  • a mid-view e.g., where the lesion spans between 5 and 25% of the image width
  • a remote view e.g., where the lesion spans less than 5% of the image width.
  • the remote view will typically show a sufficiently large body excerpt that the location of the lesion (e.g., arm, foot, hand, face) can be determined using known anatomical classification techniques.
  • the location of the lesion e.g., arm, foot, hand, face
  • Such lesion location data can then automatically be entered into the knowledge base, without requiring entry of such information by the user.
  • software can present the user with a 3D avatar on which the user virtually draws, or taps, to indicate locations of skin lesions.
  • Seeing the lesion in the context of an identifiable body part also provides context from which the size of the lesion can be estimated. E.g., the average man's palm is 3.05 inches across, permitting the size of a lesion depicted in the same frame to be deduced.
  • each view is at least 1000 pixels in width.
  • the smartphone software offers guidance to the user in capturing the images, e.g., directing that the user move the camera away from the body until the software's body part classifier is able to identify the body part in the third view. Other direction, e.g., concerning lighting and focus, can also be provided.
  • a lesion appears on a user's forearm, a second image may be submitted depicting the user's other forearm, or a skin patch that is not normally exposed to the sun - such as under the upper arm. Difference metrics can then be computed that compare the skin parameters around the lesion site with those from the other site. These data, too, can be submitted to the knowledge base, where similarities with other reference data may become evident.
  • liquid lenses e.g., marketed by Philips under the FluidFocus brand
  • Philips may soon appear on smartphones, and enable new camera close-up and topological sensing capabilities.
  • the body location from which the image is captured can be electrically sensed using small amplitude electrical waveforms inserted in the body by a wearable computer device - such as the Google Glass device, or a wrist- worn device.
  • a wearable computer device such as the Google Glass device
  • a wrist- worn device Especially if different signals are introduced into the body at two locations, their distinctive superposition at the sensing site can accurately pinpoint the location of such site.
  • Color is an important diagnostic feature in assessing dermatological conditions.
  • skin color as depicted in captured imagery, strongly depends on the "color” of the light that illuminates the skin. While dermatologists can control illumination conditions in their offices, most consumer image capture is performed under widely varying lighting conditions. To optimize performance of the detailed technologies, this variability should be mitigated.
  • ALB automatic white balance
  • One technique examines the pixels in an image, and identifies one that is the brightest. This pixel is assumed to correspond to a white or shiny feature in the image, i.e., a feature that reflects all of the incident light, without absorbing any particular color. The component color values of this pixel are then adjusted to make it truly white (e.g., adjusting an RGB representation to ⁇ 255,255,255 ⁇ ), and all other pixels in the image are remapped by similar proportions. Another technique averages all of the pixels in the image, and assumes the average should be a shade of grey (e.g., with equal red, green, and blue components - if represented in the RGB color space). A corresponding adjustment is made to all the image pixels, so that the average is remapped to a true shade of grey.
  • the former technique is ill-suited for skin photography because there is typically no white or specular pixel in the image.
  • the latter technique is ill-suited because its premise - that the average pixel value is grey - is not true for skin images.
  • a calibration card at the edge of a family group, where it can be cropped-out before printing.
  • the card includes various reference colors, including white and other known tones. Before printing, digital adjustments are made to the image to bring the depiction of colors on the calibration card to their original hues - thereby also color-compensating the portrait subject.
  • one approach to the ambient light issue is for a user to capture imagery from a calibration card, and send this image to the central system, accompanying the skin image(s). The system can then color-compensate the skin image(s), based on the depiction of colors in the calibration card image.
  • the entire envelope needn't be photographed - just a fraction will do.
  • a part of the envelope substrate is torn or cut off, and placed on the skin, within the camera's field of view. But such arrangement a single image capture can suffice.
  • the illumination-corrected, reflected color spectra from an assortment of white postal envelopes are captured and averaged, and used as reference data against which images received from end users are color-corrected.
  • the central service may, however, investigate the AWB techniques used by popular smartphone cameras. By examining the metadata that commonly is packaged with smartphone imagery, e.g., in the form of EXIF header data in an image file, the central system can determine the type of camera with which a user image was captured. If the image was captured from one of the cameras using the former AWB technique, and automated image analysis finds that the image includes an area of white next to skin tone, the system can infer that appropriate color correction has already been applied by the camera.
  • Another commonly available color reference - for those so-inclined - is oxygenated blood. Blood exhibits a consistent color spectrum despite race and other variable factors. If a drop of blood is thick enough to mask the underlying skin pigment, its color can be sensed and again used to reveal color information about the illumination.
  • Color calibration can also be performed with banknotes.
  • Banknotes are typically printed with extremely high tolerances, and consistent ink colors. Desirably, a banknote excerpt having colors near the skin tone range is employed. While US currency is commonly regarded as green, in fact the US $20 bill has areas of skin-like tones to the left and right of the Jackson portrait. (The US $10 has areas of reddish tones.)
  • the user captures images of the skin, and of a US $20 banknote, under the same illumination conditions. Both the skin image and the banknote image are then sent to the central system.
  • the central system again compares the spectrum found in the received banknote image with reference data, and determines a spectral correction function detailing variance between the received banknote image and reference data. The system then applies this correction function to the received skin image, to effect color correction.
  • Figs. 3 A and 3B show the banknote artwork, and a representative clipped region spanning most of the skin tone region.
  • This area is defined by "corner” features in the original artwork (e.g., the upper right corner of the letter E in “...PUBLIC AND PRIVATE;” the lower left corner of the A in AMERICA; etc.), and omits artwork that can vary between banknotes, i.e., the serial number.
  • the reference data is acquired by a reflectance spectroscopy technique that involves masking the banknote with a flat black mask - revealing only the clipped region - and illuminating with a light source whose spectrum is measured or otherwise known. Reflected light is sensed by a spectrometer, yielding a set of data indicating intensity as a function of wavelength. This measured data is then adjusted to compensate for the known spectrum of the light source.
  • Fig. 4 shows such a reference spectrum measured for both the Jackson portrait excerpt shown in Fig. 3B (the lower line), and for a sample white postal envelope.
  • the contemplated system may serve users in diverse countries. Desirably, suitable calibration objects are identified so that one or more is available in each of these countries.
  • the central system can examine the incoming imagery, and compare against a catalog of calibration objects to recognize which object is being used. Thus, a customer may choose to use a Mexican 100 peso note as a reference, and the central system will recognize same and apply the corresponding correction function.
  • the procedure employing a printed object in the image frame with the skin also allows the system to assess the brightness of the imaged scene.
  • Cameras have limited dynamic range. If a scene is too brightly lit, the camera's component red, blue and green sensors can no longer sense variability between different parts of the image. Instead, each outputs its full maximum signal (e.g., 255, in an 8-bit sensor). Faithful color sensing is lost. Similarly with too little illumination; differently-colored areas are again indistinguishable.
  • the object artwork also enables other information to be sleuthed, such as scale, provided the object is depicted in the same image frame as the skin condition.
  • the distance between the centers of Jackson's eyes on the US $20 banknote is 9 mm. If such a banknote is photographed next to a lesion, and the distance between Jackson's eyes spans 225 pixels, and the lesion spans 400 pixels, then the lesion is known to have a width of 16 mm. Dimensions of other features in the image can be similarly determined.
  • the pose of the camera relative to the skin can also be determined - based on apparent geometrical distortion of the object. That is, if the camera axis is not perpendicular to the skin, then perspective distortion will cause features depicted in some parts of the frame to be larger, or smaller, than would be the case with a perpendicular pose.
  • the angle from which the image was captured can be sleuthed, and a corrective counter-distortion can be applied.
  • the camera's optic function can also be considered in the analysis, to account for the expected apparent distortion of features displaced from the center of the image frame.
  • the circular seal of the US Federal Reserve System on the left side of a banknote, may be subtly distorted from round - even with a perpendicular camera pose - if the seal is not at the center of the image. Such distortion is expected, and the analysis takes such normal artifacts of perpendicular poses into account.
  • Another calibration token that can be placed on the skin for image capture is a coin.
  • a variety of different coins may be recognized by the central system - and from their known attributes, scale and pose determinations can be made - just as with the banknote arrangement described above. Also, many coins exhibit the specular reflection used by many cameras for automatic white balance.
  • Another approach to dealing with ambient light variability is to employ the smartphone's front-facing camera.
  • Smartphones are commonly equipped with two cameras - one on the front, facing the user, and one on the rear.
  • the latter is typically used for capturing skin imagery. But the former can be used to capture image data from which ambient lighting can be assessed.
  • the field of view of the front-facing camera can include a variety of subjects - making its automatic white balance determination more trustworthy than the rear-facing camera (whose field of view may be filled with skin).
  • an automatic white balance assessment is made using the front-facing camera, and resulting information is then used in AWB-processing of skin imagery captured by the rear-facing camera.
  • the light emitting diodes (LEDs) used for camera flashes have relatively consistent spectra among instances of a particular model (e.g., iPhone 5 cameras). Reference data about flash spectra for popular camera models can be compiled at the central system. Users are then instructed to capture the skin image in low ambient light conditions, with the camera flash activated. When the central system receives such imagery, it examines the header data to determine the camera model involved, and flash usage. The system then applies a color correction that corresponds to the flash spectrum for that model of camera.
  • flash is used in conjunction with ambient lighting for color correction.
  • two images are taken in quick succession - one including an LED flash, and one not.
  • Video mode can be used, but resolution is typically better in a still image capture mode.
  • Both images include the ambient light, but only one includes the
  • Still another technique for color compensation is by reference to measured norms of skin coloration. While skin comes in a variety of colors, these colors comprise a tiny fraction of the universe of possible colors. This is particularly true when skin color is represented in the CIELAB color space. This range is narrowed still further if the user's race is known, e.g., entered via the user interface of a smartphone app, or recalled from stored user profile data. (In smartphones equipped with front- and rear-facing cameras, the former can be used to capture a picture of the user - since the user typically operates the phone facing towards the screen.
  • Known techniques can assess the user's race (and gender) from facial imagery - avoiding the need for the user to enter this information. See, e.g., Lyons, et al, Automatic Classification of Single Facial Images, IEEE Trans, on Pattern Analysis and Machine
  • the race assessment can be performed by smartphone app software, so that the user's facial image is not sent from the phone.
  • a better indication of the user's normal skin color may be obtained by sampling away from the center, e.g., at the edges.
  • An average color based on samples taken from a variety of peripheral image locations, can be computed. (Samples should be checked to assure that a location does not correspond to clothing or other non-skin feature. Color consistency and/or segmentation techniques can be used.) This baseline skin color can then be checked against statistical color norms - for the user's race, if known.
  • data from a smartphone' s proximity detector can alternatively be used. Such detectors primarily rely on capacitive techniques and are presently of short range, e.g., 2 cm., but longer range sensors are under development.
  • bundle adjustment algorithms multiple images taken from different locations and/or directions are exploited to jointly produce estimates of the optical view parameters of the camera(s) and a 2D or 3D model of the scenes imaged. While bundle adjustment originated in the photogrammetry community, it has found much recent use in the computer science field, where is a fundamental component of shape from motion algorithms. Partial knowledge of the characteristics of the camera(s) can be used to improve the accuracy of the scene model. In the case of skin images captured with a smartphone, the multiple images may be made by passing the camera over the patch of skin.
  • Another scaling technique relies on known biometric norms.
  • the inter pupillary distance (the distance from the center of one eye pupil to the center of the other) is about 62mm.
  • a variety of other consistent biometric measurements are known (going back to the carpenter's "Rule of Thumb” of antiquity), or can be gathered from analysis of data.
  • Some are absolute measures e.g., the inter pupillary distance is about 62 mm
  • ratios e.g., the ratio of forearm length, to forearm plus hand length, is about 0.58.
  • Some such measures are tightly clustered, based on the user's gender and height.
  • Image classification techniques can be applied to user imagery to recognize pupils, a thumb, a fingernail, a forearm, a hand, etc. From known biometric measures, the size of a skin lesion can be inferred.
  • the pixel spacing between the depicted pupils directly correlates to the distance between the front-facing camera and the user's face. Subtracting this value from 12 inches yields the viewing distance between the smartphone and the user' s forearm. From this viewing distance, and information about the camera's optics, the size of features on the skin can be deduced.
  • the color of facial skin depicted in imagery captured by the front-facing camera can be used in assessing the color of skin depicted in imagery captured by the rear- facing camera.
  • the facial skin may be used as a reference skin color.
  • Skin recognition techniques can be applied to identify the eyes and nose, and from such information the portion of the imagery depicting cheeks and forehead can be determined. Skin facial color can be sampled from these locations.
  • eye color is a useful tool in establishing an expected skin color.
  • a grey iris is most commonly associated with people of Northern and Eastern European descent, for whom norms of skin coloration can be established.
  • Ethnic associations with other eye colors are also well known. (See, e.g., the Wikipedia article "Eye color.")
  • imagery of the subject skin condition - captured by the rear-facing camera - exhibits a skin color that is different than this reference color, such difference may be taken as a diagnostic indicia.
  • the reference facial skin color can be used in segmenting features from the skin imagery captured by the rear-facing camera.
  • the skin imaged by the rear-facing camera may be illuminated differently than the facial skin imaged by the front-facing camera.
  • the user may have oriented a fluorescent desk lamp towards their arm to provide more light.
  • such lighting changes the apparent color of the skin.
  • the skin imaged by the rear-facing camera may be within a shadow cast by the phone.
  • the evolution of a skin condition over time can be useful in its assessment. Images of a skin condition taken at different times can be shown in different manners to illustrate evolution of the condition.
  • the images are scaled and spatially aligned (i.e., registered), so that a consistently- sized and oriented frame of reference characterizes all of the images. This allows growth or other change of a lesion to be evident in the context of a generally unchanging background.
  • Images can be scaled and aligned using known techniques. Exemplary is by reference to SIFT or SURF features, in which robust feature key points that are common throughout images are identified, and the images are then warped (e.g., by an affine transform) and rotated so that these points become located at the same positions in each of the image frames. (One such arrangement is detailed in applicant's patent application 20120208592.)
  • the lesion can be masked (or flooded with a uniform color) so that the key point identification method does not identify key points from the lesion or its boundary. This reduces the key point count, and simplifies the later matching of common keypoints between the images.
  • Body hair can also be a source of many superfluous key points in the different image frames - key points that typically don't help, and may confound, the image registration process.
  • the images are desirably processed to remove hair before key points are determined.
  • Key points are then extracted from the imagery. Depending on the magnification of the images, these points may be associated with nevi, hair follicles, wrinkles, pores, pigmentation, textures, etc. If the imaging spectrum extends beyond the visible, then features from below the outermost layer of skin may be evident, and may also serve as key points. A key point matching search is next conducted to identify corresponding key points in the images.
  • One image is next selected as a reference. This may be, e.g., the most recent image.
  • the rotation and warping required to transform each of the other images to properly register with the reference image is determined.
  • These images are then transformed in accordance with such parameters so that their key points spatially align with corresponding key points in the reference image.
  • a set of transformed images results, i.e., the original reference image, and the rotated/warped counterparts to the other images.
  • each skin image would be related to the others by a simple rotation and affine transform. This is generally a useful approximation for all cases. However, due to the curvature of some skin surfaces, and the fact that skin may stretch, a more generalized transform may be employed to allow for such variations.)
  • One form by which the transformed images can be presented is as a stop-action movie.
  • the images are ordered by date, and rendered sequentially. Date metadata for each image may be visibly rendered in a corner of the image, so that the date progression is evident.
  • the sequence may progress automatically, under software control, or each image may be presented until user input (e.g., a tap on the screen) triggers the presentation to advance to the next image.
  • the software displays an image for an interval of time proportionate to the date- span until the next image. For example, if images #1-4 were captured on successive Mondays, and then two Mondays were missed before images #5-8 were captured (again on successive Mondays), then images #1-3 may be presented for one second each, and image #4 may be presented for three seconds, followed by images #5-7 presented for one second each. (Image #8 - the last image - may remain on the screen until the user takes a further action.)
  • a user interface control can be operated by the user to set the speed of rendering (e.g., the shortest interval that any image is displayed - such as one second in the foregoing example, or the total time interval over which the rendering should occur, etc.).
  • a different form by which the transformed image set may be viewed is as a transitioned presentation.
  • a video effects transition is employed to show information from two or more image frames simultaneously on the display screen.
  • image #1 (the oldest image) is displayed.
  • image #2 begins to appear - first as a faint ghosting effect (i.e., a low contrast overlay on image #1), and gradually becoming more definite (i.e., increasing contrast) until it is presented at full contrast.
  • image #3 starts to appear in like fashion.
  • the older images can fade out of view (e.g., by diminishing contrast) as newer images ghost-into view.
  • the progression can be under software, or user, control.
  • the renderings may employ the images from which hair was digitally removed (from which key points were extracted). Alternatively, the renderings may employ the images with hair undisturbed.
  • the rendering sequences can be accompanied by measurement data.
  • a textual or graphical overlay added to a corner of the presentation may indicate the width or area of the depicted lesion, e.g., an area of 12 mm 2 in the first image, 15 mm 2 in the second image, 21 mm 2 in the third image, etc.
  • the color or darkness of the lesion, or its boundary irregularity or its texture may be quantified and expressed to the user.
  • such information is not presented with each image in the series. Rather, at the end of the rendering, information is presented detailing a change in the lesion from the first frame to the last (e.g., the lesion has increased in area by 83% in 7 weeks).
  • Such statistics about the lesion, and its changes, can also be presented as a textual or graphical (e.g., with Cartesian graphs) report, e.g., for emailing to the user's physician.
  • the skin features from which the key points are extracted define a characteristic constellation of features, which permits this region of skin to be distinguished from others - a fingerprint of the skin region, so to speak - and by extension, a fingerprint of the user.
  • this characteristic fingerprint information allows the system to associate the image with the correct user.
  • This may be used as a privacy-preserving feature, once a characteristic constellation of skin features has been initially associated with a user.
  • This distinctive constellation of features can also serve as a biometric by which a person can be identified - less subject to spoofing than traditional biometrics, such as friction ridges on fingertips and iris pattern.
  • features surrounding an area of interest on the skin effectively serve as a network of anchor points by which other imagery can be scaled and oriented, and overlaid, in real time.
  • This permits an augmented reality-type functionality, in which a user views their skin with a smartphone, and a previous image of the skin is overlaid in registered alignment (e.g., ghosted), as an augmentation.
  • a user interface control allows the user to select a desired previous image from a collection of such images, which may be stored on the user device or elsewhere.
  • the phone As the user moves the phone towards or away from the skin - changing the size of the lesion depicted on the camera screen, the size of the overlaid
  • the knowledge base should enable detection of pathologies before they become evident or symptomatic. For example, a subtle change in skin condition may portend a mole's shift to melanoma. Development of a non-uniformity in the network of dermal capillaries may be a precursor to a cancerous growth. Signals revealed in skin imagery, which are too small to attract human attention, may be recognized - using machine analysis techniques - to be warning signals for soon-to-be emergent conditions. As imaging techniques advance, they provide more - and more useful - weak signals. As the knowledge base grows in size, the meanings of these weak signals become clearer.
  • the Fig. 2 data structure can be augmented by a further column (field) containing a unique identifier (UID) for each such patch of skin. All records in the data structure containing information about that patch are annotated by the same UID in this further column.
  • UID may be arbitrary, or it may be derived based on one or more elements of user-related information, such as a hash of one of the user's image file names, or based on the unique constellation of skin feature points.
  • the central system can analyze the longitudinal information to discern features (e.g., image derivatives) that correlate with later emergence of different conditions. For example, if the system finds a hundred users diagnosed with melanoma for whom - in earlier imagery - a network of capillaries developed under a mole that later become cancerous, and this network of capillaries is denser, by a factor of two- to three-times, than the density of capillaries in surrounding skin, then such correlation can be a meaningful signal. If a new user's imagery shows a similar density of capillaries developing under a mole, that user can be alerted to historical correlation of such capillary development with later emergence of melanoma. Such early warning can be key to successful treatment.
  • features e.g., image derivatives
  • the correlation is between a single signal (dense capillary development) and a cancerous consequence. Also important are combinations of signals (e.g., dense capillary development, coupled with die-off of hair in the mole region).
  • Known data mining techniques can analyze the knowledge base information to discover such foretelling signals.
  • Another way to enhance skin-patch and/or lesion registration is through teaching users of such an app to use and perfect the motion of their camera itself while gathering imagery of their skin.
  • the resultant motion of the skin regions manifested in the imagery itself often combined with the on-camera motion information data common to almost all smartphones, allows for further degrees of information to be utilized in precise millimeter and sub-millimeter scale matching of some given skin sample and the imagery of the same skin taken weeks, months or even years earlier.
  • Many detailed features change often drastically over such lengthy time periods, where the use of motion imagery and the resultant parallax data information can help to derive additional shape and perspective information, all of which assists in the generic task of stabilizing the viewing and interpretability of often quite dynamic skin conditions and lesions.
  • a user may be trying various treatments for acne for example, and they will want to be able to finger scroll between images taken when they were trying brand X, and images when they we trying brand Y.
  • CAD Computer Assisted Diagnosis
  • Canfield Imaging Systems which operates imaging centers in cities throughout the U.S. At these facilities, patients can obtain whole-body imagery, which is then passed to their physicians for review. Aspects of the Canfield technology are detailed, e.g., in patent documents 8,498,460, 8,218,862, 7,603,031, and 20090137908.
  • a basic aspect of an illustrative early melanoma screening room is simple: build a transparent phone booth (or cylinder) surrounded with cameras and synchronized lighting.
  • the booth may comprise lighting with, e.g. 16 to 32 LED spectral bands (some into the near-IR), and a dozen or two dozen RGB and/or black and white cameras.
  • Shaving, or an alcohol or other skin treatment may be employed in certain cases. People get naked or put on a bathing suit, get inside, and raise their arms - as in the TSA imaging booth. Five or fifteen seconds later they are done.
  • the computer churns for another 30 seconds conducting image analysis and comparison with reference data, and either gives a green light, or, perhaps in a non-alarming way and still "routine," a low threshold is set such that a patient is asked to go into a second room where a technician can focus in on "concern areas" using existing state-of-the-art data gathering methods on exact areas, including simple scrape biopsies. (The technician views results from the scan, with "guidance” from the software, in order to flag the patient, point out the areas of concern, and instigate the second-room screening.) Practicing clinicians also can be more or less involved in the steps. This is pap-smear, colonoscopy, cultural 101 kind of thinking... do it first when you are 25, then every 5 years, or whatever. Get it close to the pap-smear kind of test cost wise, the rooms themselves shouldn't run over $5 to $10K full manufacturing cost; no need to get too crazy on the hardware technology.
  • Fig. 5A shows a schematic sectional view looking down into a cylindrical booth (e.g., seven feet in height, and three feet in diameter, with an access door, not particularly shown).
  • a cylindrical booth e.g., seven feet in height, and three feet in diameter, with an access door, not particularly shown.
  • Arrayed around the booth are a plurality of light sources 52 and cameras 54.
  • the depicted horizontal ring array of light sources and cameras can repeat at vertical increments along the height of the booth, such as every 6 or 18 inches (or less than 6, or more than 18).
  • the lights and cameras can align with each other vertically (i.e., a vertical line through one light source passes through a series of other light sources in successive horizontal rows), or they may be staggered. Such a staggered arrangement is shown in Fig. 6A, in which successive rows of lights/cameras are offset by about 11 degrees from each other.
  • Fig. 6B shows another staggered arrangement, depicting an excerpt of the side wall, "unwrapped.”
  • successive horizontal rows of light sources 52 are offset relative to each other.
  • the light sources 52 are not centered between horizontally- neighboring cameras, but are offset.
  • the light sources needn't be interspersed with cameras, with the same number of each. Instead, there may be a greater or lesser number of light sources than cameras.
  • the light sources needn't be arrayed in the same horizontal alignment as cameras; they can be at different vertical elevations.
  • Light sources and cameras may also be positioned below the person, e.g., under a transparent floor.
  • the light sources are of the sort detailed in applications 20130308045 and
  • the cameras may be operated at a sufficiently high rate (e.g., 40 - 280 Hz) that the illumination appears white to human vision.
  • the cameras transmit their captured imagery to a computer that processes the images according to methods in the just-noted patent documents, to determine spectricity measurements (e.g., for each pixel or other region of the imagery).
  • imagery is then divided into patches (e.g., 1, 2, or 5 cm on a side) and compared against reference imagery, or applied to another form of classifier. All of the imagery can be processed in this fashion, or a human or expert system can identify patches of potential interest for analysis.
  • the patient can stand on a turntable that rotates in front of a lesser number of cameras and light sources, while frames of imagery are successively captured. Thus, a full "booth” is not required.
  • Such arrangement also captures imagery at a range of different camera-viewing and light-illuminating angles - revealing features that may not be evident in a static-pose capture of imagery.
  • a turntable also allows hyperspectral line sensors to be employed in the cameras, with 2D imagery produced from successive lines as the turntable turns. Such line sensors are available from IMEC International of Belgium, and capture 100 spectral bands in the 600-1000 nm range. Of course, 2D sensors can be used as well - including hyperspectral sensors.
  • One vendor of hyperspectral 2D sensors sometimes termed imaging spectrographs, is Spectral Imaging Ltd. of Finland.
  • 3D reconstruction techniques are applied to the captured imagery to build a digital body map.
  • SLAM 3D reconstruction techniques
  • images of the entire body surface can be recorded, allowing an examining physician to virtually fly, Google-Earth-like, over the patient' s modeled body surface, pausing at points of interest. If historical images are available, the physician can examine the time-lapse view of changes at each location, as desired.
  • the patient's body map can be morphed (stretched and tucked and squeezed, etc.) to a standardized 3D body shape/pose (of which there may be a dozen or more) to aid in automated processing and cataloging of the noted features.
  • the user software includes options that can be user-selected so that the system does not present certain types of images, e.g., of genitalia, of morbid conditions, of surgical procedures, etc. (Tags for such imagery can be maintained in the knowledge base, so that images may be filtered on this basis.)
  • the user interface can also allow the user explore imagery in the database. For example, if the system presents a reference image depicting a leg lesion that is similar to a lesion on the user's leg, the user may choose to view follow-on images of that same reference lesion, taken at later dates - showing its progression over time. Similarly, if the reference lesion was found on the leg of a prior user who also submitted imagery showing a rash on her arm, the current user may navigate from the original leg lesion reference image to view the reference image showing the prior user's arm rash.
  • Image navigation may also be based on image attribute, as judged by one or more parameters.
  • a simple parameter is color.
  • one derivative that may be computed for some or all of the images in the knowledge base is the average color of a lesion appearing at the middle of the image (or the color of the pixel at the middle of the image - if a generalized skin condition such as a rash is depicted there).
  • the user can query the database by defining such a color (e.g., by pointing to a lesion in user-submitted imagery, or by a color-picker interface such as is employed in Photoshop software), and the software then presents the image in the knowledge base that is closest in this metric.
  • the user may operate a control to continue such exploration - at each step being presented an image that is closest in this attribute to the one before it (but not previously displayed).
  • the user interface can permit user navigation of reference images based on similarity in lesion size, shape, texture, etc.
  • Hair on skin can be a useful diagnostic criterion.
  • melanoma is aggressively negative for hair; hair is rarely seen from such growths. So hair depictions should be included in the knowledge base imagery.
  • the user interface can allow the user to tap at one or more locations within a captured skin image, to identify portions about which the user is curious or concerned. This information is conveyed to the central system - avoiding ambiguity about what feature(s) in the image should be the focus of system processing.
  • the user interface can allow the user to enter annotations about that feature (e.g., "I think I first noticed this when on my Las Vegas vacation, around May 20, 2013").
  • the central system may return the image with one or more graphical indicia to signal what it has discovered. For example, it may add a colored border to a depicted lesion (e.g., in red - indicating attention is suggested, or in green), or cause an area of the screen to glow or strobe.
  • a depicted lesion e.g., in red - indicating attention is suggested, or in green
  • this image is presented to the user, and the user touches or otherwise selects the graphical indicia, information linked to that feature is presented, detailing the system's associated findings.
  • a series of such images - each with system-added graphical indicia (e.g., colored borders) - may be rendered to illustrate a time-lapse evolution of a skin condition, as detailed earlier.
  • illumination sources e.g., LEDs
  • N different spectral illumination sources combine with M different spectral detectors (e.g., the three different color filters overlaying a smartphone photodetector array) to yield up to N*M different sets of image data. From this richness of different image data, a rich set of different features can be discerned.
  • illumination at different spectral wavelengths is provided by illumination from a smartphone screen.
  • One such method captures imagery using a smartphone' s front-facing camera (i.e., the camera on the same side of the phone as the touchscreen), instead of the usual rear-facing camera.
  • the field of view captured by the front-facing camera is then illuminated - at least in part - by light from the smartphone screen.
  • This screen is software-controlled to present a sequence of different illumination patterns or colors (“rainbow mode"), during which different frames of imagery are captured.
  • Smartphone and other screens commonly emit “red,” “green” and “blue” light - each with a particular spectral profile.
  • This profile typically varies from one type of smartphone to another - due to different display technologies, and sometimes varies among smartphones of the same type due to process variations.
  • these spectral profiles never exactly match the red-, green- and blue-Bayer sensor pixel spectral profiles - giving rise to the multiplicative effect noted above.
  • OLED displays are coming into widespread use (e.g., the Samsung Galaxy SIII, S4 and S5) and offer increased brightness and wider gamut, compared with previous technologies.
  • Autostereoscopic displays (commonly including parallax barriers) can also be used, and can create structured illumination.
  • One illustrative embodiment uses rainbow mode in capturing and processing frames of image data from a user's face.
  • the motion and/or pose of a smartphone is sensed, and used to switch between data collection and data presentation modes.
  • the motion and pose of a smartphone can be discerned by reference to data from the phone's onboard accelerometers, magnetometers and gyroscopes - each commonly 3D. (Motion can also be assessed by reference to apparent movement of imagery captured by the phone camera.)
  • the user waves the phone around the head (and optionally scalp), capturing frames of imagery with the front-facing camera, from different vantage points.
  • the phone screen is displaying a sequence of different illumination patterns/colors.
  • the screen illumination can be of various types. At some update rates, human persistence of vision causes the illumination to seem uniform, e.g., all white. At slower rates, different colors or patterns flash across the screen. A simple arrangement sequentially displays screens of all-red, all-green, all-blue, in cyclical fashion. In a variant, the phone's "torch” (i.e., illumination flash) is operated in a fourth phase of the sequence, giving four different illumination states.
  • solid screen colors are still employed, but this time with combinations of the red/green/blue primaries (yielding what may be termed cyan, magenta and yellow).
  • the transitions between colors are abrupt. For example, a red screen can be maintained for a sixth of a second, and then switch to blue for the next sixth of a second, etc.
  • the transitions are blended.
  • a displayed solid color may be updated thirty times a second. At a first frame, red is presented. At the second frame, 20% of the red pixels are changed to green. And at the third frame, 20% more of the red pixels are changed to green. A seemingly-continuous smear of colors results (but is actually 15 different colors. Twice a second the display is all-red. Ditto for all-blue and all-green.
  • Capture of image frames by the camera is synchronized to the different frames of illumination.
  • the camera may capture six frames of imagery per second (i.e., two with all-red illumination, two with all-blue illumination, and two with all-green illumination).
  • Skin topology features are best revealed by illuminating the skin obliquely, at various different angles. This can be done by operating the screen to present illumination from different parts thereof, at different times. The rest of the screen can be kept dark (black).
  • Fig. 7 One such arrangement is shown in Fig. 7.
  • the display screen at the top of the figure is all-dark (black). After a fixed interval a colored band appears along the left edge of the screen. A further interval later it shifts one band- width to the right. In similar fashion the color band marches across the screen. (although six discrete steps are illustrated, a greater or lesser can be used.)
  • the sequence of colors (e.g., of the Fig. 7 bands) is sequential, e.g., red, green, blue (or red, green, blue, cyan, magenta, yellow, etc.).
  • the sequence of colors is random.
  • the direction of the band's movement is changed from one cycle to the next. For example, after the red band of Fig. 7 has marched to the right of the screen in one cycle, the next cycle may have a band of color march from the top to the bottom. Or from the right to the left. Or from the bottom to top.
  • the sequence can repeat, or random directions and/or colors can be used.
  • Fig. 8 shows a variant in which two bands - of different colors - move across the screen.
  • the color of the horizontal band in different cycles can follow a repeating pattern, or it can be selected randomly. Similarly with the direction of the horizontal band's movement (top-to-bottom, or bottom- to-top). Likewise with the vertical band.
  • Fig. 9 shows yet another variant, in which blocks of different colors appear at different positions on the screen.
  • the sequence of colors, and positions of the blocks can follow a repeating pattern, or either/both can proceed in a random sequence.
  • the device can be an iPhone 5, which has a display that is four inches in diagonal measurement, and has a 1136 x 640 pixel resolution, i.e., 326 points per inch.)
  • Fig. 10 further considers certain aspects of the illumination geometry.
  • a color band is displayed at one end of the smartphone screen. Light from this band illuminates a location 102 on the skin at an angle of 21 degrees (relative to a tangent 104 to the skin surface).
  • the band has moved to near the middle of the screen, as depicted in the middle of the figure.
  • light from this band (which may now be of a different color, or not) illuminates the skin location 102 at an angle of 40 degrees.
  • the band has moved to hear the opposite end of the screen, as depicted in the bottom of Fig. 10.
  • light from this band illuminates the skin location at an angle of 103 degrees.
  • the phone captures a frame of imagery under each of these different illumination conditions, using a camera 106.
  • Fig. 10 arrangement collects imagery of skin location 102 as illuminated from a variety of different angles. If the color of the band changes as it marches across the display, such arrangement collects imagery of the skin location with illumination that is diverse both in angle and spectrum.
  • imagery is captured having diversity in illumination angle, illumination spectrum, and viewpoint.
  • profile information can aid in classifying certain skin conditions, e.g., dryness, scaliness, age wrinkles, sunburn, etc.
  • a patch of skin having been illuminated with all these spectral combinations and from these various spatial directions can easily give rise to dozens or hundreds of bundled data attributes at millimeter by millimeter scales represented by the pixels of the camera.
  • Spatial derivatives of these data e.g.., how does the "red light from the left pixel datum" change from this pixel to ten pixels over, innately carry discrimination information between a new red rash that might be lumpy versus the less lumpy nature of a hickey, to use a deliberately out-there example.
  • boyfriends and girlfriends can quickly check for any youthful infidelities as but one tiny application for rainbow-derived data captures.
  • Sunburn analysis is another application where spatial- spectral rainbow diversity can enrich the identification and discrimination power of captured imagery.
  • audio (and other physiologic data) is collected and used in manners like the skin imagery herein.
  • one particular embodiment employs regression analysis on a set of audio data to characterize false conclusions that should not be drawn. (Culling the false helps in identifying the truth.) Another compares extracted features against templates of "normal" features, to identify anomalous signals that should be reviewed by a qualified physician.
  • Patent publications 2002085724, 2005078533, 2008046276, 2008062389, 2010045210, 2011009759, 2011021939, 2011096820, 2011076328, 2012043018, 2012090303, 2013016584, 2013061484, 2014012149, WO0209396, and W013156999 detail a variety of technologies useful in collecting and processing physiological sound information, which are suitable for use with the arrangements described herein.
  • Such a pickup can be provided in a separate unit - coupled to a portable device (e.g., smartphone) by wire (e.g., employing the headphone/mic jack) or wirelessly (e.g., employing Bluetooth).
  • a portable device e.g., smartphone
  • wire e.g., employing the headphone/mic jack
  • wirelessly e.g., employing Bluetooth
  • such a pickup can form part of the portable device, either permanently, or by an accessory unit that is removably attached to the device body.
  • An example of the latter is a downwardly- opening funnel-like member (e.g., made of plastic) that friction-fits over the lower inch or half-inch of a smartphone body, channeling sounds from the wide end of the funnel up to the microphone(s) normally used for telephone communication.
  • a downwardly- opening funnel-like member e.g., made of plastic
  • audio sensing can be done by worn microphones.
  • one or more microphones are provided in a band (e.g., a wrist- or waist-band) worn by a user (or worn by a clinician, for probing a user).
  • acoustic sensors are integrated in clothing or other garments. By positioning microphones at different locations on the body, the spatial origins of different sounds can be better determined, aiding their diagnostic significance.
  • head-worn microphones can sometimes be employed (e.g., Google Glass-like arrangements).
  • one or more other microphones can be employed to sense the ambient audio, so it can be removed from the sensed physiologic audio, using known noise cancellation techniques.
  • Smartphones are increasingly provided with multiple microphones; one or more of these can be used to enhance the sensed physiological signals.
  • a variety of diagnostically relevant acoustic signals can be sensed. These include heart sounds and other cardiovascular sounds (including murmurs, bruits, and other blood flow noises), lung and other respiratory sounds (including crackles, rales, rhonchi, wheezes, coughs, snoring and other air flow noises), bowel and digestive sounds, joint noises (e.g., pops and creaks), as well as speech and other vocalizations.
  • cardiovascular sounds including murmurs, bruits, and other blood flow noises
  • lung and other respiratory sounds including crackles, rales, rhonchi, wheezes, coughs, snoring and other air flow noises
  • bowel and digestive sounds including joint noises (e.g., pops and creaks), as well as speech and other vocalizations.
  • One processing technique characterizes and strips-out positioning sounds, e.g., when a microphone is moved and rubbed against a person's skin, before settling at a final position for data collection.
  • a classifier can be trained to recognize such positioning sounds, so that they are not used in diagnostic processing.
  • noise-cancellation e.g., as noted above, and in certain of the cited patent documents.
  • a suitable wavelet-based denoising arrangement is detailed in Messer, et al, Optimal Wavelet Denoising for Phonocardiograms, Microelectronics Journal 32.12, pp. 931-941 (2001).
  • Spectral filtering can also be employed, when same is desired based on measurement context.
  • Prediction error can be useful in detecting abnormalities in heart beats and also in isolating noise-like signals, e.g., a murmur, in the presence of a stronger heart beat signal.
  • Transient signals are not easily predictable either. So, in analyzing a wet cough signal, there may be a transient component which will be present in dry cough but in wet coughs there will also be a predictable component which will be absent in the dry coughs (based on an excitation model for coughs).
  • Spectrograms can be analyzed for high frequency vs low frequency signals - noise-like signals have higher frequency content - air flow sounds, murmurs etc.
  • Pops and crackles are transient signals and they can be detected/analyzed using short frame audio analysis
  • One example of a fingerprinting based diagnosis is to collect "healthy" audio data (when the patient is healthy) and then analyze any deviation from the signatures or fingerprints of this healthy data to determine pathologies in future diagnostic examinations.
  • Such processing can provide a variety of "features" that can be compared with reference data in assessing whether a signal is normal or anomalous (and, if the latter, used to help identify what the anomaly is - or is not).
  • One approach is to use an auto-regressive model to parameterize the sensed sounds. This is the approach employed, e.g., by Harma et al in Time- Varying autoregressive Modeling of Audio and Speech Signals, Proc. of the EUSIPCO, pp. 2037-2040, 2000.
  • Coughs, snores, Sons, and other isolatable or cyclically recurring sounds can be parameterized, in one respect, by identifying the time interval over which a threshold (e.g., 80%) portion of the spectral energy is expressed, and the smallest frequency bandwidth (characterized by low and high frequencies) within which a threshold (e.g., 90%) portion of the spectral energy is expressed. (These low and high frequency bounds can also independently serve as useful features.)
  • a threshold e.g., 80%
  • M1-M6 Feature box
  • Range range of signal energy (Q3-Q1).
  • Concentratio span encompassing loudest 50% of time envelope
  • Concentratio frequency span encompassing loudest 50% of
  • Time of Peak sec Time of single loudest spectrogram cell Time of Peak sec Time of single loudest spectrogram cell.
  • M21-M24 Amplitude and frequency modulation (variation of amplitude and frequency over time)
  • the widths may be a only a few bins
  • M25-M28 Fine features of harmonic structure, shifts in periodicity, direction of frequency change, rate of change in frequency
  • the widths may be a only a few bins wide
  • peaks means narrowband/tonal harmonics.
  • M27 ⁇ 0 frequency
  • Cells are ranked low to high and the cell at the
  • the reference data may have been collected solely from a single person over time (e.g., a longitudinal record of earlier- sensed physiologic sounds from that person), or from a collection of individuals - commonly a grouping that is demographically similar (e.g., 40-50 year old healthy males weighing between 170 and 190 pounds).
  • One particular example concerns detection of bruits in a major artery. This can be predictive of a stroke.
  • the user may periodically place a phone or other audio sensor over the femoral (or carotid) artery. Five or ten seconds of collected sounds can be converted into features, as described above, and these features can be compared with features derived from known Sons sounds. If the comparison indicates similarity beyond a threshold degree (e.g., a feature distance below a threshold value), the user can be advised to have their arteries checked for occlusion.
  • a threshold degree e.g., a feature distance below a threshold value
  • Patent publications 20070265508, 20110021939, 20110196820 and 20130261484 detail a variety of other ways that biometric signals can be processed, e.g., for comparison.
  • speech and other vocalizations also have diagnostic value.
  • Pitch, timbre, rhythm, pace, and volume are some of the vocalization attributes that can be monitored. Changes from historical norms are sometimes symptoms or precursors of different conditions, such as depression, stroke, alcohol poisoning, respiratory illness, etc.
  • Depression for example, is often accompanied by slower and quieter speech, with reduced variation in pitch. Respiratory illness can be discerned from lower-pitched speech, with rougher/coarser timbre (e.g., due to swollen vocal cords).
  • Coughs may be characterized by characteristics such as frequency and type (e.g., a dry cough - staccato in nature, with a sharp onset, short duration and dominant high frequency components; and a wet cough - commonly the opposite).
  • a user's cough may be matched to prototypical coughs (the user's own, or others) on a "fingerprint" basis, such as employing a collection of the features detailed in Table 1, above.
  • a user's current cough may be matched to a previous episode of user coughing, on a prior date. Recalling other physiologic information from around that prior date may presage upcoming symptoms, e.g., elevated body temperature, runny nose, etc. Finer classification of coughs can be achieved with a sufficiently large collection of reference data.
  • a wet cough originating from irritation of the upper airway exhibits one set of features, whereas a cough originating from the lower airway (e.g., pneumonia in the lungs) exhibits a different set of features.
  • a distance measure can be employed to assess whether the set of features characterizing a user's cough are closer to the former or the latter; the relative distances provide a confidence metric. More sophisticated classifier arrangements can also be employed.
  • a database is collected, consisting of appropriate audio captures, together with tag data indicating presence of the specific types of coughs to be identified.
  • the audio is then processed to provide a rich set of features, which form the input to a machine learning paradigm (e.g. a Support Vector Machine (SVM), or an Artificial Neural Network (ANN).
  • SVM Support Vector Machine
  • ANN Artificial Neural Network
  • the database should include metadata to indicate gender, age, height, weight, etc. This metadata is then provided as additional features to be learned.
  • the learning process automatically takes these variables into account (to the extent to which the database is varied enough to span this metadata space).
  • ANN a single network output may be provided for each cough type to be recognized; a specific cough type may then be identified by choosing the cough type with the strongest response over a predetermined threshold (assuming exclusivity among cough types).
  • SVM coughs may be classified into many (>2) classes, based on the design of several binary classifiers. Two popular methods are "one vs. rest” and "one vs. one.”
  • a user's portable device can monitor sensed sounds (and other sensed physiologic data) for such clues (e.g., comparing current sound signals with historical data), and alert the user to maladies - both present and upcoming - that the user may not recognize.
  • Collections of reference audio data are not yet as readily available as, e.g., reference collections of mole imagery. However, such reference information can be crowd- sourced in the same manners as for imagery. Since audio information is straight-forward to collect in continuous fashion, the smartphone (or other device) of each user can serve as a collection agent, and forward large amounts of audio data (or features derived from the audio data) to a cloud repository in a relatively short time.
  • the physician's medical records are not employed. Instead, a user may enter data in a personal life-log, recounting a visit to the doctor, together with the doctor's diagnosis. Such information is then associated with audio that is life-logged from the user - before and/or after the doctor visit.
  • a small corpus of training audio, and associated diagnostic conclusions, are also available from medical schools, where they are used in training new physicians.
  • Scanadu has publicized its Scout and Scanaflo offerings, which can sense temperature, blood pressure, heart rate, oximetry, ECG, heart rate variability, stress, and urine chemistry - many using optical techniques.
  • WO2013116316, WO2013116253, and WO2014025415 detail certain of the Scanadu
  • Azoi, Inc. similarly has publicized its upcoming Wello product, in which sensors are integrated in a smartphone case, and communicate by Bluetooth to a health tracker app on the phone.
  • the app logs heart rate, blood pressure, blood oxygen, respiration, heart rate variability (as an indicator of stress), ECG, temperature, and lung function (with an accessory spirometer).
  • Automated cough detection and rudimentary signal analysis is the subject of Larson, et al, Accurate and Privacy Preserving Cough Sensing Using a Low-Cost Microphone, Proc. of the 13 th Int'l Conf on Ubiquitous Computing, ACM, 2011, and Birring, et al, The Leicester Cough Monitor: Preliminary Validation of an Automated Cough Detection System in Chronic Cough, European Respiratory Journal 31.5, pp. 1013-1018 (2008).
  • Many embodiments of the present technology advantageously consider multiple physiological signals together, for their diagnostic relevance (e.g., an ensemble of plural signals that co-occur in a relevant manner) or for signal processing purposes.
  • the signals from different repeating waveform periods can be averaged, combined, correlated, compared, filtered or windowed.
  • the periodicity can be inferred from, e.g., the audio itself, it can sometimes be independently determined.
  • the periodicity can be determined by reference to electric signals sensed from the heart (e.g., the QRS complex, or the T wave of an EKG signal).
  • timing derived from electric signals can be employed in processing acoustic signals.
  • the body can be stimulated at one location with an audio signal or pressure waveform, and the signal can be sensed at another location, to discern information about the intervening transmission medium. Fluid behind the tympanic membrane can be sensed in this fashion. Dehydration can also be so-indicated, based on the degree to which the skin is stretched or loose. Enlargement of the liver and constipation can also be discerned in this way, by detection of a solid (dense) mass under the skin.
  • Percussion analysis is used by physicians in clinical diagnosis for the abdomen and thorax.
  • the resonance properties of the acoustic waveform resulting from the acoustic stimulation can be examined to classify the sounds as normal, hyper-resonant, impaired resonant or dull.
  • Formant analysis of the captured waveforms yields information about acoustic resonance.
  • Information about associated symptoms e.g., pain
  • a smartphone may serve as a rudimentary echocardiogram device - stimulating a portion of the body with ultrasonic audio, sensing the phase (and amplitude) of the returned signals, and presenting resultant information on the display screen.
  • a variety of at home testing/monitoring can be conducted, including detection of certain dilated aorta conditions.
  • hidden Markov models In classifying sensed physiologic data, hidden Markov models, artificial neural networks, and deep neural networks can be employed, borrowing techniques known from the field of pattern recognition.
  • Hidden Markov models are also known in analysis of animal vocalizations; see Ren et al, Framework for Bioacoustic Vocalization Analysis Using Hidden Markov Models, Algorithms 2, No. 4, pp. 1410-1428, 2009.
  • Classification trees, support vector machines, and other discriminatory classification techniques can also be employed.
  • Such classifiers use feature data from physiological acoustic data and known diagnoses as training sets.
  • the ensemble of parameters (or features) outlined above can also be used in unsupervised learning methods to learn complex, non-linear models of many-dimensional underlying data.
  • Examples of such techniques include deep belief networks and sparse coding. (See, e.g., Raina, et al, Large-Scale Deep Unsupervised Learning Using Graphics Processors, Int'l Conf. on Machine Learning, Vol. 9, 2009.) These techniques are suitable for high-dimensional input data and can enable inference of latent variables or conditions.
  • Such deep learning approaches can unearth new patterns or diagnostic tools using large numbers (even millions) of collected samples of various physiological acoustic data.
  • a valuable input to these techniques is the change of physiological acoustic data over time. Such data can be obtained by sensing the physiological phenomenon (e.g., heartbeats, coughs, murmurs, etc.) at different times over multiple days (or longer intervals).
  • acoustic features can be combined with other available diagnostic information (pulse rate, temperature, blood pressure, etc.) and be provided as input to automated learning and classification methods (both supervised and unsupervised).
  • a health app employs a wearable network of sensors to capture and log a history of physiologic signals, and refer related information to one or more remote processors (e.g., "the cloud") for large scale systemic analysis.
  • Many existing sensors can be employed in such an arrangement.
  • One is a wrist- worn activity tracker, such as the Fitbit Force, Basis, Larklife, Jawbone UP, and Nike Fuelband products.
  • various sensors in these devices sense heart rate, temperature, perspiration (commonly based on skin conductivity), and movement (e.g., based on
  • accelerometer magnetometer
  • gyroscope sensor data From these data, others can be derived, including calorie consumption and sleep stage.
  • Another type of sensor is a belt or strap, worn across the chest, waist or belly (typically horizontally, but alternatively vertically or diagonally), which can monitor these same
  • a band on the upper arm (e.g., across the bicep) or leg/thigh can also be employed, as can sensors deployed around the neck (e.g., in a necklace form) or finger (e.g., in ring form). Many such sensors can be concealed under clothing
  • Blood pressure, blood chemistry, skin imagery data, EKG, EEG, etc. can also be sensed. As noted, some of the sensors can be integrated into worn clothing.
  • Some sensors can be responsive - in part - to stimulus introduced into the body, such as a small electrical current, audio, vibration, etc.
  • stimulus introduced into the body such as a small electrical current, audio, vibration, etc.
  • electrical conductivity between two points on the body depends on the amount of intervening fat, muscle and water, as well as the skin contact resistance (which varies with perspiration).
  • Some systems for measuring body composition involve a user standing on two electrical pads, to sense electrical resistance; less resistance indicates less body fat, and a lower body fat percentage.
  • Audio, vibration, alternating current, and radio beam-forming arrays can used - either of emitters (e.g., piezoelectric actuators) or receptors (e.g., MEMS microphones), and employed in conjunction with one or more complementary receptors/emitters on another part of the body, to probe and localize characteristics (e.g., density, electrical conductivity, etc.) of the intermediate body mass, using phased array/synthetic aperture techniques known from other disciplines.
  • ID arrays can be used, or 2D; the sensor spacing can be spatially regular or stochastic.
  • a weigh scale may be built into the floor in front of a bathroom sink, and a camera may be positioned to view the user's face when the user looks into a bathroom mirror.
  • Still other sensors may be applied to the body, e.g., adhesively or otherwise, for short intervals, as circumstances dictate. These include EEG and EKG electrodes, piezoelectric emitters/receptors, etc. Such localized sensors may be employed to track conditions at sites of particular concern, e.g., bruising or other wound, cancer site, etc. In some instances, sensors may be implanted.
  • All such sensing apparatus are desirably wirelessly linked, e.g., to convey data to a user's smartphone, to each other, or to a monitoring service.
  • the logged parameters - or derivative information based thereon - are eventually sent to the cloud.
  • the remote service monitors this data - noting and establishing time-of-day and day-of-week baselines, for different activity scenarios (e.g., sleep, office work, walking, strenuous exercising, etc., as classified based on characteristic collections of sensor data).
  • These baselines can also be associated with different geographic locations, e.g., as determined by GPS or WiFi, or otherwise.
  • Such a monitoring service may report to the user whenever the sensed data significantly deviates from expected norms. If a person's REM sleep pulse is normally between 56 and 60 beats per second, and one night there is an episode in which the pulse varies from this range by a threshold amount (e.g., more than 5%, 15%, 30%, or 75%), then a message may be dispatched to the user (e.g., by email, text, or otherwise) noting the incident. Possible causes for the disturbance may also be communicated to the user.
  • a threshold amount e.g., more than 5%, 15%, 30%, or 75%
  • causation hypotheses can be pro forma - based on textbook understandings of the noted phenomenon (e.g., caffeine before bed) discerned from stored rule data, or they can be tailored to the user - such as taking into account other user- or ambient- sensor information that might be correlated (e.g., irregular respiration, suggesting sleep apnea or the like; or an unusually warm room - as indicated by temperature data logged by a smartphone sensor as contrasted with historical norms - leading to increased blood flow for convective body cooling). Such information is also logged in a historical data store, and may also be sent for e-charting to the user's physician.
  • other user- or ambient- sensor information e.g., irregular respiration, suggesting sleep apnea or the like; or an unusually warm room - as indicated by temperature data logged by a smartphone sensor as contrasted with historical norms - leading to increased blood flow for convective body cooling.
  • Such information is also logged in a historical data store,
  • Part of a health and wellness protocol may involve the user speaking a particular phrase every day to an associated microphone sensor (e.g., "Good morning fitband"), for collection of baseline voice information.
  • an interactive dialog may ensue, with the system (e.g., the user's smartphone) consulting the user's available biometric signals and prompting - if appropriate - "You don't sound well” (in displayed or spoken text) and asking some follow-up questions, to help determine whether the user's malady is something that needs medical attention.
  • the system e.g., the user's smartphone
  • the system may consult stored knowledge base information in advising whether the user should consult medical personnel.
  • Another application of the present technology is in sleep analysis.
  • Commercial sleep studies often involve instrumenting the patient with several different sensors, such as a band around the chest to sense respiratory effort; an 02 sensor on a finger; a sensor under the nose to detect smoothness of nasal flow; an accelerometer of the like to sense the vibration that commonly accompanies snoring, etc. All such information is sent to a data logging device. The collected information is eventually sent to a professional for analysis.
  • vibration may be sensed by positioning the phone on the bed.
  • Vibrations from the snoring will couple from the user to the bed, and then to the phone.
  • stomach and bowel sounds e.g., are those the little bowel tones associated with healthy peristalsis, or high tinkling sounds that may signal an obstruction?.
  • sounds accompanying urine flow Still another involves the sounds of blood turbulence when blood starts flowing following release in pressure from a blood pressure measurement cuff. Etc., etc. Again, lay humans typically don't discern much meaningful data from such sounds and their variations; many physicians don't do much better. But as collections of such sounds grow, and are complemented by ground truth interpretations by physicians, meaning emerges. The significant clues may not be evident in the raw audio; only by processing (e.g., by computing one or more of the features detailed in Table 1) may telling signals become apparent. But as the volume of reference data goes up, the meaning that can be derived goes up commensurately.
  • the system can advise a user that her sleeping respiration rate of 25 breaths per minute is experienced by persons of similar demographic profile only 1% of the time, and may merit professional attention.
  • each user's sensed information is desirably sent for storage in a cloud database service, which may aggregate data from thousands, or millions, of people.
  • a cloud database service which may aggregate data from thousands, or millions, of people.
  • fitness tracking watches sometimes termed “activity loggers”
  • social network components e.g., posting Facebook reports of the GPS route that users run in their daily jogs, with speed, distance traveled, calories, and heart rate, etc.
  • Default privacy settings can anonymize the uploaded data, but users may elect different settings - including to share selected categories of skin images, biometric information, activity status updates, etc., with network friends, without anonymization.
  • the sensed data is associated with location information - both the location on the body from which it was collected, and also the geolocation at which the information was sensed (e.g., latitude and longitude). Data can then be recalled and analyzed, filtered, presented, etc., based on such location information. For example, a user can compare an average of resting pulse measurements taken at work, with such an average taken at home. Sensed information can also be presented in map form. For example, a user can query archived data to obtain a map display that identifies locations at which their resting pulse exceeded 80. (A suitable map display, detailing where certain data was sensed by a user, is illustrated in applicant's published patent application 20130324161.) Again, such information and maps can be shared via social networking services.
  • Historical user data serves as a statistical chronology of the user's aging process.
  • the user's physiologic signals can be compared to those of a relevant cohort (e.g., similarly-aged people of same gender, weight, location, etc.) to reflect whether the user is aging more or less rapidly than the norm. Relative aging may be judged by relative physiological condition.
  • aging is accompanied by phenomena including higher body weight, lower body height, higher BMI, lower lean body mass (muscle and bone mass), higher blood pressure (both systolic and diastolic), lower maximum oxygen use (V02 max - Volume Maximum Oxygen consumption) under exertion (e.g., on a treadmill stress exam), decreased visual acuity (e.g., presbyopia - a lessening ability to focus on close objects), decreased hearing acuity, decreased skin tightness, etc.
  • Such phenomena can be sensed and used to assess physiological age, as contrasted with chronological age, in known fashion.
  • variance indicates the user is either physiologically younger, or older, than peers. This variance can be tracked over time. If a user is statistically judged to have a physiological age of 50 when actually 40, and later have a physiological age of 52 when actually 44, then the user is physiologically older than peers, but trending in a more healthful direction.
  • a user may interact with a user interface of the system to propose certain changes (e.g., in diet, exercise, or other lifestyle) that might be made to avoid certain undesirable predicted outcomes, and the system can predict their respective effects.
  • the audio and/or other physiologic signal sensing described herein is on-going even when the screen of the device is dark and the device is otherwise in a "sleep" mode.
  • the data collected in this mode is not streamed to the cloud, but rather is cached in a memory until the phone is next awakened.
  • the processing of the captured data to produce derivative data may be queued, e.g., waiting until the device's primary processor is again available.
  • data collection aspects of a health app may be on-going, 24 hours a day, 7 days a week.
  • a person's consultation with a physician may not be prompted by use of the present technology; rather, a consultation with a physician may prompt use of the present technology.
  • a patient may visit a physician prompted by an onset of wheezing in their respiration.
  • the physician may detect a murmur (which may be associated with the wheezing).
  • the physician may instruct the patient to capture, twice-daily, sounds of the murmur by positioning the phone in a certain location on the chest.
  • Such in-home data collection can then inform judgments by the physician about further care.
  • a physician who diagnoses possible depression in a patient can use voice data from at-home data collection to determine whether the condition is trending better or worse. (The physician needn't review the actual recorded voice. Parameters expressing volume, pitch, and variations in same, can be derived from the patient's sensed voice, and succinctly reported for physician review.)
  • One aspect of the present technology involves collecting physiologic information in a data structure, where the collected information corresponds to physiologic sensor data gathered by plural non-professional users.
  • the data structure is also collected professional evaluation information corresponding to at least some of the physiologic information.
  • query information is received, corresponding to physiologic sensor data gathered by a non-professional user.
  • the data structure is consulted in determining result information, and at least some of this result information is communicated to the non-professional user. This result information depends on correlation between the query information, the collected physiology information, and the professional evaluation information.
  • Another arrangement employs a portable device that is moved to plural different viewpoint positions relative to a skin location.
  • the skin location is illuminated with light of a first spectrum from the portable device. While so- illuminated, imagery of the skin location is captured using a camera in the portable device.
  • the skin location is illuminated with light of a second spectrum from the portable device. Again, while so-illuminated, imagery of the skin location is captured using the portable device.
  • the skin location is imaged by the portable device from plural different viewpoint positions and with plural different illumination spectra. The captured imagery may then be processed and, based on such processing, the user may be advised whether to seek a professional evaluation of the skin location.
  • the skin location is illuminated with light of a first spectrum from a first region of a display of the portable device, while imagery of the skin location is captured.
  • the skin location is illuminated with light of a second spectrum from a second region of the portable device display.
  • imagery of the skin location is captured while illuminated with this second spectrum of light.
  • Still another related method involves presenting a first illumination from a portable device display screen to a subject at a first time, and capturing a first image of the subject when it is so-illuminated.
  • second illumination is presented from the display screen to the subject at a second time, and a second image of the subject, so-illuminated, is captured.
  • the illumination does not comprise a viewfinder rendering of captured imagery.
  • Another aspect of the technology involves capturing image information from a reference subject, using a camera, and processing the captured image information to yield reference color information. Imagery is also captured with the camera, depicting a skin rash or lesion. This latter imagery is then color-corrected by use of the reference color information.
  • the reference subject can comprise, e.g., blood or a banknote.
  • Still another aspect of the technology involves receiving imagery comprising plural frames depicting a skin rash or lesion.
  • a processing operation is invoked on the plural frames, to yield an enhanced still image (e.g., (a) a super-resolution image, (b) a noise-reduced image, (c) a multi- spectral image, (d) an ambient light-compensated image, or (e) a 3D image).
  • the enhanced still image (or data derived therefrom) is submitted to a database for similarity- matching with reference images depicting skin rashes or lesions, or data derived from such reference images.
  • Such database includes reference data corresponding to enhanced still images that themselves have been processed from plural-frame imagery.
  • Yet another arrangement involves obtaining plural sets of professional data, where each set includes skin image data and patient profile data.
  • This patient profile data includes both opinion information provided by a medical professional, and factual information.
  • first feature information is extracted (e.g., using a hardware processor configured to perform such act).
  • Plural sets of lay data are also obtained, where this lay data includes skin image data and patient profile data (e.g., fact data), but lacks information provided by a medical professional.
  • second feature information is extracted. The first and second extracted feature information is then made available as reference feature information for similarity-matching with feature information extracted from query skin image data.
  • Still another aspect of the technology involves receiving data, including skin image data and associated metadata, from a party. Similarities between the received data and reference data are determined.
  • This reference data includes multiple sets of data, each including skin image data and associated metadata. Included among the reference data are sets of data that have been professionally curated, and also sets of data that have not been professionally curated.
  • a further method involves receiving a first set of information from a first submitter, where the first set of information includes imagery depicting a part of a first subject's body that evidences a symptom of a first pathological condition, and also includes drug profile data indicating drugs taken by the first subject.
  • a second set of information is received from a second submitter. This second set of information includes imagery depicting a part of a second subject's body that evidences a symptom of a second pathological condition, and also includes drug profile data indicating drugs taken by the second subject.
  • Such sets of information are received from 3d through Nth submitters. Information is then received corresponding to a query image submitted by a user.
  • One or more image parameters are computed from this query image, and compared for correspondence against such parameters computed for the imagery received from the first through Nth submitters. Information is then sent to the user, identifying one or more drugs that is correlated with skin imagery (e.g., skin symptoms) having an appearance like that in the query image.
  • skin imagery e.g., skin symptoms
  • Yet another method involves receiving a first set of information from a first submitter, where this first set of information includes imagery depicting a part of a first subject's body that evidences a symptom of a first pathological condition, and also includes a diagnosis of the first pathological condition.
  • Such information is likewise received from a second submitter, including imagery depicting part of a second subject's body that evidences a symptom of a second pathological condition, and also includes a diagnosis of the second pathological condition.
  • This is repeated for third through Nth sets of information, received from third through Nth submitters.
  • Information is then received corresponding to a query image submitted by a user.
  • One or more parameters are computed from the query image.
  • a search is conducted to identify imagery received from the first through Nth submitters having correspondence with the computed image parameter(s).
  • Information e.g., information about candidate diagnoses, or about diagnoses that are inconsistent with the available information
  • Information are then sent to the user, based on such searching of imagery.
  • Still another method involves obtaining first imagery depicting a part of a mammalian body that evidences a symptom of a possible pathological condition. This imagery is processed to derive one or more image parameter(s). A data structure is searched for reference
  • This reference information comprises information identifying one or more particular pathological conditions that is not the pathological condition evidenced by the depicted part of the body. At least some of this result information is communication to a user.
  • a further aspect of the technology involves sensing whether a mobile device is in a static, viewing pose. When it is not, frames of imagery depicting a user are collected. When it is in the static viewing pose, information is presented for user review. This presented information is based, at least in part, on the collected frames of imagery. In such arrangement, the mobile device automatically switches between data collection and data presentation modes (e.g., based on pose and/or motion).
  • a further aspect of the technology involves capturing audio sounds from a user's body, using a portable device held by the user.
  • Plural features are derived from the captured audio; these features comprise fingerprint information corresponding to the captured sounds.
  • This fingerprint information is provided to a knowledge base (data structure), which contains reference fingerprint data and associated metadata. Metadata associated with one or more of the reference fingerprint data in the knowledge base is then received back by the device.
  • Information based on this received metadata (e.g., physiologic- or health-related information) is presented to the user.
  • Still another aspect of the present technology concerns an imaging booth defined by one or more sidewalls.
  • a plurality of cameras are disposed along the one or more side walls, directed toward an interior of the booth, and these cameras are connected to an image processor.
  • the booth also includes a plurality of light sources directed toward the interior of the booth, coupled to driving electronics. Plural of these light sources is each spectrally tuned to a different wavelength, and the driving electronics are adapted so that different of the lighting elements illuminate at different times, causing the cameras to capture imagery under plural different spectral lighting conditions.
  • the image processor is adapted to produce spectricity
  • medical diagnosis often relies heavily on patient history.
  • history information can be extracted from medical records, and used in assessing the possible diagnostic relevance of different physiologic signals, or combinations of signals.
  • a physiologic signal (e.g., crackles or rales) may take on new meaning when interpreted in the context of particular DNA findings (which may indicate, e.g., a susceptibility to particular viral illnesses).
  • haptics allows data to be acquired concerning tactile information - such as the firmness, tautness, elasticity or resilience of a body part.
  • tactile information such as the firmness, tautness, elasticity or resilience of a body part.
  • haptic actuators can also be employed - applying physical forces to the body in controlled direction, strength, and temporal pattern, so that measurement data responsive to such stimulus can be sensed.
  • Face-Chek mode employed rainbow mode illumination, it will be recognized that the Face-Chek mode can also use other lighting arrangements, e.g., simple ambient light.
  • Certain of the described arrangements may capture imagery in which the body of the camera device (e.g., smartphone) casts a visible shadow. Arrangements for detailing with such shadows are detailed in applicant's patent 8,488,900.
  • indices to the database, sorted by different parameters.
  • a binary or other optimized search can be conducted in the index to quickly identify reference images with similar parameter values.
  • Reference images with remote values needn't be considered for this parameter.
  • known machine learning techniques can be applied to the reference data to discern which image derivatives are most useful as diagnostic discriminants of different conditions.
  • a query image is received, it can be tested for these discriminant parameters to more quickly perform a Bayesian evaluation of different candidate diagnosis hypotheses.
  • Bag-of-features techniques can also be mployed to ease, somewhat, the image matching operation (but such techniques “cheat” by resort to data quantization that may - in some instances - bias the results).
  • Pattern recognition techniques developed for automated mole diagnosis can likewise be adapted to identifying database images that are similar to a query image.
  • Certain embodiments of the present technology can employ existing online catalogs of imagery depicting different dermatological symptoms and associated diagnoses. Examples include DermAtlas, at www ⁇ dot>dermatlas ⁇ dot>org - a crowd-sourced effort managed by physicians at Johns Hopkins University) and DermNet NZ at www ⁇ dot>dermnetnz ⁇ dot>org - a similar effort by the New Zealand Dermatological Society.
  • Dermnet a skin disease atlas organized by a physician in New Hampshire, based on submittals from various academic institutions, www ⁇ dot>dermnet ⁇ dot>com. Also related is the website Differential Diagnosis in Dermatology, www ⁇ dot>dderm ⁇ dot>blogspot ⁇ dot>com, and the web site of the International Dermoscopy Society, www ⁇ dot>dermoscopy-ids ⁇ dot>org.
  • HIPAA patient privacy rights
  • the technology serves as an advisor to a medical professional - offering suggested diagnoses, or further testing, to consider.
  • the offered advice may be tailored in accordance with wishes of the professional, e.g., expressed in stored profile data
  • some practitioners may specify diagnostic criteria that they tend to weigh more (or less) heavily in reaching particular conclusions.
  • one physician may regard chromatic diversity across a mole as particularly relevant to a diagnosis of malignant melanoma.
  • Another physician may not hold such criterion in high regard, but may find scalloped edge contours to be highly probative for such a diagnosis.
  • stored profile data for the different professionals can indicate such preferences, and tailor system response accordingly. (In some arrangements, such preferences are not expressly specified as profile information by the physician, but rather are deduced through analysis of electronic medical records detailing previous diagnoses, and the clinical data on which they were based.)
  • Another form of metadata that may be associated with sensed user data is information indicating treatments the user has tried, and their assessment of success (e.g., on a 0-10 scale).
  • sensed user data e.g., skin images
  • assessment of success e.g., on a 0-10 scale.
  • data may reveal effective treatments for different types of rashes, acne, etc.
  • Skin also serves as a barometer of other conditions, including emotion.
  • emotion activates a different collection of bodily systems, triggering a variety of bodily responses, e.g., increased blood flow (vasocongestion) to different regions, sometimes in distinctive patterns that can be sensed to infer emotion.
  • bodily responses e.g., increased blood flow (vasocongestion) to different regions, sometimes in distinctive patterns that can be sensed to infer emotion.
  • sensed physiologic information and its derivatives are represented in the form of Linked Data - both for individual and aggregate data, and both for storage and for sharing - in order to facilitate semantic reasoning with such information.
  • Linked Data both for individual and aggregate data, and both for storage and for sharing - in order to facilitate semantic reasoning with such information.
  • imagery, image derivatives, and metadata information are stored in accordance with the DICOM standards for medical image records (see, e.g.,
  • Certain embodiments of the technology recognize the user's forearm or other body member (e.g., by classification methods), and use this information in later processing (e.g., in assessing scale of skin features).
  • analysis is applied to video information captured while the user is moving the smartphone camera into position to capture skin imagery.
  • Such "flyover" video is commonly of lower resolution, and may suffer from some blurring, but is adequate for body member-recognition purposes.
  • a hand is recognized from one or more frames of such video, and the smartphone is thereafter moved (as sensed by accelerometers and/or gyroscopes) in a manner consistent with that hand being the ultimate target for imaging (e.g., the smartphone is moved in a direction perpendicular to the plane of the phone screen - moving it closer to the hand), then the subject of the image is known to be the hand, even if the captured diagnostic image itself is a close-up from which the body location cannot be deduced.
  • an automated early warning system can be set in place where livestock passing through gates or paddocks are routinely examined for unusual skin variations that suggest closer examination is needed.
  • Livestock are often outfitted with RF tags for identification, allowing such a monitoring system to compare individual livestock over time to rule out health conditions that have already been addressed, and to note new, emerging conditions.
  • Wildlife managers can also benefit by setting up imaging systems on commonly traversed paths that are triggered by passing animals. Again, early detection and identification of contagious conditions or dangerous pests is key to maintaining healthy populations.
  • Another diagnostically useful feature is temporal observation of blood flow through the area of a skin condition.
  • Subtle color changes due to local blood pressure modulated by heartbeats can be used to distinguish between, or assess the severity of, some skin conditions.
  • One method of observing these subtle color changes is given in the Wu paper cited below ("Eulerian Video Magnification for Revealing Subtle Changes in the World"), where small differences are magnified through spatio-temporal signal processing.
  • Elasticity of a region can be measured by applying pressure (by machine or by touch) in such a way as to bend the skin. By comparing various points on the skin before and after deformation (ideally, a repeated pattern of deformation to allow for averaging), the local elastic properties of the skin can be included in the diagnosis.
  • the local 3D texture of the skin condition region can also be assessed through the use of 3D imaging technology, including light- field and plenoptic cameras.
  • 3D imaging technology including light- field and plenoptic cameras.
  • Accurate 3D information can also obtained from a single camera system by illuminating a region of interest of the patient with a structured light pattern. Distortions in the structured light pattern are used to determine the 3D structure of the region, in familiar manner.
  • the pattern may be projected, e.g., by a projector associated with the camera system.
  • a projector associated with the camera system.
  • a mobile phone or headworn apparatus can include a pico data projector.
  • Mathematical Morphology see, e.g., the Wikipedia article of that name, where the topology of an image is described in terms of spatial surface descriptions. This is used, e.g., in counting of small creatures/structures under a microscope. Such technology is well suited to counting "bumps" or other structures per area in a skin lesion. It also allows for representation by attributed relational graphs that describe a detailed relationship between structures that can be compared as graphs independent of orientation and specific configuration.
  • Such methods extract local features from patches of an image (e.g., SIFT points), and automatically cluster the features into N groups (e.g., 168 groups) - each corresponding to a prototypical local feature.
  • a vector of occurrence counts of each of the groups i.e., a histogram
  • a vector occurrence count is then determined, and serves as a reference signature for the image, or for a sub-part thereof.
  • To determine if a query image matches the reference image local features are again extracted from patches of the image, and assigned to one of the earlier-defined N-groups (e.g., based on a distance measure from the corresponding prototypical local features).
  • a vector occurrence count is again made, and checked for correlation with the reference signature.
  • SIFT Scale-Invariant Feature Transform
  • a computer vision technology pioneered by David Lowe and described in various of his papers including "Distinctive Image Features from Scale-Invariant Keypoints," International Journal of Computer Vision, 60, 2 (2004), pp. 91-110; and "Object Recognition from Local Scale-Invariant Features,” International Conference on Computer Vision, Corfu, Greece (September 1999), pp. 1150-1157, as well as in patent 6,711,293. Additional information about SIFT (and similar techniques SURF and ORB) is provided in the patent documents cited herein.
  • SIFT is referenced
  • other robust feature points may be preferred for skin imagery.
  • SIFT is typically performed on grey-scale imagery; color is ignored.
  • feature points for skin can advantageously employ color.
  • An exemplary set of feature points specific to close-up skin imagery can comprise skin pores (or hair follicles). The center of mass of each such feature is determined, and the pixel coordinates of each are then associated with the feature in a data structure.
  • 3D features can additionally or alternatively be used.
  • Features can also be drawn from those that are revealed by infrared sensing, e.g., features in the dermal layer, including blood vessel minutiae.
  • each includes one or more processors, one or more memories (e.g. RAM), storage (e.g., a disk or flash memory), a user interface (which may include, e.g., a keypad, a TFT LCD or OLED display screen, touch or other gesture sensors, a camera or other optical sensor, a compass sensor, a 3D magnetometer, a 3-axis accelerometer, a 3-axis gyroscope, one or more microphones, etc., together with software instructions for providing a graphical user interface), interconnections between these elements (e.g., buses), and an interface for communicating with other devices (which may be wireless, such as GSM, 3G, 4G, CDMA, WiFi, WiMax, Zigbee or Bluetooth, and/or wired, such as through an Ethernet local area network, a T-l internet connection, etc.).
  • memories e.g. RAM
  • storage e.g., a disk or flash memory
  • a user interface which may include, e.g.,
  • the processes and system components detailed in this specification can be implemented as instructions for computing devices, including general purpose processor instructions for a variety of programmable processors, including microprocessors (e.g., the Intel Atom, the ARM A5, and the Qualcomm Snapdragon, and the nVidia Tegra 4; the latter includes a CPU, a GPU, and nVidia' s Chimera computational photography architecture), graphics processing units (GPUs, such as the nVidia Tegra APX 2600, and the Adreno 330 - part of the Qualcomm Snapdragon processor), and digital signal processors (e.g., the Texas Instruments TMS320 and OMAP series devices), etc.
  • microprocessors e.g., the Intel Atom, the ARM A5, and the Qualcomm Snapdragon, and the nVidia Tegra 4; the latter includes a CPU, a GPU, and nVidia' s Chimera computational photography architecture
  • GPUs such as the nVidia Tegra APX 2600, and the Adreno 330 - part of the Qualcomm Snapdragon
  • processor circuitry including programmable logic devices, field programmable gate arrays (e.g., the Xilinx Virtex series devices), field programmable object arrays, and application specific circuits - including digital, analog and mixed analog/digital circuitry.
  • Execution of the instructions can be distributed among processors and/or made parallel across processors within a device or across a network of devices. Processing of signal data may also be distributed among different processor and memory devices. "Cloud" computing resources can be used as well. References to
  • processors should be understood to refer to functionality, rather than requiring a particular form of implementation.
  • Smartphones and other devices according to certain implementations of the present technology can include software modules for performing the different functions and acts.
  • Software and hardware configuration data/instructions are commonly stored as instructions in one or more data structures conveyed by tangible media, such as magnetic or optical discs, memory cards, ROM, etc., which may be accessed across a network.
  • Some embodiments may be implemented as embedded systems -special purpose computer systems in which operating system software and application software are indistinguishable to the user (e.g., as is commonly the case in basic cell phones).
  • the functionality detailed in this specification can be implemented in operating system software, application software and/or as embedded system software.
  • references data can be monolithic, or can be distributed.
  • reference data may be stored anywhere, e.g., user devices, remote device, in the cloud, divided between plural locations, etc.
  • data can be stored anywhere: local device, remote device, in the cloud, distributed, etc.
  • the present technology can be used in connection with wearable computing systems, including headworn devices.
  • wearable computing systems including headworn devices.
  • Such devices typically include one or more sensors (e.g., microphone(s), camera(s), accelerometers(s), etc.), and display technology by which computer information can be viewed by the user - either overlaid on the scene in front of the user
  • a headworn device may further include sensors for detecting electrical or magnetic activity from or near the face and scalp, such as EEG and EMG, and myoelectric signals - sometimes termed Brain Computer Interfaces, or BCIs.
  • BCIs Brain Computer Interfaces
  • Some or all such devices may communicate, e.g., wirelessly, with other computing devices (carried by the user or otherwise), or they can include self-contained processing capability. Likewise, they may incorporate other features known from existing smart phones and patent documents, including electronic compass, accelerometers, gyroscopes, camera(s), projector(s), GPS, etc.
  • embodiments of present technology can also employ neuromorphic processing techniques (sometimes termed “machine learning,” “deep learning,” or “neural network technology”).
  • machine learning Deep learning
  • neural network technology a technique that employ large arrays of artificial neurons - interconnected to mimic biological synapses.
  • These methods employ programming that is different than the traditional, von Neumann, model. In particular, connections between the circuit elements are weighted according to correlations in data that the processor has previously learned (or been taught).
  • Each artificial neuron receives a plurality of inputs and produces a single output which is calculated using a nonlinear activation function (such as the hyperbolic tangent) of a weighted sum of the neuron's inputs.
  • the neurons within an artificial neural network are interconnected in a topology chosen by the designer for the specific application.
  • the ANN consists of an ordered sequence of layers, each containing a plurality of neurons.
  • the neurons in the first, or input, layer have their inputs connected to the problem data, which can consist of image or other sensor data, or processed versions of such data.
  • Outputs of the first layer are connected to the inputs of the second layer, with each first layer neuron's output normally connected to a plurality of neurons in the second layer. This pattern repeats, with the outputs of one layer connected to the inputs of the next layer.
  • the final, or output, layer produces the ANN output.
  • a common application of ANNs is classification of the input signal into one of N classes (e.g., classifying a type of mole). In this case the output layer may consist of N neurons in one-to-one correspondence with the classes to be identified. Feedforward ANNs are commonly used, but feedback arrangements are also possible, where the output of one layer is connected to the same or to previous layers.
  • a weight which is used by the input neuron in calculating the weighted sum of its inputs.
  • the learning (or training) process is embodied in these weights, which are not chosen directly by the ANN designer, In general, this learning process involves determining the set of connection weights in the network that optimizes the output of the ANN is some respect.
  • Two main types of learning, supervised and unsupervised, involve using a training algorithm to repeatedly present input data from a training set to the ANN and adjust the connection weights accordingly.
  • the training set includes the desired ANN outputs corresponding to each input data instance, while training sets for unsupervised learning contain only input data.
  • reinforcement learning the ANN adapts on-line as it is used in an application. Combinations of learning types can be used; in feed-forward ANNs, a popular approach is to first use
  • each neuron of the input layer processes a different weighted sum of the input data.
  • certain neurons within the input layer may spike (with a high output level), while others may remain relatively idle.
  • This processed version of the input signal propagates similarly through the rest of the network, with the activity level of internal neurons of the network dependent on the weighted activity levels of predecessor neurons.
  • the output neurons present activity levels indicative of the task the ANN was trained for, e.g. pattern recognition.
  • Artisans will be familiar with the tradeoffs associated with different ANN topologies, types of learning, and specific learning algorithms, and can apply these tradeoffs to the present technology.
  • SVMs support vector machines
  • Pulse detection from wearable clothing and devices is taught, e.g., in patents 5,622,180, 6,104,947 and 7,324,841 to Polar Electro OY.
  • Cavalcanti et al, "An ICA-based method for the segmentation of pigmented skin lesions in macroscopic images, IEEE Int'l Conf on Engineering in Medicine and Biology Society, 2011;

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Epidemiology (AREA)
  • Primary Health Care (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Quality & Reliability (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Biomedical Technology (AREA)
  • Databases & Information Systems (AREA)
  • Pathology (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)

Abstract

Selon la présente invention, la disponibilité d'imageurs de haute qualité sur des téléphones intelligents et d'autres dispositifs portables facilite la création d'une importante bibliothèque de référence d'images externalisée qui représente des éruptions cutanées et d'autres états dermatologiques. Certaines images sont téléchargées, ou annotées plus tard, avec des diagnostics associés ou d'autres informations (par exemple, « cette éruption a disparu lorsque j'ai arrêté de boire du lait »). Un utilisateur télécharge dans la bibliothèque une nouvelle image d'une affection cutanée inconnue. Des techniques d'analyse d'image sont utilisées pour identifier les principales similarités entre les caractéristiques de l'image téléchargée et les caractéristiques des images de cette bibliothèque de référence. Étant donné l'important ensemble de données, des corrélations statistiquement pertinentes émergent qui identifient pour l'utilisateur certains diagnostics qui peuvent être considérés, d'autres diagnostics qui peuvent être vraisemblablement écartés et/ou des informations anecdotiques concernant des affections cutanées similaires touchant d'autres utilisateurs. Des agencements similaires peuvent utiliser des signaux audio et/ou d'autres signaux d'ordre physiologique. Une grande variété d'autres caractéristiques et d'autres agencements sont également décrits en détail.
EP14785516.7A 2013-04-18 2014-04-18 Acquisition et analyse de données physiologiques Ceased EP2987106A4 (fr)

Applications Claiming Priority (7)

Application Number Priority Date Filing Date Title
US201361813295P 2013-04-18 2013-04-18
US201361832715P 2013-06-07 2013-06-07
US201361836560P 2013-06-18 2013-06-18
US201361872494P 2013-08-30 2013-08-30
US14/206,109 US20140316235A1 (en) 2013-04-18 2014-03-12 Skin imaging and applications
US201461978632P 2014-04-11 2014-04-11
PCT/US2014/034706 WO2014172671A1 (fr) 2013-04-18 2014-04-18 Acquisition et analyse de données physiologiques

Publications (2)

Publication Number Publication Date
EP2987106A1 true EP2987106A1 (fr) 2016-02-24
EP2987106A4 EP2987106A4 (fr) 2016-12-14

Family

ID=51731884

Family Applications (1)

Application Number Title Priority Date Filing Date
EP14785516.7A Ceased EP2987106A4 (fr) 2013-04-18 2014-04-18 Acquisition et analyse de données physiologiques

Country Status (3)

Country Link
EP (1) EP2987106A4 (fr)
CN (1) CN105393252B (fr)
WO (1) WO2014172671A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10083369B2 (en) 2016-07-01 2018-09-25 Ricoh Company, Ltd. Active view planning by deep learning

Families Citing this family (75)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6003963B2 (ja) 2014-11-07 2016-10-05 カシオ計算機株式会社 診断装置並びに当該診断装置における画像処理方法及びそのプログラム
JP6003964B2 (ja) 2014-11-07 2016-10-05 カシオ計算機株式会社 診断装置並びに当該診断装置における画像処理方法及びそのプログラム
US9881368B2 (en) 2014-11-07 2018-01-30 Casio Computer Co., Ltd. Disease diagnostic apparatus, image processing method in the same apparatus, and medium storing program associated with the same method
US9687208B2 (en) * 2015-06-03 2017-06-27 iMEDI PLUS Inc. Method and system for recognizing physiological sound
CN107180067B (zh) * 2016-03-11 2022-05-13 松下电器（美国）知识产权公司 图像处理方法、图像处理装置及记录介质
EP3432228A4 (fr) * 2016-03-14 2019-04-10 Omron Corporation Dispositif de maintien d'extensibilité
EP3223237B1 (fr) 2016-03-22 2020-05-27 Tata Consultancy Services Limited Systèmes et procédés de détection et de suivi d'un marqueur
JP6650819B2 (ja) 2016-04-15 2020-02-19 株式会社 資生堂 色ムラ部位の評価方法、色ムラ部位評価装置及び色ムラ部位評価プログラム
US10251597B2 (en) 2016-04-21 2019-04-09 Viavi Solutions Inc. Health tracking device
US11134848B2 (en) * 2016-04-25 2021-10-05 Samsung Electronics Co., Ltd. Mobile hyperspectral camera system and human skin monitoring using a mobile hyperspectral camera system
US10235551B2 (en) * 2016-05-06 2019-03-19 Qualcomm Incorporated Biometric system with photoacoustic imaging
ES2710338T3 (es) 2016-05-10 2019-04-24 Von Sobbe Hans Ulrich Sistema de análisis
CA3029143A1 (fr) * 2016-06-29 2018-01-04 Koninklijke Philips N.V. Detection de changement dans des images medicales
CN106096317A (zh) * 2016-07-14 2016-11-09 王存金 一种术前麻醉访视系统
CN106328150B (zh) * 2016-08-18 2019-08-02 北京易迈医疗科技有限公司 嘈杂环境下的肠鸣音检测方法、装置及系统
EP3287914A1 (fr) * 2016-08-23 2018-02-28 Siemens Healthcare GmbH Determination de donnees de resultat en fonction de donnees de mesure medicales provenant de differentes mesures
CN106855917A (zh) * 2016-11-23 2017-06-16 北海高创电子信息孵化器有限公司 一种病理监控和诊断系统
TWI618003B (zh) * 2016-12-01 2018-03-11 財團法人資訊工業策進會 基於感測資料之估測方法及基於感測資料之估測系統
CN106548038A (zh) * 2016-12-09 2017-03-29 江西中科九峰智慧医疗科技有限公司 一种基于知识的医疗影像采集分析系统
CN106709254B (zh) * 2016-12-29 2019-06-21 天津中科智能识别产业技术研究院有限公司 一种医疗诊断机器人系统
CN106682449A (zh) * 2017-03-01 2017-05-17 中国科学院苏州生物医学工程技术研究所 基于云数据库的皮肤病图谱辅助诊断系统
CN106926277A (zh) * 2017-03-07 2017-07-07 深圳华用科技有限公司 一种用于工业机器人定位的一体化智能相机
WO2018182068A1 (fr) * 2017-03-30 2018-10-04 스노우 주식회사 Procédé et appareil de fourniture d'informations de recommandation pour un article
CN108403080A (zh) * 2017-04-18 2018-08-17 赵鹏 一种实时监测辅助诊断与治疗皮肤腐性病的仪器系统
CN107368732B (zh) * 2017-07-14 2019-07-23 南京东科优信网络安全技术研究院有限公司 一种基于设备物理指纹特征的目标识别定位系统及方法
CN107578822B (zh) * 2017-07-25 2020-12-15 广东工业大学 一种针对医疗多模态大数据的预处理及特征提取方法
CN107550458B (zh) * 2017-07-27 2020-05-12 天津大学 基于声电效应与声辐射力的生物组织多特性成像方法
CN107423573A (zh) * 2017-08-12 2017-12-01 合肥全飞信息科技有限公司 一种皮肤病识别手机app设计方法
CN107609492B (zh) * 2017-08-25 2019-06-21 西安电子科技大学 基于脑电信号的失真图像质量感知评价方法
WO2019046799A1 (fr) * 2017-08-31 2019-03-07 Zeto, Inc. Procédé d'hébergement d'accès mobile à des données d'électroencéphalographie à haute résolution
DE102017215829A1 (de) * 2017-09-07 2018-12-06 Siemens Healthcare Gmbh Verfahren und Datenverarbeitungseinheit zum Ermitteln von Klassifikationsdaten für eine Adaption eines Untersuchungsprotokolls
CN107808128B (zh) * 2017-10-16 2021-04-02 深圳市云之梦科技有限公司 一种虚拟形象重建人体五官测量的方法及系统
CN108606780A (zh) * 2018-05-15 2018-10-02 北京科莱普云技术有限公司 皮肤检测方法、装置、计算机设备和存储介质
CN108737428B (zh) * 2018-05-24 2021-07-27 中国联合网络通信集团有限公司 基于图像识别的皮肤病确定方法和装置
EP3815045A1 (fr) * 2018-06-29 2021-05-05 Zoox, Inc. Étalonnage de capteur
CN108814717B (zh) * 2018-06-29 2020-10-27 微创（上海）医疗机器人有限公司 手术机器人系统
CN109330560A (zh) * 2018-09-10 2019-02-15 天津大学 一种皮肤病识别与检测盒
CN109157231B (zh) * 2018-10-24 2021-04-16 阿呆科技（北京）有限公司 基于情绪刺激任务的便携式多通道抑郁倾向评估系统
CN109431465B (zh) * 2018-10-26 2021-06-22 南京邮电大学 一种基于射频标签技术的老年人睡眠质量监测方法
WO2020095131A1 (fr) * 2018-11-07 2020-05-14 Foss Analytical A/S Analyseur de lait pour la classification de lait
MA51643B1 (fr) * 2018-11-07 2022-09-30 Hoffmann La Roche Procédés et dispositifs pour effectuer une mesure analytique
IT201800010536A1 (it) * 2018-11-23 2020-05-23 Torino Politecnico Dispositivo e metodo per la rilevazione ed il monitoraggio di patologie cutanee
CN109511036B (zh) * 2018-12-14 2020-12-01 歌尔科技有限公司 一种耳机自动静音方法及可自动静音的耳机
CN109783709B (zh) * 2018-12-21 2023-03-28 昆明理工大学 一种基于马尔可夫决策过程和k-最近邻强化学习的排序方法
CN110349121A (zh) * 2018-12-27 2019-10-18 合刃科技（深圳）有限公司 一种基于高光谱成像与神经网络的监测方法以及终端
EP3922232B1 (fr) * 2019-02-08 2022-11-23 FUJIFILM Toyama Chemical Co., Ltd. Système d'identification de médicament, dispositif d'identification de médicament, procédé d'identification de médicament, et programme
CN109938705B (zh) * 2019-03-06 2022-03-29 智美康民(珠海)健康科技有限公司 三维脉波的显示方法、装置、计算机设备及存储介质
CN110781477B (zh) * 2019-04-02 2021-01-29 宁波潮涌道投资合伙企业(有限合伙) 基于读取大数据的反推系统
CN110084168B (zh) * 2019-04-22 2021-04-23 中国科学院自动化研究所 一种主动目标识别方法及装置
CN110279406B (zh) * 2019-05-06 2022-07-15 苏宁金融服务(上海)有限公司 一种基于摄像头的无接触式的脉率测量方法及装置
WO2020232680A1 (fr) * 2019-05-22 2020-11-26 深圳市汇顶科技股份有限公司 Procédé de reconnaissance biométrique, appareil de reconnaissance d'empreintes digitales, et dispositif électronique
CN112294253A (zh) * 2019-07-26 2021-02-02 深圳百诺明医说科技有限公司 一种基于用户语音变化的疾病诊断系统及家庭智能机器人
CN110427769B (zh) * 2019-08-12 2020-12-25 重庆大学 基于智能看护系统面向用户活动模式隐私保护实施方法
CN110504031B (zh) * 2019-08-28 2022-02-11 首都医科大学 用于健康行为干预的云端管理数据库建立方法及系统
CN110664388B (zh) * 2019-09-05 2022-03-25 广州市蜗牛互动科技有限公司 心率检测方法、装置、存储介质以及设备
US20210201008A1 (en) * 2019-12-31 2021-07-01 L'oreal High-resolution and hyperspectral imaging of skin
CN111358428A (zh) * 2020-01-20 2020-07-03 书丸子(北京)科技有限公司 一种观察能力测试评定方法和装置
CN113450906A (zh) * 2020-03-26 2021-09-28 苏州佳世达光电有限公司 临床检测诊断系统及临床检测诊断方法
US11593678B2 (en) 2020-05-26 2023-02-28 Bank Of America Corporation Green artificial intelligence implementation
CN112201351A (zh) * 2020-09-04 2021-01-08 广东科学技术职业学院 基于声音收集分析进行健康提示的方法、装置及介质
WO2022055496A1 (fr) * 2020-09-11 2022-03-17 Hewlett-Packard Development Company, L.P. Notes de santé d'utg
TWI784334B (zh) 2020-10-29 2022-11-21 國立臺灣大學 疾病檢測方法及疾病檢測系統
CN112606402A (zh) * 2020-11-03 2021-04-06 泰州芯源半导体科技有限公司 应用多参数解析的产品制造平台
TWI765420B (zh) * 2020-11-20 2022-05-21 英業達股份有限公司 心衰竭預測組件
CN112450120B (zh) * 2020-11-26 2022-08-05 小佩网络科技（上海）有限公司 宠物体重记录方法、装置、宠物厕所及介质
CN112967382B (zh) * 2021-03-09 2023-01-31 江西省儿童医院 一种肾脏内微细管道3d重构方法
CN112800245B (zh) * 2021-03-29 2021-07-16 微岩医学科技(北京)有限公司 一种病原微生物参考知识库的最大多样性聚类构建方法
CN113143217B (zh) * 2021-05-07 2024-01-26 张育青 一种预测人体发声效果的方法及系统
CN113188605B (zh) * 2021-05-12 2022-06-21 郑州大学 一种基于智能手机的物理实验方法
CN114081446B (zh) * 2021-11-19 2024-03-01 杭州师范大学 一种便秘预测方法
CN114758748B (zh) * 2022-04-07 2023-06-20 上海晨希医疗美容门诊部有限公司 一种植发环境的智能化分析处理方法及系统
CN115299937B (zh) * 2022-09-16 2023-08-11 东联信息技术有限公司 一种智能摔倒检测平台
CN116340433B (zh) * 2023-05-31 2023-07-28 中国水利水电第七工程局有限公司 施工监测信息存储计算方法、存储介质、设备及系统
CN117495384B (zh) * 2023-11-07 2024-04-26 广州准捷电子科技有限公司 一种基于ai人脸识别技术的ktv刷脸支付方法
CN118039136A (zh) * 2024-04-12 2024-05-14 中国医学科学院北京协和医院 结肠炎诊断系统、装置以及存储介质

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8543519B2 (en) * 2000-08-07 2013-09-24 Health Discovery Corporation System and method for remote melanoma screening
CA2856932C (fr) * 2005-01-19 2015-10-13 William T. Ii Christiansen Dispositifs et procedes pour identifier et surveiller les changements intervenant dans une region suspecte d'un patient
JP5245424B2 (ja) * 2008-01-25 2013-07-24 日本電気株式会社 病理組織画像撮影システム、病理組織画像撮影方法、および病理組織画像撮影プログラム
US8194952B2 (en) * 2008-06-04 2012-06-05 Raytheon Company Image processing system and methods for aligning skin features for early skin cancer detection systems
US8380533B2 (en) * 2008-11-19 2013-02-19 DR Systems Inc. System and method of providing dynamic and customizable medical examination forms
EP2409155A1 (fr) * 2009-03-15 2012-01-25 Technion Research and Development Foundation, Ltd. Complexes hla solubles utiles pour diagnostiquer des maladies
CN101756696B (zh) * 2009-12-31 2012-08-08 中国人民解放军空军总医院 多光子皮肤镜图像自动分析系统及其应用于恶性黑素瘤的诊断方法
JP5972865B2 (ja) * 2010-04-28 2016-08-17 ギブン イメージング リミテッドＧｉｖｅｎ Ｉｍａｇｉｎｇ Ｌｔｄ． 生体内画像の部分を表示するシステム及びその作動方法
US20120245952A1 (en) * 2011-03-23 2012-09-27 University Of Rochester Crowdsourcing medical expertise
CN102578995B (zh) * 2011-12-22 2014-07-30 诊断有限公司 人类和动物器官的诊断实施装置

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10083369B2 (en) 2016-07-01 2018-09-25 Ricoh Company, Ltd. Active view planning by deep learning

Also Published As

Publication number Publication date
WO2014172671A1 (fr) 2014-10-23
EP2987106A4 (fr) 2016-12-14
CN105393252B (zh) 2019-04-19
CN105393252A (zh) 2016-03-09

Similar Documents

Publication Publication Date Title
US10219736B2 (en) Methods and arrangements concerning dermatology
US20140313303A1 (en) Longitudinal dermoscopic study employing smartphone-based image registration
EP2987106A1 (fr) Acquisition et analyse de données physiologiques
US11779213B2 (en) Metaverse system
US20140316235A1 (en) Skin imaging and applications
US9445713B2 (en) Apparatuses and methods for mobile imaging and analysis
US20220047160A1 (en) Ceiling ai health monitoring apparatus and remote medical-diagnosis method using the same
US10192033B2 (en) Capturing data for individual physiological monitoring
De Greef et al. Bilicam: using mobile phones to monitor newborn jaundice
Alcón et al. Automatic imaging system with decision support for inspection of pigmented skin lesions and melanoma diagnosis
US8062220B2 (en) Monitoring physiological conditions
Casalino et al. Contact-less real-time monitoring of cardiovascular risk using video imaging and fuzzy inference rules
KR102162683B1 (ko) 비정형 피부질환 영상데이터를 활용한 판독보조장치
WO2020190648A1 (fr) Procédé et système de mesure de réflexe pupillaire à la lumière avec un téléphone mobile
Saifan et al. Six skin diseases classification using deep convolutional neural network
CN115317304A (zh) 一种基于生理特征检测的智能按摩方法及系统
KR100915922B1 (ko) 영상인식을 통한 얼굴 특징 추출과 사상체질 판별 방법 및시스템
KR102165487B1 (ko) 비정형 피부질환 영상데이터를 활용한 피부 질환 판독 시스템
Hong Spatial–Spectral–Temporal Framework for Emotion Recognition
KR102132959B1 (ko) 사용자의 신원 파악이 가능한 심박동수 모니터링 방법 및 시스템
Javvaji et al. Prediction Model for Obstructive Sleep Apnea from Facial Depth Maps using Transfer Learning
KR20230129662A (ko) 피부질환 학습을 이용한 피부질환 진단시스템
Toley et al. Facial Video Analytics: An Intelligent Approach to Heart Rate Estimation Using AI Framework
CN117672501A (zh) 一种基于人工智能的宠物健康初筛系统
Deshmukh et al. PERTIENCE OF IoT IN BIOMEDICAL DIAGNOSTIC SYSTEM–A REVIEW

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20151002

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20161111

RIC1 Information provided on ipc code assigned before grant

Ipc: G06F 19/00 20110101ALI20161107BHEP

Ipc: G06T 7/00 20060101AFI20161107BHEP

17Q First examination report despatched

Effective date: 20180406

REG Reference to a national code

Ref country code: DE

Ref legal event code: R003

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN REFUSED

18R Application refused

Effective date: 20200219