CA2711519A1

CA2711519A1 - System and method for analysis of light-matter interaction based on spectral convolution

Info

Publication number: CA2711519A1
Application number: CA2711519A
Authority: CA
Inventors: Djuro Koruga; Alexandar Tomic
Original assignee: Individual
Current assignee: MySkin Corp
Priority date: 2008-01-07
Filing date: 2009-01-07
Publication date: 2009-07-16
Also published as: AU2009204227A1; MX2010007463A; EP2243077A1; WO2009089292A1; AU2009204227B2; BRPI0905668A2; JP2011509154A; KR20100101173A; JP5771007B2; CN102027449B; JP2015221218A; ZA201004714B; EP2243077A4; JP5973626B2; KR101150184B1; CN102027449A

Abstract

In embodiments of the present invention, systems and methods of a method and algorithm for creating a unique spectral fingerprint are based on the convolution of RGB color channel spectral plots generated from digital images that have captured single and/or multi-wavelength light-matter interaction.

Description

2 PCT/US2009/030347 SYSTEM AND METHOD FOR ANALYSIS OF LIGHT-MATTER
INTERACTION BASED ON SPECTRAL CONVOLUTION

CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority to the following applications, each of which is hereby incorporated by reference in its entirety: United States Provisional Patent Application No. 61/019,440, filed January 7, 2008; United States Provisional Patent Application No. 61/061,852, filed June 16, 2008; and United States Patent Application No. 11/970,448, filed January 7, 2008.

BACKGROUND
[0002] Field:

[0003] Embodiments of the invention may include methods and apparatus for enabling the collection of dermal and non-dermal images using a non-invasive imaging device, the development of a skin state based at least in part on analysis of such images, and the monitoring of the skin state by, at least, a collection and analysis of subsequent images. Embodiments of the invention may further pertain to the field of skin care devices and systems capable of facilitating skin care decisions, more specifically the field of devices for skin condition assessment, skin care regimen recommendation, and skin care regimen effectiveness tracking.

[0004] Embodiments of the present invention may also relate to an image processing technique. More particularly, embodiments of the present invention may relate to determining a skin photo type of a captured image in Red Green Blue (RGB) color imaging system and is also applicable in classification of other skin characteristics (e.g.
elasticity, melanin, oil concentration etc.), melanoma, skin related tumors and skin related disorders.

[0005] Description of the Related Art:

[0006] Skin is the largest organ of the integumentary system which further includes skin's accessory structures, such as hair, nails, scales, feathers, sweat glands and their products. Skin comprises multiple layers of epithelial tissues that guard underlying muscles and organs. Since skin is subject to constant attack from various external and Page 1 of 122 internal factors, it can be afflicted by numerous ailments. Thus, it is critical to monitor skin health and the effect of any treatments, skin care products, or cosmetics applied to the skin.

[0001] A few known methods of skin care decision systems involving conventional photography are also known as "clinical imaging." Some of the known methods involve illuminating the skin surface with white light when the digital image is acquired by the camera. The effectiveness of clinical imaging is, however, compromised by specular (mirror-like) reflection of the skin.

[0007] The existing processes of making skin care decisions for both medical and cosmetic purposes are generally based on scattered information collected from multiple sources in different time periods. The existing processes are cumbersome and inefficient, resulting in delayed and sub-optimal skin-care decisions.
Examples of skin-care decisions that can be significantly enhanced by the principles of the invention include diagnosis of skin conditions, selection of skin-care products and regimens, and tracking the effectiveness of skin-care products and regimens over a period of time. A
skin-care regimen includes both selection of appropriate skin-care products and the entire procedure of applying the selected skin-care products, including the dosage, timing, methodology, and frequency of application of the skin-care products.

[0008] While various methods exist for determining and monitoring skin health, most require access to a dermatologist or a dermatological facility, thus, there may be difficulty, inconvenience, and prohibitively high cost in accessing the necessary resources. There is a need for a simple solution for skin health determination and monitoring that may be operable by an untrained or trained user, and, in the absence of an in-person consultation, where dermal images may be submitted to an expert, an analysis facility, or for automated analysis.

[0009] Skin characteristics are typically determined using Fitzpatrick classification. Skin phototype is categorized according to a conventional Fitzpatrick Skin Typing Test questionnaire (skin type scale), which ranges from very fair (skin type I, for example) to very dark (skin type VI, for example). Conventionally, various image processing techniques are disclosed for determining skin characteristics using captured skin images. In a conventional digital image processing technique it is often useful to Page 2 of 122 detect multiple features in the captured skin image, such as skin color. This information is used, for example, to adjust the skin colors in the image to be comfortable to perceive.
The location of skin color is also used in face detection and face recognition algorithms, automatic image retrieval algorithms, and red-eye correction algorithms.

[0010] Less work has been done on objective measurement of human skin coloration to enable its color classification. Classification of a person's skin coloration would be useful, for example, in the medical field for quantification of skin erythema, lesions, ultra-violet radiation effects, and other skin coloration phenomena.
In the field of computer graphics images of people could be rendered more accurately in video-conferencing and their appearance could be improved or altered. While the science of digital skin imaging analysis has identified various skin responses that are useful indicators of various skin condition parameters, it would still be desirable to identify and use additional specific responses of the skin that are indicative of skin conditions and skin characteristics.

[0011] Cosmetic appearance is one of the top priorities to most of the humans in the modem world. Techniques or systems existing in the known art analyze the skin conditions and suggest suitable products to improve the cosmetic appearance of a human being.

[0012] There is a need for a minimal error and speed efficient method and system to determine the phototype of skin.

SUMMARY

[0013] Real-time analysis of digitally captured skin characteristics facilitates timely skin condition assessment, skin regimen recommendation, and skin regimen effectiveness tracking.

[0014] The problem of generating a skin condition assessment in real-time is solved by having a skin condition analysis module capable of doing real-time analysis of digital skin data, acquired partly using diffused reflectance spectroscopy and/ or detecting the red-green-blue components of re-emitted white light.

[0015] In an aspect of the invention, a skin care device may include an electromagnetic radiation source capable of directing incident electromagnetic radiation Page 3 of 122 to a location on the skin of a user, a radiation detector for measuring various parameters of radiation re-emitted from the location, and a skin condition analysis module coupled to the detector, the analysis module capable of generating a skin condition assessment in real-time, based partly on at least one of RGB analysis and diffused reflectance analysis of the radiation parameters. In the device, the incident electromagnetic radiation may include radiation in at least one of the visible, near-infrared, and near-ultraviolet spectrum. The incident radiation may include white light. In the device, the radiation parameters may include at least the degree of polarization of the re-emitted radiation. In the device, the radiation source may be a set of light emitting diodes. In the device, the skin condition assessment may also be partly based on analysis of a photographic image of a skin region surrounding the location. In the device, the device may be a miniature device. Miniature may mean that no dimension of the detector exceeds six inches. The device may further comprise a memory module for storing the skin condition assessment.
The device may further comprise a user interface. The user interface may be operated using voice commands. In the device, skin assessment data of locations may be overlaid on an image of a larger skin region and displayed on the display surface. The device may further comprise an access restriction module used for restricting access to authorized users only. The access restriction module may be based on biometric access control. The device may be capable of generating alerts about abnormal skin conditions in real-time.
The device may further comprise a skin care regimen recommendation module that generates a displayable skin care regimen recommendation. The skin care regimen recommendation may be based at least partly on determination of a skin profile of the user and use of skin care regimen recommendations of persons with a similar profile.
The skin care regimen recommendation module may be linked to a product database.
The product database may include products available in a point-of-sale location. The availability of a specific product recommended by the skin care regimen recommendation module may be indicated by an audio-visual signal. The device may further comprise a skin care regimen effectiveness module that generates a displayable skin care regimen effectiveness report. The device may further comprise a communication module for communicating with a remote computer. The communication may occur wirelessly.
The Page 4 of 122 communication may occur over an internet. The remote computer may be operable by a physician. The device may be wand-shaped. The device may be wearable by the user.

[0016] In an aspect of the invention, the skin care device may include an electromagnetic radiation source capable of directing incident electromagnetic radiation to a location on the skin of a user, a detector for measuring various parameters of radiation re-emitted from the location, a skin condition analysis module coupled to the detector, the analysis module capable of generating a skin condition assessment in real-time, based partly on at least one of RGB analysis and diffused reflectance analysis of the radiation parameters, and a display panel for reflecting the image of the user. In the device, the display panel may be touch-sensitive such that touching the location in a skin region image displayed in the display panel triggers display of a magnified image of the location. The device may further comprise a camera. The camera may be integral with the display panel. The camera may be wirelessly linked to the display panel.
In the device, the display panel may be a mirror. In the device, a stored image of the user is used to automatically identify the person. The device may further comprise a user interface for controlling the skin care device. The user interface may be operated using voice commands. The device may further comprise a skin care regimen recommendation module capable of generating a displayable skin care regimen recommendation.
The skin care regimen recommendation may be based at least partly on determination of a skin profile of the user and use of skin care regimen recommendations of persons with a similar profile. The device may further comprise a skin care regimen effectiveness module capable of generating a displayable skin care regimen effectiveness report.

[0017] In aspects of the invention, an imaging device permits a user to take high magnification pictures of the skin in the vicinity of an area of concern and submit those pictures, optionally along with textual and data responses, for medical, non-medical, and cosmetic analysis, diagnosis and treatment recommendation and follow-up.

[0018] In an aspect of the invention, a method and system of a non-invasive imaging device may comprise an illumination source comprising an incident light source to direct light upon skin; and a detector for detecting the degree of polarization of light reflected from the skin. In the method and system, the illumination source may be positioned to direct light at a selected angle alpha. Varying alpha may vary the depth of Page 5 of 122 the measurement of the layers in the skin. Each depth may have a specific angle which produces a full polarized reflection. In the method and system, the incident light source may be an unpolarized light source. The unpolarized light may be white light, multiple selected wavelengths, or a single wavelength. The method and system may further comprise a sensor for capturing an image of the reflected or re-emitted light.
The method and system may further comprise an optical facility for detecting reflected or re-emitted light from the skin. The method may determine both reflected or re-emitted light, and newly emitted light, through the process of absorption and re-emission. The method and system may further comprise a communication facility for transmitting the detected information. The method and system may further comprise a storage facility for storing information collected by the device.

[0019] In an aspect of the invention, a method and system for determining a skin state may comprise illuminating skin with an incident light source, detecting the degree of polarization of light reflected from the skin, and determining a skin state based on an aspect of the polarization of the reflected or re-emitted light. In the method and system, the incident light may be directed at a selected angle alpha. Varying alpha may vary the depth of the measurement of the layers in the skin. Each depth may have a specific angle which produces a full polarized reflection. In the method and system, the incident light source may be an unpolarized light source. The unpolarized light may be white light, multiple selected wavelengths, or a single wavelength. In the method of claim, the aspect of the polarization may be at least one of an orientation, an amplitude, a phase, an angle, a shape, a degree, an amount, and the like. In the method and system, determining may be done using an algorithm. The algorithm may involve artificial neural networks, non-linear regression, genetic algorithms, fuzzy logic, fractal and multi-fractal analysis, and the like. The methods and systems may further comprise filtering the reflected or re-emitted light to obtain polarized light of at least one wavelength defined by the filter output. The algorithmic analysis may be performed on the filtered image. In the method and system, determining may involve creating an image from the difference between the reflected diffusion light and the reflected polarized light. In the method and system, determining may involve comparing the aspect of the polarization of the reflected Page 6 of 122 or re-emitted light to a calibration signal. In the method and system, determining may further comprise considering at least one of user input and a visual analysis.

[0020] In an aspect of the invention, a non-invasive imaging device may comprise an illumination source comprising an incident light source to direct light upon an area of concern and a detector for detecting the degree of polarization of light reflected from the area of concern. In the method and system, the illumination source may be positioned to direct light at a selected angle alpha. Varying alpha may vary the depth of the measurement of the layers in the skin. Each depth may have a specific angle which produces a full polarized reflection. In the method and system, the incident light source may be an unpolarized light source. The unpolarized light may be white light, multiple selected wavelengths, or a single wavelength. The method and system may further comprise a sensor for capturing an image of the reflected or re-emitted light.
The method and system may further comprise an optical facility for detecting reflected or re-emitted light from the skin. The method and system may further comprise a communication facility for transmitting the detected information. The method and system may further comprise a storage facility for storing information collected by the device.

[0021] In an aspect of the invention, a method of determining moisture levels in the skin may comprise emitting incident light towards a skin structure, detecting a degree of polarization of the light induced by the skin structure, and determining a moisture level based on the amount of polarized and reflected or re-emitted light. The method and system may further comprise combining the assessment of moisture level with skin color measurements to determine luminosity. In the method and system, the incident light may be unpolarized light. The unpolarized light may be white light, light of multiple selected wavelengths, or of a single wavelength, or one or more monochromatic lights. In the method and system, determining may involve use of an algorithm. In the method and system, determining a moisture level may be based on the ratio of polarized and reflected or re-emitted light.

[0022] In an aspect of the invention, a method and system of determining elasticity of the skin may comprise emitting incident light towards a skin structure, detecting an aspect of polarization of the light reflected by the skin structure, correlating the aspect of polarization with a concentration of elastin, and determining elasticity level Page 7 of 122 based on the concentration of elastin. In the method and system, determining may involve use of an algorithm. In the method and system, the incident light may be unpolarized light. The unpolarized light may be white light, light of multiple selected wavelengths, or a single wavelength of light.

[0023] In an aspect of the invention, a method and system of determining firmness of the skin may comprise emitting incident light towards a skin structure, detecting an aspect of polarization of the light reflected by the skin structure, correlating the aspect of polarization with a concentration of at least one of an elastin, a collagen, and an activity of a sebaceous gland, and determining firmness based on the concentration of at least one of elastin and collagen and sebaceous gland activity. In the method and system, the sebaceous gland activity may be indicated by at least one of a number of glands, percent of glands open/closed, and level of clog/ fill. In the method and system, correlating may involve use of an algorithm.

[0024] In an aspect of the invention, a method and system for obtaining dermal biophysical properties may comprise performing a spectral analysis of image data acquired from the degree of polarization of reflections and absorption and re-emission of incident light from skin structures, wherein the property is at least one of a structure, form, concentration, number, size, state, and stage of at least one of a:
melanocyte, melanin, hemoglobin, porphyrin, keratin, carotene, collagen, elastin, sebum, sebaceous gland activity, pore (sweat and sebaceous), moisture level, elasticity, luminosity, firmness, fine line, wrinkle count and stage, pore size, percent of open pores, skin elasticity, skin tension line, spot, skin color, psoriasis, allergy, red area, general skin disorder or infection, tumor, sunburn, rash, scratch, pimple, acne, insect bite, itch, bleeding, injury, inflammation, photodamage, pigmentation, tone, tattoo, percent burn/
burn classification, mole (naevi, nevus), aspect of a skin lesion (structure, color, dimensions/asymmetry), melanoma, dermally observed disorder, cutaneous lesion, cellulite, boil, blistering disease, congenital dermal syndrome, (sub)-cutaneous mycoses, melasma, vascular condition, rosacea, spider vein, texture, skin ulcer, wound healing, post-operative tracking, melanocytic lesion, non-melanocytic lesion, basal cell carcinoma, seborrhoic keratosis, sebum (oiliness), nail- and/or hair-related concern, and the like.

Page 8 of 122

[0025] In an aspect of the invention, a system and method may comprise providing an interface that includes a social networking domain or rating-and-ranking system and at least one of a skin state determination facility and a recommendation engine, and enabling users, either all or a selected few, of the interface to perform a skin state determination within the interface. In the method and system, the skin state determination facility may comprise capturing images with a non-invasive imaging device comprising an illumination source comprising an incident light source to direct light upon skin, and a detector for detecting the degree of polarization of light reflected from the skin, and determining a skin state based on an aspect of the polarization of the reflected or re-emitted light. The method and system may further comprise receiving product and regimen recommendations from the recommendation engine based on what other users with similar skin states are using as well as data regarding ingredients, effectiveness, safety, and the like. The method and system may further comprise comparing skin states, products, regimens, and recommended products or regimens with peers within the social networking domain of the interface. Comparing may comprise an analysis of similarity based on the spectral analysis of the degree of polarization of reflected or re-emitted light from users' skin. In the method and system, the interface may comprise a regimen tracker. The regimen tracker may be populated using a drag-and-drop or click-to-add functionality. In the method and system, the interface may comprise a rating facility or a product information facility. The product information facility may enable a user to obtain product information by search. Search may be a search of product identifiers, product ratings, drag-and-drop items, images, barcode scans, skin states, and profiles.

[0026] In an aspect of the invention, a method and system for determining a skin state may comprise obtaining the answers to a series of subjective questions regarding the skin, obtaining an objective skin analysis using a dermal imaging device, and combining the subjective and objective results algorithmically to obtain a skin state.

[0027] In an aspect of the invention, a system and method for providing recommendations for skin care based on a skin state and a skin care goal may comprise obtaining a skin state of an individual, categorizing the individual by skin state, and recommending products and regimens that are effective for other individuals of the Page 9 of 122 category in achieving the skin care goal. In the method and system, the system may be operable over a network. In the method and system, the skin state may be determined based on analysis of the degree of polarization of light reflected from the skin of the individual.

[0028] In an aspect of the invention, a method for tracking the effectiveness of a skin care product or regimen may comprise obtaining a baseline skin state assessment, recommending a monitoring interval based on at least one of the skin care goal, product, and regimen, obtaining a second skin state assessment, comparing the second assessment to the baseline assessment to determine progress towards a skin care goal, and optionally, optimizing the regimen or product in order to improve a skin state.
In the method and system, the skin assessment may be based on analysis of the degree of polarization of light reflected from the skin of the individual.

[0029] In an aspect of the invention, a personalized skin condition analysis system and related methods may comprise an imaging device, comprising an illumination source comprising an incident light source to direct light upon skin, and a detector for detecting the degree of polarization of light reflected from the skin, and a user interface for controlling the device. In the methods and system, the device may be adapted to interact with a physical interface to download image data to update a record of at least one of a practitioner, a spa, a salon, cosmetic sales, a cosmetics manufacturer, a clinical trials database, and a third party database. In the method and system, the illumination source may be positioned to direct light at a selected angle alpha. Varying alpha may vary the depth of the measurement of the layers in the skin. Each depth may have a specific angle which produces a full polarized reflection. In the method and system, the incident light source may be an unpolarized light source. The unpolarized light may be white light, multiple selected wavelengths, or a single wavelength. The method and system may further comprise a sensor for capturing an image of the reflected or re-emitted light.
The method and system may further comprise an optical facility for detecting reflected or re-emitted light from the skin. The method and system may further comprise a communication facility for transmitting the detected information. The method and system may further comprise a storage facility for storing information collected by the device.

Page 10 of 122

[0030] In an aspect of the invention, a non-invasive imaging device may comprise an illumination source comprising an incident light source to direct light upon skin; and a detector for detecting a characteristic of the light reflected from the skin. In the device, the illumination source may be positioned to direct light at a selected angle alpha. Varying alpha may vary the depth of the measurement of the layers in the skin.
Each depth may have a specific angle which produces a full polarized reflection. In the device, the incident light source may be a polarized light source or unpolarized light source. The unpolarized light may be at least one of white light, light of a single wavelength, and light of multiple single wavelengths. The device may further comprise a sensor for capturing an image of the reflected or re-emitted light. The device may further comprise an optical facility for detecting reflected or re-emitted light from the skin. The device may further comprise a communication facility for transmitting the detected information. The device may further comprise a storage facility for storing information collected by the device. In the device, the reflected or re-emitted light may be at least one of polarized light and unpolarized light.

[0031] In an aspect of the invention, a method and system for determining a skin state may comprise illuminating skin with an incident light source;
detecting a characteristic of the light reflected from the skin; and determining a skin state based on at least one characteristic of the reflected or re-emitted light. In the method and system, the incident light may be directed at a selected angle alpha. Varying alpha may vary the depth of the measurement of the layers in the skin. Each depth may have a specific angle which produces a full polarized reflection. In the method and system, the incident light may be unpolarized or polarized light. The unpolarized light may be at least one of white light, light of a single wavelength, and light of multiple single wavelengths.
In the method and system, the reflected or re-emitted light may be at least one of polarized light and unpolarized light. In the method and system, the characteristic may be at least one of light source, light intensity, wavelength of light, angle of light, electrical and magnetic properties of the light, and polarization state of the light. An aspect of the polarization may be at least one of an orientation, an amplitude, a phase, an angle, a shape, a degree, and an amount. In the method and system, determining may be done using an algorithm.
The algorithm may involve artificial neural networks, non-linear regression, genetic Page 11 of 122 algorithms, fuzzy logic, or fractal and multi-fractal analysis. The method and system may further comprise filtering the reflected or re-emitted light to obtain light of a wavelength defined by the filter output. The analysis may be performed on the filtered image. In the method and system, determining may involve creating an image of the difference between reflected diffusion light and reflected polarized light. In the method and system, determining may involve comparing the aspect of the polarization of the reflected or re-emitted light to a calibration signal. In the method and system, determining may further comprise considering at least one of user input and a visual analysis.

[0032] In an aspect of the invention, a non-invasive imaging device may comprise an illumination source comprising an incident light source to direct light upon an area of concern; and a detector for detecting a characteristic of the light reflected from the area of concern. In the device, the illumination source may be positioned to direct light at a selected angle alpha. Varying alpha may vary the depth of the measurement of the layers in the skin. Each depth may have a specific angle which produces a full polarized reflection. In the device, the incident light source may be a polarized light source or unpolarized light source. The unpolarized light may be at least one of white light, light of a single wavelength, and light of multiple single wavelengths.
The device may further comprise a sensor for capturing an image of the reflected or re-emitted light.
The device may further comprise an optical facility for detecting reflected or re-emitted light from the skin. The device may further comprise a communication facility for transmitting the detected information. The device may further comprise a storage facility for storing information collected by the device. In the device, the reflected or re-emitted light may be at least one of polarized light and unpolarized light.

[0033] In an aspect the invention, a system and method may be used to determine healthy and melanocytic skin. The first, reflected spectrum and/or emission spectrum from sample which is skin malformation (SM), subtract reflected spectrum from normal healthy skin (SN). The second, from obtained resulting spectral plots (SM -SN) subtract reflected spectrum from adequate comparing screen, which represents spectral plot of the light source (SO). In that path appeared pure characteristics of change generated by skin. For differentiation between melanoma, other malignant or benign Page 12 of 122 nevus and healthy skin can be used data on maxima, minima and zero positions, in wavelength scale and data on maxima and minima intensities.

[0034] In an aspect of the invention, a system and method may comprise capturing an image of a material illuminated with incident non-angled white light and angled white light, generating a normalized red and blue color channel histogram for each image, correlating the normalized red and blue color channel histograms to a wavelength scale to obtain red and blue color channel spectral plots, and convoluting the spectral plots by subtracting the spectral plot for angled light from the spectral plot for non-angled light for each color channel to generate red and blue normalized, composite color channel spectral plots, and subtracting the normalized, composite blue channel spectral plot from the normalized, composite red channel spectral plot to generate a spectral signature for the material. In the system and method, the illumination source may be positioned to direct light at a selected angle alpha. Varying alpha varies the depth of the measurement in the material. In the system and method, the unit scale on the spectral signature may be a difference of wavelength. In the system and method, the material is inorganic and/ or organic matter. In the system and method, the spectral signature may be analyzed for at least one of number of peaks and troughs, amplitude and shape of peaks and intermediate structures and patterns. In the system and method, the spectral signature may be analyzed for metal composition, identification, purity, and strength. In the system and method, the spectral signature may be analyzed for water quality, composition, and purity.
In the system and method, elements of the spectral signature may be tagged and tracked over time in order to track changes in the characteristics of the material. In the system and method, the spectral signature may be analyzed to measure, track or monitor a skin state.
In the system and method, the spectral signature may be useful for the counterfeit analysis of money. In the system and method, the spectral signature may be analyzed for at least one of sweat gland activity and anti-perspirant effectiveness. In the system and method, the spectral signature may be analyzed for Mad Cow disease. In the system, the spectral signature may be analyzed for food, all epidermal diseases, melanoma and skin cancers, rheumatoid diseases, and all diseases that show on the skin. In the system and method, the spectral signature may be useful for monitoring post-operative cosmetic concerns. In the system and method, the spectral signature may be useful for predicting Page 13 of 122 and monitoring secretion from the mammary glands of lactating women. In the system and method, the spectral signature may be fed into a recommendation engine to provide feedback and modifications to aspects of a regimen. In the system and method, the wavelength position of ideal blue in Maxwell's color triangle is aligned with the wavelength position of ideal red in Maxwell's color triangle when convoluting the composite spectral plots to obtain the spectral signature.

[0035] A method and a system are disclosed for determining skin characteristics and cosmetic features. A minimal error output is generated. In accordance with exemplary embodiments of the present invention, according to a first aspect of the present invention, a method for determining skin characteristics and cosmetic features using color analysis may include a step of analyzing color of skin images in a pixel by pixel manner in a Red Green Blue (RGB) color system for an acquired digital image. The step of analyzing color of skin images in a pixel by pixel manner in a RGB
color system for an acquired digital image may include analyzing a picture of a part of a person's skin by generating a table of most frequent colors appearing in the picture.

[0036] According to the first aspect, a method for determining skin characteristics and cosmetic features using color analysis includes a step of generating a sample of most frequent standard RGB (sRGB) colors responsive to analyzing color of skin images in a pixel by pixel manner in the RGB color system for the acquired digital image after converting colors obtained in device dependent RGB color system into device independent standard RGB color system (sRGB). The step of generating a sample of most frequent sRGB colors responsive to analyzing color of skin images in the sRGB
color system for the acquired digital image may include preserving a plurality of sRGB
color values.

[0037] In this embodiment of the invention, the sRGB color system may be used for image analysis. Determination of other skin characteristics, melanoma, skin related tumors and skin related disorders require image analysis based on other color systems such as YIQ, YCbCr, L*a*b*, L*u*v* and HSL/HSV. The enhancement of the current algorithm may include at least one of these color systems and its/their correlation with presented sRGB analysis.

Page 14 of 122

[0038] According to the first aspect, a method for determining skin characteristics and cosmetic features using color analysis includes a step of modeling the R, G and B component color distribution with Gaussian probabilistic distribution with estimated parameters (expected value and standard deviation) on the generated sRGB
color sample for the acquired digital image further including approximating colors on the generated sRGB color samples by a Gaussian normal distribution. In accordance with an exemplary embodiment of the present invention the step of approximating colors on the generated sRGB color samples by a Gaussian normal distribution comprises approximating colors on the generated sRGB color samples by a superposition of a plurality of Gaussian normal distributions.

[0039] According to the first aspect, a method for determining skin characteristics and cosmetic features using color analysis includes a step of generating a phototype of the skin through a decision tree unit responsive to the estimated distribution model parameters colors. The phototype of the skin may be generated according to a corrected Fitzpatrick classification. In accordance with an exemplary embodiment of the present invention, the step of generating phototype of the skin according to corrected Fitzpatrick classification includes generating phototype of the skin according to a skin type scale which ranges from very fair skin to very dark skin.

[0040] According to a second aspect, a system for skin phototype determination using photograph analysis may be disclosed. The system may include an image capturing device for capturing digital images of a skin. The image capturing device may include a digital camera unit.

[0041] According to the second aspect, the system for skin phototype determination using photograph analysis may include an analyzer coupled to the image capturing device for performing a pixel by pixel analysis of a picture of a part of a person's skin. The analyzer may include a quantization device for generating a look-up table of most frequent colors appearing on the picture of the part of the person's skin.

[0042] According to the second aspect, the system for skin phototype determination using photograph analysis may include a sampling device coupled to the image capturing device for generating standard Red Green Blue (sRGB) color samples for the captured digital image of the skin.

Page 15 of 122

[0043] According to the second aspect, the system for skin phototype determination using photograph analysis may include an approximating device coupled to the sampling device for approximating the color distribution parameters on the generated sRGB color samples using the estimates of expected value and standard deviation for the captured digital image of the skin. The approximating device may include at least one Gaussian normal distribution unit.

[0044] According to the second aspect, the system for skin phototype determination using photograph analysis may include a decision tree unit coupled to the approximating device for generating a phototype of the skin using Red and Blue components of the approximated colors. The decision tree unit may include a Fitzpatrick scaling unit for categorizing a skin phototype in accordance with a skin type scale which ranges from very fair skin to very dark skin.

[0045] According to the second aspect, an exemplary embodiment of the present invention discloses a scaled Gaussian normal distribution unit for approximating colors on the generated sRGB color samples using estimates of expected value and standard deviation for the captured digital image of the skin.

[0046] According to the second aspect of the present invention, the system for skin phototype determination using photograph analysis may include a subsystem for determination of cosmetic features for a human element and a veterinary element. The cosmetic features may further include features pertaining to hair, nail and skin.

[0047] In another aspect the system may include a sampling device for generating standard Red Green Blue color samples of the captured digital image of the skin, the generated samples of standard Red Green Blue are in the range of values between 0 and 255 and they are preserved for further processing.

[0048] In another aspect the system may include an approximating device coupled to the sampling device for approximating the color distribution parameters on the generated sRGB color samples in the range of values between 0 and 255 by Gaussian normal distribution using the estimates of expected value and standard deviation for the captured digital image of the skin.

[0049] In another aspect the system may further include a decision tree unit coupled to the approximating device for generating a phototype of the skin using standard Page 16 of 122 Red and Blue components of the approximated colors, the decision tree unit with an algorithm equates estimates of expected values and standard deviation for the captured image of the skin to the Fitzpatrick notation of skin analysis for determination of skin phototype.

[0050] In another aspect the system may automatically adjust lighting intensity and wavelengths and angles in order to assess various factors of the skin.

[0051] In yet another aspect of the system skin phototype may be determined using photograph analysis for use in cosmetics and surgical industry.

[0052] In an aspect of the invention, a skin care device may include an electromagnetic radiation source capable of directing incident electromagnetic radiation to a location on the skin of a user, a radiation detector for measuring various parameters of radiation re-emitted from the location, and a skin condition analysis module coupled to the detector, the analysis module capable of generating a skin condition assessment in real-time, based partly on at least one of RGB analysis and diffused reflectance analysis of the radiation parameters. In the device, incident electromagnetic radiation may include radiation in at least one of the visible, near-infrared, and near-ultraviolet spectrum. The incident radiation may be white light. In the device, the radiation parameters include at least the degree of polarization of the re-emitted radiation. In the device, the radiation source may be a set of light emitting diodes. In the device, the skin condition assessment may be also partly based on analysis of a photographic image of a skin region surrounding the location. In the device, the device may be a miniature device.
Miniature may mean that no dimension of the detector exceeds six inches. The device may further include a memory module for storing the skin condition assessment. The device may further include a user interface. The device may further include a display surface. The skin assessment data of locations may be overlaid on an image of a larger skin region and displayed on the display surface. The device may further include an access restriction module used for restricting access to authorized users only. The access restriction module may be based on biometric access control. The device may be capable of generating alerts about abnormal skin conditions in real-time. The user interface may be operated using voice and/or eye movement commands. The device may further include a skin care regimen recommendation module that generates a displayable skin care regimen Page 17 of 122 recommendation. The skin care regimen recommendation may be based at least partly on determination of a skin profile of the user and use of skin care regimen recommendations of persons with a similar profile. The skin care regimen recommendation module may be linked to a product database. The product database may include products available in a point-of-sale location. The availability of a specific product recommended by the skin care regimen recommendation module may be indicated by an audio-visual signal.
The device may further include a skin care regimen effectiveness module that generates a displayable skin care regimen effectiveness report. The device may further include a communication module for communicating with a remote computer. The communication may occur wirelessly. The communication may occur over an internet. The remote computer may be operable by a physician. The device may be wand-shaped. The device may be wearable by the user.

[0053] In an aspect of the invention, the device an electromagnetic radiation source capable of directing incident electromagnetic radiation to a location on the skin of a user, a detector for measuring various parameters of radiation re-emitted from the location,a skin condition analysis module coupled to the detector, the analysis module capable of generating a skin condition assessment in real-time, based partly on at least one of RGB analysis and diffused reflectance analysis of the radiation parameters, and a display panel for reflecting the image of the user. In the device, the display panel may be touch-sensitive such that touching the location in a skin region image displayed in the display panel triggers display of a magnified image of the location. The skin care device may further include a camera. The camera may be integral with the display panel. The camera may be wirelessly linked to the display panel. In the device, the display panel may be a mirror. In the device, a stored image of the user may be used to automatically identify the person. The device may further include a user interface for controlling the skin care device. The user interface may be operated using voice and/or eye movement commands. The device may further include a skin care regimen recommendation module capable of generating a displayable skin care regimen recommendation. The skin care regimen recommendation may be based at least partly on determination of a skin profile of the user and use of skin care regimen recommendations of persons with a similar Page 18 of 122 profile. The device may further include a skin care regimen effectiveness module capable of generating a displayable skin care regimen effectiveness report.

[0054] In an aspect of the invention, a system and method for moving information objects available on a website to a receptacle to communicate with a plurality of people in a controlled access community network may include enabling movement of a plurality of information objects from a predetermined website to a web based network responsive to a regimen of a person, a routine of a person, a purpose of use of an information object of the plurality of information objects and a degree of affinity of a first person towards a second person, initiating at least one customized action from the actions including a drop down movement; a drag and drop movement for populating data;
and a pop-up movement in a Graphical User Interface (GUI) responsive to enabling movement of a plurality of information objects from a predetermined healthcare website, and enabling transportation of the plurality of information objects across a plurality of websites. In the system and method, the plurality of information objects may pertain to a questionnaire on at least one of a human skin condition, product information, an article, a blog posting, an image, a video, an individual message, a forum posting, and a veterinary skin condition. In the system and method, the plurality of information objects pertains to a questionnaire on human cosmetic parameters and veterinary cosmetic parameters. The questionnaire on human cosmetic parameters and veterinary cosmetic parameters may include questions on at least one of a human nail and a veterinary nail. The questionnaire on human cosmetic parameters and veterinary cosmetic parameters may include questions on at least one of a human hair and a veterinary hair. In the system and method, the purpose of use of the information object may pertain to controlling at least one of cleansing, protection, repair, moisturizing, elasticity, firmness, glow, luminosity, anti-inflammatory properties, anti-itch properties, anti-wrinkle properties, firming, exfoliating, anti-redness properties, oil controlling, anti-aging properties and shine of a human skin. In the system and method, the degree of affinity of a first person towards a second person comprises at least one of a relationship of friendship between the first person and the second person; a genetic similarity between the first person and the second person; a similarity of lifestyle between the first person and the second person; a climatic similarity between a first residential environment and a second residential environment;

Page 19 of 122 and a skin type similarity between the first person and the second person. In the system and method, the step of enabling transportation of the plurality of information objects across a plurality of websites may include a sub-step of dragging an item of user interest off a website of the plurality of websites in a predetermined format and transferring through an electronic signal to affiliates of a user accessing the website.
The affiliates of the user may be friends and relatives of the user or associated experts. In the system and method, the step of enabling movement of a plurality of information objects from a predetermined website to a web based network may include a sub-step of enabling drop down menus on the Graphical User Interface (GUI) responsive to a plurality of end user convenience and requirement parameters. In the system and device, the plurality of people in a web based network includes a plurality of people in an online friendship network. In the system and device, the plurality of people in a web based network includes a plurality of people in an online social network.

[0055] In an aspect of the invention, an interface including a social networking domain and at least one skin health assessment and recommendation unit for enabling users of the interface to perform a skin health assessment within the interface and to receive product and regimen recommendations from a recommendation engine based on a predetermined usage of health assessment and maintenance data may include a regimen tracker populated using a drag and drop facility, a rating unit for rating a plurality of healthcare facilities, and a product information unit for enabling a user to obtain product information by conducting a web based search of a plurality of web based drag and drop products, web based images and bar code scans. In the interface, the regimen tracker includes a diet tracking unit. In the interface, the plurality of healthcare facilities comprises at least one of skin cleansing, skin protection, skin moisture control, skin repair, skin elasticity, skin luminosity, skin firmness, skin wrinkles, pore size on skin, spots on skin , glow on skin, hair color, hair type, age and life stage further including marriage, pregnancy, dating and social life. In the interface, the product information comprises at least one of a product type, a product function, a product format, a product appropriateness level, a regimen information, product articles, product blogs, product safety, product toxicity, a product effectiveness index, a product cost information, and a product timeliness information. In the interface, the interface is a Page 20 of 122 multiple language and customized interface for: web based applications; mobile phone applications; touch screen applications; and personal digital assistant applications. In the interface, the interface is seamlessly coupled with a dermal imaging device for customized web based access, control and maintenance of spectral analysis of image data acquired from a degree of polarization of reflections and re-emission of incident light from skin structures. The degree of polarization of reflections and/or re-emissions of incident light from skin structures is derived from at least one of a Red Green Blue (RGB) color analysis of a plurality of digital images; and an analysis from spectroscopic data image analysis.

[0056] In an aspect of the invention, a system and method for determining a health state may include obtaining the answers to a series of subjective questions regarding health conditions, obtaining an objective health assessment report through a dermal imaging device, and generating a combination of answers to the series of subjective questions and the objective health assessment report to thereby generate a health state output and a real skin type output. In the system and method, a real skin type output is generated based on biophysical properties generated by at least one of a person seeking skin health monitoring, a spa, and a cosmetic advisor. In the system and method, the objective health assessment report may include an objective skin health assessment report on at least one of systemic hydration, skin hydration, skin firmness, skin wrinkles, pore size on skin, spots on skin, glow on skin, melanocyte, melanin, hemoglobin, porphyrin, keratin, carotene, collagen, elastin, sebum, sebaceous gland activity, sweat pore, sebaceous pore, moisture level, elasticity, luminosity, firmness, fine line, wrinkle count, pore size, percent of open pores, skin elasticity, skin tension line, spots, viscosity, epidermal, dermal sebum levels, skin color, psoriasis, allergy, red area, general skin disorder, infection, tumor, sunburn, rash, scratch, pimple, acne, insect bite, itch, bleeding, injury, inflammation, photodamage, pigmentation, tone, tattoo, percent burn, burn classification, mole, aspect of a skin lesion, melanoma, dermally observed disorder, cutaneous lesion, cellulite, boil, blistering disease, congenital dermal syndrome, cutaneous mycoses, melasma, vascular condition, rosacea, spider vein, texture, skin ulcer, wound healing, post-operative tracking, melanocytic lesion, nonmelanocytic lesion, basal cell carcinoma, seborrhoic keratosis, sebum hair color, hair type, nail condition, and age Page 21 of 122 and life stage further including marriage, pregnancy, dating and social life.
In the system and method, the objective health assessment report is sent to an end user through at least one of email, SMS, MMS, mobile phone, a graphical user interface (GUI) of an internet connected device, and a touch screen enabled personal digital assistant. The system and method may further include obtaining health assessment and maintenance data from a physiologically polarized light data. The step of obtaining health assessment and maintenance data from a physiologically polarized light data comprises obtaining health assessment and maintenance data from a Red Green Blue (RGB) color analysis device, wherein the data comprise at least one of a white light data, a blue light data, and an ultra violet light data. The step may further comprise obtaining at least one of the white light data, the blue light data, and the ultra violet light data by reading and recording conditions of at least one of the dermis and epidermis. Obtaining health assessment and maintenance data from a physiologically polarized light data comprises obtaining data pertaining to age, geography and demography for a person subjected to health monitoring.

[0057] In an aspect of the invention, a web-enabled health tracking method and system may include a camera comprising a photo guide unit for generating notes for each photograph captured, an interface coupled between the camera and a web-enabled computing system for uploading the photograph captured by the camera, a graphical user interface unit included in the web-enabled computing system for generating a frequently asked questionnaire unit further comprising a self answer guide module, a scoring module coupled to the frequently asked questionnaire unit, a comparison module coupled to the scoring module for comparing: a color parameter; a symmetry parameter;
and a border parameter, an automation unit coupled to the graphical user interface for enabling a time-based synchronization of the frequently asked questionnaire unit, the scoring module, and the comparison module, and a learning unit coupled to the automation unit for activating: a user training module, an article module coupled to the user training module, a blogging unit coupled to the user training module and the article module, and a report unit including an email unit for emailing health related information.
In the system and method, the camera comprises a tracking unit for tracking at least one of skin spots over time, laser treatment effectiveness, cellulite content in skin, condition of veins Page 22 of 122 and capillaries, botox treatment effectiveness, anti-aging treatment effectiveness, anti-acne treatment effectiveness, and a pictorial history of skin to be given to the doctor. The skin spots over time include at least one of blemishes, scars, rashes, lesions, and moles.
In the system and method, the web-enabled computing system for uploading the photograph captured by the camera further includes a walkthrough module for walking through features of a skin health record of a first time user of the system, a personal skin photo album for reviewing pictorial history of a regular user of the system, and a product quality menu for tracking product expiration dates. In the system and method, the interface for uploading the photograph further includes a reminder unit for next photo for a regular user of the system; and a cosmetic status unit coupled to the reminder unit for displaying a current usage of a cosmetic for the regular user of the system.
The current usage comprises a usage of at least one of a moisturizer, an antiseptic, a toner, a laser, and a botox. The system and method may further include a photo review unit for date based reviewing of at least one of a condition of a predetermined body part, a current usage status of a cosmetic, and a recommended usage list of cosmetics. In the system and method, the report unit may further include a secure transmission unit for sending a health assessment report to a medical practitioner, an affinity unit for discussing health assessment data with a friend, and a printing unit for printing health assessment data.

[0058] In an aspect of the invention, a mobile device-based health assessment system and method may include a photograph capturing device for capturing a skin image of a mobile device user, a transmission unit coupled with the photograph capturing device for uploading the captured skin image to a network location, a global positioning device coupled to the photograph capturing device for determining a location of the photograph capturing device, and a weather estimation device coupled to the photograph capturing device to determine a weather condition at a location of the mobile device user to thereby obtain a remote diagnosis report. In the system and method, the photograph capturing device further comprises at least one of a skin photograph assessment unit, a nail photograph assessment unit, and a hair photograph assessment unit. In the system and method, the global positioning device comprises a location tracker for answering user raised questions pertaining to geographical positioning of the user. In the system and method, the location tracker includes a database pertaining to weather intensive Page 23 of 122 cosmetics. The system and method may further include a phone number tracker for enabling a mobile device user to contact health assessment and cosmetic outlets.

[0059] In an aspect of the invention, a system and method for estimation of skin type and skin features to create a unique spectral signature may include convoluting data from a first image captured in incident diffuse white light, wherein the data relate to reflected and/or re-emitted polarized or white light, convoluting data from a second image captured in incident polarized light, wherein the data relate to reflected and/or re-emitted polarized light, comparing extreme positions of at least two unique convolutions generated by convoluting data from the first image and the second image, and determining a distance between minimum and maximum intensity positions in convoluted red minus blue spectral plots from the at least two unique convolutions for generating a numerical skin type output. In the system and method, the physiological white light comprises three spectral intervals including a width less than 100 nanometer.
The three spectral intervals pertain to red, green, and blue (RGB) colors. The three spectral intervals provide a natural white light sensation to a human eye. In the system and method, the step of comparing extreme positions of at least two unique convolutions comprises comparing a component (R-B)(W-P) for the reflected and/or re-emitted polarized light, and a component (R-B)W for the white light. The two unique convolutions in white light and polarized light further include a White Red component (WR), a White Blue component (WB), a reflected and/or re-emitted Polarized Blue component (PB) and a reflected and/or re-emitted Polarized Red component (PR).
The two unique convolutions are based on a numerical value difference correlating to medical standards. The system and method may further include a spectral convolution scheme wherein multiple combinations of subtraction of blue spectrum from red, in white light and polarized white light are determined, wherein the spectral interval is expressed in a wavelength scale interval of 100 nanometers to 300 nanometers.

[0060] In an aspect of the invention, a system and method for creating a unique spectral signature of skin features may include a RGB (Red Green Blue) color channel spectral plot generated from digital images including single wavelength light matter interaction thereby generating skin type characterization output, skin moisture conductivity and skin elasticity in numerical and descriptive standards. In the system and Page 24 of 122 method, the RGB (Red Green Blue) color channel spectral plots generated from digital images include multi-wavelength light matter interaction.

[0061] In an aspect of the invention, a system and method to track and store movement parameters of an imaging device moving over a subject area may include the steps of capturing an image of the subject area at a plurality of locations, identifying a direction of movement of the imaging device using an image processing technique for at least one captured frame, recognizing the direction of movement of the imaging device by comparing each frame with at least three distinct features captured to thereby triangulate a location of the imaging device, and comparing data of the captured image with a predetermined image database to store the image of the subject area and to store placement parameters of the imaging device. In the system and method, the step of capturing the image of the subject area at a plurality of locations comprises a sub step of capturing a continuous video image of the subject area. In the system and method, the step of capturing the image of the subject area at a plurality of locations comprises a sub step of capturing a frame by frame sequence of images of the subject area. In the system and method, the step of identifying a direction of movement of the imaging device using an image processing technique comprises a sub-step of a frame by frame comparison of the captured image to identify movement parameters of the imaging device. In the system and method, the step of recognizing the direction of movement of the imaging device by comparing each frame with at least three distinct features captured to triangulate a location of the imaging device comprises a sub-step of capturing a direction of movement of the imaging device by comparing three or more distinct positions across different frames.

[0062] In an aspect of the invention, an automated location tracking and data storage method and system for an imaging device may include an image capturing unit, a positioning unit coupled to the image capturing unit for positioning the imaging device on a subject area, and an image processing unit for enabling a frame by frame comparison of the captured image and for enabling the imaging device to capture three or more distinct points to triangulate a location of the imaging device to identify a direction of movement of the imaging device. In the system and method, the image capturing unit comprises a digital camera. In the system and method, the image capturing unit Page 25 of 122 comprises at least one of a mobile device and a Personal Digital Assistant (PDA). In the system and method, the image processing unit comprises a comparison unit for comparing positions of three or more distinct points across different frames to capture direction of movement of the imaging device. The system and method may further include a sub-system for measuring lateral motion of the image capturing unit from a predetermined point to a new location on the subject area.

[0063] In an aspect of the invention, a system and method for determining a surgical excision margin may include illuminating a melanocytic lesion skin with an incident light source, detecting a characteristic of the light reflected and/or re-emitted from the melanocytic lesion, and determining a border between the melanocytic lesion and surrounding healthy tissue based on at least one characteristic of the reflected and/or re-emitted light. In the system and method, the incident light is directed at a selected angle alpha. In the system and method, varying alpha varies the depth of the measurement of the layers in the melanocytic lesion. Each depth has a specific angle which produces a full polarized reflection. In the system and method, the incident light is unpolarized light. The unpolarized light is at least one of white light, light of a single wavelength, and light of multiple single wavelengths. In the system and method, the incident light is polarized light. In the system and method, the reflected and/or re-emitted light is at least one of polarized light and unpolarized light. In the system and method, the characteristic is at least one of light source, light intensity, wavelength of light, angle of light, electrical and magnetic properties of the light, and polarization state of the light.
An aspect of the polarization is at least one of an orientation, an amplitude, a phase, an angle, a shape, a degree, and an amount. In the system and method, determining is done using an algorithm. The algorithm involves at least one of artificial neural networks, fuzzy logic, fractal and multi-fractal analysis, non-linear regression, a genetic algorithm, white light analysis and RGB color analysis. The system and method may further include filtering the reflected and/or re-emitted light to obtain light of a wavelength defined by the filter output. Algorithmic analysis is performed on the filtered image. In the system and method, determining involves creating an image of the difference between reflected diffusion light and reflected polarized light. In the system and method, determining involves comparing the aspect of the polarization of the reflected and/or re-emitted light Page 26 of 122 to a calibration signal. In the system and method, determining further comprises considering at least one of user input and a visual analysis.

[0064] These and other systems, methods, objects, features, and advantages of the present invention will be apparent to those skilled in the art from the following detailed description of the preferred embodiment and the drawings. All documents mentioned herein are hereby incorporated in their entirety by reference.

BRIEF DESCRIPTION OF THE FIGURES

The invention and the following detailed description of certain embodiments thereof may be understood by reference to the following figures:

[0065] Fig. 1 depicts a skin care system for skin health analysis and monitoring, and skin care assessment and recommendation.

[0066] Fig. 2 depicts a mechanism for light polarization by a skin structure.

[0067] Fig. 3 depicts a process for skin care examination.

[0068] Fig. 4A & B depict a front and back view of a dermal imaging device.

[0069] Fig. 5 depicts a skin health monitoring page of a skin care system.

[0070] Fig. 6 depicts an interactive modeling tool of a skin care system.

[0071] Fig. 7 depicts a recommendations page of a skin care system.

[0072] Fig. 8 depicts a user interface of a skin care system.

[0073] Fig. 9 depicts a welcome page of a skin care system.

[0074] Fig. 10 depicts a questionnaire page of a skin care system.

[0075] Fig. 11 depicts a skin image capture page of a skin care system.

[0076] Fig. 12 depicts a results page with bar graphs of a skin care system.

[0077] Fig. 13 depicts a results page with line graphs of a skin care system.

[0078] Fig. 14 depicts a summary screen of a skin care system.

[0079] Fig. 15 depicts an elasticity summary screen of a skin care system.

[0080] Fig. 16 depicts a summary screen of a skin care system.

[0081] Fig. 17 depicts an elasticity summary screen of a skin care system.

[0082] Fig. 18 depicts a map of a user interface for a skin care system.

[0083] Fig. 19 depicts a review page of a skin care system.

Page 27 of 122

[0084] Fig. 20 depicts a review page of a skin care system.

[0085] Fig. 21 depicts a My Experience page of a skin care system.

[0086] Fig. 22 depicts a What Works page of a skin care system.

[0087] Fig. 23 depicts an Info For Me page of a skin care system.

[0088] Fig. 24 depicts an example of a skin care shelf portion of a user interface of a skin care system.

[0089] Fig. 25 depicts an example of a skin care shelf portion of a user interface of a skin care system.

[0090] Fig. 26 depicts a user interface of a skin care system.

[0091] Fig. 27 depicts a registration page of a skin care system.

[0092] Fig. 28 depicts a recommendation page of a skin care system.

[0093] Fig. 29 depicts a mobile content map for a mobile user interface of a skin care system.

[0094] Fig. 30 depicts a How Good Is This Product message flow.

[0095] Fig. 31 depicts a What Should I Look For? message flow

[0096] Fig. 32 depicts a Suncheck message flow.

[0097] Fig. 33 depicts an Alert message flow.

[0098] Fig. 34 depicts an Options message flow.

[0099] Fig. 35 depicts an algorithm and method for analyzing materials.

[00100] Fig. 36 depicts the reflection and capture of white light and reflected polarized light from a specimen based on varying angles.

[00101] Figs. 37A&B depict color coordinate systems that can be used in digital image processing.

[00102] Fig. 38 depicts a histogram of color density.

[00103] Fig. 39 depicts a normalized color channel histogram correlated to wavelength scale.

[00104] Figs. 40A&B depicts overlaid, normalized color channel histograms.

[00105] Figs. 41A&B depicts a convolution of individual color channel histograms.

[00106] Fig. 42 depicts the combination of the two convolutions of the two color channel histograms.

Page 28 of 122

[00107] Fig. 43 depicts a mathematical modeling of a portion of Maxwell's color triangle.

[00108] Figs. 44A & B depict the resulting spectral signatures for light and dark skin.

[00109] Figs. 45A - C depict the resulting spectral signatures for pure and alloy metals.

[00110] Figs. 46A & B depict the resulting spectral signatures for different types of water.

[00111] Fig. 47 depicts a block diagram of a skin care device embodiment.

[00112] Fig. 48 depicts a wand-shaped skin care device embodiment.

[00113] Fig. 49 depicts a vertical display panel including skin care device.

[00114] Fig. 50 depicts an embodiment of a wearable skin care device.

[00115] Fig. 51 depicts positive and negative intensities on a waveform as a function of emission and absorption of specific wavelengths within skin tissue.

[00116] Fig. 52 depicts the comparison between spectral signatures of healthy skin and malignant skin around a reference wavelength.

[00117] Fig. 53 depicts malignant pigmented skin in white and physiologically polarized white light.

[00118] Fig. 54 depicts the comparison of convolutions between healthy, benign and malignant skin lesions.

[00119] Fig. 55 depicts a system for tracking and targeting an image.

[00120] Fig. 56 depicts a system for determining an excision margin.

[00121] Fig. 57 depicts a system for determining an excision margin.

[00122] Fig. 58 is a flowchart illustrating a process for RGB color analysis.

[00123] Fig. 59 is a diagram depicting a pixel view of an acquired digital image of a sample of person's skin.

[00124] Fig. 60 is a diagram depicting a pixel view of the acquired digital image of a sample of person's skin after quantization.

[00125] Fig. 61 is a diagram depicting a Histogram / Distribution of standard R, G and B colors on one of the taken photographs of a patient whose skin phototype is classified as type III by Fitzpatrick, and their Gaussian normal approximation / hull.

Page 29 of 122

[00126] Fig. 62 is a diagram depicting a Histogram / Distribution of standard R, G and B colors on one of the patient's photographs whose skin phototype is classified as type VI by Fitzpatrick, and their Gaussian normal approximation / hull.

[00127] Fig. 63 is a flowchart illustrating an algorithm for determining the skin phototype according to the estimated values of mathematical expectation for R
and B
colors in a standard RGB color system.

[00128] Fig. 64 depicts an embodiment of a friend toolbar.

[00129] Fig. 65 depicts the auto-scroll feature of the friend toolbar.

[00130] Fig. 66 depicts the drag-and-drop share functionality of the friend toolbar.

[00131] Fig. 67 depicts the drag-and-drop share functionality of the friend toolbar.

[00132] Fig. 68 depicts sharing skin data as a data object with friends.

[00133] Fig. 69 depicts posting skin care data as a data object on a blog or forum where users may discuss the data.

[00134] Fig. 70 depicts sharing skin data as a data object where the data object becomes part of the content that a user may wish to discuss.

DETAILED DESCRIPTION

[00135] Provided herein may be methods, systems, and a device for dermal and non-dermal imaging. Throughout this disclosure the phrase "such as" means "such as and without limitation". Throughout this disclosure the phrase "for example"
means "for example and without limitation". Throughout this disclosure the phrase "in an example"
means "in an example and without limitation". Throughout this disclosure, the term "product" refers to any medical, non-medical, cosmetic, skin, hair, or nail care product.
Generally, any and all examples may be provided for the purpose of illustration and not limitation.

[00136] Real-time analysis of digitally captured skin-related and other information may facilitate real-time skin condition assessment, real-time skin regimen recommendation, and real-time evaluation of the effectiveness of a selected skin regimen.
Real-time analysis of digitally captured data may be performed by using a skin care Page 30 of 122 device embodying the principles of the invention disclosed herein. A skin care device embodying the principles of the invention may include, for example, an electromagnetic radiation source capable of directing incident electromagnetic radiation, a radiation detector for measuring various parameters of the re-emitted radiation, and a skin condition analysis module capable of generating a skin condition assessment in real-time.

[00137] The skin condition assessment may be cosmetic and/or medical in nature. By way of example, and in no way limiting the scope of the invention, the skin condition assessment may include any one of an acne condition assessment, a pore condition assessment, a wrinkle condition assessment, a skin elasticity assessment, a skin oiliness assessment, a skin moisture assessment, a skin luminosity assessment, a skin sebum assessment, a skin redness assessment, a skin inflammation assessment, a skin texture assessment, a skin color assessment or any combination of the listed assessments.
For example, the pore condition assessment can help in determining whether the pores are clean, open and of optimal health.

[00138] Skin-condition data may be acquired, for example, by directing incident electromagnetic radiation to a location, such as a pin-point location, on the skin of a person and detecting the re-emitted radiation from the location by using a radiation detector. The effectiveness of generating high-quality, real-time skin condition assessments may be enhanced in some embodiments by using a skin condition analysis module that bases its analysis at least partly on diffused reflectance spectroscopy. The quality of real-time skin condition assessments may be further enhanced in other embodiments by using white light as the incident radiation and by detecting the red-green-blue components of the re-emitted light.

[00139] Referring to Fig. 1, a system for skin health analysis, monitoring, and recommendation may comprise host hardware 108, such as an imaging device 108, for capturing biophysical skin properties such as in a skin health test 160, performing pre-diagnosis 162, and performing remote monitoring 164 using a light source 127;
a user interface 102 interfacing with the host hardware 108, an online platform 120, or a mobile platform 124 for capturing demographic information, additional anecdotal information on skin health, current skin care regimen 118, rankings and ratings 138 of current skin care Page 31 of 122 products and regimen, populating a skin care shelf 114, and accessing a skin cycle monitor 140, health and/or wellness information 142, games 148, a gift guide 144, a wishlist 119, a Daily Report 134, simulation tools 132, a type determination engine 130, a shopping cart 113, and the like; a host system 104 for processing and analyzing captured information such as by employing an algorithm 150, obtaining an expert consultation 128, data integration 152, and analysis tools/API's 154 to define a skin state 158; other inputs 112 to the host system 104, which may be subject to ranking/ rating feedback 138, for providing additional granularity in identifying, monitoring, and adjusting a skin state 158, such as a wearable monitor 182, a mobile communications device 184, a social network 188, product information 190, wellness information 192, a plug-in (web capture) 194, a barcode scan 198, conventional information/ questionnaire answers 101, a query/search 103, third part experts 105, third party hardware 109, third part service providers 111, and the like; and data storage 110 for storing data from the host hardware 108, host system 104, user interface 102, and other inputs 112, such as hardware 168, removable memory 170, a wireless communication device 174, a computer 178, a practitioner record 180 such as a dermatologist, general physician, aesthetician, spa employee, salon employee, cosmetic salesperson, and the like, a personalized manufacturing record 172, and the like. While dermal embodiments are contemplated throughout this disclosure, except where context prohibits such embodiments should be understood to encompass non-dermal embodiments, such as and without limitation any hair, nail, agricultural, veterinary, internal, biological and non-biological embodiments.

[00140] An imaging device 108 may be used to capture images of skin structures to obtain biophysical skin properties such as in a skin health test 160, a pre-diagnosis 162, remote monitoring 164, and the like. The imaging device 108 may also be adapted to capture images of non-dermal structures, such as hair, nails, teeth, eyes, internal organs and structures, and the like. The imaging device 108 may use an internal or external light source 127 to provide a specific sequence of irradiation using unpolarized light, such as diffusion light, white light, monochromatic light, light of multiple single wavelengths, and the like, then polarized light in order to obtain data on skin structures. In embodiments, the incident light may be polarized or unpolarized and the reflected or re-emitted light may be polarized or unpolarized. The polarized light Page 32 of 122 may result from the reflection on the skin and is not polarized from the light source. The capture and storage of the reflections enables the imaging and analysis of skin lesions, as well as all types of skin diseases, skin problems, and cosmetic concerns and indications.
Analysis of polarized reflections may enable obtaining thermal, electrical, and magnetic properties of the imaged skin area. The images may be transmitted to an analysis facility 154, analyst, practitioner and the like, which may also include assessment with patient questionnaires, to determine a final analysis of skin health. The device 108 may also employ specific targeted wavelengths, such as in the red, green, and blue areas, to identify key features, based on spectroscopic and quantitative analysis of skin lesions.
The device 108 may be used with diffused reflectance techniques, as well as with color imaging analysis based on indirect results from spectroscopic techniques (DR, SF, etc).
In embodiment, the device 108 may be adapted to emit polarized light. The device 108 may be adapted to emit more than one type of light and may be able to switch among or combine various light sources 127. The skin health analysis may be compared with a previous user skin health analysis, other users' skin health analysis, other users' experience data, and ingredient, product, and regimen characteristics to provide a recommendation for and track the effectiveness of a product or regimen 108.

[00141] Referring now to Fig. 2, in an embodiment, the imaging device 108 may comprise an illumination source 127 to direct unpolarized light, diffusion light, white light, monochromatic light, light of multiple single wavelengths, polarized light, and the like, upon the skin at an angle alpha, a sensor for detecting reflected or re-emitted light from a skin structure, and an image storage device for storing and transmitting the captured images. A skin structure may be at least one of a cell, a molecule, a group of cells, a group of molecules, an epidermis and sublayers, a basement membrane, a dermis, a subcutis, a gland, a stratum, a follicle, a pore, a vascular component, and the like resident within the skin. In an embodiment, the light source may be white light for generating reflected or re-emitted light and diffuse emission, such as polarized light, to measure the electrical and magnetic components of the skin. White light may be emitted as a combination of wavelengths of light across the spectrum of visible light.
Incident unpolarized light may be directed at its target at a defined angle `alpha' from vertical. As the value of alpha changes, such as and without limitation over a range of 0 to 90 degrees Page 33 of 122 from vertical, incident unpolarized light may interact with different structural elements of the skin since varying the angle of incidence affects the depth of penetration. The angle alpha may be changed by changing the position of the light source, either manually, through a remote control, through a user interface 102, and the like. The relationship between depth of penetration and alpha may be defined by the formula depth =
f(alpha).
For each skin structure which may correspond to a particular known depth within the skin, there may be a specific angle of incidence which produces a full polarized reflection. By analyzing the reflected or re-emitted light and/or diffuse emission, either polarized and/or diffusion, information on the underlying skin structures responsible for the reflection and/or re-emission may be obtained. The diffuse emission occurs because there is scattering and absorption that occurs from light bouncing around in the substructures. The polarization of the light may be due to classical / quantum effects of skin structures interacting water. That is, skin structures possess enough of a magnetic and electric field to be able to alter the polarization of light as it strikes the structures and to affect the wavelength of light as it strikes the structures. An aspect of the polarization of the reflected or re-emitted light, such as an orientation, an amplitude, a phase, an angle, a shape, a degree, an amount, and the like, may correlate with various measures associated with the particular skin structures targeted, and ultimately, a skin state 158.
For example, a lesion present in a particular skin structure may cause the diffusion of a portion of the reflected or re-emitted light resulting in reflected or re-emitted light that is partially polarized and partially diffused. For example, collagen structures are one indicator of a biological difference between a benign and a malignant melanocytic skin lesion. The collagenous differences may affect the polarization state of reflected or re-emitted light, and the resultant images may indicate locations of tumor center and tumor periphery. Such images may aid a practitioner in visualizing excision margins, as will be further described herein. Because melanocytes are located at the lower part of the epidermis, the appropriate wavelength may be selected for this depth as well as for the chromophores within the various types of nevi.

[00142] If incident light is polarized, only the electrical properties of skin will be apparent but unpolarized incident light may reveal both the electrical and magnetic properties of skin. While using polarized light may generate improved induction of Page 34 of 122 optical activity, the data sets generated may be of less value as compared to the data sets captured using incident unpolarized light, such as white light, a monochromatic light, light of multiple single wavelengths, and the like. By measuring the effects between 1 OE-34 and 1 OE-30 Js, one can make measurements at the border area of quantum and classical physics effects on the skin and as a difference of action of electrical and magnetic forces of valence electrons of skin's biomolecules.

[00143] In an embodiment, the wavelength and/or intensity of the incident light may be modified in order to measure the presence of specific molecules, such as collagen, elastin, cadherin, hemoglobin, and the like. Certain molecules possess the property of endogenous fluorescence. For example, if incident light is limited to a particular wavelength, such as 325 nm, collagen may be detected at an emission wavelength of 400 nm and 405 nm. Table 1 lists certain illustrative examples of excitation and emission maxima of biological molecules that exhibit endogenous fluorescence, such as amino acids, structural proteins, enzymes and coenzymes, vitamins and vitamin derivates, lipids, porphyrins, and the like. To detect the presence of specific molecules in the skin, a user may shine a light of a specified wavelength, such as and without limitation those shown in the excitation maxima column, onto the skin and collect reflected or re-emitted light to identify the presence of specific emission wavelengths in the reflections. It may be understood by one knowledgeable in the art that many different single wavelengths and combinations of wavelengths of light may be used to illuminate the skin.

Endogenous fluorescence Excitation Emission maxima (nm) maxima (nm) Amino acids Tryptophan 280 350 Tyrosine 275 300 Phenylalanine 260 280 Structural proteins Collagen 325 400, 405 Elastin 290, 325 340, 400 Enzymes and coenzymes FAD, flavins 450 535 NADH 290,351 440,460 Page 35 of 122 Endogenous fluorescence Excitation Emission maxima (nm) maxima (nm) Vitamins Vitamin A 327 510 Vitamin K 335 480 Vitamin D 390 480 Vitamin B6 compounds Pyridoxine 332,340 400 Pyridoxamine 335 400 Pyridoxal 330 385 Pyridoxic acid 315 425 Pyridoxal 5o-phosphate 330 400 Vitamin B12 275 305 Lipids Phospholipids 436 540, 560 Lipofuscin 340-395 540, 430-460 Ceroid 340-395 430-460, 540 Porphyrins 400-450 630, 690 FAD, flavin adenine dinucleotide; NADH, reduced nicotinamide adenine dinucleotide; AND(P)H, reduced nicotinamide adenine dinucleotide phosphate.

[00144] In an embodiment, light may be emitted at any wavelength, such as across the range from 280 nm to 3800 nm. Incident light may be blue, yellow, orange, red, or some other light.

[00145] Continuing to refer to Fig. 1, in an embodiment, the light source may be integral to the device 108 or provided from an associated source. The light source may be a light-emitting or laser diode (LED) of any wavelength, such as and without limitation 280, 340, 360, 385, 405, 395, 400, or 480 nm incident excitation wavelengths, as well as infrared and near-infrared. Wavelengths in the ultraviolet and infrared ranges may also be emitted by the device 108. The light source may be diffusion light, white light, monochromatic light, light of multiple single wavelengths, incandescent, electroluminescent, fluorescent, halogen, ultraviolet, polarized light, collimated light, light provided by a wireless communications device, light provided by a fiber optic cable, and the like. In an embodiment, the light source may comprise a diffuser to provide diffuse incident light.

Page 36 of 122

[00146] In an embodiment, a sensor for detecting reflected or re-emitted light from the skin may be embodied in optics resident in a CCD camera, CMOS-based imaging system, digital camera, webcam, camera embedded in a communications device such as a cell phone or iPhone, PDA (Personal Digital Assistant), a watch or other wearable device for continuous monitoring of the skin as in a sports-type indication, a third party device, a scanner, and the like. The sensor may be adapted to absorb any wavelength of light, such as near IR or visible wavelengths. The sensor may be adapted to automatically filter out particular wavelengths. The sensor may be adapted to image any size area, such as a small portion of the skin, the full face, a complete cutaneous examination, and the like. The sensor may be adapted to operate without any intervening fluids between the device 108 and the area of concern, or may be used with an oil-like application or other reflective media to the area of concern. The sensor may be adapted to detect reflected or re-emitted light, from any distance from the area or when in contact with the area of concern, which may be used for subsequent visual and/or algorithmic analysis. The images generated from this reflected or re-emitted light may be considered both visual as well as spectroscopically resolved images or electromagnetic skin maps.
The sensor may have an internal calibration scale that enables measuring the size of the region being imaged as well as the distance from the imaged area. In an embodiment, a lens may focus the reflected or re-emitted light from the detection optics onto a visible-NIR sensitive CCD, CMOS, or other sensory device. In an embodiment, the sensor may be adapted to acquire images at a high frame rate. In an embodiment, the device may possess a high magnification lens.

[00147] In an embodiment, the device 108 may store captured images for analysis and/or transmittal to an analysis facility 154. The analysis facility 154 may be a practitioner, an automated analysis tool, a practitioner employing analysis tools, and the like. Data storage 110 may occur manually when image capture is initiated, may occur automatically upon contact with the skin, may be remotely controlled, and the like. Data may be stored in an internal device memory 168 or may be stored externally in memory media 170 such as USB memory, an external hard drive, a mass storage device, and the like. The device may be able to connect externally, either through a wired connection or wirelessly, to a computer, such as a laptop, kiosk, desktop computer, central server, and Page 37 of 122 the like. For example, the connection may be a direct USB connection. When the device 108 is connected to the computer, captured data may be downloaded or transmitted either automatically or upon manual initiation from the device 108 to the computer.
For example, the device 108 may have a cradle in connection with a computer. When the device 108 is placed in the cradle, data may be transmitted or downloaded from the device 108. Additionally, the device 108 may receive software updates when connected to the computer, such as through the cradle. In embodiments, the device 108 may have no internal storage and may only be able to transmit or store data externally through a persistent hard-wired or wireless connection. Data transmittal and storage may be a fully automated process or may be manually operated. Data may be transmitted over a wireless network connection, a cellular connection, a wired connection, a Bluetooth connection, and the like. Data transmittal from the device 108 may enable remote assessment techniques. In an embodiment, non-image data may also be stored and/or transmitted by the device 108 as described herein, such as voice responses, text responses, video data, and the like. The device 108 may have an internal microphone to record audio, a video camera to record video, a keyboard input to record text responses, and the like. In an embodiment, the device 108 may use externally available audio and video.

[00148] In an embodiment, data storage may be in a skin health record 121.
The skin health record 121 may be an object or database or repository for an individual that contains information on key medical, non-medical, and cosmetic indications related to a user's skin. This may comprise images, graphics, icons, written history, personal demographic information, levels of cosmetic conditions such as moisture, elasticity, firmness, texture, color level, or non-medical conditions such as inflammation, and the like. A user may self-populate the record 121 with data from any device 108, 109 or input 112. The record 121 may contain a history of skin concerns, comments, a user blog, and the like. In an embodiment, the skin health record 121 may auto-populate upon acquisition of an image. For example, when a user submits their first image for analysis, a record 121 may be automatically created and populated with information, which may be edited, derived from the image and its analysis.

Page 38 of 122

[00149] In an embodiment, data storage 110 may occur in a practitioner record 180. A practitioner record 180 may be a repository of key health characteristics including background demographic data, personal information, information on diet, skin health record 121 and the like. It may have embedded images, links to other image data files, tracking effectiveness of personal skin products, medical products, and OTC
products and the like and their historical impact on key parameters. It may also capture community data or data of selected individuals who may be similar to the patient or user and may include rankings and comments and the like

[00150] In an embodiment, data storage 110 may be in a personalized manufacturing record 172. Based on the skin health measurement 160, product ingredients to obtain a desired effect to make the skin healthy may be selected. This ingredient selection may be achieved by analyzing and tracking the change of various skin health parameters through the application of various products and ingredients through using the device 108 and tracking the change of the skin health over time through a personalized manufacturing record 172. Once the selected product ingredients are identified, they may be mixed to create a product best suited for the individual's skin characteristics and/or desired goals (such as improved moisturization). Thus a personalized product may be developed for the user. Additionally, this same process could be used for creation of specific customized skin products and ingredients for medical and non-medical purposes and conditions.

[00151] In an embodiment, the form of the data captured may be compatible with any standard image processing and manipulation software and techniques, word processing software, slideshow presentation, spreadsheet applications, and the like. For example, the captured data may be in any suitable image format, such as jpeg, tiff, pict, png, bmp, gif, pdf, and the like. In an embodiment, multiple images may be captured as a movie or a movie may be constructed from combining multiple images.

[00152] In an embodiment, the device 108 may be powered by any suitable source, such as an electric power plug, a battery, solar power, USB power, and the like.
A user may initiate power to the device 108 in order to begin acquiring images.
Acquisition may commence automatically, may commence when the device 108 is placed Page 39 of 122 against the skin, may commence when a trigger, such as a button, is actuated by a user, and the like.

[00153] The device 108 may have a display for viewing the area to be imaged.
For example, a user may use the display with positioning tools to obtain exact images over time, such as a series of images taken over different days. The display may be integral to the device 108 or may be a separate display. For example, the device 108 may be connected to a monitor, such as that of a computer, using a wired connection or a wireless connection. In an embodiment, a user interface 102 to the device 108 may display a real time view of the imaging.

[00154] The positioning tools may enable tracking and targeting. Referring to Fig. 55, a method of tracking and targeting is depicted. The positioning tools may be used to track and store movement parameters of the imaging device 108 moving over a subject area. First, the device may capture an image of the subject area at a plurality of locations. Then, the device 108 may identify a direction of movement of the imaging device 108 using an image processing technique for at least one captured frame. The image processing technique may recognize the direction of movement of the imaging device by comparing each frame with at least three distinct features captured to thereby triangulate a location of the imaging device, as shown in Fig. 55. The data of the captured image may be compared with a predetermined image database to store the image of the subject area and to store placement parameters of the imaging device 108.
If no entry exists in the database, a new entry may be made. The step of capturing the image of the subject area at a plurality of locations may include a sub-step of capturing a continuous video image of the subject area. The step of capturing the image of the subject area at a plurality of locations may include a sub-step of capturing a frame by frame sequence of images of the subject area. The step of identifying a direction of movement of the imaging device using an image processing technique may include a sub-step of a frame by frame comparison of the captured image to identify movement parameters of the imaging device. The step of recognizing the direction of movement of the imaging device by comparing each frame with at least three distinct features captured to triangulate a location of the imaging device may include a sub-step of capturing a direction of movement of the imaging device by comparing three or more distinct Page 40 of 122 positions across different frames. The positioning tools may be an automated location tracking and data storage system for the imaging device 108, including an image capturing unit, a positioning unit coupled to the image capturing unit for positioning the imaging device on a subject area, and an image processing unit for enabling a frame by frame comparison of the captured image and for enabling the imaging device to capture three or more distinct points to triangulate a location of the imaging device to identify a direction of movement of the imaging device. The image capturing unit may include a digital camera. The image capturing unit may include at least one of a mobile device and a Personal Digital Assistant (PDA). The image processing unit may include a comparison unit for comparing positions of three or more distinct points across different frames to capture direction of movement of the imaging device. The automated location tracking and data storage system may further include a sub-system for measuring lateral motion of the image capturing unit from a predetermined point to a new location on the subject area.

[00155] In an embodiment, the device 108 may have security features in order to protect the privacy of user data. For example, the device 108 may have a unique MAC-ID with encryption technology.

[00156] In an embodiment, the device 108 may be associated with peripherals or other functional attachments. For example, the device 108 may be associated with a blood pressure monitor or sensor, a heart rate monitor or sensor, and the like. For example, the device 108 may be used to perform a pre-diagnosis 162 of a skin lesion while also monitoring other endpoints such as blood pressure, heart rate, and the like in order to assess other aspects of health in addition to skin health.

[00157] In an embodiment, the device 108 may be sized to permit a user to operate the device 108 in a handheld fashion. The device 108 may sized for portability.
The device 108 may adapted for single-handed operation. For example, the device may be embodied as in Fig. 4 A & B, but it may have multiple other embodiments in any shape and/or size, such as a mirror, a large device adapted to image a large area, a PDA, a scanner, a mobile communication device, and the like. In Fig. 4 A, the illumination source is visible as a ring of LED's around a central detection area. In both images, the size, handheld nature, and portability are clearly demonstrated. The ease of operation Page 41 of 122 enables even an inexperienced user, such as a home user connected to a laptop, to employ the device 108. The device 108 may be a self-contained unit and not part of a larger camera system. In an embodiment, the device 108 may be designed for one handed ergonomic holding. In an embodiment, the device 108 may be used with or without application of reflective media. In an embodiment, the device 108 may be used to capture images at a distance, close-up, in direct contact, and the like. For example, software loaded on a computer interfaced with the device 108 may prompt for near distance and far distance image capture.

[00158] In an embodiment, the device 108 may also be a standalone, non-hand-held version, which may be used to take images or particular body components or materials.

[00159] In some embodiments of the skin care device, the device may be a miniature one, enabling portability and hand-held use. Some embodiments of the skin care device may be in the form of a hand-held and portable wand that can be conveniently moved across a skin region to be examined. Some other embodiments of the skin care device may be so miniaturized that no dimension of the skin care device exceeds six inches. Such skin care devices may be embedded in wearable accessories, for example, bracelets, necklaces, ear-rings, and the like. Some embodiments of the skin care device may have a convenient user interface and/ or a display surface. In some embodiments of the skin care device, the device may be coupled to or embedded in a vertical display panel, for example but not limited to, a mirror, an LCD
screen, a plasma screen, and the like.

[00160] Referring to Fig. 47, an exemplary skin care device 4700 embodying the principles of the invention is shown in a block diagram. The skin care device 4700 may include an electromagnetic radiation source 4702, a radiation detector 4704, and a skin condition analysis module 4708.

[00161] The electromagnetic radiation source 4702 may be capable of directing incident electromagnetic radiation to one or more locations on the skin of a person. For example, and not by way of limitation, the radiation source 4702 may be a set of light emitting diodes (LEDs). In certain embodiments, the incident radiation emitted by the radiation source 4702 may include radiation in the visible, near-infrared (NIR) and near-Page 42 of 122 ultraviolet (NUV) spectrum. In certain other embodiments, the incident radiation may include white light.

[00162] As depicted in Fig. 47, the electromagnetic radiation source 4702 may be coupled to the radiation detector 4704. The radiation detector 4704 may be capable of detecting the radiation re-emitted from the location and measuring various radiation parameters of the re-emitted radiation. As shown in the Fig. 47, the radiation detector 4704 may be coupled to the skin condition analysis module 4708. A variety of radiation parameters may be detected by the radiation detector, including, for example but not limited to, degree of polarization, intensity of the radiation at different wave-lengths, and the like. The electromagnetic radiation sources, radiation detectors, and the skin condition analysis module have been previously described herein.

[00163] The skin condition analysis module 4708 may be capable of analyzing the radiation parameters of the reflected radiation and other information to generate a skin condition assessment. The skin condition analysis module 4708 may be adapted to generate the skin condition assessment in real-time. In some embodiments, the radiation detector 4704 measures diffused reflectance. In some other embodiments, the incident radiation may be white light and the radiation detector 4704 may measure the red, green, and blue components of the re-emitted light.

[00164] In certain embodiments, the skin condition assessment may also be partly based on analysis of a photographic image of the skin location.

[00165] As used in the specification and the appended claims, the term "diffused reflectance" may refer to radiation, sometimes loosely referred to as light, scattered in many directions from target samples. Diffused reflectance is the complement to specular, or mirror-like, reflection. If a surface is completely non-specular, the reflected or re-emitted light will be evenly spread over the hemisphere surrounding the surface. Diffused reflectance stems from tiny irregularities on surfaces of targets and is the reflection of incident light from uneven or granular surfaces of targets such that incident light strikes the targets and is scattered over wide angles.

[00166] Some embodiments of the skin care device may have a memory module for storing the skin condition assessments and other data, such as with timestamps. Some embodiments of the skin care device may have a communication Page 43 of 122 module for communicating the skin condition assessments and other data with timestamps to a remote computer. The communication of data may occur, for example, over a wire, wirelessly, using an internet, and the like. The skin condition assessments and other data may also be accessed in remote locations via mobile devices and/or computers. Such remote access may be particularly convenient for service providers, such as for example, dermatologists.

[00167] Some embodiments of the skin care device may have a user interface to enable a user to interact with the skin care device. The user interface may enable a user to give instructions to the device, for example, to analyze the available information to generate a real-time skin condition assessment of a skin location or a larger skin region. In some other embodiments, the user interface may be voice-operated providing the facility to give commands to the skin care device through speech commands.
Other examples of user interfaces that may be used in the skin care device are graphical user interface (GUI), web-based user interface (W UI), command line interface, touch interface, and any combination of the above.

[00168] In certain embodiments, the user interface may also provide alerts to a user if any abnormal skin condition, such as for example, a clogged pore, is detected. The alerts may be in the form of a light signal, a beep, an email alert, an SMS
alert, and the like. There may be other methods, such as a small electric tingle, a mark, a sound, and a light, a heat emitting signal, and the like, to alert users about skin conditions requiring user attention.

[00169] Some embodiments of the skin care device may have also have a display surface either for a more convenient and intuitive user interface and/or for viewing an image of a skin region and/or for viewing some useful skin-related information, for example, a skin condition assessment report, a skin regimen recommendation report, and/or a skin regimen effectiveness report. In some embodiments, the display surface and/or the user interface may be touch-sensitive to enable touch-control of the device.

[00170] In some embodiments, the skin condition assessment data of locations may be overlaid on an image of a larger skin region displayed on the display surface, providing a useful picture of the health of the entire skin region in a single view.

Page 44 of 122

[00171] Some embodiments of the skin care device may also have an access restriction module restricting access to patient data to authorized users only. The access restriction module may be based on a user name and password feature and/or biometric access control, for example, fingerprint recognition, facial recognition, retina recognition, and the like.

[00172] In some embodiments, the skin condition analysis module 4708 may have access to user information like age, gender, ethnic group, and the like, and such information may be used to build a user profile and used in analysis of the skin condition.

[00173] The skin care device 4700 may be used in a user's home, a user's bathroom, a cosmetic store, a provider's office, a mobile location, and the like. The skin care device 4700 may be used at any time of the day, such as before going to bed, before or after using a cleanser on the skin, and the like.

[00174] The skin care device 4700 may have a skin care regimen recommendation module 4710 capable of generating a displayable skin care regimen recommendation. The skin care regimen recommendation may include information not only about the most appropriate skin-care products, but also information about the best way of applying the product, the timing, amount, and frequency of application, and the like. The skin care regimen recommendation module 4710 may be linked to the skin condition analysis module 4708 so that the skin care regimen recommendation is personalized to the skin condition of each person. The skin care regimen recommendation may be generated in real-time based on skin condition assessments generated by the skin condition analysis module 4708, product information, and other relevant information analyzed using algorithms, as described herein. In some embodiments, the skin care regimen recommendations generated by the skin care regimen recommendation module 4710 may be displayed to the user in real-time, for example, on a display surface attached with the skin care device 4700.

[00175] In some embodiments, it may be possible to print the skin care regimen recommendations generated by the skin care regimen recommendation module 4710.

[00176] In some embodiments, the skin care regimen recommendations generated by the skin care regimen recommendation module 4710 are based at least Page 45 of 122 partly on determination of a skin profile, or skin state 158, of the user and use of skin care regimen recommendations of persons with a similar profile.

[00177] In some other embodiments, the skin care regimen recommendation module 4710 is coupled to a skin-care product database 190. If the products recommended by the skin care regimen recommendation module 4710 are available in the product database 190, the user may be informed and given an option to purchase the product immediately. In some embodiments, the user may operate the skin care device 4700 in a point-of-sale location, for example, a retail store, and the availability of a product recommended by the skin care regimen recommendation module 4710 may be indicated by an audio-visual signal, such as for example by lighting up the shelf in which the product is located.

[00178] A user practicing a specific skin care regimen, for example, use of a skin-care product in a prescribed manner, may be interested in tracking the effectiveness of the skin care regimen over a period of time. The skin care device 4700 may have a skin care regimen effectiveness module 4712. The skin care regimen effectiveness module 4712 may be coupled with the skin condition analysis module 4708. The skin condition of the user may be tracked at different points of time using the skin care device 4700 and may be displayed to the user on a display surface. The device could also help track changes by various activities - exercise, food, smoking, work, and the like.

[00179] Fig. 48 shows an embodiment of a skin care device 4700 in which the skin care device is wand-shaped. For example, a user may switch on the wand-shaped device 4800 and move the device over her face. The wand-shaped device may have a grip 4802, a radiation detector 4808, an indicator 4804 that may provide an indication such as with light, warmth, sound, and the like, an LED light 4810, and a power source 4812.

[00180] The wand-shaped device 4800 is functionally similar to the skin care device 4700 described earlier. The wand-shaped device 4800 may comprise an electromagnetic radiation source, a radiation detector, and a skin condition analysis module. The wand-shaped device 4800 may be miniature, hand-held, and portable.

[00181] In some embodiments of the wand-shaped device, the electromagnetic radiation source may be one or more LEDs. Each of the LEDs may have unique Page 46 of 122 predetermined frequencies. In some embodiments, the one or more LEDs may be arranged in a line to form a light strip.

[00182] In some embodiments, the wand-shaped device 4800 may be powered via a USB coupled to an external power source or through built-in batteries, or other similar power source.

[00183] As the wand is moved over the skin, light is emitted from the radiation source 4702. Then, the radiation detector 4704 detects re-emitted light and sends information back to the skin condition analysis module 4708. The module 4708 employs an algorithm for skin condition analysis.

[00184] Fig. 49 shows another embodiment of a vertical panel-including skin care device 4900, in which the skin care device comprises an electromagnetic radiation source 4702, a radiation detector 4704, a skin condition analysis module 4708, a user interface 4714, and a vertical display panel 4902.

[00185] The vertical display panel 4902 may have the user interface 4714 on the sides of the vertical display panel 4902. In some embodiments, the display panel may be touch-sensitive and in such cases, the vertical panel itself may be part of the user interface. An image of a skin region may be displayed in the display panel. A
user may touch a location on an image and this may trigger display of a magnified image either on the display panel or on another screen. A menu bar may show up in the user interface 4714, and the user may be able to view various reports, for example, a skin condition assessment report, a skin regimen recommendation report, a skin regimen effectiveness tracking report, and the like.

[00186] The user interface 4714 may enable a user to give instructions to the device, for example, to analyze the available information to generate a real-time skin condition assessment of a skin location or a larger skin region. In some other embodiments, the user interface may be voice-operated providing the facility to give commands to the skin care device 4900 through normal speech commands. Other examples of user interfaces that may be used in the skin care device 4900 are graphical user interface (GUI), web-based user interface (W UI), command line interface, touch interface, and any combination of the above.

Page 47 of 122

[00187] The basic functioning of the vertical panel-including skin care device 4900 is similar in many respects to the skin care device 4700. The electromagnetic radiation source 4702 is capable of directing incident electromagnetic radiation to one or more locations on the skin of a person. For example, and not by way of limitation, the radiation source 4702 may be a set of light emitting diodes (LEDs). In certain embodiments, the incident radiation emitted by the radiation source 4702 may include radiation in the visible, near-infrared (NIR) and near-ultraviolet (NUV) spectrum. In certain other embodiments, the incident radiation may include white light.

[00188] As depicted in Fig. 49, the electromagnetic radiation source 4702 may be coupled to the radiation detector 4704. A variety of radiation parameters may be detected by the radiation detector 4704, including, for example but not limited to, degree of polarization, intensity of the radiation at different wave-lengths, and the like.

[00189] In certain embodiments of the vertical panel-including skin care device, the skin condition assessment may also be partly based on analysis of a photographic image of the skin location.

[00190] Some embodiments of the vertical panel-including skin care device may have a memory module for storing the skin condition assessments and other data, such as with timestamps.

[00191] Some embodiments of the vertical panel-including skin care device may have a communication module for communicating the skin condition assessments and other data with timestamps to a remote computer. The communication of data may occur, for example but not limited to, over a wire, wirelessly, using an internet, and the like. The skin condition assessments and other data may also be accessed in remote locations via mobile devices and/or computers. Such remote access may be particularly convenient for service providers, such as for example, dermatologists.

[00192] In certain embodiments, the user interface 4714 may also provide alerts to a user if any abnormal skin condition (for example, a clogged pore) is detected.
The alerts may be in the form of a light signal, a beep, an email alert, an SMS alert, etc.
There may be other methods e.g. a small electric tingle, a mark, a sound, and a light, a heat emitting signal, etc. to alert users about skin conditions requiring user attention.

Page 48 of 122

[00193] In some embodiments, the skin condition assessment data of locations may be overlaid on an image of a larger skin region displayed on the vertical display panel 4902, providing a useful picture of the health of the entire skin region in a single view.

[00194] Some embodiments of the vertical panel-including skin care device may also have an access restriction module restricting access to private information to authorized users only. The access restriction module may be based on a user name and password feature and/or biometric access control, for example, fingerprint recognition, facial recognition, retina recognition, etc.

[00195] In some embodiments, the skin condition analysis module 4708 may have access to user information like age, gender, ethnic group, etc., and such information may be used to build a user profile and used in analysis of the skin condition.

[00196] The vertical panel-including skin care device 4900 may be used in a consumer's home, a consumer's bathroom, a cosmetic store, a provider's office and/or a mobile location. The vertical panel-including skin care device 4900 may be used at any time of the day, such as before going to bed, before or after using a cleanser on the skin.

[00197] In some embodiments of the vertical panel-including skin care device, the device may include or be coupled with a skin care regimen recommendation module capable of generating a displayable skin care regimen recommendation.

[00198] In some other embodiments of the vertical panel-including skin care device, the device may include or be coupled with a skin care regimen effectiveness module capable of generating a displayable skin care regimen effectiveness report.

[00199] In some embodiments of the vertical panel-including skin care device, the vertical display panel is a mirror.

[00200] In some embodiments of the vertical panel-including skin care device, the vertical display panel is an LCD panel or a plasma screen.

[00201] In some embodiments of the skin care device, the device also includes or is coupled with a camera for taking photographic images of a skin region.

[00202] In certain embodiments of the skin care device, the camera is integrally attached to the display surface or display panel. In certain other embodiments, Page 49 of 122 the camera is either wired to the display surface or display panel. In other embodiments, the camera is wirelessly coupled to the display surface or display panel.

[00203] In certain embodiments of the vertical panel-including skin care device, the user interface 4714 may have one or more buttons (not shown explicitly) for doing a skin scan and/or analysis. The buttons may be of different types, for example push buttons, hard wired buttons, or a combination of both. The user may touch a button on the display panel for doing a skin scan, while she may touch another button for directing the machine to do a skin analysis.

[00204] Fig. 50 shows an embodiment of a wearable skin care device 5000, in which the device is in the form of a wearable device. The wearable device can be worn by a user in the form of necklace, ear-rings, bracelets, a patch, or as a sensor attached to a strap, and the like. Such wearable devices can be persistent, personalized skin care monitors.

[00205] The wearable skincare device 5000 is functionally similar to the skin care device 4700 described earlier. Similar to the skin care device 4700, the wearable skincare device 5000 comprises an electromagnetic radiation source, a radiation detector, and a skin condition analysis module. Preferably, the wearable skincare device 5000 is miniature, hand-held, and portable, and no dimension of the device exceeds six inches.

[00206] In some embodiments of the wearable skincare device, the electromagnetic radiation source may be one or more LEDs. Each of the LEDs may have unique predetermined frequencies. In some embodiments, the one or more LEDs may be arranged in a line to form a light strip.

[00207] In some embodiments, the wearable skincare device 5000 may be powered via a USB coupled to an external power source or through built-in batteries, motion power, solar power, or other similar power source

[00208] Embodiments of the wearable skincare device may also have sensors for measuring various body and environmental parameters. Examples of body parameters that could be measured by the wearable skincare device are body temperature, hemoglobin antioxidant level, etc. Examples of environmental parameters that could be measured by the wearable skincare device are air cleanliness, humidity, temperature, UV
index, external air quality, smoke index, etc Page 50 of 122

[00209] In an embodiment, the device 108 may be adapted for use as a component of a minimally invasive medical device associated with laparoscopy, cytoscopy, ureteroscopy, arthroscopy, endoscopy, dermoscopy, gynecology, urology, dentistry, natural orifice insertion analysis such as through ears, mouth, anus, nose, and external breast cancer analysis through the skin, and the like. For example, the system may be able to process the data and to appear on a video monitor or other display in a surgical suite or other medical setting. A medical professional may be able to select a viewing mode, such as still image capture or video capture, and may be able to manually adjust the parameters of the light source, sensor and display to assist in observation, identification, and monitoring with the device 108. In an embodiment, the system may be pre-programmed with various protocols for the various types of medical procedures and tissues types that a medical professional may encounter such that the system may automatically handle the device 108 based on the medical professional's indication of the type of procedure and tissue being examined.

[00210] For example, the device 108 may be used as part of a system and method for distinguishing between healthy and suspect tissue in real or near-real time on a patient. The imaging device 108 allows a surgeon or other practitioner to precisely determine the border area around a surgical intervention for primary cutaneous melanoma, skin cancers, and other skin diseases that require excision around the skin.
Generally, the surgical excision of suspect tissue, such as cutaneous melanoma, may be determined either by a surgeon's experience or through a Breslow scale and punch biopsy that determines the thickness of a melanoma and hence generally agreed-to border areas.
The device 108 allows an automatic determination of the excision margin for primary cutaneous melanoma based on the optical characteristics of the surrounding skin. By precisely defining where there is healthy tissue and where there is suspect tissue, a surgeon could leave a larger amount of healthy tissue around a site, decrease recurrence and decrease micrometastasis in surrounding skin while enabling minimal surgical morbidity and improved cosmetic appearance. The device 108 and associated algorithms 150 and analysis techniques, such as the convolution technique and RGB color analysis discussed later herein, embodied in software, may be employed to image a particular site, and determine border area, suspect tissue, either before surgery, in pre-surgery, or during Page 51 of 122 surgery. The software could also show post surgical analysis of affected skin tissue.
Using the device 108 allows more precise determination of the border area instead of relying on subjective experience or fixed tables as noted in medical journals and other published works. The advantage of this method is better isolated suspect tissue and retaining a greater degree of healthier tissue. Referring now to Fig. 56, a melanocytic lesion is displayed. The visible melanoma 5602 or suspect tissue is surrounded by normal looking skin, but which may contain unhealthy /diseased tissue that must be excised 5604 (pseudo-normal skin 5604). The device 108 may be able to visualize the border between healthy and non healthy tissue 5608, thereby allowing the surgeon to spare healthy tissue 5610 that should remain intact. The device 108 may perform an estimation and provide an outlined area 5612 indicating where the surgeon should cut the tissue. In Fig. 57, an embodiment of a user interface for visualizing a melanocytic lesion is displayed along with access to tools for analyzing an image of the lesion 5702, manually selecting a border 5704, automatically selecting a border 5708, drawing a border area 5710, and the like.

[00211] In an embodiment, the device 108 may enable a skin health test 160.
The imaging device 108 may be used to perform a skin health test 160 to learn the characteristics of the skin and to obtain a diagnosis. The hardware device may capture an image and enable analysis of the image. The imaging components within the device 108 may enable measuring various skin health characteristics like color, age, damage, collagen, elastin, pores and types, keratin, and the like. The skin health test 160 may be performed in the home, in a spa, clinic, hospital, from a mobile phone at any location, and the like. The skin health test 160 may be used in conjunction with specific background information through questionnaires, image upload, genetic testing, DNA
samples, and lifestyle habits to determine a skin state 158. The test 160 would respond with specific information related to the biophysical health of the skin, a portion of which would be physical and genetic disposition to certain medical or non-medical or cosmetic problems or conditions.

[00212] In an embodiment, the device 108 may enable a pre-diagnosis 162.
This is a system of pre-diagnosis where a practitioner (such as the user, a dermatologist, medical practitioner, aesthetician, and the like) may receive or request from a user to take Page 52 of 122 an image and/questionnaire of a skin concern or the like and receive a pre-diagnosis based on algorithmic analysis of pre-existing conditions. The user may submit a questionnaire and image with a pre-diagnosis of conditions prior to going to see a practitioner and allow a follow-up. Images captured by the device may be submitted to obtain a preliminary diagnosis to enable effectively referring the case to the best practitioner. The pre-diagnosis 162 may be performed by software algorithms on the images, manual analysis, a combination thereof, and the like. The pre-diagnosis 162 may include the preliminary assessment as well as indicate the time required and the steps required for the final diagnosis or assessment. This pre-diagnosis 162 feature may enable effective scheduling of the practitioner. The pre-diagnosis 162 could also help screen for particular skin issues as well as identify users with certain issues.

[00213] In an embodiment, the device 108 may enable remote monitoring 164.
The user may use the device in the privacy of their home, work, or any other location to perform remote monitoring 164 and submit images to track progress of their skin's health or medical conditions. A practitioner may be able to remotely guide changes in treatment or guide on prevention factors. Remote diagnosis may greatly increase efficiency of progress monitoring since users will not have to make a physician trip to the provider, and the provider could conveniently select a time during the day to observe the patients change. The monitored data may be viewed as a recording or in real time.

[00214] In an aspect of the invention, the imaging device 108 may illuminate an area of concern at a known angle of incidence with unpolarized light. To obtain a spectral diagram based on the magnetic properties of the area only, the reflected polarized light, which possesses the electrical properties of the area of concern, may be subtracted from any reflected diffusion light, which possesses electromagnetic properties of the area of concern. The distribution of pixels in the image corresponding to the diffusion light and reflected polarized light may be determined and indicated by any conventional means. For a known image sensor, a one-to-one mapping of pixel image distribution between the diffusion light image, corresponding to an electromagnetic signal, and reflected polarized light, corresponding to an electrical signal image, may be made with a distribution of the intensity of the spectroscopic data for the same area. A
magnetic gradient image of the area may be made by equipment such as an AFM-MMR (Atomic Page 53 of 122 Force Microscopy in Magnetic Mode Regime) and from the one-to-one correspondence, a skin state 158 may be based on the gradient image, diffusion light image, and reflected polarized light image.

[00215] In an embodiment, the device 108 may be an imaging device 108 for performing digital spectroscopic imaging of the skin. Incident unpolarized light may be delivered, either vertically or on an angle alpha from vertical, from an unpolarized light source associated with the device 108, such as a white light, diffuse light, monochromatic light, light of multiple single wavelengths, and the like, to a target skin structure. White light, which possesses both electrical and magnetic properties, when incident onto a skin structure at a particular angle interacts with the structure's components and leads to the reflected or re-emitted light having a polarized light component. In embodiments, the incident light may be polarized. Unpolarized light reflected by skin structures may become polarized, at least in part. The reflected or re-emitted light, either polarized or diffusion light, may be captured by the device 108. Such multispectral skin imaging may be used to develop an electromagnetic skin topography. By measuring aspects of the polarization of the reflected or re-emitted light such as an orientation, an amplitude, a phase, an angle, a shape, a degree, and an amount, and the wavelength of the reflected or re-emitted light, the biophysical properties of skin structures may be obtained. A skin state 158 may be determined from the aggregate biophysical data obtained from one or more skin structures as well as a visual analysis of the captured images and any additional data obtained from the user anecdotally. For example, the skin state 158 may encompass data on moisture, wrinkles, pores, elasticity, luminosity, and any of a number of measures, as described herein. By varying alpha, the angle of incident white light, the depth of penetration of the light to skin structures may be varied. Each depth within the skin corresponds to different skin structures. For each skin structure or depth, there may be a specific angle which produces a full polarized reflection. For example, a certain angle of incidence may be used to obtain data for skin structures within the epidermis, however, the angle of incidence may need to be changed in order to obtain data on skin structures within the subcutis which resides at a different depth within the skin. The angle of incidence may be modified to penetrate the skin anywhere from a few microns up to a few centimeters, thus enabling the capture of reflections from other non-dermal Page 54 of 122 structures. For example, the device 108 may be used as a non-invasive imaging tool, such as to image tumors, breast cancer, melanoma, and the like. In an embodiment, the area to be imaged may be any biological tissue that may have normal or pathologic variations in its structure, such as variations in the tissue's birefringent properties. For example, scars, keloids, hypertrophic scars, and stria all have organizations of collagen fibers that are different from normal skin. Since collagen is a primary determinant of cutaneous wound repair, it may be of interest to monitor changes in collagen structure and concentration. For example, the stage of healing may be determined by the size of collagen bundles which may increase as healing progresses, by the organization of collagen structures at the molecular or small-fibril level which may increase as healing progresses, by the return or increase of birefringence, and the like. Since collagen structures are polarization-sensitive, changes that occur in the structures may be monitored using a polarization-based technique during scar formation, the healing process, and treatment of scars, as has been and will be further described herein.

[00216] Being able to measure the electrical and magnetic properties of various skin structures may enable the differentiation between healthy and non-healthy skin structures. Normal or healthy skin structures exhibit a unique conformation that differs from the conformation exhibited by equivalent structures when unhealthy or abnormal.
These conformational changes can be detected by differences in an aspect of the light reflected off of skin, re-emitted light, or amount of absorption in the skin, such as an aspect of the polarization of the reflected or re-emitted light. The aspect of polarization may be the wavelength of the light, an orientation, an amplitude, a phase, an angle, a shape, a degree, an amount of polarization of the light, and the like.
According to Maxwell's equations, light can be described as comprising an electric field and a magnetic field which can be described as two vectors, E and B, which behave as waves.
The vectors are perpendicular to the propagation direction of the light, and they are orthogonal to each other. Furthermore, given the electric field E, B can be determined via Maxwell's equations, and vice versa. Thus, by measuring the electrical component of the light reflected, re-emitted, or absorbed by the skin structures, the magnetic component or the degree of polarization/polarization state may be determined.
Alternatively, the light may spread to other wavelengths that can be measured. By comparing those Page 55 of 122 electrical and magnetic readings from the polarized component of reflected or re-emitted light and non-polarized white light to that of normal or healthy skin structures incident with light at the same or similar angles, changes may be detected in the skin structure and its molecular or structural conformation. Based on the amount or other aspect of both electrical and magnetic determination, specific defects such as cancer, skin diseases, cosmetic indications and the like, may be detected, since each range of measurements may correspond to a particular defective conformation. If any other molecules, cell, or structure are now incident with the same type of light at the same angle, the strength of certain wavelengths of the reflected component may enable the measurement of the intensity of the difference in conformation states of the measured component.
The polarization state of the reflected or re-emitted light may be described by a number of parameters. The polarization state may be described in terms of the polarization ellipse, specifically its orientation and elongation. Parameters which may be used to describe the polarization state may include the azimuth angle (w) which is the angle between the major semi-axis of the ellipse and the x-axis, the ellipticity (E) which is the ratio of the two semi-axes, the ellipticity angle which is the arctangent of the ellipticity, the eccentricity, the amplitude and phase of oscillations in two components of the electric field vector in the plane of polarization, and the like. For example, an ellipticity of zero corresponds to linear polarization and an ellipticity of 1 corresponds to circular polarization. The polarization of the reflected or re-emitted light may be at least one of elliptical, linear, circular, left-circular, right-circular and any potential combinations thereof.

[00217] In an embodiment, determining a skin state 158 may comprise processing and analyzing 154 the reflected or re-emitted light to obtain images for visual and spectroscopic analysis. Analysis 154 may be facilitated by examining the wavelength and other characteristics of the reflected or re-emitted light. For example, if the incident light is white light, the reflected or re-emitted light may be filtered to examine a collection of wavelengths or a single wavelength and, ultimately, a specific skin structure fluorescence. In another example, monochromatic or semi-monochromatic light, such as provided by an LED may be used to excite targeted fluorophores and chromophores. In this example, fluorescence of deeper layers may be extracted.
The Page 56 of 122 reflected or re-emitted light in this example may also be filtered to isolate a specific fluorescence. In another example, varying the wavelength of the illuminating light may enable detection of biophysical properties from various depths within the skin. In addition, certain chromophores, such as the various forms of hemoglobin found in blood, have specific absorption bands; thus processing of data created with different color light may yield information about chromophore distribution that may be polarization-sensitive.
The wavelength dependence may be obtained in several ways: 1) illuminate sequentially with light of a single wavelength or multiple single wavelengths and collect each resultant image separately; or 2) illuminate with white light and examine the reflected or re-emitted light for individual wavelengths or a collection of individual wavelengths either during detection or during processing. Algorithms 150 may be used to obtain information from data obtained by either method by processing and analyzing one or more wavelengths of light to form a spectroscopic, polarization-based image.
In an embodiment, the combination of both techniques may enable the elimination of the reflection from the surface of the skin.

[00218] In an embodiment, filtering may be employed to filter out a range of wavelengths, such as those belonging to the ultraviolet, infrared, near infrared, visible, and the like. The filter may be a digital or an analog filter. For example, captured images may be processed by software that may be able to employ digital filter techniques to process the images for analysis. For example, using software, any digital filter parameter may be selected such as a particular cutoff wavelength, a set of single wavelengths, a sampling interval, and the like. For example and without limitation, a digital filter may be used to isolate reflections of 405, 458, 488, 532, 580, and 633 nm wavelengths. In another example, an analog filter may be employed to filter the images as they are captured, such as a filter that is integral to the optics of the device 108, or as they are stored, transmitted, manipulated, processed, and the like, such as with an external analog filter. Filtering the images may result in obtaining images of underlying structures and/or a specific pattern of polarization. Filtering the images may result in the separation of the electrical and magnetic components of the reflected or re-emitted light.
Filtered images may be subjected to algorithmic analysis. Filtering may eliminate reflections due to skin surface reflections by isolating specific wavelengths of light. For Page 57 of 122 example, sebaceous glands may appear as bright spots in an image when only a certain wavelength of light is isolated for analysis, while isolation of a different wavelength of light enables the visualization of all the pores in the imaged area. Thus, the fluorescence from deeper layers may be isolated. Image processing may be used to count and measure changes in the sebaceous glands and pores, including count, size, activity of gland, quantity of sebum/other materials inside the sebaceous gland, quantity of sebum/other materials inside the pore, age of the contents within the gland, age of contents within the pore, amount of inflammatory processes surrounding the gland, and the like.
Multiple images from different image sources may be combined for the analysis. The analysis results in function, diagnosis, prognosis of skin health, such as disposition to acne, oilyness, shine, viscosity, and the like. The analysis may be combined with color image processing (RGB analysis, for example) to determine other skin characteristics.

[00219] In an aspect of the invention, a host system 104 may comprise algorithms 150, data integration 152, analysis tools/ API's 154, a skin state 158, an expert consult 128, and the like. The skin state 158 may be a data object or characterization of skin based on tests 160, pre-diagnoses 162, and monitoring 164 performed by a device 108, user input, expert consult 128, other inputs 112, analysis 154, algorithms 150, and the like. The skin state 158 along with all of the underlying data and user information may be stored in a skin health record 121. In an embodiment, the host system 104 may comprise server architecture. The host system may be technology agnostic. The host system 104 may comprise one or more cloud computing, service-oriented architecture, distributed objects, and the like.

[00220] In an embodiment, expert consult 128 may provide analysis, recommendations, assessment advice, and the like. The skin image data collected as well as the pre-diagnosis, in addition with any other allied data such as physician's diagnosis, insurance, blood analysis, and the like may be referred to an expert either by the user or a practitioner, or by other users to obtain an analysis, recommendation or assessment advice. Experts could be located in geographically distant locations, and may have very different skills. For example, the skin image data and analysis may be shared at the request of another user with an herbal specialist in India, or the user may request the image data to be shared with an aging expert in France to learn of best suited skin care Page 58 of 122 treatment from their experience. The expert's consultation analysis may be maintained on the host system 104 as part of the skin history record 121 and may be accessed by the user at their convenience, or shared with other users.

[00221] In an embodiment, the system 104 may be a home-based, in clinical or medical settings, at spas and salons, at a cosmetics counter and in cosmetics sales, and the like to perform skin analysis discretely and accurately in a low cost, rapid, and secure fashion. In embodiments, the device 108 may integrate with a user interface 102, online platform 129, mobile platform 124 and the like to perform analysis 154, skin state 158 record keeping, obtain referrals/analysis from a remote practitioner or algorithm 150, and the like. The home-based system 104 may allow a practitioner, who may be any qualified or unqualified person to give advice, to analyze cosmetic or non-cosmetic conditions that may be captured by an imaging device 108 or third party device 109 and give advice and recommendations on products, regimen, diet, lifestyle and the like based on inputs from questionnaires, uploaded images, and the like. The system may consist of a starter website that may be customizable for a personal business where the practitioner could organize clients' cosmetic skin health, track their regimens, recommend products, be their online advisor, and the like. This would leverage the analysis and device platform to allow a practitioner to analyze comments, images, questions, and/or concerns and the like and give advice, consultation on lifestyle improvement and tracking. A spa/
salon based system may enable personalized skin assets. For example, the spa may own the device, the device may capture images to feed a large scale display adapted to present a skin condition, and then a practitioner may be able to simulate the effect of treatment.
Users may compare a skin state 158 with peers or other spa goers and generate recommendations based on what worked for them or what they bought. Desired improvements may be correlated to ingredients and most effective products /
regimens 118 for the users' skin. The regimen 118 may be a feature that enables users to learn what product sequence would work best for their skin, based on a hardware-led personalized skin care assessment 122 and / or type determination 130 for the skin and product experience sharing via ranking and rating 138 and / or comments regarding product effectiveness and experience (e.g. smell, taste , feel, texture, color, etc.) collection. The regimen 118 may be a dynamic recommendation based on users' Page 59 of 122 collective inputs as well as experts' inputs on products that would best suit the user's individual needs.

[00222] The spa / salon based system 104 may generate product/ service recommendations based on a skin state 158, offer one-click shopping based on recommendations and enable SKU tracking, offer wellness packages such as through a contractual relationship, provide the ability to port regimen from spa to spa, from home to spa, and the like, enable optimization of regimens/ advising such as helping practitioners tailor the length of a procedure, enable development of targeted therapies, enable clear, visual communication to clients, generate effectiveness of products/services reports, and the like. Reports may be based on or comprise correlation with other users, feedback on regimen 118, modifications of a regimen 118, skin cycle monitoring, and the like. A
medical practitioner based system, such as a dermatologist, general physician, metabolist, and the like, may enable pre-diagnosis, may link to the practitioner's scheduling system, may enable pre-pricing of services, may enable follow-up tracking, and the like. A
cosmetic sales or retail based system 104 may enable integration with inventory of product enabling clearing of inventory. A handheld/portable device 108 may be used at a makeup counter, in a drugstore, at a home or trade makeup show/party, and the like.
Users may purchase peripherals/accessories for the device, such as a holster, charger, and the like. Users may pay-per-scan or may have a subscription scanning service and the like. The system 104 may be based in health clubs, gyms, resorts, and the like. A
cosmetics manufacturing / testing based system may enable skin state-based product design, targeting skin care samples to particular consumers, and the like. The system 104 may be veterinarian based to monitor veterinary dermal- and non-dermal concerns. The system 104 may be based in a hospital, ER, military setting, and the like to enable rapid assessment of medical conditions, triaging urgent skin care, and the like. The system 104 may be agriculturally based to enable application to fruits, vegetables, and other such agricultural products. The system 104 may be used in a battlefield scenario or in an austere environment, such as in space flight, air flight, underwater, submarine, and the like, to enable wound management, battlefield diagnosis and triage, and the like. The system 104 may be research based to enable comparing any materials and their specific Page 60 of 122 composition. Based on using the reading of the electrical property of the light, a user may be able to determine a similarity or difference between imaged material.

[00223] In an embodiment, determining a skin state 158 may comprise employing an analysis 154. In an embodiment, the acquired data may be analyzed by a practitioner, such as a physician, dermatologist, spa employee, clinical trial practitioner, aesthetician, cosmetologist, nutritionist, cosmetic salesperson, and the like.
The practitioner may analyze the data upon acquisition, visually, with the assistance of an algorithm 150, expert consult 128, database 115, and the like. In an embodiment, the practitioner may be remote from the location of data acquisition. In an embodiment, an algorithm 150 may be used to process and analyze 154 the reflected or re-emitted light to obtain spectroscopically resolved images, either automatically or under the control of a user, practitioner, and the like. For example, to obtain a spectroscopic image of the magnetic properties of the area only, an algorithm 150 may be used to generate an image of an area of concern using the difference between the reflected polarized light, which possesses the electrical properties of the area, and the reflected diffusion light, which possesses the electromagnetic properties of the area of concern. Algorithms 150 may be rules-based software and processes to 1) analyze imaging evidence to obtain skin health, 2) correlate skin health with ingredients, medicaments, and/or products that may be best suited for the determined skin health, 3) correlate skin health with peers in a skin health community, and 4) recommend and design personalized products based on skin health and/or other like users usage experience, 5) observe measurable changes in skin health, and the like. Algorithms 150 may be automated. Algorithms 150 may be used to analyze 154 medical concerns, such as degree of suspicion of cancer, rash analysis, and the like.
Algorithms 150 may be used to analyze 154 non-medical concerns, such as the effectiveness of a medical, non-medical, or cosmetic regimen 118, a pimple avoidance regimen 118, a sun-protection effectiveness, an itch prevention cream, and the like.
Algorithms 150 may be useful for correlating desired improvements with ingredients and most effective products for improving or maintaining the user's skin health.
The algorithm 150 may utilize a calibration scale to determine the skin structures imaged based on the angle of incidence, wavelength and intensity of the light source, an aspect of the reflected or re-emitted light, filter parameters, and the like. Algorithms 150 may be Page 61 of 122 useful for determining a dermascopic effect, a luminescence effect, a spectroscopic effect, and the like. For all algorithms 150, there may be an input, an output, and functional parameters to modulate the algorithm 150. In an embodiment, analysis 154 may comprise examining at least one of: physical data and/or an image of the material using diffusion white light; physical data and/or an image of material using light of a single wavelength or multiple single wavelengths; physical data and/or an image of the material using polarized, reflected or re-emitted light of a certain angle;
physical data and/or an image of the material generated using the difference between diffusion white light and polarized reflected or re-emitted light of a certain angle; physical data and/or an image of the material generated using the difference between light of a single or multiple wavelengths and polarized, reflected or re-emitted light of a certain angle;
and the like.
Algorithms 150 may be used with data and images generated by the device 108 or third party hardware 109. Algorithms 150 may be used with data and mages captured using any image capture device or technique, employing any kind of incident light, such as unpolarized light, polarized light, monochromatic light, diffuse light, white light, multiple single wavelength light, and the like. In embodiments, any captured data or image may be subjected to algorithmic analysis, as described herein.

[00224] In an embodiment, the algorithm 150 may be based on artificial neural networks, non-linear regression, or fuzzy logic. For example, the algorithm 150 may be used in skin lesion diagnosis based on a probabilistic framework for classification. Two kinds of data may be inputs to the neural network or to non-linear regression:
numerical data such as intensity, size, numbers, and the like, and descriptive data such as white, gray, dark, and the like. Fuzzy logic may directly encode structured descriptive data in a numerical framework. Based on associative memories, learning algorithms 150, and adaptive control system behavior, neural and fuzzy machine intelligence may enable correspondence between input data taken from collected images and a biophysical skin state 15 8.

[00225] In an embodiment, the algorithm 150 may be based on fractal and multi-fractal analysis of images based on biophysical and spatio-temporal data. Both digital image data and spectroscopic data of skin may be analyzed using Hausdorff dimensions (fractal property) and Kolmogorov's entropy (K-entropy). Then, Page 62 of 122 spectroscopic data may be divided into spatio-temporal cells and analyzed as multi-fractal objects, yielding information about a level of functional disharmony of skin structures (epidermal and dermal). Structural data of these two analyses can be correlated to determinate a one-to-one correspondence between them. Once fractal correlations between digital image data and spectroscopic data of skin are established, it may be possible to obtain information about a functional state of skin structures through multi-fractal analysis of digital image data.

[00226] In an embodiment, an algorithm 150 may be for the analysis 154 of data integrity. For example, an algorithm 150 may be able to determine if the image has been captured in high enough detail to render subsequent analyses reliable.

[00227] In an embodiment, an algorithm 150 may be useful for the analysis of skin characteristics, obtaining the biophysical properties of the skin, and determining a skin state 158. The skin state 158 may capture a combination of underlying skin structure with time-based variance. Some variation may be predictable but some may be based on a transient condition like infection, sunburn, hormonal imbalance, and the like. The algorithm 150 may be able to measure aspects such as the structure, form, concentration, number, size, state, stage, and the like of melanocytes/ melanin, hemoglobin, porphyrin, keratin, carotene, collagen, elastin, sebum, sebaceous gland activity, pores (sweat and sebaceous), wrinkles, moisture, elasticity, luminosity, all forms of the aforementioned, such as derivatives, salts, complexes, and the like. The algorithm 150 may be used to make a quantitative assessment of clinical, medical, non-medical, and cosmetic indications, such as moisture level, firmness, fine lines, wrinkle count and stage, pore size, percent of open pores, skin elasticity, skin tension lines, spots, skin color, psoriasis, allergies, red areas, general skin disorders and infections, or other skin related concerns for the user such as tumors, sunburns, rashes, scratches, pimples, acne, insect bites, itches, bleeding, injury, inflammation, photodamage, pigmentation, tone, tattoos, percent burn/ burn classification, moles (naevi, nevus), aspects of skin lesions (structure, color, dimensions/asymmetry), melanoma, dermally observed disorders and cutaneous lesions, cellulite, boils, blistering diseases, management of congenital dermal syndromes, (sub)-cutaneous mycoses, melasma, vascular conditions, rosacea, spider veins, texture, skin ulcers, wound healing, post-operative tracking, melanocytic lesions, non-melanocytic Page 63 of 122 lesions, basal cell carcinoma, seborrhoic keratosis, sebum (oiliness), nail-and/or hair-related concerns, and the like. The algorithm 150 may also be useful for the analysis of and obtaining the physical properties and composition of hair, nails, biological substances, gaseous substances, food, wine, water, liquid, metal, non-metals, plastics, polymers, and the like. Either manually or as determined by an algorithm 150, a targeted wavelength or wavelengths may be employed for specific endpoint measurements.

[00228] Either a specific wavelength or multiple wavelengths may be chosen for the incident light or a specific wavelength or wavelengths may be isolated by filtering, as described herein. An algorithm 150 may determine the presence, absence, structure, form, and the like of particular skin structures based on the properties of the reflected or re-emitted light. For example, an algorithm 150 may detect which axes/
angle the light is polarized on and compare this to signature emission spectra of individual proteins/ underlying skin structures. Each skin structure may have a unique signature pattern based on the electrical and magnetic contributions of molecule(s) present in the skin structure. The algorithms 150 may identify, analyze and separate the electrical and magnetic components of the unique polarization signal, as described herein.
The signals may correlate with the aggregate conformation state of molecules in the skin structure. By comparing this signal to a standard calibration signal, aspects of the underlying skin structures may be determined. The standard calibration signal may be provided by a catalog of skin structures/ molecules and their specific wavelength of observation. The catalog may be developed by the technique described herein or any other spectroscopic technique. For example, to determine moisture levels in the skin, an algorithm 150 may determine a ratio of the reflected polarized light and reflected diffusion light and correlate the ratio with a moisture level. Ideally, close to 100%
polarized light may be generated from reflections, however if a portion of the reflected or re-emitted light is diffusion light, such as 95% polarized, 5% diffusion, the amount of diffused light may be correlated with a level of moisture. Incident unpolarized light may interact with a skin structure and lead to varying amounts of polarization of the reflected or refracted light. This polarized reflected or refracted light strength may be measured.
This polarization may be as much as 100 percent, however, the reflected polarized strength may even be less than 100% in some cases. The incident angle and the imaged Page 64 of 122 material would help determine the maximum strength possible for the polarization of the reflected or re-emitted light. It should be understood that there may be a maximum amount of polarization with a maximum of 100% for a particular incident angle, but any amount of polarization ranging from 0 to 100% polarized may be expected from the light reflected by any skin structure. The underlying cause for the differences in reflection may be due to the ratio of the captured and free water in the skin. To determine elasticity, an algorithm 150 may determine the concentration of elastin per area of concern. To determine luminosity, an algorithm 150 may combine moisture levels and skin color into a single, objective assessment. Objective measures may be correlated with an expert grading scale or other external measure. To determine firmness/
tightness, an algorithm 150 may combine an assessment of collagen and elastin concentrations in an area of concern along with the activity of sebaceous glands (as measured by number of glands, percent open/closed, level of clog/ fill). The algorithm 150 may be able to overlay varying wavelengths and intensities and spectroscopic techniques, such as reflectance, excitation/emission, and the like. The algorithm 150 may be able to process and analyze 154 images collected by the device 108 or any other imaging device using unpolarized light, polarized light, or a combination thereof. The algorithm 150 may be able to process and analyze 154 many different types of images, such as thermoelectromagnetic (TEM) images or electromagnetic (EM) images, images collected with incident polarized light, traditional dermoscopy images, spectroscopically resolved images, conventional images, harmonized light images, and the like. The algorithm 150 may be able to calculate a variance measurement of skin state 158 over time.
Determining a skin state 158 may also include, in addition to the processing and analysis of images of the skin for various measures and endpoints as described herein, a visual analysis of the images, user entered information, and third party information, such as lifestyle, smoking history, exercise habits, diet, allergies, and the like.
For example, a user may enter anecdotal information, such as medication they may be taking, recent overexposure to sun, stage in a menstrual cycle, and the like.

[00229] Referring to Fig. 35, in an embodiment, an algorithm 150 comprise spectral convolution of digital images taken with: 1) "angled white light", or white light incident on an angle sufficient to produce a polarized reflection; and 2) "non-angled Page 65 of 122 white light", or white light incident on an angle that produces substantially no polarized reflections. While the foregoing discussion will focus on skin as the primary specimen, it should be understood that any specimen, such as material characterized by covalence effects, ionic effects, and hydrogen bond effects, including skin, hair, biological materials, foodstuffs, liquid, wine, metallic materials, non-metallic materials, and the like may be specimens for the algorithm 150. Briefly, a digital image of a specimen is captured with non-angled light 3502 and angled light 3504, blue and red color channel histograms are generated for each image 3508, 3510 and are normalized to the relative intensity of the light, and the color channel histograms are correlated to a wavelength scale 3512, 2514. The spectral convolution proceeds in two steps. The first step involves subtracting, for each of the red and blue color channels, the color channel histogram for angled light from the color channel histogram for non-angled light 3518. Two composite histograms are generated, the blue color channel composite histogram and the red color channel composite histogram. The second step of the spectral convolution involves subtracting the blue channel composite histogram from the red channel composite histogram 3520. Continuing to refer to Fig. 35 throughout the discussion of Figs. 36 through 43, the various steps of the algorithm will now be described in greater detail.

[00230] Referring now to Fig. 36, a specimen 3604, which may be any suitable material for imaging as described previously, may be illuminated with non-angled white light 3608 and angled white light 3610. As described previously herein, varying the angle of incidence affects the depth of penetration of the light to various skin structures.
For each skin structure which may correspond to a particular known depth within the skin, there may be an angle of incidence which produces a polarized reflection. By analyzing the reflected or re-emitted light, either polarized 3614 and/or diffusion 3612, captured by an imaging device 3602, information on the underlying skin structures responsible for the reflection maybe obtained. The term "angled white light"
3610 refers to incident white light that is directed towards the specimen at an angle sufficient to produce a polarized reflection. The term "non-angled white light" refers to incident white light that is not directed at a specific angle towards the specimen and is diffuse. In this case, the non-angled white light may produce reflected white light, polarized light, or a combination thereof. In an embodiment, reflected polarized light generated by non-Page 66 of 122 angled white light may be of a different characteristic than polarized light generated by angled white light.

[00231] Referring now to Fig. 37, Maxwell's color triangle, in Fig. 37B, may facilitate an understanding of the nature of white light. Maxwell's color triangle depicts the complete visible color spectrum, with reference to specific wavelengths.
In order to establish a mathematical coordinate system for the RGB color space, a simplified version is used with straight lines, shown in Fig. 37A. Each of the vertices of the outer triangle corresponds to an ideal color, either ideal green, red, or blue going clockwise from the top. Along the sides of a Maxwell triangle mixing of two of the three color components occurs with every possible proportion. As one travels from the side towards the center, the third primary color becomes increasingly important. Near the center at the "equal energy" point, E, a true white is seen, with radial axes extending to each of the three vertices. Mixing of the full intensity of red, green, and blue gives this true white. Thus, every point on the triangle is a result of a mixture of at least one of red, green, and blue, including the point representing white light. For example, the solid circle represents a point in color that is between pure/dark blue and pure white.
Similarly, the dashed circle 3704 represents a point in color that is between pure/dark red and pure white. Using digital photos of white paper, the coordinate system may be validated, as represented by the internal triangle 3708. The internal triangle 3708 validates the system when the sides are parallel to the limits of the color space lines of the original coordinate system. If they are not parallel, then the coordinate system is not valid.

[00232] Referring now to Fig. 38, an RGB histogram for each color channel is generated for each of the images. An RGB digital image has three color channels: red, green, and blue. Each of these channels may be examined and analyzed separately. A
blue color channel histogram is generated for the image taken with non-angled white light and another blue color channel histogram is generated for the image taken with angled white light. Similarly, a red color channel histogram is generated for the image taken with non-angled white light and another red color channel histogram is generated for the image taken with angled white light. For example, an automated system may be used to generate the histograms for each color channel, as shown in Fig. 38.
By simply specifying which channel 3804 a user may wish to examine, a histogram 3802 may be Page 67 of 122 generated for that channel. The histogram may be normalized to the relative intensity of the light. Normalizing the histograms to the intensity of incident light is important to be able to process the histograms generated from different images. Referring now to Fig.
39, the RGB color channel histograms are then correlated to a specific wavelength scale to generate RGB color channel spectral plots.

[00233] Referring now to Figs. 40A and B, the data from the pair of images are then combined mathematically in two steps. In the first step, the blue color channel spectral plot generated from the image taken with angled white light 4004 is subtracted from the blue color channel spectral plot generated from the image taken with non-angled white light 4002 to generate a blue color channel composite spectral plot. The two spectral plots 4002, 4004 are shown first overlaid in Fig. 40A and then subtracted in Fig.
41A. Similarly, the red color channel spectral plot generated from the image taken with angled white light 4008 is subtracted from the red color channel spectral plot generated from the image taken with non-angled white light 4010 to generate a red color channel composite spectral plot. The two spectral plots 4008, 4010 are shown first overlaid in Fig. 40B and then subtracted in Fig. 41B. Subtraction maybe facilitated by aligning the spectral plots by wavelength and mathematically subtracted the normalized intensities at each wavelength. For example, if the intensity is 0.005 at 470 nm for the blue channel spectral plot from angled white light and the intensity at the same wavelength of the blue channel spectral plot from non-angled white light is 0.003, the resultant spectral plot would comprise an intensity of -0.002 at 470 nm. The specific intensities and wavelengths in the spectral plots reflect the specific properties of the underlying material and the angle at which the material was exposed to light.

[00234] Referring now to Fig. 42, the two color channel composite, normalized spectral plots are then combined to create a unique spectral signature of the specimen.
The normalized, composite blue channel spectral plot is subtracted from the normalized, composite red channel spectral plot. The scale is determined as a difference in wavelengths between the red and blue color images, starting from the darkest point in both colors. This scale is based on the mathematical coordinate system for Maxwell's color triangle. For example, and referring to Fig. 43, the lower part of Maxwell's color triangle is shown plotted out in Fig. 43B, with arrows indicating the correspondence in Page 68 of 122 the plot with the position on the color triangle shown in Fig. 43A. Position 1 in the plot corresponds to ideal blue in Maxwell's color triangle, position 2 corresponds to true white, and position 3 corresponds to ideal red. Points 1 and 3 are aligned when convoluting the composite spectral plots to obtain the spectral signature, hence the unit scale on the convoluted histogram is a difference of wavelength (e.g. 500-400nm to 700-400nm).

[00235] The spectral signature obtained may be analyzed for a number of characteristics, such as number of peaks and troughs, amplitude and shape of peaks and intermediate structures and patterns, and the like. Various mathematical, visual, and algorithm processing techniques may be used to process and analyze the spectral signatures. The spectral signatures obtained for various specimens may be unique, for example, the spectral signature in Fig. 44A is for light skin while the spectral signature in Fig. 44B is for dark skin.

[00236] In an embodiment, the algorithm may be used for identifying metal composition, purity, strength, and the like. For example, the spectral signature may be used to distinguish between metals. The spectral signature in Fig. 45A is for a pure metal, aluminum, while the spectral signature in Fig. 45B is for an alloy of metals, PbMnTe. The spectral signature may also be used to distinguish between similar substances with different compositions. For example, the spectral signatures in Fig. 45B
and Fig. 45C are both for the PbMnTe alloy but the alloy of Fig, 45B is of a different composition as compared to the one in Fig. 45A.

[00237] In an embodiment, the algorithm 150 may be used to analyze water quality, composition, purity, and the like. For example, the spectral signature for filtered water is shown in Fig. 46A in comparison with the spectral signature for highly purified water, shown in Fig. 46B.

[00238] The spectral signature may further be enhanced by subtracting the spectral contribution attributable to the source light from the reflected light spectrum in order to normalize the spectral signature to specific skin conditions. For example the spectral signatures in Figures 51 through 54 may be normalized by subtracting the source spectral signature from the reflected light spectral signature. By subtracting the source spectral signature, the resulting spectral waveform is normalized to only the changes in Page 69 of 122 the skin from the interaction with incident light. In this way, specific type of incident light may be used which may be more amenable to detecting certain structures, compositions, or conditions. In some embodiments, a spectral signature for the subtraction of RGB histograms for angled light from non-angled light may be calculated and used to subtract from the final spectral signature for the material.

[00239] Other convolutions may be possible, such as for a yellow color channel or some other color channel. Additionally, pre-determined convolutions may also be possible.

[00240] Referring now to Figure 51, positive intensities 5101 represent a net reflection or emission at specific wavelengths based on material characteristics while negative intensities 5102 represent a net absorption from the source light's spectral signature. Negative intensity 5102 indicates no absorption of source light at specific wavelengths based on material characteristics. The source may be selected for use in examining specific biophysical or material criteria in order to produce a specific waveform for analysis.

[00241] Referring now to Figure 52, it is possible to determine changes in skin state 158 using spectral characteristics of specifically selected light sources based on specific biophysical criteria. Figure 52 shows a comparison of PB(S-O) signatures showing an example for differences between benign/healthy expected tissues and diseased tissue. Changes, such as in the 462nm-485nm range in Fig. 52, such as absorption or emission within the spectral diagram may correspond to additional changes in tissue processes, tissue activity, or presence of other molecules that indicate a changed state of skin. By measuring these changes, it is possible to determine healthy and diseased or disturbed states of the skin. The characterization of healthy tissue based on emission and or absorption may be determined at a specific reference wavelength 5209 that is based on the source light selection. For example, the spectral signature of healthy skin 5201 using a specific source light shows little or no absorption or emission in the spectral range 5205. The spectral diagram shows normal spectral characteristics 5206 right of the reference wavelength at line 5203. Additionally, characteristics in the area 5207 to the left of the reference wavelength at the line 5204 indicate diseased characteristics due to re-emission or emission 5211, while the area 5208 to the right of Page 70 of 122 the line 5204 indicates absorption 5210. The area 5207 corresponding to wavelengths 462nm-485nm shows additional activity due to additional changes in tissue processes, activity, or presence of other molecules that indicated a changed state of skin. The size and shape of peaks, troughs, curves, frequency, spacing, specific sections of wavelength differences, and the like may also correspond to concentrations of molecules, stages of disease progression, skin characteristics, and the like.

[00242] In an embodiment, the algorithm 150 may only use reflected polarized light due to increased selectivity for specific biophysical or material characteristics. For example and referring to Fig 53, the reflected polarized and/or emitted polarized light spectral signature 5302 may be much more sensitive to certain biophysical characteristics than simple white light convolution 5301. Figure 53 depicts the spectral signatures for malignant melanocytic lesions. The spectral diagram showing emission 5305 in the polarized 5302 spectral signature is much taller than the spectral diagram showing emission 5303 in the non-polarized 5301 spectral signature. Similarly, the spectral diagram showing absorption 5306 in the polarized 5302 spectral signature is much deeper than the spectral diagram showing emission 5304 in the non-polarized 5301 spectral signature.

[00243] In an embodiment, the algorithm 150 may be used to analyze healthy and non-healthy or malignant skin. For example, the spectral signatures for healthy, non-pigmented skin 5401 and 5402, healthy pigmented skin 5403 and 5404, and malignant pigmented skin 5405 and 5406 are shown in Fig. 54. Both polarized (bottom) and white light (top) spectral signature convolutions are shown for purposes of comparison. The spectral signature of normal, healthy skin 5401 and 5402 shows very little absorption or emission relative to the source light spectrum around referent wavelength 485nm.
Similarly, the healthy, benign pigmented skin lesion 5403 and 5404 shows very little absorption or emission to the left or right of the reference wavelength 485nm.
The malignant tissue, however, clearly shows absorption and emission effects around the referent wavelengths with higher amplitudes and shifting of the spectral diagram peaks and valleys.

[00244] In embodiments, these spectroscopic techniques may be useful for a variety of analytical tests where the test substrate comprises a light-sensitive component.
Page 71 of 122

[00245] In an embodiment, elements of the waveform may be tagged and tracked over time in order to track changes in the characteristics of the material or specimen, such as peaks, troughs, curves, frequency, spacing, specific sections of wavelength differences, and the like.

[00246] In an embodiment, the algorithm 150 may be incorporated for automated measurement as part of an integrated device that conducts surface analysis, such as a skin imaging device or metal testing device. In an embodiment, the algorithm 150 may be part of a remote analysis system whereby a surface imaging device may capture images and send them to a processing center where the algorithmic computations may be made.

[00247] In an embodiment, the algorithm 150 may be used for the analysis of hair in order to determine the health of hair follicles, composition, and the like.

[00248] In an embodiment, the algorithm 150 may be used for the counterfeit analysis of money. For example, a unique signature may be created for each series of appointment and/or issue.

[00249] In an embodiment, the algorithm 150 may be useful for the analysis of anti-perspirant effectiveness. In certain cases, axillary odor may be an indication of sickness or some other medical condition, such as lymphoma, apocrine gland sweating, hyperhidrosis, hydradenitis suppurativa, or other sweat related medical problems. The algorithm 150 may be useful in determining a scale of deodorant effectiveness based on an individual's specific sweat gland activity and type. The algorithm 150 may enable measuring the activity of sweat glands located in the axilla, feet, palms, and the like. The algorithmic analysis may enable the classification of sweat glands and may enable the suggestion of appropriate products/ingredients for treatment. The algorithm 150 may be able to determine the effectiveness of an anti-perspirant based on the impact on sweat gland activity.

[00250] In an embodiment, the algorithm 150 may be useful for determining a veterinary condition, such as Mad Cow disease. For example, imaging the tongue of a cow or any mucosal or dermal area where the disease may manifest may allow for the detection of a disease state using the algorithm 150. White light imaging, as described Page 72 of 122 herein, in combination with UV imaging may facilitate detection of a Mad Cow disease state.

[00251] In an embodiment, the algorithm 150 may be useful for monitoring post-operative cosmetic concerns, such as stretch mark progression and diminishment, and the like.

[00252] In an embodiment, the algorithm 150 may be useful for predicting and monitoring secretion from the mammary glands of lactating women. If milk production is predicted to be low based on the algorithmic analysis, suggestions may be made to increase milk production.

[00253] In an embodiment, an algorithm 150 for determining a skin state 158 may facilitate measuring, tracking, and monitoring a skin state 158 as well as the effectiveness of a regimen 118, topical and/or systemic therapies, avoidance routines, diet, and the like. For example, the skin state 158 may be measured at intervals and current measurements may be compared to previous measurements to determine skin health changes. As will be further described herein, the results from the algorithm 150 may feed into a recommendation engine to provide feedback and modifications to aspects of the regimen 118.

[00254] In an embodiment, an algorithm 150 for determining a skin state 158 may enable a diagnosis. The diagnosis may be an early diagnosis by distinguishing between critical and non-critical indications. For example, the algorithm 150 may be able to distinguish between a minor sunburn and a third degree sunburn requiring medical attention. Use of the device 108 to capture images enables a user to readily transmit the images to any practitioner for remote assessment, to track progression of a skin condition, rapidly compare images to previous images, other user images or third party images, such as images in a dermascopic database 115, and the like, and to make an immediate assessment with no need for historical knowledge, and the like. Historical data and the results of modeling tools 132 may be compared to the images to assist in analysis, either by an algorithm 150, a practitioner, or a practitioner employing an algorithm.
Also, in addition to images, user input in the form of audio, video, or text anecdotes describing the issue, such as a level of pain, a sensation of heat, an itchiness, and the like, may be useful in analyzing the images to determine a diagnosis. The algorithm 150 may enable non-Page 73 of 122 linear regression, such as principal component analysis (PCA), which may be a biomedical analysis used in conjunction with spectrometric analysis for analyzing medical and health conditions. The algorithm 150 may enable a simple pattern analysis for diagnosis. The algorithm 150 may be able to determine the thermo- and electroconductivity conditions of skin lesions. In an embodiment, the algorithm 150 may be able to diagnose a melanocytic lesion by examining the images for the relationship of changes in collagen and porphyrin, as a change in collagen but not porphyrin may indicate a change from a normal lesion to a dysplastic lesion. The skin state 158 may be compared with a table of indicators for various types of lesions. In an embodiment, the algorithm 150 may be able to diagnose UV damage. UV damage may be difficult to assess from a conventional superficial view as UV damage may be present even in wrinkle-free skin. However, UV damage may be assessed by examining skin structures for an increase in melanin production; global distribution, damage and count of superficial blood vessels; change in hemoglobin count: changes in the thickness of the epidermis; changes in the quantity and global distribution of collagen, and the like. In an embodiment, diagnosis may not require processing the border of the lesion, as it may not be a key factor in final analysis of the skin lesion. In an embodiment, the algorithm 150 may be able to diagnose oral cancer.

[00255] In an embodiment, an algorithm 150 for determining a skin state 158 may enable cosmetics manufacturing validation or cutaneous clinical trials.
For example, a skin state 158 may be determined prior to medical, non-medical, skin care product or cosmetics application and a time lapse series of images may be acquired to track the medical, non-medical, skin care product, and cosmetics effectiveness.

[00256] In an embodiment, there may be methods for storing, handling, integrating, and analyzing a skin state 158. The skin state 158 may be stored in the device 108 itself, on a PC, in a central server, a salon record, an e-medicine record, a medical repository, a cosmetic clinical studies database 115, a mobile device, and the like. The device 108 may communicate with a user interface 102, an online platform 120, a mobile platform 124, and the like to upload, deliver, share, and/or port images, analysis 154, skin states 158, data, track history, user profiles, and the like, as will be further described herein. For example, a user may use a device 108 embodied in a mobile Page 74 of 122 device to capture an image of the skin and upload it to a mobile platform 124 for analysis 154 to determine a skin state 158. In response, the user may receive a personalized regimen 118 for sun protection given the user's skin state 158. Other factors that may be used to determine the regimen 118 may be the current UV Index, time of day, location, kind of sun protection product the user prefers, and the like. In the same example, the user may have already obtained a skin state 158 determination and they need not upload a new image but simply request a regimen 118 recommendation from the mobile platform 124 given the already determined and stored skin state 158. Once a skin state 158 is determined, it may be accessible by and/ or integrated with any element of the user interface 102, online platform 120, mobile platform 124 and the like. A user may choose to share the skin state 158 as part of a practitioner record 180.

[00257] In an embodiment, an algorithm 150 for determining a skin state 158 may enable an analysis of differences and similarities among peers. The algorithm 150 may determine peers of a user who may be most like them in terms of skin state 158 or other criteria such as gender, age, ethnicity, behaviors such as smoking, working outdoors, and the like, diet, regimen 118, and any other identifying factors.
The algorithm 150 may be able to interface with an online platform 120, third party database 115, or third party service provider 111 to access skin states 158 and demographic information for comparison. For example, a user may wish to know what other women in their mid-30's of the same skin color are using for foundation. By employing the algorithm 150, a user may be able to determine their own skin color, identify peers according to the search criteria, and view details on their peers' regimen 118 or the results of the specific search query 103. The algorithm 150 may enable grading of the skin relative to a peer group. Using the algorithm 150, a user's skin state 158 may be compared to a previously defined skin state 158 in order to monitor the skin state 158 over time. A user's skin state 158 may also be compared to the skin state 158 of other individuals or groups of individuals to identify peers whose skin state 158 is closest to the user. Once a peer, such as a similar individual or group, is identified, the system may display the skin care products and/or skin care regimen that is effective for the peer.
Similarly, any comparison among users may be made by the system, such as a comparison of at least one of age, gender, location, climate, skin color, ethnicity, and the Page 75 of 122 like, to identify a peer. In an embodiment, as the device 108 captures data from users and determines skin states 158, the information may be fed back into the algorithm 150 to further enhance the peer identification and product recommendation process.

[00258] In an embodiment, an algorithm 150 for determining a skin state 158 may enable prediction/simulation tools 132. Having determined a skin state 158, an algorithm 150 may be able to simulate progression of aging, simulate skin care treatment effects and skin care and cosmetic regimens 118, simulate progression of a skin condition, and the like. Referring to Fig. 6, a user may use a user interface 102 to access the simulation tools 132. In the example, the image of an entire face may be used but it should be understood that simulation tools 132 may be used to generate simulations for any size area of concern. After selecting or capturing a starting image, a user may indicate the kind of simulation they would like to perform. For example, the user may like to perform a simulation of aging only, or a simulation of aging and treatment effects.
The simulation tool 132 may return data on overall appearance, wrinkle count, elasticity, luminosity, moisture, product usage simulation, and the like. For example, the output may also include a split image with the original face on one half and a new simulated output on the other half.

[00259] In an embodiment, an algorithm 150 for determining a skin state 158 may enable skin cycle monitoring 140. By monitoring skin at determined intervals, skin conditions with a cyclical nature may be monitored, predicted, pre-empted and the like.
For example, skin conditions associated with a season, weather, pollen count, hormone level, environmental condition and the like may be identified and monitored by a skin cycle monitor 140.

[00260] In an embodiment, an algorithm 150 may be used to generate searchable and/or indexable tags to associate with images and may take advantage of image tagging. Images may be tagged with information relating to the content of the image, such as information relating to a skin state, a skin condition, a gender, an ethnicity, an age, a regimen, a treatment, and the like. The information may be gathered by algorithmic analysis, user input, visual inspection of the image, and the like. An algorithm 150 may be used to perform a search 103 using the information associated with the image as a search term. In embodiments, the information may be stored separately Page 76 of 122 from the image, such as an entry in a user profile, or may be stored in association with an image. In an embodiment, a search 103 may be performed against information or images from other users' or a third party database 115 to identify similarities or differences in images or information. For example, a user may use information to search for peers with a similar skin condition in order to determine what to expect as the condition progresses.
In another embodiment, the search 103 or query for advice or recommendation from experts may be performed against product information 190, wellness information 192, skin care regimens 118, third party experts 105, and the like. For example, a user may use information to search for product information 190 indicating an effectiveness of a product for the user's skin condition. In an embodiment, the search 103 may be performed to determine an availability of a product, an inventory of a product, a price of a product, and the like. For example, a user may use the information to search a store catalog for a specific product that may be effective for the user. In the example, the user may be pale skinned and be interested in identifying an inventory of a self-tanning product formulated specifically for pale skin. In an embodiment, the image itself may be used as a search query 103. For example, the image itself may be used to search a database 115 of skin images. In an embodiment, images and information entered into the system 104 may be leveraged to develop new algorithms 150 for enhanced diagnosis.
For example, algorithms 150 may be developed for non-skin specific diseases with dermal manifestations, such as rheumatoid arthritis.

[00261] In an embodiment, an algorithm 150 may be useful for analyzing product characteristics. For example, an algorithm 150 may be able to take product ingredients and match the product up with a projected effectiveness on a particular skin state 15 8.

[00262] In an embodiment, an algorithm 150 may use RGB color analysis.
The algorithm may employ standard RGB analysis and correlation with skin structures in determining skin phototype. The calculation of parameters for determining skin phototype is fast and the skin phototype can be found in a very short period of time using a simple skin and cosmetic parameters classification routine.

[00263] Exemplary embodiments of the present invention are directed to a method and system for determining skin characteristics and cosmetic features.
The Page 77 of 122 method and system provide a minimal error and speed efficient skin analysis.
The present technique describes a method and a system for determining a skin phototype of acquired digital image in a Red Green Blue (RGB) color system.

[00264] In an exemplary embodiment of the present invention, a method for determining skin characteristics and cosmetic features using color analysis includes a step of analyzing color of skin images in a pixel by pixel manner in a Red Green Blue (RGB) color system for an acquired digital image. The colors obtained in device dependent RGB
color system are then converted into device independent standard RGB color system (sRGB) which will be used in subsequent color analysis. The step of analyzing color of skin images in a pixel by pixel manner in a sRGB color system for an acquired digital image comprises analyzing a picture of a part of a person's skin by generating a table of most frequent colors appearing in the picture.

[00265] In this embodiment of the invention, the sRGB color system has been used for image analysis. Determination of other skin characteristics (e.g.
elasticity, melanin, oil concentration etc.), melanoma, skin related tumors and skin related disorders may require image analysis based on various color systems such as YIQ, YCbCr, L*a*b*, L*u*v* and HSL/HSV. The enhancement of the current algorithm 150 may include at least one of these color systems and its/their correlation with presented sRGB
analysis. This will most likely lead to in-depth refinement and overall accuracy of the current results as well as further embodiments of the present invention. Apart for the human skin related issues, this method of image analysis is also applicable to any content whether it be animals, products, plants or any other material whose surface needs to be analyzed by a digital image.

[00266] A method for determining skin characteristics and cosmetic features using color analysis includes a step of generating a sample of most frequent sRGB colors responsive to analyzing color of skin images in a pixel by pixel manner in the RGB color system for the acquired digital image after converting colors obtained in a device-dependent RGB color system into a device-independent standard RGB color system (sRGB). The step of generating a sample of most frequent sRGB colors responsive to analyzing color of skin images in the sRGB color system for the acquired digital image comprises preserving a plurality of sRGB color values.

Page 78 of 122

[00267] A method for determining skin characteristics and cosmetic features using color analysis includes a step of modeling the standard R, G and B
component color distribution with Gaussian probabilistic distribution with estimated parameters (expected value and standard deviation) on the generated sRGB color sample for the acquired digital image further including approximating colors on the generated sRGB
color samples by a Gaussian normal distribution. In accordance with an exemplary embodiment of the present invention the step of approximating colors on the generated sRGB color samples by a Gaussian normal distribution comprises approximating colors on the generated sRGB color samples by a superposition of a plurality of Gaussian normal distributions.

[00268] A method for determining skin characteristics and cosmetic features using color analysis includes a step of generating a phototype of the skin through a decision tree unit responsive to the estimated distribution model parameters colors. The phototype of the skin is generated according to a corrected Fitzpatrick classification, or any other applicable color classifier. In accordance with an exemplary embodiment of the present invention, the step of generating a phototype of the skin according to corrected Fitzpatrick classification includes generating a phototype of the skin according to a skin type scale which ranges from very fair skin to very dark skin.

[00269] According to an exemplary embodiment of the present invention, the system for skin phototype determination using photograph analysis includes a subsystem for determination of cosmetic features for a human element and a veterinary element. The cosmetic features further include features pertaining to hair, nail and skin.

[00270] According to an exemplary embodiment of the present invention, the image of the skin sample of a person's body can be captured by any digital camera. The acquired digital image sample of the person's skin may be analyzed in a pixel by pixel manner in the RGB color system. After the conversion of colors from a device-dependent RGB color system into a device-independent standard RGB color system (sRGB), a table of most frequent sRGB colors which appear on the image may be generated.
According to an example, the generated table may consist of 256 most frequent colors which appear on the image of the person's skin. The color samples obtained from the image may be approximated by a Gaussian normal distribution (or a (scaled) superposition of few Page 79 of 122 Gaussian normal distributions). Therefore the estimates of expected value (using weighted mean) and standard deviation (using unbiased (n-1) method as the precise expected value is unknown / estimated) for each of the acquired digital images may be evaluated. The phototype of the skin may be determined through a decision tree with the estimated expected value and standard deviation. Fitzpatrick classification may be used for categorizing a skin phototype in accordance with a skin type scale which ranges from very fair skin to very dark skin.

[00271] Referring to Fig. 58 is a flowchart 5800 illustrating a process for determining a skin phototype of an acquired digital image of a part of a person's skin is shown. The process starts at block 5810 where in an image of a part of a person's skin is captured. The image capturing device may be a digital camera or the like.
Processing flow continues to logical block 5820 wherein analysis of acquired digital image is done in a pixel by pixel manner in a RGB color system. After converting all colors from the device-dependent RGB color system into a device-independent standard RGB color system (sRGB), a table of most frequent colors which appear on the acquired digital image may be generated using a quantization technique at block 5830. In accordance with an example embodiment of the invention, at block 5840 a plurality of sRGB
color values/samples generated between a range of values 0 and 255 may be preserved for further analysis. This range of values has been proven to be more convenient for skin type determination than the one between 0 and 1. The transformation from one to another can be done simply by dividing the values with 255 and vice versa. In the next stage 5850 and 5860 approximations of colors on the samples are done by Gaussian normal distribution, at block 5860 the estimates expected value and standard deviation are evaluated. Finally at block 5870, the phototype of skin of the acquired digital image is determined according to corrected Fitzpatrick classification using decision tree.

[00272] According to an exemplary embodiment of the present invention, the decision tree may be an algorithm wherein the estimated expected value and standard deviation are equated to the values of Fitzpatrick classification/notation values in determining the phototype of the skin. The effectiveness of this approach may be seen in research regarding parametric skin distribution modeling for skin segmentation /
detection.

Page 80 of 122

[00273] Referring to Fig. 59, a diagram depicting a pixel view of an acquired digital image of a sample of person's skin is shown. The image of a sample of a person's skin is captured under white emitting light. The image may be captured by any digital camera and the like under white emitting light. An analyzer coupled to the image capturing device may analyze the acquired digital image in a pixel by pixel manner in the RGB color system. The analysis of the acquired digital image in a pixel by pixel manner in the sRGB (after RGB to sRGB color system conversion) is not only limited for determining skin phototype but also may be useful for other purposes like classification of other skin characteristics (e.g. elasticity, melanin, oil concentration etc.), melanomas and other skin tumors/disorders and the like.

[00274] Digital images captured from a sample of person's skin are usually given in the RGB color system. The present technique employing an algorithm 150 for determining skin phototype in one aspect is dependent on this color system, although device independent due to conversion to sRGB color system. The calibration of the image capturing device like digital camera or the like should be taken into consideration carefully, so that the eventual color offset could be corrected. The color offset correction in the present technique can be implemented from any known techniques in the previous art and color offset correction can also be implemented in software used in the present technique in determining skin phototype.

[00275] Referring to Fig. 60, a diagram depicting a pixel view of the acquired digital image of a part of person's skin after quantization is shown. The image of the sample of the person's skin is captured under the white emitting light. The image may be captured by any digital camera and the like under white emitting light. The analyzer coupled to the image capturing device analyzes the acquired digital image in a pixel by pixel manner in the RGB color system. The analysis of acquired digital image in a pixel by pixel manner in the sRGB (after RGB to sRGB color system conversion) is not only limited for determining skin phototype but also may be useful for other purposes like classification of other skin characteristics (e.g. elasticity, melanin, oil concentration etc.), melanomas and other skin tumors/disorders and the like. Color quantization or color image quantization is a process that reduces the number of distinct colors used in an image, usually with the intention that the new image should be as visually similar as Page 81 of 122 possible to the original image. Color quantization is critical for displaying images with many colors on devices that can only display a limited number of colors, usually due to memory limitations, and enables efficient compression of certain types of images.

[00276] An image quantization technique may be applied to the captured image. A table of 256 most frequent colors which appear on the acquired digital image of the part of person's skin may be generated using a sampling device coupled to the analyzer. The acquired color samples from a digital image may be preserved in the sRGB color system. In accordance with an example embodiment of the present invention, the generated color samples may be preserved in their range of values between 0 and 255 in the sRGB color system. This range of values has been proven to be more convenient for skin type determination than the interval ranging between 0 and 1.

[00277] Accordingly colors on the samples may be approximated by Gaussian normal distribution (or a (scaled) superposition of few Gaussian normal distributions) through an approximating device coupled to the sampling device. Further the estimates of expected value (using weighted mean) and standard deviation (using unbiased (n-1) method as the precise expected value is unknown / estimated) for each of the acquired digital image may be calculated with approximating device coupled to the sampling device.

[00278] Usage of an algorithm 150 of the present technique is depicted in Fig.
61 and Fig. 62 and the algorithm 150 for RGB color analysis is depicted in Fig. 63.

[00279] Referring to Fig. 61, a diagram depicting a Histogram / Distribution of standard R, G and B colors on one of the taken photographs of a patient whose skin phototype is classified as type III by Fitzpatrick, and their Gaussian normal approximation / hull is shown. The relevant estimates are R (expected value of red) =171.1304 and B (expected value of blue) =135.3047 , for example. The estimates are compared with the decision tree described below for determining skin phototype. The phototype of skin is determined according to corrected Fitzpatrick classification.
Fitzpatrick Skin Typing Test questionnaire (skin type scale) which ranges from very fair (skin type I) to very dark (skin type VI) is often used to determine skin phototype.

[00280] Dermatologists use the Fitzpatrick Classification Scale to classify a person's complexion and tolerance to sunlight. In accordance with an exemplary Page 82 of 122 embodiment of the present invention, the Fitzpatrick scale classifies skin types from Ito VI.

[00281] Type I - Very white or freckled skin, always bums with sun exposure (very fair; often in people with red or blond hair and blue eyes)

[00282] Type II - White skin, usually bums with sun exposure (fair; often in people with red or blond hair and blue, green, or hazel eyes)

[00283] Type III - White or olive skin tone, sometime bums with sun exposure (fair; seen in people with any hair or eye color)

[00284] Type IV - Brown skin, rarely bums with sun exposure (common in people of Mediterranean descent)

[00285] Type V - Dark brown skin, very rarely bums with sun exposure (common in people of Middle-Eastern descent)

[00286] Type VI - Black skin, never bums with sun exposure

[00287] The images of skin are captured under white emitting light with an image capturing device, such as a digital camera, video camera or the like. An analyzer analyzes the captured image pixel by pixel of a part/sample of a person's skin. A
sampling device coupled to the analyzer generates a table of 256 most frequently occurring colors in the captured image. The acquired color samples from digital image are preserved in the sRGB color system. The generated color samples are preserved in their range of values between 0 and 255 in the sRGB color system. An approximating device coupled to the sampling device may calculate the estimates of expected value (using weighted mean) and standard deviation (using unbiased (n-1) method as the precise expected value is unknown / estimated) for each of the acquired digital images. A
decision tree coupled to the approximating device determines the skin phototype. This imaging turns out that expected values of R and B may be sufficient for determining skin phototype according to following decision tree. An exemplary embodiment of the present invention illustrates below.

Page 83 of 122 1, (uR <_MR)A(ue MBU) 2, (MRI <_ uR < MRU) A (Ma' <_ P,3 < MeU ) 3, (MRI <_ PR < MRU) A (Ma' <_ ue < MeU ) 4, (MRI <_ PR MRU) A (Me" <_ u8 <_ MeU) 5, (MRI < PR < MRU) A (Me' < P,3 < MeU ) Phototype = 6, (MRI <_ PR) A (Pe MB'U ) 1/2, (MR2'I < ,U, < MR2'u) A (Ma 2'I < ,u8 < M82'u ) 2/3, (MR 3"C 1RMRSu)A(Me3"<_!~8Mesu) 3/4, (MR4,I <- PR MR4u)A(M84I </u8 ~M84u) 4/5, (MR5,I <uR MR5u)A(M85I </u8 ~M85u) 5/6, (MR6,I <- PR MR6u)A(M86I </u8 ~M86u) Further examination, all other cases Mn'LOr' n=1,2,3,4,5,6,1/2,2/3,3/4,4/5,5/6

[00288] The values R,e have been determined from the images analyzed by using the programmed neural network.

[00289] Fig. 62 is a diagram depicting a Histogram / Distribution of R, G and B colors on one of the patient's photographs whose skin phototype is classified as type VI by Fitzpatrick, and their Gaussian normal approximation / hull. Here the relevant estimates are R (expected value of red) =189.7173 and B (expected value of blue) =103.537, in accordance with an example embodiment of the present invention.
The estimates are compared with the decision tree mentioned above for determining the phototype of the skin.

[00290] Referring to Fig. 63, a flowchart 6300 illustrating an algorithm 150 for determining the skin phototype according to the estimated values of mathematical expectation for standard R and B color in sRGB color system is shown. The flow chart describes the algorithm 150 developed in accordance with the present technique wherein the photograph of a part of person's skin is captured with a digital camera or the like under white emitting light at logical block 6310. At logical block 6320 the captured digital image is analyzed in a pixel by pixel manner in the RGB color system.
A
quantization technique is employed for analyzing the captured image in a pixel by pixel Page 84 of 122 manner in the sRGB color system at logical block 6330. The color samples obtained from the image can be approximated by a Gaussian normal distribution (or a (scaled) superposition of few Gaussian normal distributions). Therefore the estimates of expected value (using weighted mean) and standard deviation unbiased (n-1) method (as the precise expected value is unknown / estimated) for each of the acquired digital images may be evaluated. Now at logical block 6330 the phototype of the skin is determined according to the decision tree.

[00291] As will be appreciated by a person skilled in the art, the various implementations of the present technique provides a variety of advantages.
Firstly, the present technique determines skin phototype using regular low-cost digital photography equipment under standard environmental conditions. Secondly, the analysis performed on the captured digital image may be useful in recommendation of cosmetic product and medical or surgical purposes. Thirdly, the picture quantization algorithm and calculation of estimates expected value and standard deviation are fast, this makes it easier to determine skin phototype in a short span of time using a simple routine.
Fourthly, the analysis performed may be useful for classification of other skin characteristics (e.g.
elasticity, melanin, oil concentration etc.), melanomas, skin tumors or disorders and the like.

[00292] In an embodiment, new algorithm 150 development by practitioners, users, service providers 111, and the like may be enabled by a software development kit that anyone could use to develop new algorithms 150 and APIs 154 for the device 108.

[00293] Referring now to Fig. 3, in an embodiment, a process for collecting images, performing skin analysis, communicating findings and scheduling follow up, if required may commence with image capture by a user using a device 108. The user may also answer questions or provide additional details regarding a user-entered imaging, cosmetic regimen, area of concern, or the like. Using the user interface 102, the data may be communicated to an analyst 304 or a computer for analysis 154 by any communication method, such as over a network, the Internet, wirelessly, and the like. In certain embodiments, as the data are collected or communicated, a payment system 302 may be accessed by the user. In the example shown, an insurance company may access the data, however, payment may be effected or requested by any interested entity such as a one-Page 85 of 122 time payment by the user, a subscription by the user, a third party service provider 111, a platform 120,124, a practitioner, and the like. The entered data may be analyzed by the analyst, by software in real-time, by analysts assisted by software assistance, and the like.
An initial analysis may be to determine data integrity. In instances where the data do not pass the integrity test, it may be communicated back to the user. The analyst's assessment may be assisted by software that uses an algorithm to determine type of condition and/or recommended care/treatment. Historical analysis and data, and modeling tools may be used to assist the analyst's assessment. Relevant parties (company personnel, payment providers, physicians, medical personnel, users, amongst others) may receive the analysis and/or user specific details for follow up or other actions that may be required. The analysis 154 may be stored 308 by the system and/or submitted to a practitioner for approval 310. In embodiments, storage 308 may require practitioner approval 310. A test of the severity 312 may determine the selection of an appropriate method of communication with the user. If the result of the test 312 is positive, the user may be notified immediately by a preferred communication method, such as telephone, instant message, and the like. If the result of the test 312 is negative, the user may similarly be notified, however, the notification may take a less urgent route, such as by email or postal mail. In any event, the software tool may recommend an appropriate communication method and media, based on the assessment and may populate preset templates with the information/message to be communicated. In addition, notification by any means may also include a notification of practitioner availability. The analysis 154 may trigger a practitioner availability /
scheduling tool.
For example, prior to transmitting the results on severity 312 to the user, a practitioner availability may be assessed and transmitted simultaneously. The user may access availability and scheduling tools in order to obtain and confirm an appointment time.

[00294] In an embodiment, a user interface 102 for a skin analysis system 104 may be used to interface with the device 108, store images, deploy algorithms 150, track a skin state 158 by keeping track of images from any number of areas of concern, the interval between image capture, a projected next image capture date, communicate findings to a practitioner, interact with simulation tools 132, skin type determination tools 130, a skin cycle monitor 140, practitioner availability / scheduling tools, and the like.

Page 86 of 122

[00295] In embodiments, the user interface 102 may be operable as an application running on a device 108, a computer, server, kiosk, or the like, on an online platform 120, on a mobile platform 124, and the like. Any and all aspects of the user interface 102 described herein may be applicable to the user interface 102 running in any environment.

[00296] In an embodiment, the user interface 102 for the device, as will be further described herein, may be integral with the device 108, such as embodied in the keypad of a communications device or a series of buttons, switches, keys and the like disposed on the device 108, or may be external to the device 108, such as software running on a computer, on the Internet, on an intranet, on a mobile communications device, on an online platform 102, on a mobile platform 124, and the like. The user interface 102 may be used to modify a setting of the device 108, such as the magnification, light source, light intensity, wavelength of light, angle of light, electrical and magnetic properties of the light, positioning of sensor, duration of image capture, image size, data storage, data transmittal, and the like.

[00297] Referring now to Fig. 5, the user interface 102 may organize and index images captured by date, area of concern, skin state, and the like. For example and without limitation, as seen in the Fig. 5, four images captured from the same area of concern are indexed by their number within the series. In an embodiment, the user interface 102 may show in real time the field of view on the skin being imaged as well as populate the user interface 102 with the images once taken or once submitted by the user.
The user interface 102 may keep track of the first image, latest image, next image, and the like. The user interface 102 may allow users to shuffle through image s and use the images as a basis for simulation 132, as described herein. The user interface 102 may be used to set a reminder for next image capture. The user interface 102 may be used to create a report of the images and skin state 158. The user interface 102 may be used to transmit the report to a practitioner. In an embodiment, the user interface 102 may be used to launch a skin type test. In an embodiment, the user interface 102 may depict a form of a body. As a user interacts with the depiction of the body, such as with an indicating device, the portions of the body that have been imaged may be linked with the images such that the images may pop-up or be otherwise accessed. The user interface Page 87 of 122 102 may be adapted to collect data from the user in response to prompts. The user interface 102 may employ an algorithm 150 to check the integrity of the captured images.
The user interface 102 may guide the user in capturing images and providing user input in association with the images.

[00298] In an embodiment, the user interface 102 may interface with host hardware 108 or third party hardware 109. Hardware 108, 109 may comprise an imaging device that may connect with a computer, online platform 120, mobile platform 124, and the like via the user interface 102 and enable users to capture an image that enables measure various skin health, condition and type parameters. The hardware device 108,109 may be a standalone device or connect via or be embodied in a computing device of either medical or non-medical use. The user interface 102 may guide the connection process for the hardware device 108, 109. The device 108, 109 may store images, reports and recommendations generated and maintain a repository of the image, all as part of a skin health record 121. It may enable a systematic storing of the skin health record 121.
Third party hardware 109 may comprise devices such as moisture sensors, cosmetic analysis machines, dermascopes, cameras, x-ray machines, MRIs, medical record providers and software, web cameras, communication devices, and the like.
Third party hardware 109 may connect to the system 104 seamlessly to enable the user to gain a better analysis, and share such sets of data with other experts or users.

[00299] In an embodiment, the user interface 102 may enable type determination 130. Characteristics may be captured to determine the skin characteristics and the skin state 158 of the users' skin. Broad genetic parameters, such as ethnicity, skin color, location factors, environmental factors (such as pollen count, weather, etc.) , and lifestyle factors may be collected in addition to image and skin health data to determine the users' skin state 158. This skin state 158 may be correlated with product experience ranking and ratings 138 to enable providing a recommendation for most effective products.

[00300] The user interface 102 may display a regimen 118. The regimen 118 may be a feature that enables users to learn what products and product usage pattern would work best for their skin based on a hardware- or community-led personalized skin care assessment 160 and / or type determination 130 and product experience sharing via Page 88 of 122 ranking and rating 138 and / or comments regarding product effectiveness and experience (such as smell, taste, feel, texture, color, and the like). The regimen 118 may be a dynamic recommendation based on users' collective inputs as well as experts' inputs on products that would best suit the user's individual needs.

[00301] In an embodiment, the user interface 102 may enable simulation tools 132. Users may be able to upload an image and model various skin parameters (such as moisture level in skin, collagen level, age, and the like.) and observe changes in the image. Additionally, users may be able to model the impact of various products and regimens 118 (skin care, cosmetic, medical, nail care, hair care, and the like) on the image. Simulation tools 132 may enable users to view changes on the entire image or split half of the image to show a comparison of modeled change with current image. The user's images could also be automatically or manually optimized for the best look and the products or regimen 118 to obtain that look may be provided. Simulation tools 132 may also enable consumers to model the skin characteristics or state 158 of other selected users or non-users, such as celebrities, luminaries, average users, and the like.

[00302] In an embodiment, the user interface 102 may enable a daily report 134. The daily report 134 may be a report that provides the user information largely customized and most relevant to the user based on their skin state 158. The daily report 134 may list skin care regimen 118 to be followed based on the environmental and lifestyle factors relevant to the user, may indicate new product information 190, show the current skin care shelf 114 and rankings 138 or change in rankings 138, feedback from users or experts 105 on products most relevant to the user, and the like. The daily report 134 may include information about clinical trials and upcoming results, new product releases and status, events, various factors affecting the skin such as the day's weather forecast, UV index, temperature, pollen count, and the like, and other data to provide value to the user. The daily report 134 may report on whether a product is nearing its shelf life or may require replenishment based on a recommended usage protocol.
The daily report 134 may be provided to the user by the user interface 102, paper, email, SMS, RSS, video or any other communication media.

[00303] In an embodiment, the user interface 102 may enable a wishlist 134.
The wishlist 134 may be a function that a user could select and add products to a part of Page 89 of 122 the skin care shelf 114 using drag and drop functionality or other selection mechanism as they surf the web or otherwise access product information 190. They could share this function with other users, friends and/ or family so that other people could see the wish list 134. Other users could then select the products off the wish list 134 and purchase and send the product to the user.

[00304] In an embodiment, the user interface 102 may enable ranking and rating 138. Ranking and rating 138 may be performed for various product characteristics as well as on the various raters and rankers. Product experience may be collected from users in simple ranking and rating 138 format as well as textual comment data to be stored in a database. This ranking and rating 138 may be real time, and may be synthesized to show what is most relevant to the user based on like users or peers, such as users with any of the following characteristics: same age, same sex, same skin type, same ethnicity, geography, moisture levels, and the like. These ranking and ratings 138 may be dynamic ranking and ratings 138. The users may be shown either the total number of rankers / raters and/or the weighted percent score ranking or rating 138. The ranking and rating 138 may comprise any of the following characteristics: perceived effectiveness, smell, touch, feel, texture, ability to absorb product, stains left by product, ease of use, and the like. Users may also be able to upload their images and obtain effectiveness/look ranking and rating 138 for different product recommendations from other users or experts 105. For example and without limitation, a user may upload data and/or images and request rating and feedback on better products from an herbal expert in India, aging expert in Japan, and the like. Users providing ranking and rating 138 for various products may themselves be rated by other users. This may enable selection of the most effective and unbiased users and help identify potential experts 105. A small select group of highly ranked users may be offered exclusive writing / publishing and ranking / rating privileges.

[00305] In an embodiment, the user interface 102 may enable a skin cycle monitor 140. The skin cycle monitor 140 may indicate when the last image was collected and countdown to the next scan based on a time interval, such as the time required to replenish the skin or any other interval. Currently, it is believed that the skin replenishes Page 90 of 122 itself every 28 days. The skin cycle monitor 140 may take into consideration age, environmental changes, and other factors to indicate the upcoming scan schedule.

[00306] In an embodiment, the user interface 102 may enable wellness/ health 142. The user interface 102 may collect lifestyle data and also provide lifestyle (such as sleep, rest, exercise, and the like) and health (such as vitamins, food, products usage, and the like) recommendations based on the users particular skin state 158 and characteristics.
The wellness and health module 142 may enable the user to obtain a personalized best fit health and wellness schedule and regimen 118.

[00307] In an embodiment, the user interface 102 may enable games 148.
Users may be able to play games 148 that may enable users to model various products, try different hairstyles, model different hairstyles and clothes, and the like. Users may interact with other users or the computer to make the product selection a fun process.
This process could also be used to collect information on user preferences and looks.

[00308] In an embodiment, the user interface 102 may enable a gift guide 144.
Based on the user's skin state 158, personalized gift advice may be provided to others in the user's network.

[00309] In an embodiment, the user interface 102 may be embodied in touch screen user navigation. A touch screen system may be employed to enable the user to obtain a visual look and navigate to various parts of the user interface 102, such as navigate to the simulation tools 132, change picture orientation, drag and drop, and the like. Touch screen navigation may be particularly helpful as the hardware device 108 is connected to a computing platform. The user interface 102 may also enable collecting and coordinating information from other devices 109 and/or assessments, such as a dermascope, blood report, biopsy report, and the like to provide additional information for the skin record 121.

[00310] In an embodiment, the user interface 102 may enable a purchase/
sample portal. The user interface 102 may include a purchase/sample portal that may enable the user to select products and complete a purchase or request a sample to be delivered to a pre-entered address. The portal may be available in various social networking platforms 188 as well as over various computing platforms, such as an online Page 91 of 122 platform 120, mobile platform 124, computer, laptop, mobile phones, and other mobile devices, medical-use devices, and the like.

[00311] In an embodiment, the user interface 102 may enable scheduling and data sharing functionality. A user may be able to schedule online a meeting with a particular expert or practitioner and, if willing, then share a skin state 158 or specific parts of the skin record 121 and history in part or its entirety with the expert or practitioner. Ranked experts and practitioners, availability, and other criteria to aid the selection and scheduling process may be indicated to the user. Experts may also be able to share particular sets of data amongst themselves, such as among practitioners, physician to another physician, physician to spa, spa to spa, and the like.

[00312] Other inputs 112, such as devices, features and data, may be used to augment the data submitted by the user or as the primary data to obtain a personalized assessment regarding the users' beauty, cosmetic, or medical concerns related to skin, hair, nails, and the like. For example, certain devices may be available commercially off the shelf, purchased, proprietary, and the like.

[00313] In an embodiment, a wearable monitor 182 may be an input 112 to the system 104 and user interface 102. Wearable skin health monitors 182 may enable real time tracking of changes in the environment and the skins health. These devices could be worn directly on the body, or integrated into clothing, apparel and / or accessories carried by the user. An example would be a user having a device that monitors the UV
level, and provides a warning if the sun protection level accorded by a product used by the user falls below a set target level. These wearable monitors 182 may have independent user interfaces 102 or can be programmed for personalized parameters using other input devices. Wearable monitors 182 may also capture various physical parameters like heart rate, blood pressure, exercise rate, water consumption, fat counter, calorie meter, and the like. The monitors 182 may be able to assess hydration levels.

[00314] In an embodiment, a social network 188 may be an input 112 to the system 104 and user interface 102. The beauty social network 188 may be a collection of users interested in knowing and sharing information on beauty or medical concerns in a personal, private, and social interactive setting. The intent may be to create a beauty social network 188 where users invite and link to other users to discuss such concerns;

Page 92 of 122 obtain information 190, 192; perform ranking, rating, and review of products, regimens, experts, practitioners, other rankers/raters, and the like; complete purchases; access a wishlist 119; access a gift guide 144; play a game 148; review their daily report 134; and the like, all the while sharing experiences with other users in their network.

[00315] In an embodiment, product information 190 may be an input 112 to the system 104 and user interface 102. A database of product information 190 may comprise product, name, claims, manufacturer information, ranking and ratings 138, packaging information, images, usage parameters, product development history or forecast, special handling, upcoming changes, safety information, effectiveness information, smell, taste, color, texture, price, geography of manufacturing, brand information, consumer feedback and experiences, and other such parameters that may be obtained and/ or maintained to assist in the selection of the best product suited to the users' individual preferences or conditions to obtain the best beauty or medical outcome for their skin, hair, nails, and the like. Additionally, similar information on service oriented products such as massages, facials, hair toning, and the like may also be captured as well as information on procedures such as liposuction, Botox treatments, laser hair removal and other beauty, cosmetic and/ or medical procedures related to helping the user look good, improve or maintain a skin state 158, and the like. Manufacturers may register product information 190, contribute information on procedures, products in the pipeline, products in clinical trials, and the like. Users may rank and rate 138 products. A database update utility may update the database with new product information 190, store inventory, and the like.

[00316] In an embodiment, wellness information 192 may be an input 112 to the system 104 and user interface 102. Health and wellness information 192 may be captured, such as the impact of various products, primarily but not limited to non-prescription medications, supplements and other consumables that assist and maintain health and wellness (such as vitamins, protein shakes, supplements, and the like).
Additionally, information on lifestyle recommendations (such as sleep, rest, diet and exercise recommendations for particular age groups/ ethnicities, etc.) may be collected and correlated with user preferences and characteristics to enable and provide a holistic health, wellness, and beauty/ cosmetic optimal personalized solution and service.

Page 93 of 122

[00317] In an embodiment, a plug-in web capture 194 may be an input 112 to the system 104 and user interface 102. A software component-plug in for internet web browsers and basket or repository may recognize graphic objects on any browsed web page and allow the user to select, and drag-and-drop the graphic object onto a basket or repository onto a page of the web browser, such as a page comprising the skin care shelf 114. The graphic objects would be recognized through a standard reference table that would be accessed remotely or reside on the user's PC as part of the plug-in module 194, or as part of a resident software program on the computing platform. Graphic objects may include images for commercial products, such as skin care products or creams, or other objects that are part of any web e-commerce site. Once recognized, the plug-in 194 may highlight the picture, notifying the user that is it recognized, or provide additional information or reference. The plug-in 194 may also recognize brand names, trade names, generic pharmaceutical names, trademarks, and the like.

[00318] In an embodiment, barcode scan 198 may be an input 112 to the system 104 and user interface 102. Bar code information on various products may be captured to assist tracking, identification, price determination and correlation with other product information 190 for identifying similar substitute products, or other allied product information, usage recommendation, other user experience, pricing and delivery information, amongst other relevant sets of data. The bar code scanner 198 could be part of the hand held user device 108, a standalone system, a manual entry mechanism, and the like.

[00319] In an embodiment, conventional information/ questionnaires 101 may be an input 112 to the system 104 and user interface 102. Information 101 on the users and products may be captured via dynamic and static questions. Information such as age, sex, location, personal lifestyle traits, smoking habits, sleep patterns, skin dryness /
oiliness and moisture levels, product likes and dislikes, experiences with other products along parameters such as smell, taste, absorption, staining propensity, and the like may be captured in a fun manner using questions and answers, games and other interactive tools interspersed at various points of the users' interaction with the service product, system 104, or user interface 102. Information 101 may be captured directly form the user or via an intermediary, and augmented automatically via computer data population, as an output Page 94 of 122 of an algorithm 150 or by experts based on their assessment. Information 101 may be obtained by quizzes, badge- and widget-based forms, on-the-fly, through adaptive, investigative questioning, and the like. Information 101 may be obtained through questionnaires, such as How often do you go shopping?, When do you shop for cosmetics?, Where do you typically go? Why that spot?, Who do you shop with?
Why?, What do you ask your friends when asking for advice?, Where do you go for new products/ information about cosmetics?, When do you have to go to a dept store, vs.
buying online?, When would you want to know something immediately from your friends?, What do you ask from your friends?, How do you choose a mobile phone?, What do you care about menus on a cell phone?, When do you get a new cell phone?, and the like.

[00320] In an embodiment, third party experts 105 may be an input 112 to the system 104 and user interface 102. The system 104 may connect various experts such as practitioners, physicians, medical experts, aestheticians, schedulers, product ingredient experts, cosmetologists, herbal, ayurvedic and homeopathic experts, health and wellness experts, media experts, photograph enhancement experts, and the like with users and one another. Users may be able to direct questions to such experts 105 who may be located at different places geographically over the system to obtain personalized advice.
The experts 105 may be provided with users' data and characteristics collected and a record of the experts assessment may be retained in the record 121. The recommendation provided by the expert may be offered to the user for purchase / sample request, and the like.
Experts may also be able to flag certain cases or sets of data for discussion or referrals within the expert community or with users.

[00321] In an embodiment, third party hardware 109 may be an input 112 to the system 104 and user interface 102. The system may connect with various third party hardware 109, such as existing imaging solutions, camera devices, computers, lighting systems, sports devices such as pedometers, and the like.

[00322] In an embodiment, third party service providers 111 may be an input 112 to the system 104 and user interface 102. Third party service providers 111 may be integrated into the system 104 to enable users to make the best personalized product or service selection for their hair, skin, nails, and the like for medical or cosmetic / beauty Page 95 of 122 needs, and the like. Third party service providers 111 may include hospitals, physicians, spas, salons, aestheticians, beauticians, cosmetic counters, drug stores, cosmetics sales representatives and websites, ranking and rating services, product information databases, testing laboratories, magazines and information providers, insurance companies, social networking sites, health and wellness services, photograph enhancement services, and the like. For example, based on a skin concern, the scheduling system for a physician may be integrated and scheduling options offered online to users, while also connecting with insurance providers to confirm coverage with the user. In addition, pre-assessments on the condition, availability of historical medical and/or cosmetic products prescribed either over the counter or by medical prescription, and / or recommended services may be captured to make the selection process for the user convenient and easy.

[00323] Referring to Fig. 7, a system for providing recommendations for skin care based on a skin state 158, a skin care goal, and environmental factors affecting the skin may comprise obtaining a skin state 158 of an individual, categorizing the individual by skin state 158, and recommending products and regimens that may be effective in achieving a skin care goal. The system may be computer-based, Internet based, network based, and the like. The system may be a community-led provision of skin services. In an embodiment, the recommendation may be made on the basis of identifying other users with similar skin states and identifying a product or regimen that is effective for them. In an embodiment, the recommendation may be made on the basis of product information 190, wellness information 192, a third party database 115, an expert 105, a service provider 111, and the like. As seen in Fig.7, a user may acquire an initial image and perform an analysis for a specific endpoint, such as moisture in this case.
The system may automatically recommend certain products based on the moisture level that may be effective given the moisture level, a skin state 158, and the like.
Additionally, the system may perform a projection of skin state 158 based on various skin care regimens 118, such as maximum care, normal care, or poor care. In an embodiment, the images may be captured using the device 108 or third party hardware 109. Images may be captured using any image capture device or technique, employing any kind of incident light, such as unpolarized light, polarized light, monochromatic light, diffuse light, white light, Page 96 of 122 multiple single wavelength light, and the like. Any captured image may be used to obtain a skin state 15 8.

[00324] An embodiment of a skin care recommendation page of a skin care system may include a report of products the user is currently using, user input to obtain a skin state 158, a recommendation request, and the like. The report on the products the user is currently using may include ranking or ratings 138. For example, when a user accesses the user interface 102, they may access an adaptive questionnaire to determine their experience with their current regimen 118, current products or therapies used, or any products or regimens 118 used in the past. For example, the user may be asked to respond to questions such as How effective is it?, How is its fragrance?, How does it absorb?, Does it cause breakouts?, How does it feel?, Do you think this product is of good value?, and the like. Of course, rankings and ratings need not be prompted by questions but may simply be anecdotal, deployed in a non-question format, deployed in a drop down menu, and the like. To obtain a skin state 158, the user may enter data relating to aspects such as gender, age, ethnicity, location, skin color, environmental factors, and the like. In embodiments, analysis 154 of images obtained from the device 108 or third party hardware 109 may also be used to determine a skin state 158. Based on the skin state 158, either derived from user input, analysis of images, or a combination thereof, users may be able to determine products and regimens 118 that may work best for their skin state 158 by connecting to a database containing wellness 192, regimen 118, expert 105, service provider 111, and product information 190, wherein the information may comprise product ingredients, product claims, product indications, product pairing, product usage protocol, product ratings and rankings 138, and the like. By including rankings and ratings 138, community-led recommendations may be made for skin related products adjusted for age, skin color, location, ethnicity, environmental factors, and the like. In an embodiment, the user may perform a recommendation request which may involve selecting a skin goal, such as moisturize, protect, cleanse, tone, beautify, anti-aging, wrinkle protection, skin tightening, deep cleanse, pore diminishing, treat rosacea, exfoliate, lighten skin, tan, sun protect, self-tan, treat acne, avoid pimples, improve luminosity, skin rejuvenation, treat spots, treat Crow's feet, hair removal, scar treatment, and the like. In embodiments, a skin goal may be automatically selected by the system Page 97 of 122 104. Automatic selection may be based on an aspect of the skin state 158. For example, if analysis 154 reveals that the skin is severely dry, the system may recommend moisturizing products for severely dry skin, or the system may recommend ingredients to look for in a product. The user may be able to purchase products directly from the recommendations page, such as by placing the product in an electronic shopping cart 113, or may be directed to another site for purchase. In an embodiment, the user may add the product to a wishlist 119 for future purchasing. In an embodiment, the user may add the product to a skin care shelf 114, which may be an interface to or depiction of a regimen 118 that enables users to organize their products and regimen 118 in a logical fashion based on the user's specific skin characteristics 130, by usage scenario (e.g.
Morning, afternoon, night, etc.), intent (e.g. work, fun, etc.). The beauty shelf 114 may have multiple screens for recommendations by various bodies (e.g. Physicians, dermatologists, aestheticians, spa specialists, overall users, experts, people most like you, etc.). The beauty shelf 114 may be a personalized arrangement of products. Users may drag and drop products (or select to add) as they are surfing the web and discover new products as well as having auto-populated recommendations. The functionality may include a program that will highlight products of interest while surfing the web. The beauty shelf 114 may be an application that can also sit independently on social networking sites and other personal pages and or toolbars. The beauty shelf 114 may also indicate purchase date and purchase history, product expiration alerts and other usage updates.
A purchase made off the website will automatically add to the user's beauty shelf 114, while manual entries for offline purchases may also be possible.

[00325] . In an embodiment, the user may be able to obtain samples of recommended or non-recommended products directly from the recommendations page.
The shopping cart 113 may be a functionality that integrates with the skin care shelf 114.
Users may be able to use the personalized recommendations and select products either for purchase, or for sample delivery. The user may be prompted for personal information such as address, shipping method, credit card number and the like, and that information may be retained by the shopping cart 113. The shopping cart 113 may be an independent program, in similar fashion to the skin care shelf 114, that may reside in a toolbar, as part of a user interface 102 or as a program on a webpage, so that products could be Page 98 of 122 highlighted and dragged into the shopping cart 113 for later purchase.
Dragging the product into the cart 113 may also initiate queries across the database and across various websites for best price, location and availability of product, consumer experience, rankings and ratings and the like.

[00326] Referring to Fig. 9, a product rating page of a skin care system is depicted. To obtain recommendations, users may be asked to respond to their medical, non-medical, cosmetic and skin care product experiences, thereby scaling data collection inexpensively. For example, a user may identify a product and provide an effectiveness assessment, rankings and ratings 138 for the product, anecdotal information, usage information, and the like. This information may be stored in a wellness 192, regimen 118, and product information 190 database in order to refine future recommendations. In an embodiment, user responses to product experiences may be shared with friends and/or other users automatically or upon request.

[00327] Referring to Fig. 10, a user interface 102 home page 1000 of a skin care system 104 is depicted. The user may be prompted to input demographic information such as name, gender, age, occupation, ID, address, telephone number, email address, payment information, new related users, and the like, which may be stored in a user profile or as part of a skin record 121. The home page may show a skin record 121, or a listing of areas imaged, date imaged, and status of analysis. Once a task is complete in the skin history/record 121, an icon may be displayed near the Status. The user may be able to launch a new Skin Health Test from the home page 1000 or submit a new skin concern. The user may be able to forward the analysis 154 to an interested party; Ask an Expert a question regarding an aspect of the skin, skin history/record 121, image analysis, and the like; view payment information and history; and the like.

[00328] Referring to Fig. 11, a welcome page 1100 of a skin health test is depicted. The welcome page may provide information on the skin health test, what endpoints will be tested for, such as elasticity, wrinkles/ fine lines, sun damage, glow /
luminosity, and the like. Using the analysis of the skin health test, the system may provide a personalized assessment of the user's skin regimen 118. The user may initiate the skin health test from the welcome page 1100.

Page 99 of 122

[00329] Referring to Fig. 12, a questionnaire page 1200 of a skin care system is depicted. The questionnaire may capture relevant skin history that may be useful for subsequent image analysis. The questions may be asked in multiple choice fashion or as open-ended questions. For example, a question may be `Where do you use your product?' with responses including face, hands, neck, legs, torso, and the like. Another question may be `Why are you using your product?' with responses including to protect, repair, moisturize, and any other skin care goal. Another question may be, `Why are/will you be using your product?' with responses including reduce wrinkles / fine lines, increase shine / luminosity, increase softness / elasticity, and any other skin care goal.
Other questions may include, `How long have you been using your product?', `How often do you apply your product?', `When do you apply your product?', and the like, with responses including stated intervals of time. Other information gathered may be how the user prefers notification, where products were purchased, if the user employs a seasonal usage of products, and the like. From the questionnaire page 1200, the user may launch the skin health test.

[00330] Referring to Fig. 13, a skin image capture page 1300 of a skin care system is depicted. In the example, the user interface 102 may access a device 108 in order to capture images, however, it should be understood that other devices 109 may be conveniently used in the system. The page 1300 may show a real time view of the area being imaged. The user may be able to employ positioning tools to be able to take an exact image of an area previously imaged. Once an image has been captured and submitted, an algorithm 150 may verify the integrity of the image. Once an image suitable for analysis has been captured, the user may proceed to an analysis page 1400.

[00331] Referring to Fig. 14, a results page of a skin care system with bar graphs is depicted. Algorithms 150 may be used to analyze the image and provide measurements of wrinkles, elasticity, luminosity, firmness, tightness, and the like, as described previously herein. In an embodiment, the measurements may be quantitative measurements. The first analysis may be considered a baseline for purposes of tracking.
For each measure, the user may be compared against the baseline for their age, skin state, gender, ethnicity, or any other category. For example, the graph depicts the reading for the user in the first bar on each graph and the average baseline for people of the same age Page 100 of 122 in the second bar. It is apparent from visual inspection that the user is better than average, in this case. These results may be color-coded for ease of interpretation. The results page 1400 may include a description of each measure. The user may be able to request More Information for each of the measures, such as why a certain condition is caused and hints and tips on how to improve a skin condition. The user may be given instructions on when to re-scan the area, which products to use, which regimen 118 to employ, and the like. Desired improvements may be correlated to ingredients and most effective products for the user's skin may be recommended. The user may access and/or edit a skin record 121, which may contain information about the user, images, a chronology of images, information derived from the images, recommendations, products, regimen 118, and the like. The user may access a report facility to obtain a report.

[00332] Referring to Fig. 15, a results page of a skin care system with trend analysis is depicted. A method for tracking the effectiveness of a skin care product or regimen may comprise obtaining a baseline skin health assessment; recommending a monitoring interval based on at least one of the skin care goal, product, and regimen;
obtaining a second skin health assessment; comparing the second assessment to the baseline assessment to determine progress towards a skin care goal; and, optionally, optimizing the regimen 118 or product in order to improve a skin health assessment.
When a subsequent image is acquired and submitted to the system 104, a trend analysis may be performed. Subsequent images may be used to track effectiveness of products and/or regimens 118 and, ultimately, advise the user on and optimize their skin regimen 118, product and/or condition. The trend analysis 1502 may be useful for determining an intermediate skin state 158 during a regimen 118. The trend analysis 1502 may show a baseline reading, an average reading for healthy skin for someone of the user's age, and individual measurements for each type of skin condition. Progress may be shown over time. A time series of images, such as over a twenty-eight day skin cycle, over a treatment timeframe, seasonally, periodically over a year and the like may be captured in order to track progress of a skin state 158. The data may be presented in a pictorial view with data on the picture, graphical view, trend view, numerical view, text view, and the like. Progress may be sorted by the concerns / skin care goals that the user may have indicated at the beginning of the test. The user may be told when to take the next image, Page 101 of 122 how much longer to continue with a regimen 118, how to modify the regimen 118, be reassured about the effectiveness of a product or regimen 118, receive useful tips, and the like. The user may view and/or edit a skin record 121. The user may be able to view past images and perform a simulation 132 of future progress. The user may access a report facility to obtain a report.

[00333] Referring to Fig. 16, a summary screen of a skin care system is depicted. An overall analysis for a time interval may be shown, current measurements, progress towards reaching a skin care goal, a product assessment, a regimen assessment, advice on continuing, modifying, or terminating a regimen 118 or product usage, and the like. The user may view a step-by-step analysis or obtain a full report. At an interval, such as at the end of a suggested regimen 118, a report may include information on how the user's skin state 158 changed over time, if the user's skin is healthier than when they started the regimen 118, if the product or regimen 118 met their initial goals, feedback on regimen 118/ product effectiveness, and the like.
Given the current skin state 158, a new product or regimen 118 may be recommended. For example, the system may recommend specific ingredients to look for in order to increase a user's luminosity given a current skin state 158. Reports may be on-screen, printed, custom, and the like. Reports may be shared with a practitioner for ongoing treatment and consultation.

[00334] Referring to Fig. 17, an elasticity summary page 1700 of a skin care system is depicted. A step-by-step analysis of each indicator may be performed. For example, a step-by-step analysis of the elasticity measurement is shown in Fig. 17. The summary page 1700 may depict all of the data captured over an interval, such as in a bar graph, for each indicator on separate summary pages 1700. It should be understood that while Fig. 17 depicts an elasticity summary page, the summary page may summarize data related to any and all concerns. Progress towards meeting a skin care goal may be indicated by the data and its analysis or from user input. An assessment of a user's product or regimen 118 in meeting the skin care goal may be made. Products or regimens 118 that may enable meeting future needs may be indicated. The system may also indicate products used or regimens 118 employed by other users in meeting the stated skin care goal.

Page 102 of 122

[00335] In an embodiment, the data acquired at a single time point or over a time interval may be shared with other users of the skin care system, practitioners, and the like. In an embodiment, the data may be shared as a data object with users of an online platform 120 or mobile platform 124 of the skin care system, posted to blogs, e-mailed to third parties, and the like. In some embodiments, the data may be a drag-and-droppable data object. For example, the wrinkle trend analysis 1502 shown in Fig. 15 may be shared with friends as in Fig. 68, posted on a blog or forum where users may discuss the data as in Fig. 69, become part of the content that a user may wish to discuss as in Fig. 70, and the like.

[00336] In embodiments, a system for providing recommendations for skin care based on a skin state 158, a skin care goal, and environmental factors affecting the skin may comprise interaction with tools and algorithms 150 on an online platform 120, a mobile platform 124, a social networking interface, and the like to receive product and regimen recommendations and track product and regimen 118 effectiveness. The system may be a communication platform, online 120 or mobile 124, that connects geographically separate consumers, manufacturers, product information, experts, service providers and others related to or allied to the beauty and medical field to provide personalized assessment regarding the consumers skin, hair, or nails queries and concerns. The user interface 102 may reside on an online platform 120, mobile platform 124, or social networking interface. In some embodiments, a skin care assessment may be provided by algorithms 150 operating on an online platform 120 without the use of images or data from a device 108, that is, a user need not have data from a device 108 to participate in the online platform 120. The online platform 120 may be a standalone skin health assessment and skin care recommendation tool. However, in embodiments, image data may also be used by the online platform 120 to provide skin health assessments and skin care recommendations. A user interface 102 may interface with the online platform 120. For example, a user may access an online platform 120 of the system for skin health analysis, monitoring, and recommendation to: monitor skin health, download, process, analyze, track, and store data from an imaging device 108 or other device 109 or monitor 182, receive product and /or regimen recommendations from an analysis/ API 154 or from peers, compare skin state 158 and regimen 118 with peers, receive product Page 103 of 122 information 190, purchase products; add recommendations to a skin care shelf 114;
organize a skin care shelf 114 by regimen 118, rankings, expiration date, cost, skin care goal, time of day, frequency, friends, and the like; view community ratings, rankings and comments on products/ regimen in a skin care shelf 114; rank/rate products;
leave comments on products, regimens, peers products and/or regimens; and the like, receive new product alerts or product recalls, receive a daily report 134, interact with a social network 188, and the like. The user interface 102 may enable users to conveniently take and submit images, enter data, track history, obtain recommendations and analysis and perform a purchase regarding their skin, hair, and/or nail's beauty/cosmetic or medical concern. The user interface 102 may reside on an online platform 120 and guide the user while also serving as a data repository to maintain a skin record 121 and history tracking tool, and may help the user organize information relevant to their condition in a logical fashion.

[00337] In an embodiment, the user interface may comprise a skin care shelf 114. The skin care shelf 114 may be a structure that enables users to organize their products and regimen 118 in a logical fashion based on users' specific skin characteristics 130 / skin state 158 by usage scenario (such as morning, afternoon, night, and the like), intent (such as work, fun, etc.), skin care goal (such as moisture, glow, protect, and the like), and the like. The skin care shelf 114 may have multiple "pages" for recommendations by various entities (such as practitioner, physicians, dermatologists, aestheticians, spa specialists, overall users, experts, people most like you, and the like).
The skin care shelf 114 may be a personalized arrangement of products, regimen 118, and/or information 190, 192. Users may drag and drop products (or select to add) as they are surfing the web and discover new products as well as having auto populated recommendations. The functionality may include a facility that may highlight products of interest while surfing the web. For example, a plug-in 194 may be used to allow a user to capture information from any location on the Internet. For example, a user may access a web page for a makeover article in a beauty magazine and wish to include the products from the makeover in their skin care shelf 114 and/or shopping cart. 113. The user may click on the product name and drag it over to at least one of the skin care shelf 114 and shopping cart 113 to obtain additional product information 190, include in their regimen Page 104 of 122 118, purchase, request samples, and the like. The skin care shelf 114 may an application that may also sit independently on social networking sites 188 and other personal pages and or toolbars. The skin care shelf 114 may also indicate purchase date and purchase history, product expiration alerts and other usage updates. In an embodiment, a purchase made off a website may automatically add to the users' shelf 114, while manual entries for offline purchases may also be possible.

[00338] In an embodiment, the user interface 102 may interface with a mobile platform 124. The user interface 102 may support plug and play with various mobile devices 184 such as mobile phones, laptops, digital cameras, medical-use devices, and the like. For example, the mobile phone may have an attachment or an integrated feature that may enable a user to take an image of the skin and input/ capture data and have it connect via the web, wirelessly or via cable, to the user interface 102 and enable seamless connectivity and data transfer. The mobile device could be used to take images and data at various locations for obtaining various information from the community (such as at the beach to measure effectiveness of sun screen, an image of a specific location, a product image or a bar code image to get product feedback, best price, nearest physical selling location, coupons, and the like). Users may also be able to share data / ask questions regarding products instantaneously to other users. The mobile device could have an internal lens system that may be internally charges or an independently attached lens system that would enable using the battery power and light source of the device to take an image and use the in-built communication method for submitting the image.

[00339] Referring to Fig. 18, the user interface for the online platform 120 may be depicted as a map. The home page may have a different theme or feel depending on the user profile, the user preference, or any other criteria. For example, it may be fun, serious, clinical, and the like. From the user interface, a user may review products, contribute anecdotes, report, review reports, review blogs by product, skin type, and the like, visit their beauty shelf 114, and the like. Information may be accessed freely, with registration, or only partially freely and partly with registration. All products and pages may link through the beauty shelf 114.

[00340] For example, Fig. 19 depicts a review page of the user interface of a skin care system. The menu across the top of the user interface may enable a user to Page 105 of 122 access Reviews, Experience, Recommendation, Info For Me, Checkout, and the like. The user interface may depict a portion of the user profile, such as the age, gender, location, skin type, skin color, skin goal, picture, and the like for the user. The user interface may also depict what products or regimen 118 the user may be using and any associated review, rating, or comments of the product. Other users accessing a user profile may make comments on the regimen 118 or products in use, give the products or regimen 118 a rating, recommend a different product or regimen 118, and the like. The user interface may present tools to aid a user in selecting a product or regimen 118. For example, the tools may be in the form a questionnaire or wizard guising the user to describe their skin.
The user may provide age, gender, skin type (oiliness, sensitivity), skin color, goal, current brand or product, current regimen 118 and the like. In some embodiments, the skin type and/ or color may be detected automatically if the user interface is interfaced with an imaging device 108. The user may also access their beauty shelf 114 from the user interface.

[00341] Referring to Fig. 20, a review page of a user interface of a skin care system is depicted. The review page is shown in a different layout than the compact view depicted in Fig. 19.

[00342] Referring to Fig. 21, an experience page of a user interface of a skin care system is depicted. The experience page allows users to provide a detailed report of experience with a product or regimen 118. For example, the user may note the effectiveness of a product or regimen 118, such as by answering questions. For example, the questions may be "How effective is it?", "How does it feel?", "How is its fragrance?", "How does it absorb?", "Does it cause breakouts?", and the like. The experience page may also allow a user to update a user profile with age, gender, nickname, location, a photo, skin type, skin color, goal, and the like. The user may be able to query other users for their experience or make a general inquiry by submitting a request to an email, MMS, SMS, phone number, mobile device, social network, and the like.

[00343] Referring to Fig. 22, a recommendation page of a user interface of a skin care system is depicted. Given the goal, various products or regimens 118 that may be effective in meeting the goal may be shown on the recommendation page. The brand and product or regimen 118 may be shown along with a rating from the community of Page 106 of 122 users, comments from users, the ability to indicate of the user believes the product may better than the current product or regimen 118 in use, and the like. If the user believes the product or regimen 118 may be better than what they are currently using, the product or regimen 118 may be stored for future consideration on the beauty shelf 114.

[00344] Referring to Fig. 23, an Info For Me page of a user interface of a skin care system is shown. A People Like Me algorithm 150 may be used to sort the community of users of the skin care system. Given the aspects of the user profile, the algorithm 150 may determine which other users are most similar along all criteria, along custom-selected criteria, along a combination of skin color and skin type, and the like.
Once the algorithm 150 has determined a subset of the community of users who are most like the user, the user can view data for the community. For example, the user can find out which products work best for the subset generally, for a specific issue, for a specific time of day, for a specific season, and the like. The Info for Me page may also depict the weather for the location given in the user profile and a UV rating and any specific tips given the location / weather/ environment. The Info for Me page may also alert the users of new products being launched. The user may sort the products according to effectiveness.

[00345] Referring to Fig. 24, an example of a beauty shelf 114 portion of a user interface of a skin care system is shown. Products or regimens 118 used by the user may be categorized by time of day use, specific effectiveness, cost, expiration, and the like.
Each item may be clicked on to pop-up additional details about the product or regimen 118, such as effectiveness, ingredients, suggested use, expiration date, a link to purchase more, a link to blog about the product or regimen 118, a link to write a review or read reviews, a link to the manufacturer's site, a link to an in-store coupon, and the like. Fig.
25 depicts another example of a beauty shelf 114 portion of a user interface of a skin care system. Fig. 26 depicts an alternate view of the beauty shelf 114 of the user interface of a skin care system. In this example, friends have the ability to comment on the products or regimen 118 and suggest an alternative product or regimen 118. The user also has the option to receive price alerts, new product launch alerts, new user comment alerts, and the like.

Page 107 of 122

[00346] Referring to Fig. 27, a registration page of a user interface of a skin care system is depicted. Information may be entered by the user, goals may be indicated, a security code may be entered, skin concerns, color, and/or type may be entered, samples may be registered for, and the like. Additionally, the user may indicate that the want to add a feed from the skin care system to their RSS feed, and application from the skin care system to a social networking site, and the like. The user may have the option to opt-in to alerts, to be notified of samples and products, and the like.

[00347] Referring to Fig. 28, another embodiment of a recommendation page of a user interface of a skin care system is shown. This page may show people in the user's category, such as number of people of the same gender, same age group, with similar skin type, with similar concerns, and the like. For each stated goal, a product may be recommended that is most popular, has the most buzz, has been reviewed, has been rated, has been blogged about, and the like.

[00348] Referring to Fig. 64, the user interface may include a friend toolbar.
The friend toolbar may float over a current website, or any website, such as by using a plug-in. Friends may upload images and the images 6408 may be displayed on the friend toolbar 6402. A home key 6404 may be part of the toolbar 6402, where the whole toolbar can be reduced to just the home key 6404. When an alert is associated with a friend, such as a new product being added to their beauty shelf 114 or a new review being written, a flag alert 6410 may pop-up next to their image on the toolbar 6402.
A bottom bar 6412 may be used for shuffling friends or accessing other options related to the toolbar 6402. Referring to Fig. 65, the toolbar 6402 may auto-scroll 6502 as the user scrolls the webpage they are viewing. Referring to Fig. 66, objects may be shared with friends in the friends' toolbar 6402 using a drag-and-drop functionality 6602.
For example, a blog posting may be shared as in Fig. 66 by dragging and dropping the blog title onto a friend's image. Similarly, products may be recommended to a friend by dragging and dropping 6702 the product into the friends' image, as in Fig, 67.
Rolling over a friends' image may result in a pop-up, dialog box or other manifestation of additional information about the friend, such as a view of their user profile, beauty shelf 114, reviews, blogs, and the like.

Page 108 of 122

[00349] Referring to Fig. 29, a mobile content map for a mobile user interface of a skin care system on a mobile platform 124 is depicted. The content map depicted shows an example of content that can be accessed from a mobile platform 124 home page. For example, starting from the home page, a product may be scanned or identified from a list and searched for using the internet on the mobile device. For example, a bar code may be scanned for a product and prices, reviews, ratings and the like for the product may be returned. The user may be helped to find something, such as an item for themselves, a gift for a friend, and the like. The product may be searched for based on a goal, an issue, a skin type, a skin color, and the like. The mobile skin care system may return a list of products, such as the top 10 products, and information about the products such as rating, impact on goals, safety, reviews, and the like. The user may access a Suncheck application to be given UV information by location and advice, as well as based on an image captured by an imaging device 108 embodied in a mobile device, as described previously herein.

[00350] Referring to Fig. 30, a How Good Is This Product message flow is depicted. In the example, a bar code may be scanned to obtain product info, the bar code numbers may be manually entered, or the product may be chosen from a list. The system may return product information such as the product name, rating, ingredients, a general rating, a rating for a specific concern, a friend's rating, a price, where the product can be found, and the like. If the mobile device is enabled, a purchase may be initiated on the mobile platform 124.

[00351] Referring to Fig. 31, a What Should I Look For? message flow is depicted. The message flow may begin by giving the user the option to indicate if the item searched for is a gift, for the user, to update a pick list, and the like. For gifts, a recipient may be selected from a pre-populated list or a new recipient may be indicated.
An occasion may be indicated. Based on the recipient and occasion and any other criteria entered, products may be recommended along with any information associated with the product, a price, a location, and an option to purchase on the mobile platform 124. In looking for something for the user, the user may indicate a goal, such as from a drop down menu, and receive a list of recommended products. Once a product is selected, the Page 109 of 122 user may request to locate the product at a store or initiate a purchase on the mobile platform 124, or the like.

[00352] Referring to Fig. 32, a Suncheck message flow is depicted. The initial message may contain information about the user's location, the weather, a UV
index, a sun impact rating, an indication of the maximum exposure time, and a timer for measuring the current time in the sun. Advice may be generated based on the information, such as what level of sun protection factor to apply, a maximum recommended time of exposure, and the like.

[00353] Referring to Fig. 33, an Alert message flow is depicted. The user may be linked to other users on the mobile platform 124 so that when another user requests a review or rating of a product, an alert may be sent to the user. The user may respond with a review, a rating, a chat message, an SMS, an MMS, a phone call, a voicemail, and the like.

[00354] Referring to Fig. 34, an Options message flow is depicted. From the mobile platform 124 home page 3402, Options may be selected. Options 3404 may be a friend list, a pick list, alerts, address/location, and the like. For example, a friend list 3408 may be accessed to pick and choose friends to follow, receive alerts from and the like. The friends list may indicate if the friend is online. Alerts 3410 may also be set on the mobile platform 124, for example to notify the user when their friends buy something new, notify the user when a new product that is good for them is available, and the like.
Address / location / payment setup may allow the user to initiate purchases from the mobile platform 124.

[00355] The methods and systems described herein may be deployed in part or in whole through a machine that executes computer software, program codes, and/or instructions on a processor. The processor may be part of a server, client, network infrastructure, mobile computing platform, stationary computing platform, or other computing platform. A processor may be any kind of computational or processing device capable of executing program instructions, codes, binary instructions and the like. The processor may be or include a signal processor, digital processor, embedded processor, microprocessor or any variant such as a co-processor (math co-processor, graphic co-processor, communication co-processor and the like) and the like that may directly or Page 110 of 122 indirectly facilitate execution of program code or program instructions stored thereon. In addition, the processor may enable execution of multiple programs, threads, and codes.
The threads may be executed simultaneously to enhance the performance of the processor and to facilitate simultaneous operations of the application. By way of implementation, methods, program codes, program instructions and the like described herein may be implemented in one or more thread. The thread may spawn other threads that may have assigned priorities associated with them; the processor may execute these threads based on priority or any other order based on instructions provided in the program code. The processor may include memory that stores methods, codes, instructions and programs as described herein and elsewhere. The processor may access a storage medium through an interface that may store methods, codes, and instructions as described herein and elsewhere. The storage medium associated with the processor for storing methods, programs, codes, program instructions or other type of instructions capable of being executed by the computing or processing device may include but may not be limited to one or more of a CD-ROM, DVD, memory, hard disk, flash drive, RAM, ROM, cache and the like.

[00356] A processor may include one or more cores that may enhance speed and performance of a multiprocessor. In embodiments, the process may be a dual core processor, quad core processors, other chip-level multiprocessor and the like that combine two or more independent cores (called a die).

[00357] The methods and systems described herein may be deployed in part or in whole through a machine that executes computer software on a server, client, firewall, gateway, hub, router, or other such computer and/or networking hardware. The software program may be associated with a server that may include a file server, print server, domain server, internet server, intranet server and other variants such as secondary server, host server, distributed server and the like. The server may include one or more of memories, processors, computer readable media, storage media, ports (physical and virtual), communication devices, and interfaces capable of accessing other servers, clients, machines, and devices through a wired or a wireless medium, and the like. The methods, programs or codes as described herein and elsewhere may be executed by the Page 111 of 122 server. In addition, other devices required for execution of methods as described in this application may be considered as a part of the infrastructure associated with the server.

[00358] The server may provide an interface to other devices including, without limitation, clients, other servers, printers, database servers, print servers, file servers, communication servers, distributed servers and the like.
Additionally, this coupling and/or connection may facilitate remote execution of program across the network. The networking of some or all of these devices may facilitate parallel processing of a program or method at one or more location without deviating from the scope of the invention. In addition, any of the devices attached to the server through an interface may include at least one storage medium capable of storing methods, programs, code and/or instructions. A central repository may provide program instructions to be executed on different devices. In this implementation, the remote repository may act as a storage medium for program code, instructions, and programs.

[00359] The software program may be associated with a client that may include a file client, print client, domain client, internet client, intranet client and other variants such as secondary client, host client, distributed client and the like. The client may include one or more of memories, processors, computer readable media, storage media, ports (physical and virtual), communication devices, and interfaces capable of accessing other clients, servers, machines, and devices through a wired or a wireless medium, and the like. The methods, programs or codes as described herein and elsewhere may be executed by the client. In addition, other devices required for execution of methods as described in this application may be considered as a part of the infrastructure associated with the client.

[00360] The client may provide an interface to other devices including, without limitation, servers, other clients, printers, database servers, print servers, file servers, communication servers, distributed servers and the like. Additionally, this coupling and/or connection may facilitate remote execution of program across the network. The networking of some or all of these devices may facilitate parallel processing of a program or method at one or more location without deviating from the scope of the invention. In addition, any of the devices attached to the client through an interface may include at least one storage medium capable of storing methods, programs, applications, code Page 112 of 122 and/or instructions. A central repository may provide program instructions to be executed on different devices. In this implementation, the remote repository may act as a storage medium for program code, instructions, and programs.

[00361] The methods and systems described herein may be deployed in part or in whole through network infrastructures. The network infrastructure may include elements such as computing devices, servers, routers, hubs, firewalls, clients, personal computers, communication devices, routing devices and other active and passive devices, modules and/or components as known in the art. The computing and/or non-computing device(s) associated with the network infrastructure may include, apart from other components, a storage medium such as flash memory, buffer, stack, RAM, ROM and the like. The processes, methods, program codes, instructions described herein and elsewhere may be executed by one or more of the network infrastructural elements.

[00362] The methods, program codes, and instructions described herein and elsewhere may be implemented on a cellular network having multiple cells. The cellular network may either be frequency division multiple access (FDMA) network or code division multiple access (CDMA) network. The cellular network may include mobile devices, cell sites, base stations, repeaters, antennas, towers, and the like.
The cell network may be a GSM, GPRS, 3G, EVDO, mesh, or other networks types.

[00363] The methods, programs codes, and instructions described herein and elsewhere may be implemented on or through mobile devices. The mobile devices may include navigation devices, cell phones, mobile phones, mobile personal digital assistants, laptops, palmtops, netbooks, pagers, electronic books readers, music players and the like. These devices may include, apart from other components, a storage medium such as a flash memory, buffer, RAM, ROM and one or more computing devices.
The computing devices associated with mobile devices may be enabled to execute program codes, methods, and instructions stored thereon. Alternatively, the mobile devices may be configured to execute instructions in collaboration with other devices. The mobile devices may communicate with base stations interfaced with servers and configured to execute program codes. The mobile devices may communicate on a peer to peer network, mesh network, or other communications network. The program code may be stored on the storage medium associated with the server and executed by a computing Page 113 of 122 device embedded within the server. The base station may include a computing device and a storage medium. The storage device may store program codes and instructions executed by the computing devices associated with the base station.

[00364] The computer software, program codes, and/or instructions may be stored and/or accessed on machine readable media that may include: computer components, devices, and recording media that retain digital data used for computing for some interval of time; semiconductor storage known as random access memory (RAM);
mass storage typically for more permanent storage, such as optical discs, forms of magnetic storage like hard disks, tapes, drums, cards and other types;
processor registers, cache memory, volatile memory, non-volatile memory; optical storage such as CD, DVD;
removable media such as flash memory (e.g. USB sticks or keys), floppy disks, magnetic tape, paper tape, punch cards, standalone RAM disks, Zip drives, removable mass storage, off-line, and the like; other computer memory such as dynamic memory, static memory, read/write storage, mutable storage, read only, random access, sequential access, location addressable, file addressable, content addressable, network attached storage, storage area network, bar codes, magnetic ink, and the like.

[00365] The methods and systems described herein may transform physical and/or or intangible items from one state to another. The methods and systems described herein may also transform data representing physical and/or intangible items from one state to another.

[00366] The elements described and depicted herein, including in flow charts and block diagrams throughout the figures, imply logical boundaries between the elements. However, according to software or hardware engineering practices, the depicted elements and the functions thereof may be implemented on machines through computer executable media having a processor capable of executing program instructions stored thereon as a monolithic software structure, as standalone software modules, or as modules that employ external routines, code, services, and so forth, or any combination of these, and all such implementations may be within the scope of the present disclosure.
Examples of such machines may include, but may not be limited to, personal digital assistants, laptops, personal computers, mobile phones, other handheld computing devices, medical equipment, wired or wireless communication devices, transducers, Page 114 of 122 chips, calculators, satellites, tablet PCs, electronic books, gadgets, electronic devices, devices having artificial intelligence, computing devices, networking equipments, servers, routers and the like. Furthermore, the elements depicted in the flow chart and block diagrams or any other logical component may be implemented on a machine capable of executing program instructions. Thus, while the foregoing drawings and descriptions set forth functional aspects of the disclosed systems, no particular arrangement of software for implementing these functional aspects should be inferred from these descriptions unless explicitly stated or otherwise clear from the context.
Similarly, it will be appreciated that the various steps identified and described above may be varied, and that the order of steps may be adapted to particular applications of the techniques disclosed herein. All such variations and modifications are intended to fall within the scope of this disclosure. As such, the depiction and/or description of an order for various steps should not be understood to require a particular order of execution for those steps, unless required by a particular application, or explicitly stated or otherwise clear from the context.

[00367] The methods and/or processes described above, and steps thereof, may be realized in hardware, software or any combination of hardware and software suitable for a particular application. The hardware may include a general purpose computer and/or dedicated computing device or specific computing device or particular aspect or component of a specific computing device. The processes may be realized in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable device, along with internal and/or external memory. The processes may also, or instead, be embodied in an application specific integrated circuit, a programmable gate array, programmable array logic, or any other device or combination of devices that may be configured to process electronic signals. It will further be appreciated that one or more of the processes may be realized as a computer executable code capable of being executed on a machine readable medium.

[00368] The computer executable code may be created using a structured programming language such as C, an object oriented programming language such as C++, or any other high-level or low-level programming language (including assembly languages, hardware description languages, and database programming languages and Page 115 of 122 technologies) that may be stored, compiled or interpreted to run on one of the above devices, as well as heterogeneous combinations of processors, processor architectures, or combinations of different hardware and software, or any other machine capable of executing program instructions.

[00369] Thus, in one aspect, each method described above and combinations thereof may be embodied in computer executable code that, when executing on one or more computing devices, performs the steps thereof. In another aspect, the methods may be embodied in systems that perform the steps thereof, and may be distributed across devices in a number of ways, or all of the functionality may be integrated into a dedicated, standalone device or other hardware. In another aspect, the means for performing the steps associated with the processes described above may include any of the hardware and/or software described above. All such permutations and combinations are intended to fall within the scope of the present disclosure.

[00370] Furthermore, embodiments may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied in the storage medium. Some embodiments may take the form of web-implemented computer software. Any suitable computer-readable storage medium may be utilized including hard disks, CD-ROMs, optical storage devices, magnetic storage devices, or the like.

[00371] It will be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, respectively, can be implemented by computer program instructions. These computer program instructions may be loaded onto a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions which execute on the computer or other programmable data processing apparatus create a means for implementing the functions specified in the flowchart block or blocks.

[00372] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including computer-Page 116 of 122 readable instructions for implementing the function specified in the flowchart block or blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions that execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[00373] Accordingly, blocks of the block diagrams and flowchart illustrations support combinations of means for performing the specified functions, combinations of steps for performing the specified functions, program instruction means for performing the specified functions, and so on. It will also be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, can be implemented by special purpose hardware-based computer systems that perform the specified functions or steps, or combinations of special purpose hardware and computer instructions.

[00374] While the invention has been disclosed in connection with the preferred embodiments shown and described in detail, various modifications and improvements thereon will become readily apparent to those skilled in the art.
Accordingly, the spirit and scope of the present invention is not to be limited by the foregoing examples, but is to be understood in the broadest sense allowable by law.

[00375] All documents referenced herein are hereby incorporated by reference.
Page 117 of 122

Claims

What is claimed is:

1. A method for estimation of skin type and skin features to create a unique spectral signature, the method comprising:
convoluting data from a first image captured in incident diffuse white light, wherein the data relate to reflected and/or re-emitted polarized or white light;
convoluting data from a second image captured in incident polarized light, wherein the data relate to reflected and/or re-emitted polarized light;
comparing extreme positions of at least two unique convolutions generated by convoluting data from the first image and the second image; and determining a distance between minimum and maximum intensity positions in convoluted red minus blue spectral plots from the at least two unique convolutions for generating a numerical skin type output.

2. The method of claim 1, wherein the physiological white light comprises three spectral intervals including a width less than 100 nanometer.

3. The method of claim 2, wherein the three spectral intervals pertain to red, green, and blue (RGB) colors.

4. The method of claim 3, wherein the three spectral intervals provide a natural white light sensation to a human eye.

5. The method of claim 1, wherein the step of comparing extreme positions of at least two unique convolutions comprises comparing a component (R-B)(W-P) for the reflected and/or re-emitted polarized light, and a component (R-B)W for the white light.

6. The method of claim 5, wherein the two unique convolutions in white light and polarized light further include a White Red component (WR), a White Blue component (WB), a reflected and/or re-emitted Polarized Blue component (PB) and a reflected and/or re-emitted Polarized Red component (PR).

7. The method of claim 6, wherein the two unique convolutions are based on a numerical value difference correlating to medical standards.

8. The method of claim 1, further comprising a spectral convolution scheme wherein multiple combinations of subtraction of blue spectrum from red, in white light and polarized white light are determined, wherein the spectral interval is expressed in a wavelength scale interval of 100 nanometers to 300 nanometers.

9. A machine readable medium, the machine readable medium having program instructions stored thereon executable by a processing unit, the program instructions comprising the steps of:
capturing an image of a material illuminated with incident non-angled white light and an image of a material illuminated with incident angled white light;
generating a normalized red and a normalized blue color channel histogram for a reflected and/or emitted light in each image;
correlating the normalized red and normalized blue color channel histograms to a wavelength scale to obtain red and blue color channel spectral plots; and combining the red and blue color channel spectral plots by subtracting the spectral plot for angled light from the spectral plot for non-angled light for each color channel to generate red and blue normalized, composite color channel spectral plots of a specific wavelength scale and subtracting the normalized, composite blue channel spectral plot from the normalized, composite red channel spectral plot to generate a spectral signature for the material.

10. The medium of claim 9, further comprising subtracting a normalized blue channel spectral plot for the incident light from the normalized, composite blue channel spectral plot before generating the spectral signature for the material.

11. The medium of claim 9, further comprising subtracting a normalized red channel spectral plot for the incident light from the normalized, composite red channel spectral plot before generating the spectral signature for the material.

12. The medium of claim 9, wherein the illumination source is positioned to direct light at a selected angle alpha.

13. The medium of claim 9, wherein varying alpha varies the depth of the measurement in the material.

14. The medium of claim 9, wherein the unit scale on the spectral signature is a difference of wavelength.

15. The medium of claim 9, wherein the material is inorganic matter.

16. The medium of claim 9, wherein the material is organic matter.

17. The medium of claim 9, wherein the spectral signature is analyzed for at least one of number of peaks and troughs, amplitude and shape of peaks and intermediate structures and patterns.

18. The medium of claim 9, wherein the spectral signature is analyzed for metal composition, identification, purity, and strength.

19. The medium of claim 9, wherein the spectral signature is analyzed for water quality, composition, and purity.

20. The medium of claim 9, wherein elements of the spectral signature are tagged and tracked over time in order to track changes in the characteristics of the material.

21. The medium of claim 9, wherein the spectral signature is analyzed to measure, track, or monitor a skin state.

22. The medium of claim 9, wherein the spectral signature is fed into a recommendation engine to provide feedback and modifications to aspects of a regimen.

23. The medium of claim 9, wherein the wavelength position of ideal blue in Maxwell's color triangle is aligned with the wavelength position of ideal red in Maxwell's color triangle when convoluting the composite spectral plots to obtain the spectral signature.

24. The medium of claim 9 wherein a spectral signature is calculated for the incident light and is subtracted from the spectral signature generated from the reflected and/or emitted light.

25. The medium of claim 24, wherein only a polarized component is used when the incident light spectral signature is subtracted.

26. The medium of claim 24, wherein only the white component is used when the incident light spectral signature is subtracted.

27. The medium of claim 24, wherein both a polarized component and non-polarized components are used to subtract from the source light spectral signature.