WO2014047712A1

WO2014047712A1 - Imaging device of facial topography with multiple light source flash photography and method of blending same

Info

Publication number: WO2014047712A1
Application number: PCT/CA2012/000886
Authority: WO
Inventors: Jerry K.L. Tan; Brian NHAN; Alden LING
Original assignee: Windsor Clinical Research Inc.
Priority date: 2012-09-26
Filing date: 2012-09-26
Publication date: 2014-04-03
Also published as: CA2885775A1; CA2885775C

Abstract

An image capturing device that consists of one or multiple light sources and a image capturing device for imaging of a human face. The device enhances topographic information by capturing a multitude of digital images of a face in a fixed orientation with varying light source positions and combining the images with a blending algorithm. There may be multiple light sources or one light source that is repositioned with each image capture such that light source positions are not coincident. The blending algorithm can be applied to any number of a set of images in a fixed orientation where only the light source direction is varied. The device may include a support device to maintain the human face in a fixed orientation during image capture.

Description

IMAGING DEVICE OF FACIAL TOPOGRAPHY WITH MULTIPLE LIGHT SOURCE FLASH PHOTOGRAPHY AND METHOD OF BLENDING SAME

FIELD OF THE INVENTION

The present invention relates to imaging device of facial topography with multiple light source flash photography and method of blending same which combines a set of images of a fixed human face where light source position is varied between images. This blending method enhances highlights and shadows of facial topographic details that are present in various dermatological conditions, such as acne, moles, keloids, scarring from trauma, accidents and/or infections, etc.

BACKGROUND OF THE INVENTION

The use of two-dimensional digital photography in dermatology has been well documented. Techniques that have been pursued include: flash, polarized light, and fluorescence digital photography. Flash photography approaches employ single image captures in semi-standard facial positioning defined by a trained clinical photographer. The digital flash is positioned such that salient details in a given single shot are enhanced. Polarized light photography encompasses approaches utilizing parallel polarized, perpendicular polarized and cross polarized filtering. The use of polarizing filters aims to reduce ambient glare reflected from the facial skin surface to enhance visualization of details below. Fluorescence digital photography utilizes light filtering to capture certain wavelengths of light emitted by areas of interest on the face (i.e. acne lesions) while ignoring light emission from other facial areas.

For example, US Patent Nos. 6,177,984 and 6,437,856 to Jacques propose, among other things, a video imaging device including a light source, a detector, and an optical polarization system for video imaging of superficial biological tissue layers. The device relies on taking a set of measurement at different polarization orientations so as to render a new image that is independent of the light reflected from the surface of a tissue sample and that is independent of light scattered from deep tissue layers. However, such removal of specular reflection based on polarization is not suitable for capturing skin details, such as acne, moles, keloids, scarring from trauma, accidents, and/or infections, etc, in a two-dimensional image / video.

US Patent No. 7,764,303 to Pote et al. (or "Pote") proposes an imaging station for taking a plurality of digital images of a subject under a plurality of illuminating conditions and storing and analyzing the digital images. However, Pote is silent about the analysis or processing of captured images.

US Patent Application Publication No. 2011/0304705 by Kantor et al. (or "Kantor") proposes an optical device for imaging and measuring characteristics of an objects surface shape, surface spectral reflectance and structure of sub-surface layers. For doing so, Kantor makes various measurements to calculate surface elevation based on light intensity, and then reconstruct a three-dimensional image therefrom. However, for making such measurement, specular reflections are detriment to its calculation and thus are considered as "noise" that need to be filtered out; which would increase complexity in its image capturing system/process, measurement and/or its computation.

As shown above, many of the proposed techniques or processes are not suitable or cost effective for characterising skin details, i.e. acne, moles, keloids and other surface growth, and scarring from trauma, accidents, infections etc, on a two- dimensional or on a flat image. As suggested in the literature, various attempts have been made for standardizing methods of capturing images; however, no gold standard has yet been established.

Therefore, there exists a need for developing an effective approach to imaging facial surface details including elevations and depressions, that provides as good or better topographic information than current practice would allow practitioners to obtain consistent images of patients that can be easily compared over time.

SUMMARY OF THE INVENTION

The present invention provides imaging of facial topography with multiple light source flash photography and method of blending same. According to one aspect of the invention, it provides an imaging device of facial topography, comprising a light source for illuminating a facial surface of interest from at least three different positions circumferentially around the facial surface of interest, and said at least three different positions are equally

circumferentially distanced around the facial surface of interest, an image capturing device, capturing images of the facial surface from a fixed position above the facial surface, each captured images having the light source illuminating the facial surface from different one of said at least three different positions, and an image blending device for blending said at least three images.

According to another aspect of the invention, it provides a method of imaging facial topography, the method comprising the steps of capturing at least three images of a facial surface of interest from a fixed position, each of said at least three images having a light source illuminated from different one of at least three positions which are equally circumferentially distanced from each other, and overlaying said at least three images by blending to produce a blended image.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will now be described in more detail with reference to the accompanying drawings, in which:

Figure 1A illustrates a schematic view of a preferred embodiment of the present invention;

Figure IB illustrates a flow diagram of a preferred method of overlaying captured images by blending in accordance with the present invention;

Figure 2 A illustrates a side plan view of a three-point support device 10 of a preferred embodiment of the present invention, being applied to a person of interest S' to be imaged;

Figure 2B illustrates a perspective view of the three point support device 10;

Figure 3 illustrates a schematic view of the prototype of the image capturing device 6 used to create FIGS. 4 to 7; Figure 4 illustrates three captured images (original images are in color, the images have been grey-scaled) of a first representative facial surface with three light sources at 0, 120 and 240 degrees and one blended image created from the three captured images using the darken blending mode of Adobe Photoshop;

Figure 4A illustrates a blended image using lighten blending mode of Adobe Photoshop based on the three captured images shown in Figure 4;

Figure 5 illustrates three captured images (original images are in color, images have been grey-scaled) of a second representative facial surface with three light sources at 0, 120 and 240 degrees and the blended image created from the three captured images using the darken blending mode of Adobe Photoshop;

Figure 5A illustrates a blended image using lighten blending mode of Adobe Photoshop based on the three captured images shown in Figure 5;

Figures 6 illustrates three captured images (original images are in color, images have been grey-scaled) of a third representative facial surface with three light sources at 0, 120 and 240 degrees and the blended image created from the three captured images using the darken blending mode of Adobe Photoshop;

Figure 6A illustrates a blended image using lighten blending mode of Adobe Photoshop based on the three captured images shown in Figure 6;

Figures 7 illustrates three captured images (original images are in color, images have been grey-scaled) of a fourth representative facial surface with three light sources at 0, 120 and 240 degrees and the blended image created from the three captured images using the darken blending mode of Adobe Photoshop; and

Figure 7A illustrates a blended image using lighten blending mode of Adobe Photoshop based on the three captured images shown in Figure 7.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides an image capturing and blending method that also accentuates salient topographic information using a superposition of images captured with varying light source directions. This method for blending images represents a new approach to obtaining pseudo-three dimensional information of a facial skin surface in a two dimensional image through blending.

Blending images have been well described in digital image editing. Raw images are represented numerically, the currently most common numerical representation of colors is the one used in RGB (red, green, blue) values, each of red, green and blue can take values between 0 and 255, indicating the intensity of red, green and blue contained by a pixel in the image. Basic blend modes are commercially available in image editing applications like Adobe Photoshop ® and GIMP™ (or GNU Image Manipulation Program) or a like.

The present invention utilizes a computer implemented method of blending images to combine a set of images captured by an image capturing device (i.e. a digital camera) in which, while the position of the image capturing device being fixed in respect of the facial surface of interest / the subject being captured, light illumination at a different position is applied in each image. The image capturing device consists of a digital camera synchronized with one repositionable light source circumferentially around the facial surface of interest or multiple light sources positioned at a plurality of positions circumferentially around the facial surface of interest. In a preferred embodiment of the present invention, there are at least three different positions, and these positions are circumferentially evenly distanced around the facial surface of interest (i.e. at 0, 120 and 240 degrees with the equal distance or radius from the centre of the surface). In compiling a set of images, the capturing device's position relative to the facial surface of interest and the facial orientation at large are fixed.

To maintain the position of the face being captured, a three-point support device resting on the chin, occiput and shoulders of the individual to be imaged may be employed. This three point support device would sit on top of the individual's shoulders and consists of adjustable arms to be in contact with the individual's chin and occiput.

Each image in the set consists of the same facial surface area of interest illuminated by light from a different angle. This set of images taken with different combinations of light source position are combined using a blending algorithm to enhance light and shadow details that provide topographic information in the facial surface area of interest. The present invention utilizes a computer-implemented arithmetic blending method or mode(s), lighten and/or darken to combine the images in the set.

For example, each pixel of a captured raw digital image in the set would have an RGB value (red, green, blue) between 0 and 255. The lighten blending algorithm combines the images in the set such that the final overlaid / combined image consists of the 'lightest' (i.e. largest RGB values) pixels from each individual image. The darken blending algorithm overlays / combines the images in the set such that the final overlaid / combined image consists of the 'darkest' (i.e. smallest RGB values) pixels from each individual image. The lighten blending typically enhances highlights around a feature while the darken blending typically enhances shadows around a feature.

A prototype for the image capturing device of the present invention consists of a Canon ® EOS 60D Digital SLR digital camera with a Canon EF 100 mm f/2.8 USM lens and a Canon Speedlite ® 430EX II as the repositionable light source. Images are captured using the following camera settings: exposure 1/100s, aperture f/10, focal length 100mm, ISO speed 100. The light source is set to ¼ flash intensity. A constant distance of 72 cm is maintained from the lens to the surface of interest. The light source is oriented at 0, 120, and 240 degrees around the surface of interest in an arc with a radius of 30.5 cm from the centre of the surface of interest. The light source is oriented to provide tangential lighting to the surface of interest during imaging. The light source is synchronized to the image capture by the detector. Ambient lighting is reduced to a minimum and images are captured in a light controlled room. Four surfaces of interest were constructed from flesh coloured plasticine representative of a rounded human cheek. These surfaces of interest are approximately 15.24 cm in diameter. These surfaces have 20 randomly distributed lesions (10 elevations and 10 depressions of random size).

In a preferred embodiment of the present invention, the image set for each surface of interest would contain at least three captured images with the light source in three different positions as described above. The three images are overlaid and combined in an image editing application software, for example, Adobe Photoshop ® using the darken blending algorithm to enhance the shadowing around elevations and depressions on the surface. The darken blending mode overlays and combines the images such that the final image consists of the darkest pixel from the three separate images.

In another embodiment of the present invention, when more images with more light sources are added, a fuzzy approach may be preferable and be utilized to reconstruct or blend an image therefrom. For example, thresholds can be selected such that an average of the pixels with RGB intensity values above (i.e. in a fuzzy lighten approach) or below (i.e. in a fuzzy darken approach) the threshold are used in the final blended image.

FIG. 1 A illustrates a schematic view of a preferred embodiment of the present invention. A person 4 is positioned with multiple light sources 1, 2, and 3 circumferentially around to illuminate the surface of interest S. First, only the first light source 1 is used to illuminate the surface of interest S via the first illuminated light 1A synchronized to capture an image by an image capturing device 6. Raw image data is transferred from the image capturing device 6 via communication means 5 (i.e. wired or wireless communication or wired or wireless communication network) or stored in a data storage medium (i.e. memory card, hard disc, CD, DVD, etc, not shown) and is accessible from a computing device 7 (such as personal computer, handheld computing device, embedded or dedicated computing device or module, etc) as an image set as a first image IB. Then, only the second light source 2 is used to illuminate the surface of interest S by second light 2A to produce the second image 2B in the set. Finally, only the third light source 3 is used to create image 3B via illumination from third light 3A. A multitude of light sources in a multitude of positions may be included to produce more images in the set where the positions of the light sources are not coincident. The images in the set are combined by the computing device 7 using a blending mode (or computer-implemented image blending method) to produce a final combined or blended image 8 that enhances light and shadow detail of topographic information on the surface of interest S.

FIG. IB illustrates a flow diagram of a preferred method of overlaying images by blending in accordance with the present invention. A plurality of images captured using the image capturing device 6, each of which having the light source 1, 2 or 3 (or other, not shown) illuminating the facial surface of interest S from different position from others are taken at steps 100-1, 100-2 ... 100-m. Then, each of these images are converted into digital format / representations, such as RGB values, 201-1, 201-2, ... 201-m by a known method or equivalent processed by a computing device (not shown). Those RGB values 201-1, 201-2, ... 201-m are then manipulated by the computing device for overlaying the images 100-1, 100-2 ... 100-m by blending. In a preferred embodiment of the present invention, a darken blending mode, in which the lowest RGB value among 201-1, 201-2, ... 201-m at every pixel position are selected at step 202. In another preferred embodiment of the present invention, lighten blending mode (i.e. selecting largest RGB values) may be used instead of darkening blending mode. In yet another preferred embodiment of the present invention, a predetermined threshold is set by a user, an average light intensity (or an average RGB value) is calculated with the pixels of images in the given image set above said predetermined threshold (i.e. in a fuzzy lighten blending mode) or below said predetermined threshold (i.e. in a fuzzy darken blending mode) to produce a combined RGB value in the blended image. Once RGB values are calculated and collected at step 203, the blended image is produced at step 300.

FIG. 2 A illustrates a side plan view of a three-point support device 10 of a preferred embodiment of the present invention, being applied to a person of interest S' to be imaged. The person of interest S' is seated in a chair 14. The support device's shoulder support 13 is attached to the back rest of the chair 14. The shoulder support 13 contacts and rests on the shoulders of the person of interest S'. Occiput support 11 is adjusted to be in contact with and to rest on the occiput of the person of interest S' by manipulating the length of the adjustable support arm 11 A. The adjustable support arm 11 A is extendable and retractable, and may further comprise fastening means (i.e. handle, screws or latching device, not shown) for releasably fixing the length of the adjustable support arm 11 A at the desirable length. Chin support 12 may be adjusted to be in contact with and to rest on the chin of the person of interest S' by manipulating the adjustable support arm 12A. Similarly, length of the adjustable support arm 12A is extendable and retractable, and may further comprise fastening means (not shown) for fastening the arm 12 A at a desirable length. The three points of contact by the support device 10 on the person of interest S' prevent or minimize movement and head turning of the person of interest S' while attachment of the shoulder support 13 to the chair 14 prevents and/or minimizes any other movement.

FIG. 2B illustrates a perspective view of the three point support device 10 of a preferred embodiment of the present invention. The three point support device 10 includes occiput support 11 being attached to an adjustable occiput support arm 11 A. The adjustable occiput support arm 11 A having a threaded inserting member 21 and a receiving member 22, where the threaded inserting member 21 slidably engaged with the receiving member 22, such that the adjustable occiput support arm 11 A longitudinally extends or retracts along its length. The receiving member 22 is connected to and extends from the shoulder support 13. The receiving member 22 further comprises a fastening means 25, including a threaded aperture 24 and a screw 23. Once the threaded inserting member 21 is adjusted to a desirable length, the screw 23 is driven to engage with a thread on the threaded inserting member 21 for releasably attaching or holding the threaded inserting member 21 to the receiving member 22. Various other means or structure for adjustable occiput support arm 11 A than the one shown in the present drawing may be used by a person ordinary skilled in the pertinent art.

Similarly, a chin support 12 being attached to an adjustable chin support arm 12 A. The adjustable chin support arm 12 A having a threaded inserting member 31 and a receiving member 32, where the threaded inserting member 31 slidably engaged with the receiving member 32, such that the adjustable chin support arm 12A longitudinally extends or retracts along its length. The receiving member 32 is connected to and extends from the shoulder support 33. The receiving member 32 further comprises a fastening means 35, including a threaded aperture 34 and a screw 33. Once the threaded inserting member 31 is adjusted to a desirable length, the screw 33 is driven to engage with a thread on the threaded inserting member 31 for releasably attaching or holding the threaded inserting member 31 to the receiving member 32. Various other means or structure for adjustable chin support arm 12A than the one shown in the present drawing may be used by a person ordinary skilled in the pertinent art.

FIG. 3 illustrates a schematic view of the prototype of the image capturing device 6 used to create FIGS. 4 to 7. A second light source 2 is used to illuminate the surface of interest S via light 2A. The light is synchronized with the image capturing device 6 to capture the raw image data. Similarly, a third light source 3 is used to illuminate the surface of interest S via light 3 A for image capture by image capturing device 6. Additional light sources can be included circumferentially around the surface of interest S such that the light source positions are not coincident. The prototype consisted of three such light sources at positions of 0, 120 and 240 degrees circumferentially around the surface of interest S.

FIG. 4 shows three captured images (original images are in color, images have been grey-scaled) of a first representative facial surface with three light sources at 0, 120 and 240 degrees and the blended image created from the three captured images using the darken blending mode of Adobe Photoshop ®. The first representative facial surface of interest has ten (10) elevations and ten (10) depressions of random size and distribution.

FIG. 4A shows a blended image by lighten blending mode of Adobe Photoshop based on the three captured images shown in FIG. 4.

FIG. 5 shows three captured images (original images are in color, images have been grey-scaled) of a second representative facial surface with three light sources at 0, 120 and 240 degrees and the blended image using the darken blending mode of Adobe Photoshop ®. The second representative facial surface of interest has ten (10) elevations and ten (10) depressions of random size and distribution.

FIG. 5A shows a blended image by lighten blending mode of Adobe Photoshop based on the three captured images shown in FIG. 5.

FIG. 6 shows three captured images (original images are in color, images have been grey-scaled) of a third representative facial surface with three light sources at 0, 120 and 240 degrees and the blended image using the darken blending mode of Adobe Photoshop ®. The third representative facial surface of interest has ten (10) elevations and ten (10) depressions of random size and distribution.

FIG. 6A shows a blended image by lighten blending mode of Adobe Photoshop based on the three captured images shown in FIG. 6.

FIG. 7 shows three captured images (original images are in color, images have been grey-scaled) of a fourth representative facial surface with three light sources at 0, 120 and 240 degrees and the blended image using the darken blending mode of Adobe Photoshop ®. The fourth representative facial surface of interest has ten (10) elevations and ten (10) depressions of random size and distribution.

FIG. 7A shows a blended image by lighten blending mode of Adobe Photoshop based on the three captured images shown in FIG. 7. It is to be understood that the embodiments and variations shown and described herein are merely illustrations of the principles of this invention and that various modifications may be implemented by those skilled in the art. For example, an image capturing device (i.e. web camera, digital camera, etc) and repositionable or multiple lighting device(s) may be in communication with a computing device (i.e. a personal computer, embedded computing device, or a like) via a communication means (i.e. wired or wireless communication, wired or wireless network, etc), where the computing device controls the image device and the lighting device(s) for capturing images and processing captured images.

Claims

THE EMBODIMENTS OF THE INVENTION IN WHICH AN EXCLUSIVE PROPERTY OR PRIVILEGE IS CLAIMED ARE DEFINED AS FOLLOWS:

1. An imaging device of facial topography, comprising:

(i) a light source for illuminating a facial surface of interest;

(ii) an image capturing device, capturing at least three images of the facial surface from a fixed position above the facial surface, each captured image having the light source illuminating the facial surface from a different position, circumferentially around the facial surface of interest; and

(iii) an image blending device that blends said at least three images.

2. The imaging device as recited in claim 1, wherein said image blending device blends said at least three images using darken blending.

3. The imaging device as recited in claim 1, wherein said image blending device blends said at least three images using lighten blending.

4. The imaging device as recited in claim 1 , wherein said image blending device compares the digital values at pixel locations of said at least three images with a predetermined threshold, and blends said at least three images by averaging the digital values at a pixel location of said at least three images if the digital values are less than or equal to the predetermined threshold (in the darken blending embodiment) or greater than or equal to the predetermined threshold (in the lighten blending embodiment) for constructing blended image.

5. The imaging device as recited in claim 1, wherein said light source comprises multiple light sources positioned circumferentially evenly around the facial surface of interest.

6. The imaging device as recited in claim 1, wherein said different positions are circumferentially evenly distanced around the facial surface of interest.

7. The imaging device as recited in claim 1, wherein said light source provides tangential lighting illuminating the facial surface of interest.

8. The imaging device as recited in claim 1 further comprises a three-point support device, wherein said three-point support device comprising:

(i) a shoulder support for being contact with the shoulder of a person;

(ii) an occiput support to contact with the occiput of the person;

(iii) an adjustable occiput support arm extended from the shoulder support and connected to the occiput support, being extendible and retractable for adjusting the length of the adjustable occiput support arm;

(iv) a chin support to contact with the chin of the person; and

(v) an adjustable chin support arm extended from the shoulder support and connected to the chin support, being extendible and retractable for adjusting the length of the adjustable chin support arm.

9. A method of imaging facial topography, the method comprising the steps of:

(i) capturing at least three images of a facial surface of interest from a fixed position, each of said at least three images having a light source illuminated from different one of at least three positions which are equally circumferentially distanced around the facial surface of interest;

(ii) overlaying said plurality of images by blending to produce a blended image.

10. The method as recited in claim 9, wherein said blending is darken blending.

11. The method as recited in claim 9, wherein said blending is lighten blending.

12. The method as recited in claim 9, wherein said blending further comprises the steps of, for every pixel position of said at least three images:

(i) comparing the light intensity at said pixel position of said at least three images with a predetermined threshold;

(ii) producing the blended image by averaging the light intensity at said pixel position of said at least three images if the light intensity at said pixel position is less than or equal to the said predetermined threshold in a darken blending approach; or

(iii) producing the blended image by averaging the light intensity at said pixel position of said at least three images if the light intensity at said pixel position is greater than or equal to the said predetermined threshold in a lighten blending approach.

The method as recited in claim 9 comprises further steps of:

(i) prior to capturing said at least three images, fixing the position of the facial surface of interest by using a three-point support device, said three-point support device comprises: a. a shoulder support for being contact with the shoulder of a person; b. an occiput support to contact with the occiput of the person;

c. an adjustable occiput support arm extended from the shoulder

support and connected to the occiput support, being extendible and retractable for adjusting the length of the adjustable occiput support arm; d. a chin support to contact with the chin of the person; and e. an adjustable chin support arm extended from the shoulder support and connected to the chin support, being extendible and retractable for adjusting the length of the adjustable chin support arm.