Face detection and/or recognition
FIELD OF THE INVENTION
The present invention relates to face detection and/or recognition, and in particular, face detection and/or recognition using visible light and infrared light.
BACKGROUND OF THE INVENTION
One of the most successful and growing applications of image analysis is face recognition, especially in recent times when security applications are being developed. A prerequisite to reliable face recognition is face detection, whereby face detection is used to find a face in a scene, while face recognition is used to identify a face. Various techniques have been used in the process of face detection. One technique involves the use of visible light, whereby color-based detection is used to detect a face. Another known technique uses infrared imagery, whereby thermal information is used to detect faces. Both of these techniques have various disadvantages, as will be explained below.
Face detection using visible light relies on being able to accurately detect skin color.
RGB is the most common color space. It is three-dimensional and is shaped like a cube. It consists of three channels, each having one of the three colors: Red, Green and Blue. Each channel represents one of the three axes of the cube. The values of each channel may vary from 0 to 255. Thus, each different color of the 3-dimensional cube can be represented as a mixture of Red, Green and Blue color quantities.
The YUV color space is also a three-dimensional cube, as shown in Fig. 1. The Y is the luminance and it is the brightness of a color image, as it would be displayed in a black and white monitor. The U and V are just components of the color signal so that the color image can be reconstructed. The Y value can vary from 0 to 255 whereas the U and the V can have values from — 128 to +128. In YUV, the Black and White colors exist in the center of every UV rectangle, and then dark colors appear and get lighter until they reach the outer perimeter of the UV rectangle. It is noted that not all possible combinations of YUV in this cube lead to a valid value in the RGB space.
Furthermore, it is very cumbersome to use RGB color space for segmentation because of its complex 3-dimensional structure. Skin color is rather unique, and to indicate skin color on the RGB spectrum requires certain calculations because of the complex 3- dimensional structure. As a result, segmenting a color in RGB requires working in 3- dimensional space, which in turn has the disadvantage of requiring a large amount of calculation time. To avoid this problem, an RGB pattern is normally converted into YUV where Y is the intensity and UV purely indicates the colors.
Skin color comprises a region in the YUV space, and is generally more difficult to indicate on the YUV spectrum than solid colors such as red, orange, blue, green, etc.
Fig. 2 shows a schematic diagram of face detection using a visible light camera 21, which provides information to a face detection apparatus 23, which is passed to a face recognition apparatus 25 for identifying a face. Face detection using visible light is not reliable for a number of reasons. First, as indicated above, skin color is difficult to indicate on the U and V pattern because of its large color range. This makes the detection more complicated because other colors can also be detected (e.g. the color red). Secondly, skin color is light dependent; which means that if there is less light emitted on the face, the detection will deteriorate. For example, a fluorescent lamps emits more green light which means that, if too much light is emitted, the face will change color. This drawback makes it difficult to find optimum emitted light every time face detection is enabled. Thirdly, to detect darker skin correctly, the values of U and V have to be changed because the skin tone is slightly different. All these factors make the decision interval of U & V larger, and increase the probability of false detection.
As discussed above, another alternative is to use an infrared camera for face detection. Infrared face detection uses thermal information from faces. As a result, infrared is impervious to variations in illumination. Infrared also makes segmentation easier against "cold" backgrounds. However, infrared-based face detection also has its disadvantages, in that infrared is totally reflected by glass, thus causing problems when having to detect the faces of people wearing spectacles. Also, temperature profiles vary drastically based on a person's activity level or the time of day, thus making face detection using infrared unreliable.
SUMMARY OF THE INVENTION
The aim of the present invention is therefore to provide an improved face detection and/or recognition.
The invention is defined by the independent claims. The dependent claims define advantageous embodiments.
According to a first aspect of the invention, there is provided an apparatus for detecting and/or recognizing a face in a scene. The apparatus comprises a visible light sensor for producing a visible light signal of the scene, and an infrared sensor for producing an infrared signal of the scene. The apparatus is configured to dynamically control whether the visible light signal and/or the infrared signal is used for detecting and/or recognizing the face, based on real-time measurements taken in the apparatus.
The invention has the advantage of being able to dynamically determine whether the visual light or infrared light, or both, is used for face detection and/or recognition depending on real time measurements taken from the apparatus. According to another aspect of the invention, there is provided a method of detecting and/or recognizing a face in a scene. The method comprises the steps of providing a visible light sensor for producing a visible light signal of the scene, and providing an infrared sensor for producing an infrared signal of the scene. The method involves dynamically controlling whether the visible light signal and/or the infrared signal is used for detecting and/or recognizing the face, based on real-time measurements taken in the apparatus.
In one embodiment, face detection and/or recognition are performed under varying light and temperature conditions.
BRIEF DESCRIPTION OF THE DRAWINGS For a better understanding of the invention, and to show more clearly how the invention may be carried into effect, reference will now be made, by way of example only, to the following drawings in which:
Fig. 1 shows a YUV color space;
Fig. 2 shows a conventional visible light face detection and recognition system;
Fig. 3 shows a face detection and recognition system according to the present invention;
Fig. 4 shows a face detection and recognition system according to another embodiment of the present invention; and
Fig. 5 shows a face detection and recognition system according to a further embodiment of the present invention.
DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT OF THE PRESENT INVENTION
Fig. 3 shows a face detection and/or recognition apparatus 30 according to the present invention. The face detection and/or recognition apparatus 30 comprises a visual light sensor or camera 31 and an infrared sensor or camera 33. A mixer 35 receives a visual light signal 32 from the visual light camera 31, and an infrared signal 34 from the infrared camera 33. The output of the mixer is provided to a face detection unit 36, which provides a face detection signal 40. The face detection signal 40 can be used, for example, by a face recognition unit 37 for identifying a particular face.
The face detection unit 36 is also connected to a feedback control unit 38, which provides a feedback signal 39 for controlling the mixer 35. In other words, based on the success of the face detection being carried out in the face detection unit 36, the feedback control unit 38 dynamically varies how the visible light signal 32 and infrared signal 34 are combined in the mixer 35.
By using a visible light camera 31 and an infrared camera 33, not only will the detection improve but also the recognition. In other words, the recognition depends on the detection quality. If the detection is not reliable the recognition can also be considered as unreliable.
The feedback control unit 38 influences the mixing or combining of the mixer 35 based on measurements taken in the face detection unit 36. The mixer can be controlled to perform an AND operation, an OR operation or a weighted combination of the visible light signal 32 and the infrared signal 33.
For example, a simple test from the face detection unit 36 could be: IF (the spectrum of visual light does not permit robust detection, i.e. there is not enough Blue, Green or Red light)
THEN More Mix-in of the Infrared signal ELSE More Mix-in of the visual light signal
ENDIF
Optionally, an output from the face recognition unit 37 can also be connected to the feedback control unit 38, such that the operation of the face recognition unit 37 can
also be used to dynamically influence how the visible light signal 32 and the infrared signal 34 are combined.
For example, the recognition unit 37 could rely on features in the infrared domain: IF (the face in the visual light domain cannot be recognized)
THEN Switch to the infrared domain (partly) to carry out face recognition using infrared features ENDIF
With the aid of the visible and infrared cameras, the invention enables skin based colors and heat emitted from face(s) to be detected, thus allowing face detection and/or recognition to be made more robust. The detection is reliable because two different cameras are detecting, one is detecting the desired visible light conditions and the other on heat glowing surfaces. With the combination of the visible light camera and infrared camera, face detection and/or recognition becomes more precise. If the visible light sensor is detecting on skin color and infrared on the heat emitted from a face, the detection quality increases. The detection feasibility can be adapted on the demands of the user.
As described above, it is possible to detect with both sensors in the "AND" or "OR" modes. For example, if the Infrared sensor is not detecting anything while the visible light sensor is detecting skin color because of a certain circumstance, the processor or the user can decide to place the infrared detection in a standby mode. Also, the invention can be operated such that both of the sensors are used for detection at the same time, thus allowing' precise detection. The detection and/or recognition is also safer because "real" faces have to be applied and not mere photographs. Preferably, the face detection and/or recognition apparatus is configured to use the infrared signal whenever possible, since the infrared camera can detect a human body/face without changing the RGB space into U and V to detect a skin color. This feature requires no additional calculation time. Using an infrared camera to detect a face is easier than using a visible light camera, because the infrared signal is light independent. This means that light does not have any influence on the detection. The output value of an IR camera shows the temperature of the object observed. Therefore, humans can be found by scanning in a temperature region. Using an infrared camera is also advantageous because the temperature of a human face is above 29 0C and below 42 0C. Furthermore, darker skin colors
are detected easier because the detection is based on the observed temperature of the face and not the skin color.
Fig. 4 shows a face detection and/or recognition apparatus according to another embodiment of the present invention. As with Fig. 3, a visual light sensor or camera 31 produces a visual light signal 32, while an infrared sensor or camera 33 produces an infrared signal 34. However, according to this embodiment, separate face detection units 36a and 36b are provided for the visible light signal 32 and the infrared signal 34, respectively. The face detection signals 40a and 40b from the face detection units 36a and 36b are connected to a mixer 35, which generates an output under the control of a feedback control unit 38. The feedback control unit 38 is connected to receive measurements from the visual light face detection unit 36a and the infrared face detection unit 36b, such that it can control the mixer 35 according to the operation of the face detection units 36a, 36b.
In other words, if the face detection unit 36b based on the infrared signal 34 is providing an accurate output, then the feedback control unit 38 controls the mixer 35 such that the signal 40b passes to the face recognition unit 37. As before, the mixer 35 can operate in an AND, OR or weighted combination of the face detection signals 40a and 40b.
Optionally, the apparatus can also be configured such that the feedback control unit 38 receives a feedback signal from the face recognition unit 37, for assisting in controlling the mixer 35. Fig. 5 shows a face detection and/or recognition apparatus according to another embodiment of the present invention. As with Figs. 3 and 4, a visual light sensor or camera 31 produces a visual light signal 32, while an infrared sensor or camera 33 produces an infrared signal 34. As with Fig. 4, separate face detection units 36a and 36b are provided for the visible light signal 32 and the infrared signal 34, respectively. However, according to this embodiment, the face detection signals 40a and 40b from the face detection units 36a and 36b are connected to separate face recognition units 37a and 37b. The face recognition units 37a and 37b provide face recognition signals 47a and 47b to a mixer 35, which generates an output signal under the control of a feedback control unit 38. The feedback control unit is connected to receive measurements from the visual light face detection unit 36a and the infrared face detection unit 36b, such that it can control the mixer 35 according to the operation of the face detection units 36a, 36b.
In other words, if the face detection unit 36b based on the infrared signal is providing an accurate output, then the feedback control unit 38 controls the mixer 35 such
that the signal 47b passes to the output. As before, the mixer can operate in a AND, OR or weighted combination of the face recognition signals 47a and 47b.
The apparatus can also be configured such that the feedback control unit 38 receives feedback signals from the separate face recognition units 37a and 37b, for assisting in controlling the mixer 35.
Optionally, in this embodiment the face recognition unit 37a for visible light can be provided with visible light features 45 for enhancing the face recognition process. Also, the face recognition unit 37b for infrared can be provided with infrared features 43 for enhancing the face recognition process. Both the embodiments shown in Figs. 4 and 5 have the advantage of being able to prevent the use of photographs to mislead the system. Furthermore, an increased face temperature will indicate activity, mischief or illness, which can be used to selectively look at people. Fig. 5 also offers recognition in both the infrared and visual light domains, which is beneficial to distinguish similar looking people more accurately. - The invention described above enables skin based colors and heat emitted from face(s) to be detected, thus allowing face detection to be made more robust. The detection is reliable because two different cameras are detecting, one is detecting the desired visible light conditions, while the other is detecting heat glowing surfaces. With the combination of the visible light camera and infrared camera, face detection becomes more precise and robust. If the visible light sensor is detecting on skin color and infrared on the heat emitted from face, the detection quality increases. The detection feasibility can be dynamically controlled using the feedback control unit.
Furthermore, infrared technology can be used to focus on recognizing a living person, thus preventing photograph fraud. Although the embodiments described in Figs. 4 and 5 have been described as using feedback signals from both the face detection units 36a and 36b for controlling the mixer 35, it will be appreciated that the feedback can also be controlled based on just one of these signals. Similarly, although the embodiment of Fig. 5 has been shown using the signals from both the face recognition units 37a and 37b in the feedback control unit 38, it will be appreciated that just one of these signals could be used.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. The word
'comprising' does not exclude the presence of elements or steps other than those listed in a claim.
In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.