JP2010531074A - Imaging system and method using desired facial expression recognition - Google Patents

Abstract

  Disclosed is a photography system (12) and method using recognition of desirable facial expressions to improve the quality of portraits. The system and method recognize when a human subject in a photograph has one or more desirable facial expressions for the desired photograph. The system and method can be applied to individual photographs and group photographs, and can be applied in various modes depending on the number of subjects having desirable facial expressions. The system and method may further operate in a “best picture mode” in which a plurality of photographs are taken and the best shot determined as the photograph with the highest percentage of human subjects with desirable facial expressions is selected for storage. it can.

Description

  The technology to which this specification relates generally relates to the field of photography, and more particularly to a photography system and method based on recognition of desirable facial expressions. Desirable facial expressions may include parameters for considering whether a photograph is preferred, such as when the subject person of the photograph is smiling and / or not blinking.

  In addition to stand-alone models, digital cameras are increasingly being incorporated into other electronic devices such as mobile phones, media players, and personal digital assistants (PDAs). As the use of digital cameras increases, users continue to look for new ways to improve their photographs, especially those of people, whether for professional or leisure purposes. Sometimes a photo is taken at the exact moment when the subject is not laughing or blinking. If such a photo was taken with a digital camera, you can see the photo immediately, and you must delete it and retake it. Also, the resolution of the display may be insufficient to determine whether all the subjects in the photo are smiling or open eyes. This is inconvenient and wastes time. When pictures are taken with a conventional film camera, the difficulty is greater because the poor quality pictures are noticed after the film is developed. In such a case, the opportunity for shooting is lost.

  The difficulty of photographing a person is further increased in group photographs. The larger the group, the more difficult it is to get a photo that everyone is smiling or not blinking. The difficulty of group shooting of people is even greater when the people in the group tend not to cooperate like small children.

  In this technical field, a face detection function is known. In face detection, the shape, orientation, and / or color of general facial features are used to detect whether a face is present in an image. However, such techniques have not been successfully incorporated into cameras to ensure that photographs are taken with the desired quality.

  Various camera options are known for improving the quality of photos. For example, the camera operator can adjust settings such as shutter speed, resolution, red-eye prevention and zoom range to increase the probability of a preferred photo. However, there was no camera function that could capture specific facial expressions such as someone laughing or blinking. Timer settings are also known.

  The timer setting allows the camera operator to take a picture with a delay from when the shutter is activated until the picture is taken. In such a function, the delay time is the only factor that defines when the picture is taken. Thus, regardless of whether the resulting photograph is preferred, the photograph is taken at the end of the delay.

  In order to improve the quality of portraits, the subject person to be photographed must satisfy or have a predetermined facial expression desirable for a desired photo, such as when laughing and / or not blinking. What is needed is a system and method for recognition. The system and method according to the present invention can be implemented by a method of notifying an operator when one or more subject persons have a desired facial expression. The system and method can be applied to both personal photos and group photos.

  The system and method of the present invention can operate in a “full mode” where a photograph is taken when all human subjects satisfy or have a desired facial expression. The system and method can also operate in a “proportional mode” where a picture is taken when a predetermined percentage of the human subject is a desirable facial expression. The system and method may further operate in a “best picture mode” in which a plurality of photographs are taken and the best shot determined as the photograph with the highest percentage of human subjects with desirable facial expressions is selected for storage. it can. The system and method may also incorporate a timer setting that allows the camera user to take a picture.

  Thus, according to a first aspect of the present invention, a first method for taking a photograph with a digital camera comprises setting a desired facial expression for a desired photograph and searching for at least one face within the field of view of the camera. And a step of determining whether the face has the desired expression and a step of taking a photo in response to the determination that the face is the desired expression.

  According to one embodiment of the first method, the desired facial expression includes a smiling face in the field of view of the camera.

  According to one embodiment of the first method, the desired facial expression includes open eyes of the face in the field of view of the camera.

  According to one embodiment of the first method, the field of view includes two or more faces, and the method comprises the step of taking a picture when all of the plurality of faces have the desired facial expression.

  According to one embodiment of the first method, the field of view includes two or more faces, and the method includes the step of taking a picture when a predetermined percentage of the plurality of faces is the desired facial expression. Have.

  According to one embodiment of the first method, the digital camera comprises a display, the method comprising indicating to the display the presence of each of the plurality of faces in the field of view; When the face includes only a human subject intended to be photographed, a step of locking on the indicated plurality of faces as a target, and a predetermined ratio of the plurality of faces determined as the target are A step of tracking a plurality of faces determined as the object to determine when the desired expression is present, and a photograph when a predetermined ratio of the plurality of faces determined as the object is the desired expression Photographing.

  According to one embodiment of the first method, the field of view includes more than one face, the digital camera has a display, and the method indicates to the display the presence of each of the faces in the field of view. Determining a plurality of the indicated faces as a target when the plurality of indicated faces include only a human subject intended for photographing; and a plurality of the plurality of faces including the plurality of faces determined as the targets Taking a plurality of photos, displaying the plurality of photos on the display, and selecting which of the plurality of photos has the highest proportion of the desired facial expression.

  According to one embodiment of the first method, the method further comprises the step of deleting a photo other than the photo with the highest proportion of faces with the desired facial expression.

  According to another aspect of the present invention, a second method for taking a photograph with a digital camera comprises setting a desired facial expression for a desired photograph, receiving a delay time, and following the delay time, Searching for at least one face in the field of view of the camera; determining whether the face is the desired expression; and taking a photo in response to determining that the face is the desired expression. According to one embodiment of the second method, the method further comprises the step of alerting the user that a picture has been taken.

  According to one embodiment of the second method, the field of view includes two or more faces, and the method uses the plurality of faces to determine when all of the plurality of faces are the desired facial expression. A step of following, and a step of taking a picture when all of the plurality of faces have the desired expression.

  According to one embodiment of the second method, the field of view includes two or more faces, and the method includes determining the time when a predetermined percentage of the plurality of faces is the desired facial expression. The method further includes the steps of following a plurality of faces, and taking a picture when a predetermined ratio of the plurality of faces is the desired expression.

  According to one embodiment of the second method, the field of view includes two or more faces, the method comprising: taking a plurality of photos including the plurality of faces; and displaying the plurality of photos to the display And a step of selecting which of the plurality of photographs has the highest proportion of the desired facial expression.

  According to one embodiment of the second method, the method further comprises the step of deleting a photo other than the photo with the highest proportion of faces with the desired facial expression.

  According to another aspect of the invention, a camera assembly for taking a field of view and receiving an input of a desired facial expression for a desired photograph and searching for at least one face from the output of the sensor representing the field of view of the camera. And determining whether the face has the desired expression and taking a picture in response to determining that the face has the desired expression.

  According to one embodiment of the camera assembly, the camera assembly further comprises a memory that stores executable code executed by the controller.

  According to one embodiment of the camera assembly, the camera assembly further comprises a display, and the controller executes code to indicate to the display the presence of each of the plurality of faces in the field of view, the controller Further, when the indicated plurality of faces include only a human subject intended to be photographed, the plurality of indicated faces are determined as targets, and the plurality of faces determined as the targets are predetermined. A plurality of faces determined as the target are tracked to determine when a ratio is the desired facial expression, and a photograph is obtained when a predetermined ratio of the plurality of faces determined as the target is the desired facial expression. Run the code to shoot.

  According to one embodiment of the camera assembly, the camera assembly has a display, the controller executes code to indicate to the display the presence of each face in the field of view, the controller further comprising: When the plurality of indicated faces include only a human subject intended to be photographed, the plurality of indicated faces are determined as targets, and a plurality of photographs including the plurality of faces determined as the targets are taken. , Displaying the plurality of photographs on the display, and executing code for selecting which of the plurality of photographs is the photograph having the highest proportion of the face with the desired facial expression.

  According to one embodiment of the camera assembly, the controller further executes code for deleting a photo other than the photo with the highest percentage of the desired facial expression.

  According to one embodiment of the camera assembly, the camera assembly further comprises a timer that delays the execution of the code by the controller and alerts the user that a picture has been taken. And an alarm for receiving.

  These and other features of the present invention will be apparent with reference to the following description and attached drawings. In the following description and drawings, specific embodiments of the present invention are disclosed in detail to illustrate some of the ways in which the principles of the invention may be used, but the scope of the invention is intended to be specific embodiments. It is understood that the present invention is not limited to. Rather, the invention includes all changes, modifications and equivalents within the spirit and terminology of the appended claims.

  Features described and / or illustrated with respect to certain embodiments may be used in the same or similar manner, in combination with or instead of one or more other embodiments and / or functions of other embodiments. it can.

  In this specification, “having” and “comprising” are interpreted as specifying the presence of the described feature, integer, step or part, but one or more other features, integer, step, Emphasize not excluding the existence and addition of parts and their collection.

It is a typical front view of the mobile telephone which has a camera assembly as an example electronic device which concerns on embodiment of this invention. It is a typical back view of the mobile phone of FIG. It is a typical block diagram of the operation part of the mobile telephone of FIG. FIG. 2 is a schematic diagram of a communication system in which the mobile phone of FIG. 1 can operate. 3 is a flowchart illustrating an exemplary imaging method according to the present invention. , , It is a typical figure which shows the example menu system used with the mobile telephone which concerns on this invention. 4 is a flowchart representing an exemplary method for taking a group photo in “full mode” or “proportional mode” in accordance with the present invention. 4 is a flowchart representing an exemplary method for taking a group photo in “best picture mode” in accordance with the present invention. 6 is a flowchart illustrating an exemplary method for taking a group photo using a timer in “full mode” or “proportional mode” in accordance with the present invention. 6 is a flowchart illustrating an exemplary method for taking a group photo using a timer in “best picture mode” in accordance with the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings, wherein like reference numerals denote like elements throughout. It will be understood that the drawings are not necessarily drawn to scale.

  The aspects of the method and system invention relate to photography. The technique described here can be applied to photography using a digital still camera, but the technique described here can be used for a film camera as long as it has a suitable sensor as described here. It will be understood that

  The techniques described herein can be performed by any type of electronic device including a camera. For example, a dedicated still camera may be configured according to the present invention. As another example, many cell phones with cameras can be configured in accordance with the present invention. The following description will be given with respect to a mobile phone with a camera assembly. However, the present invention is not intended to be limited to the content associated with mobile phones, and may relate to any type of suitable electronic equipment. Examples of such an electronic device include a dedicated camera, a media player equipped with a camera, a game device equipped with a camera, and a computer equipped with a camera. For convenience of explanation, the synonyms “electronic device” and “electronic device” include portable wireless communication devices. In the following, “mobile wireless communication device”, sometimes referred to as “mobile wireless terminal”, includes mobile phones, pagers, communicators, electronic notebooks, personal digital assistants (PDAs), smartphones, and any portable communication devices. Including.

  Referring first to FIGS. 1-3, an electronic device device 10 is shown. The electronic device 10 includes a camera assembly 12 that is configured to perform face detection, including smile and blink detection, within the camera's field of view. Further details and operation of the camera assembly 12 will be described later. The detection techniques described herein may exist in the electronic device 10 and may be implemented as code executable by the electronic device 10. In one embodiment, camera assembly 12 and / or electronic device 10 may have a controller or processor that executes a program stored on a computer or apparatus readable medium. The program may be a stand-alone software application or may form part of a software application that performs additional tasks for the electronic device 10.

  The electronic device of the illustrated embodiment is a mobile phone and is therefore called mobile phone 10. Although the mobile phone 10 is illustrated as having a “brick” or “block” shaped housing, other types of housings such as shell-type housings (eg, “flip” open housings) and sliding housings may be used. It will be understood that it is available.

  The mobile phone 10 can include a display 14. The display 14 displays information that allows the user to use various functions of the mobile phone 10 such as an operation state, time, telephone number, contact information, and various menus. Display 14 may also be used to visually display content received by mobile phone 10 and / or read from memory 16 (FIG. 3) of mobile phone 10. Display 14 may be used to present images, video and other graphics, such as photos, mobile TV content and video associated with games.

  The display 14 may also be used as an electronic viewfinder for the camera assembly 12. Furthermore, the mobile phone may include a separate optical viewfinder or electronic viewfinder (not shown in this example), like a stand-alone camera. As an exemplary embodiment, the present description describes a situation where the display 14 is used as a camera viewfinder. It will be appreciated that a separate viewfinder can be used in place of the display 14 within the scope of the present invention.

  The display 14 can also display various indicators 15 that provide information to the user regarding various aspects of the camera function. Such indicators can include items such as battery meter 15a and memory space meter 15b. Display 14 may further display one or more box indicators 15c for use with the present invention. As will be described later, the box indicator 15c can indicate any face within the field of view.

  Keypad 18 provides various user input operations. For example, keypad 18 typically includes alphanumeric keys that allow entry of alphanumeric information such as phone numbers, phone books, contacts, and notes. In addition, the keypad 18 includes special function keys such as a “call send” key for calling and answering and a “call end” key for ending or “hanging up” the call. It is common to include. Special function keys may further include menu navigation and selection keys to facilitate navigation through menus displayed on display 14. Special function keys may include audiovisual content playback keys for starting, stopping, pausing, skipping or repeating tracks, etc. Other keys associated with the mobile phone may include a volume key, a mute key, a power on / off key, a web browser activation key, and the like. Some or all of these keys may be used as soft keys with the display. A key or a function similar to the key may be realized as a touch screen attached to the display 14. When the camera assembly 12 is activated, the keys on the keypad 18 may be used to control the operation of the camera assembly 12. For example, one key may function as the shutter button 17 and the other keys may control the zoom of the camera assembly.

  The mobile phone 10 has a calling circuit / calling device, which is usually another mobile phone or a fixed phone, and a call circuit for the mobile phone 10 to establish a call and / or exchange signals. However, the called / calling device need not be another telephone, and may be another device such as an Internet web server, a content providing server, or the like. The call can take any suitable form. For example, even if the call is a conventional call established over a cellular circuit switched network, a call using the Internet protocol (VoIP) established over the packet switched communication capability of the cellular network or other packet switched network It may be a call. Other packet switching networks include WiFi, WiMax and the like. Another example includes a video call established over a cellular network or other network.

  The mobile phone 10 is used for text messages (for example, colloquially called “SMS”, which is an abbreviation for simple message service), e-mail messages, multimedia messages (for example, “MMS is an abbreviation for multimedia message service). Send, receive and / or process data such as image files, video files, audio files, ringtones, streaming audio, streaming video, data feeds (including podcasts) It may be constituted as follows. Such data processing includes data storage in memory 16, execution of applications that allow the user to manipulate the data, display of video and / or image content associated with the data, audible audio associated with the data. Output may be included.

  FIG. 3 shows a functional block diagram of the mobile phone 10. For the sake of brevity, the general conventional functions of the mobile phone 10 will not be described in detail. The mobile phone 10 includes a main control circuit 20 configured to perform overall control of functions and operations of the mobile phone 10. The control circuit 20 can include a processing device 22 such as a CPU, microcontroller or microprocessor. The processing device 22 executes code stored in an internal memory (not shown) of the control circuit 20 and / or a separate memory such as the memory 16 in order to perform the operation of the mobile phone 10. The memory 16 may be, for example, one or more of a buffer, flash memory, hard disk drive, removable media, volatile memory, non-volatile memory, or other suitable device.

  Still referring to FIGS. 1 to 3, the mobile phone 10 has an antenna 24 connected to a radio circuit 26. The radio circuit 26 includes a radio frequency transmitter and a receiver for transmitting and receiving signals via the antenna 24 as in the related art. The radio circuit 26 may be configured to operate within a mobile communication system and may be used to transmit and receive data and / or audiovisual content. Receiving formats that interact with mobile radio networks and / or broadcast networks include GSM, CDMA5, WCDMA, GPRS, WiFi, WiMax, DVB-H, ISDB-T, etc. It is not limited to.

  The cellular phone 10 further includes an audio signal processing circuit 28 for processing an audio signal transmitted and received by the radio circuit 26. The speaker 30 and the microphone 32 connected to the audio processing circuit 28 allow the user to listen and speak via the mobile phone 10 as before. The radio circuit 26 and the audio processing circuit 28 are each connected to the control circuit 20 so as to execute the entire operation. The audio data may be passed from the control circuit 20 to the audio signal processing circuit 28 for reproduction to the user. The audio data may include, for example, audio data from an audio file stored by the memory 16 and read by the control circuit 20, or received audio data such as a streaming audio data format from a mobile radio service. The audio processing circuit 28 can include any suitable buffer, decoder, amplifier, and the like.

  The display 14 may be connected to the control circuit 20 via a video processing circuit 34 that converts the video data into video data used to drive the display 14. Video processing circuit 34 may include any suitable buffer, decoder, video data processor, and the like. The video data may be generated by the control circuit 20, read from a video file stored in the memory 16, or obtained from an incoming video data stream received by the wireless circuit 28, or any other suitable May be obtained by various methods.

  The mobile phone 10 may have one or more I / O interfaces 36. The I / O interface 36 may be in the form of a typical mobile phone I / O interface and may include one or more electrical connectors. As is often the case, the I / O interface 36 may be used to connect the mobile phone 10 to a charger to charge the battery of a power supply unit (PSU) 38 inside the mobile phone 10. Additionally or alternatively, the I / O interface 36 may provide a function of connecting the mobile phone 10 to a headset assembly (eg, a personal hands-free (PHF) device) that has a wired interface with the mobile phone 10. Further, the I / O interface 36 may provide a function of connecting the mobile phone 10 to a personal computer or other device via a data cable for data exchange. The mobile phone 10 may receive operating power when connected to a car battery adapter or an AC adapter.

  The mobile phone 10 may further include a system clock 40 that provides a clock to various components of the mobile phone 10 such as the control circuit 20. The cellular phone 10 may further include a position data receiver 42 such as a GPS (Global Positioning System) receiver, a Galileo satellite system receiver, or the like.

  The mobile phone 10 further includes a local wireless interface 44, such as an infrared transceiver and / or RF adapter (eg, a Bluetooth adapter) for establishing communications with accessories, other mobile wireless terminals, computers or other devices. May be. For example, the local wireless interface 44 may operatively connect a headset assembly (eg, a PHF device) having a corresponding wireless interface in one embodiment with the mobile phone 10.

  With further reference to FIG. 4, the mobile phone 10 may be configured to operate as part of the communication system 46. The system 46 manages a call originated by the mobile phone 10 and a call directed to the mobile phone 10, sends data to the mobile phone 10, and performs one or more servers 50 that perform other support functions. You may have a communications network 48 that you have. Server 50 communicates with mobile phone 10 via a transmission medium. The transmission medium may be any suitable device or assembly including, for example, a communication tower (eg, a base station), other mobile phones, wireless access points, satellites, and the like. Part of the network may have a wireless transmission path. The network 48 may support communication activities of multiple mobile phones 10 and other types of end user devices. As will be understood later, the server 50 may be configured as a normal computer system used to perform server functions, and is configured to execute software including logical instructions that implement the functions of the server 50. A processor and a memory for storing such software may be included.

  With continued reference to FIGS. 1-3, the camera assembly 12 may have a controller 52, such as a digital signal processor (DSP). In other embodiments, some or all of the control functions of the camera assembly 12 described as being performed by the controller 52 may be performed by the control circuit 20, in which case the controller 52 may be reduced or unnecessary. Good. Controller 52 controls various aspects of camera assembly 12 operation. Such operations include, but are not limited to, photometric operations and exposure setting management. The controller 52 can coordinate the storage of photographs and / or video data taken with the camera assembly 12. For example, photo and / or video data may be stored in memory 16 in a corresponding photo and / or video file.

  The controller 52 can execute code to perform camera assembly 12 control functions. Those skilled in the art of computer programming, and in particular those of application programming for cameras, mobile phones or other electronic devices, may use a camera and / or to operate and perform logic functions related to the control of the camera assembly 12. It will be clear how to program the mobile phone. Therefore, for the sake of brevity, details regarding specific program codes are omitted. Also, such code is executed by the controller 52 according to an exemplary embodiment, but performs such functions through dedicated hardware, firmware, software, and combinations thereof within the scope of the present invention. You can also.

  In the illustrated embodiment, the camera assembly 12 is a digital camera that uses a sensor 54 to image the field of view determined by the imaging optics 56 of the camera assembly. The exposure meter 58 can detect the illumination state of the visual field. A flash 60 may be present to provide auxiliary illumination during shooting.

  Referring to FIGS. 5, 7, 8, 9 and 10, logic operations for implementing an exemplary method of operation of camera assembly 12 are shown. An exemplary method can be performed by the controller 52 executing code. Accordingly, the flowcharts of FIGS. 5, 7, 8, 9 and 10 can be considered to represent the method steps performed by the camera assembly 12. 5, 7, 8, 9 and 10 show a specific execution order of functional logic blocks, the execution order of the blocks may be changed with respect to the order shown. Also, two or more consecutive blocks shown in the figure may be executed simultaneously or partially simultaneously. It is understood that all such variations are within the scope of the present disclosure.

  FIG. 5 shows an embodiment of the method of the present invention relating to the shooting of a single individual. At step 65, one or more desired facial expressions for the photo to be taken are set. The desired facial expression represents a parameter that the user would consider acceptable photograph characteristics. In one embodiment, the desired facial expression includes the presence of a smile. In another embodiment, the desired facial expression includes no blinking. In the third embodiment, desirable facial expressions include the presence of a smile and the absence of blinking.

  As an example, the method of the present invention will be described primarily with respect to taking pictures where one or more human subjects are smiling as much as possible. However, it will be appreciated that the desired facial expression may comprise any facial expression or combination thereof that the photographer deems desirable. Desirable facial expressions may further include environmental factors such as specific highlight levels or lack of movement. Therefore, various desirable facial expressions can be envisaged within the scope of the present invention.

  There are various ways in which a desirable facial expression can be set. In the exemplary embodiment, the mobile phone 10 has selectable operating modes and the user can select from a general main menu. As an example, a mode representing the function described here is named “time to smile”. The smile shooting mode can be selected from a general menu displayed on the display 14. Alternatively, the smile shooting mode may be entered by pressing one or more buttons on the keypad 18 that are assigned exclusively. When the smile shooting mode is entered, a desired facial expression can be selected. In one embodiment, the desired facial expression is present and fixed at the time of manufacture. Alternatively, as shown in FIG. 6A, by selecting the smile shooting mode, the mobile phone 10 displays a submenu 300 including options of desired facial expressions on the display 14, and the user selects the submenu options according to the situation. May be. In FIG. 6A, exemplary choices include smile mode, blink mode, smile plus blink, and others (for user defined desired facial expressions). For purposes of this example, the user selects the smile mode to maximize the probability that a photo will be taken when the person subject is laughing.

  Referring again to FIG. 5, once the desired facial expression is set, the method proceeds to step 70 where the camera searches the field of view to detect the face. In an exemplary embodiment, the user may initiate face detection by pressing the shutter button 17 (see FIG. 1) halfway. In this technical field, half-pressing is used to start predetermined camera functions such as red-eye prevention and autofocus. Similarly, face detection may be started. In one aspect of the embodiment, when a face is detected in the field of view, a box indicator 15 c (see FIG. 1) is generated around the face of the subject on the display 14. The frame display may be colored green or red to improve visibility.

  When a face is detected, the user can fully press the shutter button 17 to start smile detection for the face of the subject, and step 80 in FIG. 5 is started. To lock on a subject's face means that smile detection is performed only for that specific subject. The determination of the target is visually indicated on the display by the fact that the frame display 15c remains around the determined face, but no new frame display is displayed. Therefore, faces other than the target subject are not tracked by smile detection. For example, even if a person in the immediate vicinity enters the background of the field of view, the person's face is not subject to smile detection. In step 80, the camera determines whether the subject's face has the desired facial expression set in step 65. In the context of this example, the camera determines whether the human subject is laughing.

  If no smile is detected, the camera stays at step 80 and continues monitoring and tracking until a smile is detected. If so, the method proceeds to step 90 where a picture is taken. In one embodiment, the method may incorporate a “time out” function. Using this function, the method is aborted if no smile is detected within a certain number of seconds.

  Face and facial expression (for example, smile and / or blink) detection can be implemented as follows. As described with respect to FIG. 3, when implemented, the camera assembly 12 includes a sensor 54 for photographing the field of view of the imaging optical system 56. Face and facial expression detection can be realized by modifying known face detection software to identify whether the subject is smiling, blinking, or has other desirable facial expressions. This software may be programmed into the phone and stored as executable code in the memory 16 and executed by the camera controller 52. Alternatively, these functions may be executed by the control circuit 20 of the mobile phone 10.

  Although the present invention has been described with respect to personal photography, the method may also be useful in improving group photography. An embodiment of the method applied to the group photo is shown in FIG. As in the previous example, as shown in step 100, the user first enters the smile shooting mode and sets a desired facial expression. In step 110, the user can press the shutter button 17 halfway, and the camera starts face detection to search for a face in the field of view. Here, the intended photograph includes two or more faces.

  It is often desirable to take a group photo in a crowded place such as a tourist spot or a park. In such a place, a person other than the intended person subject may enter the field of view. Camera users will want to wait until such a person goes out of sight, and when in the background, they don't care if they are looking at the camera or laughing. Therefore, it is necessary to identify the intended subject and a third party.

  Therefore, in step 120, when the shutter button 17 is half-pressed, the display shows each detected face. As described above, this display can be realized, for example, by generating a frame display 15 c around each detected face on the display 14. In this example, the frame display 15c is green. When the green frame is displayed only around the intended human subject, the camera user can press the shutter button fully and proceed to step 130. In step 130, the camera determines a face in the green frame as a target. That is, the camera process is applied to the intended human subject, excluding third persons who may be in the field of view. Similar to the above-described shooting by an individual, determination of a target for a plurality of faces in a group photo means that smile detection will be performed for those target subjects. The determination of the object is visually indicated on the display by the fact that the frame display 15c remains around the face determined as the object, but no new green frame is displayed. Therefore, faces other than the target subject are not tracked for smile detection. For example, even if a third person enters the background of the field of view, the third person's face is not subject to smile detection.

  In step 140, the camera tracks the face determined for the object and performs smile detection to determine if the face determined for the object is laughing (the desired facial expression set in this example). If it is determined that all are laughing, the method proceeds to step 150 and a photo is taken.

  If not impossible, it is unlikely that everyone in the group will have the desired facial expression at the same time. This may be the case when the group is very large or a group of individuals that are not cooperative, such as small children. In such a situation, it would be preferable to be able to take a picture even if not everyone in the group is fully satisfied with the desired facial expression. In one embodiment, step 140 requires that the desired facial expression be fully satisfied, and this function is referred to as “full mode”.

  As an alternative to the full mode step 140 in FIG. 7, the method may require that a certain percentage of human subjects meet the desired facial expression, which is referred to as “proportional mode”. In the proportional mode, step 140a is executed instead of step 140, as indicated by a dotted line in FIG. In step 140a, the face determined as the object is tracked to determine when a predetermined percentage of the subject has a desired facial expression. In the exemplary embodiment, the proportional mode may be an option in the smiley shooting submenu 310, as shown in FIG. 6B. To set the ratio, the mobile phone 10 can generate a prompt on the display 14 requesting the ratio (the user may enter the ratio using the keypad 18). The percentage can be any amount less than the number of faces being tracked. The percentage may be a common percentage such as 50 percent, 75 percent. Alternatively, the total number may be set as a part (subset) of the green frame in the display 14. For example, when the display 14 displays 10 green frames, 10 faces are tracked, but the user may input a ratio such as 7,8. In this way, the user can take a picture without waiting until everyone in the group has the desired facial expression. It should be understood that the actual method for setting the ratio may vary from the method described above within the scope of the present invention.

  FIG. 8 shows another embodiment for taking a group photo. This embodiment is referred to as “best picture mode”. In contrast to the proportional mode, in the best picture mode, no picture is taken even if it is detected that a predetermined ratio satisfies a desired facial expression. A plurality of photographs are taken and displayed, and a photograph containing the highest desired expression can be selected from them. The remaining photos may be deleted.

  Referring to FIG. 8 in more detail, the method of this embodiment proceeds in the same manner as the above-described embodiment. As shown in step 200, the user first enters a smile shooting mode and selects a desired facial expression. At step 210, the user can press the shutter button halfway and the camera begins face detection to search for a face in the field of view. In step 220, the display shows each detected face with the shutter button half pressed. As described above, this display can be realized, for example, by generating a frame display 15c around each detected face on the display 14 by the camera. If it is confirmed that a frame display appears around the human subject intended by the camera user, the method proceeds to step 230. Step 230 may be initiated by the user pressing the shutter button 17 fully. In step 230, the camera determines the face in the frame display, i.e. the intended human subject, to exclude third parties who may enter the field of view.

  The best picture mode that is the embodiment of FIG. 8 is different from the proportional mode that is the embodiment described above from step 240. Rather than tracking the face determined as the object, in the best picture mode, the camera takes a plurality of pictures of the scene including the face determined as the object. At step 250, a plurality of photos are displayed on the display 14. Then, at step 260, smile detection may be applied to the multiple photos for determination and selection of “best picture”. A “best picture” may be defined as the highest percentage of human subjects that are laughing (or have one or more selected desired facial expressions). The selected best picture may be held by storing it in the memory 16 or the like. In one embodiment, the best picture may be automatically selected by the camera assembly 12 based solely on the percentage that satisfies the desired facial expression. Alternatively, the user may view a plurality of photos on the display 14 and manually select the best picture.

  Referring to FIGS. 9 and 10, a time delay 410 or 510 is added to the method described herein. Timer mode is known in the art, which generally allows camera users to be included in a group photo. The timer function may be executed in cooperation with the controller 52 of the camera assembly 12 and the system clock 40 of the mobile phone 10. In other embodiments, a separate timer (not shown) may be included as part of the camera assembly 12. The user can set a delay time. When the shutter button 17 is pressed, the timer starts counting down. Alternatively, the user can use a conventional remote control to activate the shutter mechanism or start a timer countdown. During the delay, the camera user may determine his position so that he is included in the picture. With the conventional timer, a picture is taken when the delay time elapses. Such prior art timer methods have the disadvantage that shooting takes place at the end of the delay time, regardless of whether the resulting photograph is desirable.

  In the exemplary embodiment, the user can enter the timer mode from the smiley shooting submenu 320 (see FIG. 6C) or through another menu (step 320). Following the setting of the desired facial expression, the user can enter a delay time, for example, the number of delay seconds using the keypad 18. Alternatively, the user can use a conventional remote control to activate the shutter mechanism or start a timer countdown. During the delay, the user can take a position such that he is included in the photograph taken. In contrast to the traditional timer method, the camera does not take a picture at the end of the delay time. Instead, face detection is started and the method proceeds according to steps similar to those described above.

  FIG. 9 illustrates the use of the timer function as part of full mode or proportional mode. The user sets a desired facial expression (step 400) and inputs a delay time (step 410). After pressing the shutter button 17 or the remote control button to start the timer countdown, in step 420 the camera searches for a face in the field of view. When using the timer, the camera user can leave the camera once the shutter mechanism is activated. Therefore, the display step and the object determination step (corresponding to step 120 and step 130 in FIG. 7) may not be used. In full mode, at step 430, the face is tracked until everyone has the desired facial expression. On the other hand, as shown by a dotted line in FIG. 9, step 430a can be used in the proportional mode, and the face is tracked until a predetermined ratio of the face becomes a desirable expression. In step 440, a photograph is taken. In step 450, a warning function may notify the user (which will be in front of the camera) that a picture has been taken. As an example, the alarm may be emitted from a camera flash 60, an independent LED indicator (not shown), or may be a sound from a speaker 30.

  FIG. 10 shows the use of the timer function as part of the best picture mode. The user sets a desired facial expression (step 500) and inputs a delay time (step 510). After pressing the shutter button 17 or the remote control to start the timer countdown, the camera searches for a face in the field of view at step 520. Again, when using the timer, the camera user can leave the camera once the shutter mechanism is activated. Therefore, the display step and the object determination step (corresponding to step 220 and step 230 in FIG. 8) may not be used. Multiple photos are generated at step 530. Thereafter, in step 540, the alarm function may inform the user that a photo has been generated. In step 550, a plurality of photos are displayed. The user then returns to the camera to see a plurality of photos and selects the best photo (step 560). The user may delete photos other than the best photo (step 570). Alternatively, as described above, the camera assembly 12 may automatically select the best photo.

  6A-6C illustrate an exemplary scroll down menu system for selecting various variations of the methods described herein. The user can start the operation by selecting “smile shooting” from a previous main menu (not shown) or by pressing a dedicated button on the keypad 18. The display 14 will then display a submenu 300 (FIG. 6A) for selecting options for the desired facial expression. The above example has been described as the user selecting the “smile mode”. FIG. 6A illustrates other possibilities for desirable facial expressions, such as “blink mode” and “smile and blink”. An “Other” category is shown, which would allow a user-defined desired facial expression. After the desired facial expression is selected, the display 14 may display the second submenu 310 (FIG. 6B). Through the second submenu, the user can select “full” for full satisfaction of the desired facial expression, “proportional” for proportional mode, or “best” for best picture mode. After this selection is made, the third submenu 320 (FIG. 6C) may provide the user with an option to use the timer.

  It should be understood that other menu systems can be used within the scope of this disclosure. Alternatively, one or more buttons on the keypad 18 can be assigned to various combinations of possible modes. Such dedicated assignment can be preset at the time of manufacture and / or defined by the user.

  While the invention has been illustrated and described with respect to specific embodiments, it will be understood that, upon reading and understanding this specification, those skilled in the art can implement equivalents and variations. All such equivalents and modifications are included in the present invention, and the present invention is limited only by the scope of the claims.

Claims (10)

A method of taking a picture using a digital camera (12),
Setting a desired facial expression for the desired photo;
Searching for at least one face in the field of view of the camera;
Determining whether a face has the desired facial expression;
Taking a picture when it is determined that the face has the desired facial expression. The field of view includes a plurality of faces;
The method of claim 1, further comprising the step of taking a picture when a predetermined percentage of the plurality of faces has the desired facial expression. The digital camera (12) has a display (14);
Said method comprises
Displaying the presence of each of the plurality of faces in the field of view on the display;
Determining the plurality of displayed faces as targets when the displayed plurality of faces includes only the face of a human subject intended for shooting;
Tracking the plurality of faces determined for the object to determine when a predetermined percentage of the plurality of faces determined for the object has the desired facial expression;
The method of claim 2, further comprising: taking a picture when a predetermined proportion of the plurality of faces determined as the target has the desired facial expression. The field of view includes a plurality of faces;
The digital camera (12) has a display (14);
Said method comprises
Displaying the presence of each of the plurality of faces in the field of view on the display;
Determining the plurality of displayed faces as targets when the displayed plurality of faces includes only the face of a human subject intended for shooting;
Taking a plurality of photographs including a plurality of faces determined as the object;
Displaying the plurality of photos on the display;
The method according to claim 1, further comprising: selecting a photograph having the highest proportion of faces having the desired facial expression from the plurality of photographs. A method of taking a picture using a digital camera (12),
Setting a desired facial expression for the desired photo;
Accepting a delay time;
Following the delay time, searching for at least one face in the field of view of the camera;
Determining whether a face has the desired facial expression;
Taking a picture when it is determined that the face has the desired facial expression. The field of view includes a plurality of faces;
Said method comprises
Tracking the plurality of faces to determine when a predetermined percentage of the plurality of faces has the desired facial expression;
6. The method of claim 5, further comprising the step of taking a picture when a predetermined percentage of the plurality of faces has the desired facial expression. A sensor (54) for photographing the field of view;
Receiving an input of a desired expression for a desired photograph, searching for at least one face from an output of the sensor representing the field of view of the camera, determining whether a face has the desired expression, and determining whether the face has the desired expression A camera assembly (12), comprising: a controller (52) for taking a picture in response to the determination of having the camera.   The camera assembly (12) of claim 7, further comprising a memory (16) for holding executable code executed by the controller (52). It further has a display (14),
The controller (52) executes the code to display on the display the presence of each of the plurality of faces in the field of view;
The controller (52) further determines the plurality of displayed faces as targets when the plurality of displayed faces include only faces of human subjects intended to be photographed, and is determined as the target. In order to determine when a predetermined ratio of a plurality of faces has the desired expression, the plurality of faces determined as the target are tracked, and a predetermined ratio of the plurality of faces determined as the target 9. A camera assembly (12) according to claim 7 or claim 8, wherein said code is executed to take a picture when said has the desired facial expression. It further has a display (14),
The controller (52) executes the code to display on the display the presence of each of the plurality of faces in the field of view;
The controller (52) further determines the plurality of displayed faces as targets when the plurality of displayed faces include only faces of human subjects intended to be photographed, and is determined as the target. Taking a plurality of photographs including a plurality of faces, displaying the plurality of photographs on the display, and selecting the code for selecting the one having the highest proportion of faces having the desired facial expression from the plurality of photographs. Camera assembly (12) according to claim 7 or 8, characterized in that it is implemented.

Images