JP4877762B2 - Facial expression guidance device, facial expression guidance method, and facial expression guidance system - Google Patents

Description

  The present invention relates to a user interface technology based on facial expression recognition.

When shooting a person, the scene of making a shout like “laughing” or “high, cheese” is common. This can be said to be a technique for taking a picture of a person who becomes a subject (hereinafter referred to as “photographed person”) with a suitable expression.
JP 2005-293539 A

  In the first place, human expression should appear as a natural expression of emotion. The present inventor has conceived that more suitable shooting is possible if the camera is provided with a mechanism for naturally guiding the facial expression of the subject.

  The present invention has been made in view of the above-mentioned attention of the inventor, and its main purpose is to realize a user interface that draws out a natural expression of the subject.

One embodiment of the present invention is a facial expression guidance device.
When imaging a user's face, this apparatus displays a captured image of the user on a screen that can be visually recognized by the captured user, and captures a guided object for guiding the user's facial expression to a desired facial expression. Display above. While displaying the guidance object, the user's facial expression is detected, and the degree of matching between the user's facial expression and the desired facial expression is notified on the screen.

  The guiding object here is not character information for directly inducing a facial expression such as “Please laugh” but computer graphics for indirectly inducing a facial expression by a character or a symbol. The guiding object that produces the joy of shooting naturally guides the photographer's facial expression to a desired facial expression. If the photo opportunity is guided to a desired facial expression, the subject can be photographed with the desired facial expression.

  In addition, what expressed this invention with the method, the system, the program, and the recording medium is also effective as an aspect of this invention.

  ADVANTAGE OF THE INVENTION According to this invention, the user interface which draws out a to-be-photographed person's natural expression can be provided.

FIG. 1 is a hardware configuration diagram of the facial expression guidance device 100.
A camera 102 and a monitor 104 are connected to the facial expression guidance device 100. The monitor 104 is a general television monitor. The camera 102 is installed on the upper part of the monitor 104 and images the user 106. In the figure, a user 106 is a photographer and a subject. The facial expression guidance device 100 displays the captured image on the monitor 104. Therefore, the user 106 who is the subject can view his / her captured image on the monitor 104. The camera 102 periodically images the user 106 at a predetermined time interval, for example, at intervals of 1/30 second to 1/60 second. In this way, the camera 102 captures the user 106 as a plurality of captured images. In other words, the user 106 is captured as a moving image.

  In this embodiment, the subject is continuously imaged for a predetermined imaging period, for example, 30 seconds. One captured image is selected from a plurality of captured images acquired during this imaging period, and is definitely recorded in an auxiliary storage device such as a hard disk. A captured image recorded as a still image in this way, in other words, a captured image recorded as a photograph is hereinafter referred to as a “captured image”. In this embodiment, the captured image is a moving image that is temporarily recorded in the built-in memory of the facial expression guidance device 100, and the captured image is a still image that is definitely recorded on the internal hard disk of the facial expression guidance device 100.

  The facial expression guidance device 100 displays the guidance object 180 on the monitor 104 when the user 106 is imaged by controlling the camera 102. The guidance object 180 is associated with various facial expressions. For example, a guidance object 180 that imitates “? (Question mark)” is associated with a “suspicious expression”. The user 106 changes his / her facial expression in response to the guidance object 180. The facial expression guidance apparatus 100 matches the facial expression suggested by the guidance object 180 (hereinafter referred to as “suggested facial expression”, in which “suspicious facial expression” is the suggested facial expression) and the actual facial expression of the user 106. Determine. The facial expression guidance device 100 notifies the degree of fitness (hereinafter referred to as “expression similarity”) via the monitor 104. For example, the display mode of the guidance object 180 is changed according to the facial expression similarity. The display mode of the guidance object 180 will be described in detail with reference to FIG.

  The facial expression guidance device 100 selects a captured image when the facial expression similarity exceeds a predetermined threshold as a “captured image”. With such a processing method, it is possible to acquire a photographed image after guiding the facial expression of the user 106 to the suggested facial expression. In this embodiment, the facial expression guidance device 100 will be described as a stationary device that guides the facial expression of the user 106 by controlling the monitor 104 and the camera 102 at home. As another example, the function of the facial expression guidance device 100 may be incorporated in an existing digital camera. In that case, the digital camera may have the monitor 104 built-in, or the captured image or the guiding object 180 may be displayed on the external monitor 104.

FIG. 2 is a functional block diagram of the facial expression guidance device 100.
Each block shown here can be realized in hardware by an element such as a CPU of a computer or a mechanical device, and in software it is realized by a computer program or the like. Draw functional blocks. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by a combination of hardware and software.
Here, with regard to the functions to be exhibited by the respective blocks, specific actions thereof will be described with reference to FIG.

The facial expression guidance device 100 includes a user interface processing unit 110, a data processing unit 130, and an image holding unit 170.
The user interface processing unit 110 is responsible for processing related to the entire user interface, such as input processing from the user of the facial expression guidance device 100 and information display to the user. The data processing unit 130 executes various data processing based on an input operation via the user interface processing unit 110. The data processing unit 130 also serves as an interface between the user interface processing unit 110 and the image holding unit 170. The image holding unit 170 stores various image data such as captured images and captured images received from the data processing unit 130. The image holding unit 170 is realized by functions of a main storage device such as a memory and an auxiliary storage device such as a hard disk.

The user interface processing unit 110 includes an input unit 112, an imaging unit 114, and a display unit 120. The input unit 112 receives an operation input from the user. The imaging unit 114 periodically acquires captured images via the camera 102. The display unit 120 displays various information to the user via the monitor 104. The display unit 120 includes a guidance display unit 122 and a captured image display unit 124. The guidance display unit 122 displays a guidance object 180 described later on the monitor 104. The captured image display unit 124 displays the captured image on the monitor 104.
In addition, the user interface processing unit 110 also provides voice input, voice output, input via a button interface, and the like as a type of user interface of the facial expression guidance device 100.

  The data processing unit 130 includes a facial expression processing unit 140, a guidance processing unit 150, and an image recording unit 160. The facial expression processing unit 140 is in charge of the facial expression recognition process of the subject. The guidance processing unit 150 is in charge of processing for guiding the user's facial expression to the suggested facial expression. The image recording unit 160 selects a captured image from the captured image and records it in the image holding unit 170.

  The facial expression processing unit 140 includes a head specifying unit 142 and an expression determining unit 144. The head specifying unit 142 specifies an image area corresponding to the face of the subject in the captured image. Hereinafter, an image area corresponding to a face in a captured image is simply referred to as a “face area”. In addition, image data corresponding to the face area in the captured image is simply referred to as a “face image”. An existing technique may be applied to specify the face area. For example, a reference image obtained by synthesizing several tens to several hundreds of face images is prepared in advance. The head specifying unit 142 sets a rectangular area in the captured image, compares the reference image with the image of the rectangular area, and digitizes the similarity as an image. By appropriately calculating the similarity while setting various rectangular areas in the captured image, the “face-like” area in the captured image is specified as the face area.

  The facial expression determination unit 144 detects the facial expression of the subject. The image holding unit 170 is an image corresponding to various human facial expressions such as “angry facial expression”, “sad facial expression”, “laughing facial expression”, “confused facial expression”, “disgusting facial expression”, etc. Holding. These images are called “expression patterns”. The facial expression determination unit 144 calculates the “similarity as an image” between the facial region of the subject and the facial expression pattern of the suggested facial expression as the “expression similarity”. In this way, the similarity between the subject's facial expression and the suggested facial expression is indexed as the facial expression similarity. As a method for calculating the expression similarity, an existing technique for image matching may be applied. Hereinafter, the facial expression pattern corresponding to the suggested facial expression is referred to as a “suggested facial expression pattern”.

  The facial expression pattern may be an image typified for each facial expression as described above, but is data indicating the characteristics of each facial expression (hereinafter referred to as “facial expression feature information”). May be. For example, an “angry expression” may be characterized by wrinkles between the eyebrows, and a “laughing expression” may be characterized by lowering the corners of the eyes. In this way, facial expression feature information is numerical information indexed for various items such as the angle of the eyebrows, the angle of the corner of the eye, and the shape of the mouth. Therefore, how much the facial expression of the subject is matched with the facial expression feature information of the suggested facial expression may be quantified as the facial expression similarity.

  The guidance processing unit 150 includes an object selection unit 152 and an object adjustment unit 154. The object selection unit 152 selects the guidance object 180 corresponding to any one of a plurality of types of suggested facial expressions. The guidance display unit 122 displays the selected guidance object 180 on the monitor 104 so as to overlap the captured image. Instead of the object selection unit 152, the user may select the guidance object 180 to be displayed.

  The object selection unit 152 may select a plurality of types of guidance objects 180 at random. Alternatively, the guiding object 180 may be selected according to the imaging condition. The imaging conditions here are conditions relating to the number, sex, shooting date and time, and shooting environment of the person to be shot. For example, when a plurality of face images are detected from the captured image, that is, when there are a plurality of subjects, the guiding object 180 may be selected according to the number of subjects. Depending on the number of subjects, such as “suspicious expression” when there is one subject, “laughing facial expression” when there are three or more subjects, and “angry facial expression” when there are five or more subjects. To select the guidance object 180. Further, when there are a plurality of subjects, the guidance object 180 may be selected according to the sex ratio. Alternatively, the object selection unit 152 may select the guidance object 180 according to the brightness of the captured image. For example, a “facial expression of laughter” may be used as the suggested facial expression when the captured image is bright, and a “sad expression” when the captured image is dark.

  The subject changes his / her facial expression according to the guidance object 180. A facial expression similarity between the facial expression of the subject and the suggested facial expression pattern is calculated by the facial expression determination unit 144. The object adjustment unit 154 determines the display mode of the guidance object 180 according to the calculated facial expression similarity. For example, when the facial expression similarity is large, that is, when the subject has a facial expression close to the suggested facial expression, the guidance object 180 is displayed large. The guidance display unit 122 appropriately changes the display mode of the guidance object 180 in accordance with an instruction from the object adjustment unit 154. Specific display modes will also be described in detail with reference to FIG.

  The image recording unit 160 includes an image quality control unit 162 and a synthesis unit 164. The image quality control unit 162 controls the image quality of the captured image, particularly the image quality of the face area. In addition, when there are a plurality of subjects, the synthesis unit 164 generates a single shot image by combining the face images of each person. The image quality control will be described later in detail with reference to FIGS. 5A and 5B, and the face image synthesis will be described later with reference to FIG.

FIG. 3 is a screen diagram when the guiding object 180 is displayed on the captured image.
FIG. 3A is a screen diagram when the guidance object 180 is displayed. When the imaging unit 114 acquires a captured image of the subject via the camera 102, the captured image display unit 124 causes the monitor 104 to display the captured image on the screen. At this time, the guidance display unit 122 displays the guidance object 180 shown in the figure near the face of the subject. Here, it is assumed that the object selection unit 152 randomly selects the guidance object 180. A guiding object 180 in the figure is a graphic imitating “?”. In general, “?” Is a symbol reminiscent of “Troublesome” such as “I don't know” or “What is it?”. The subject captures the meaning contained in the guiding object 180 and changes his / her facial expression. The suggested facial expression of the guiding object 180 shown in the figure is a “suspicious expression” and is associated with the suggested facial expression pattern of “suspicious expression”.

  FIG. 3B is a screen diagram when the photographed person changes his / her facial expression in accordance with the guiding object 180. The head specifying unit 142 specifies a face area from the captured image after the guidance object 180 is displayed. The facial expression determination unit 144 compares the facial area with the suggested facial expression pattern to calculate the facial expression similarity. When the facial expression similarity exceeds a predetermined threshold, that is, when the subject's facial expression matches the suggested facial expression, the object adjustment unit 154 changes the display color of the guiding object 180 to a dark color. 122 is instructed. In the figure, since the subject can successfully create a suspicious expression with a neck that corresponds to the guidance object 180, the guidance display unit 122 changes the color of the guidance object 180 to a dark color. As a result, the photographer recognizes that he / she successfully made a facial expression.

  As another method for notifying that the facial expression creation is successful, the shape and position of the guiding object 180 may be changed. As the shape change, a method of changing the size of the guiding object 180, and a method of changing the position may be a method of rotating the guiding object 180. On the other hand, if the facial expression similarity does not exceed a predetermined threshold, the color of the guiding object 180 changes to a light color. As a result, the subject recognizes that the facial expression has failed. Alternatively, the failure to make a facial expression may be notified by changing the guiding object 180 to a dark color or displaying it large. In this way, the subject object 180 may be alerted by changing the guidance object 180 to a display mode that is easy to visually recognize.

In this way, since the guidance object 180 just like a manga symbol expression changes the atmosphere of the entire captured image, the photographer's facial expression can be naturally led to a suggested facial expression. The guiding object 180 need not be limited to the symbols shown in the figure, and may be a manga character image. When taking a picture, the photographer often requests the subject to smile, such as “high, laugh”. However, the smiles that are made when “Laughter” is ordered must be artificial smiles. Also, in such shooting, it is unlikely that an instruction such as “high, angry” or “high, with a strange face” will be given. Even if such an instruction is actually given, it is not easy for the subject to make an angry expression or a suspicious expression.
In contrast, the guiding object 180 can be guided to various suggested facial expressions without difficulty. In particular, when a child is a subject, various facial expressions can be naturally extracted. The guidance object 180 and the facial expression determination mechanism improve the interactiveness in shooting, and fun shooting that stimulates a sense of play is realized.

  As a modification, the guidance processing unit 150 may change the display mode itself of the captured image instead of the guidance object 180 or in addition to the guidance object 180. For example, when the suggested facial expression is “angry facial expression”, the guidance processing unit 150 may instruct the guidance display unit 122 to display the entire captured image in red. When the suggested facial expression is a “laughing facial expression”, the guidance processing unit 150 may display the entire captured image in a bright color such as yellow. Further, the guidance processing unit 150 may guide the subject's facial expression to a “surprising facial expression” by performing an effect that the subject does not expect, such as distorting the entire captured image. The display mode of the captured image itself may be changed according to the expression similarity. For example, the guidance processing unit 150 may display the captured image at a higher resolution as the facial expression similarity is higher, and may display the captured image at a lower resolution as the facial expression similarity is lower.

FIG. 4 is an example of a flowchart showing a process of obtaining a photographed image by guiding the facial expression of the subject.
The process shown in the figure is a loop process periodically executed at every acquisition timing of the captured image during the imaging period. The imaging unit 114 acquires a captured image of the subject from the camera 102 (S10). The captured image display unit 124 causes the user 106 to display the captured image (S12).

  The head specifying unit 142 specifies the face area of the subject from the captured image (S14). Here, it is assumed that there is one person to be photographed. The guidance display unit 122 displays the guidance object 180 on the captured image (S16). It is assumed that the guidance object 180 to be displayed is previously selected by the object selection unit 152 according to the imaging conditions. The display mode at this time is as shown in FIG. The facial expression determination unit 144 compares the face area of the subject and the suggested facial expression pattern, and calculates the facial expression similarity (S18). If the facial expression similarity is equal to or greater than a predetermined threshold and the subject's facial expression matches the suggested facial expression (Y in S20), the image recording unit 160 uses the captured image acquired in S10 as a captured image as the image holding unit 170. (S22). If not (N in S20), the process returns to S10.

  During the imaging period, the processes from S10 to S20 are repeatedly executed at predetermined time intervals. If the subject's facial expression matches the suggested facial expression within the imaging period, the photographed image is recorded, and the process ends. Alternatively, a captured image when the facial expression similarity is the highest among a plurality of captured images obtained during the imaging period may be recorded as a captured image.

  The image quality control unit 162 may increase or decrease the resolution of the captured image in conjunction with the level of facial expression similarity. Such a processing method motivates the photographed person to make a good expression because “a good face is a good picture”. Not limited to the resolution, the image quality control unit 162 may change the background image in the captured image according to the expression similarity and add various character objects to the captured image. As described above, the image quality control unit 162 changes the content of the captured image according to the facial expression similarity, thereby making it easier to guide the subject's facial expression to the suggested facial expression.

  The image recording unit 160 may record a captured image including the guidance object 180. For example, if a photographed image including the guiding object 180 is recorded when a sticker photograph such as a photo booth (registered trademark) is photographed, the subject's interest in photographing can be further deepened. When there are a plurality of subjects, a captured image obtained when the average value or the minimum value of the facial expression similarity of each subject exceeds a predetermined threshold may be recorded as a captured image.

FIG. 5 is an example of a screen diagram when the guiding object 180 is displayed on the captured image when there are a plurality of subjects.
FIG. 5A is a screen diagram when the guidance object 180 is displayed. In the figure, five subjects A to E are subjects of imaging. The head identifying unit 142 detects the face areas of five people. The object selection unit 152 selects the guiding object 180 for each of the four subjects other than the subject E. The guidance display unit 122 displays the selected guidance object 180 in the vicinity of the face areas of the four people A to D. The subject A is displayed with a guidance object 180 that suggests a “disappointed expression”. In addition, a guidance object 180 that suggests a “facial expression” is displayed on the subject B. Thus, the guidance object 180 corresponding to general manga expression guides the photographed person to the suggested facial expression. The subject changes his / her facial expression according to the guidance object 180 for the subject.

  FIG. 5B is a screen view when the photographed person changes his / her facial expression in accordance with the guiding object 180. Here, the facial expressions of the subjects A, C, and D match the suggested facial expressions, but the facial expressions of the subject B do not match the suggested facial expressions. In other words, the facial expression similarity between the facial expression of the subject B and the suggested facial expression pattern does not exceed the threshold value. As described above, when there is only one photographed person whose facial expression similarity does not exceed the threshold among the plurality of photographed persons, the guidance display unit 122 displays the prompt object 182 for the corresponding photographed person B. . According to such a processing method, it is possible to introduce a play sensation of competing for facial expressions between subjects when photographing subjects as a group photo.

  The guidance display unit 122 may display a predetermined object (hereinafter referred to as “applicable object”) to a subject who has the highest expression similarity among a plurality of subjects. Or you may notify the magnitude | size of a facial expression similarity by the number and movement of a conformity object. Further, the guidance object 180 may be displayed only to the subject who has a low expression similarity. For example, in FIG. 5B, the guiding object 180 may be displayed only on the subject B. According to such a display method, the guiding object 180 can be used as a prompt for guiding a subject who has not made a good expression to a predetermined expression.

  Prior to shooting, the user may register the number of subjects. Imaging or photographing may be started on the condition that the number of registered face areas has been detected by the head identifying unit 142. When the number of detected faces is less than the registered number, the display unit 120 may display a message prompting the gathering on the screen.

FIG. 6 is another example of a screen view when the guiding object 180 is displayed on the captured image when there are a plurality of subjects.
FIG. 6A is a screen diagram when the guidance object 180 is displayed. In the figure, five subjects A to E are subjects of imaging. The head identifying unit 142 detects the face areas of five people. The object selection unit 152 selects the guidance object 180 that imitates a star mark for all subjects. The star mark guiding object 180 has a “smiling facial expression” as a suggested facial expression. The guidance display unit 122 causes a plurality of guidance objects 180 to float over the entire captured image. When the guide object 180 imitating a star is displayed on the entire screen, each photographed person changes his / her facial expression to a laughing facial expression.

  FIG. 6B is a screen diagram when the photographed person changes his / her facial expression in accordance with the guidance object 180. Here, it is assumed that, among the subjects A to E, the facial expression similarity of the subject C is the highest, and the facial expression similarity of the subject D is the lowest. Then, the guidance display unit 122 moves the guidance object 180 so as to gather more around the face of the subject C. On the other hand, the guiding object 180 moves away from around the subject D. Even with such a processing method, it is possible to introduce a play sensation of competing for facial expressions among subjects.

  The image quality control unit 162 may record the photographed image with the face of the photographed person having a large expression similarity among the plurality of photographed persons being large and the face of the photographed person having a small expression similarity being small. The image quality control unit 162 may record a photographed image with a high resolution for the face of the subject having a high expression similarity and a low resolution for the subject having a low expression similarity. As described above, the image quality control unit 162 changes the content of the captured image according to the facial expression similarity, so that it is possible to introduce a play sensation of competing for facial expressions among subjects.

FIG. 7 is another example of a flowchart showing a process of obtaining a photographed image by guiding the facial expression of the subject.
In the figure, the processing contents shown in S10 to S16 are the same as the processing contents denoted by the same reference numerals in FIG. The process shown in the figure is also a loop process that is repeated at predetermined time intervals. First, it is assumed that there is one person to be photographed. In S18, the facial expression determination unit 144 calculates the facial expression similarity and then records the facial expression similarity. Further, the captured image at this time is temporarily recorded in the image holding unit 170. When a new captured image is acquired in S10 of the next loop, the facial expression similarity is calculated again in S18. At this time, the facial expression determination unit 144 determines whether the facial expression similarity calculated for the new captured image is greater than the facial expression similarity already recorded (S30). In other words, it is determined whether the captured image newly acquired is closer to the suggested facial expression than the previously acquired captured image. If the maximum value of facial expression similarity is not updated (N in S30), the process returns to S10, and analysis for the next captured image is started. If updated (Y in S30), the newly acquired captured image is overwritten and recorded as a captured image over the previously recorded captured image (S32). When such processing is repeated and the imaging time has elapsed (Y in S34), the processing ends.
With the above processing, it is possible to automatically select a captured image when the facial expression similarity is the highest during the imaging period, that is, a captured image when the facial expression of the subject is closest to the suggested facial expression.

  When there are a plurality of subjects, the process shown in FIG. 7 may be executed for each subject. The image recording unit 160 selects a captured image when the average value of the facial expression similarity of all members is the largest as a captured image. The image recording unit 160 may synthesize a face image that is the best shot for each person to be photographed to generate a single photographed image. Finally, a method for synthesizing such an image will be described.

FIG. 8 is a time chart for explaining the process of synthesizing one photographed image from a plurality of face images of subjects.
Here, the imaging over shooting period _t 0 ~t _n the target participant A-E. Specifically, a total of n + 1 captured images of t ₀ , t ₁ , t ₂ ,..., T _n are acquired. That is, the process shown in FIG. 7 is executed for each person to be photographed n + 1 times.

Of n + 1 of captured images, approaches the expression is most suggested expression of the photographer A is obtained captured image at time t _2. The image recording unit 160 temporarily records the face image of the photographer A from the captured image at the time t _2. The face of the subject B at time t _i (3 <i <j <n), the subject C at time t ₃ , the subject D at time t _n , and the subject E at time t _j Images are temporarily recorded. In this way, the facial image that is the best shot when the facial expressions of the five subjects are closest to the suggested facial expression is acquired. However, the acquisition timing of each face image does not exactly match.

Combining unit 164, any time during the imaging period, for example, among the captured image at time t _n, by replacing the face region of each person to be photographed and best shot of the face image to generate a composite image. Specifically, among the captured image at time t _n, to detect the face region of the person to be photographed A, replacing the face area and the photographer A face image at the time t _2. Similarly, the face areas of the subjects B, C, and E are detected, and the face areas are replaced with the face images of the subjects B, C, and E at the times t _i , t ₃ , and t _j . Face image of the photographer E are replaced because the captured image is the best shot at time t _n is not executed. With such a processing method, the best shot for each subject can be reflected in the group photo. Conventionally, when taking a group photo, it is common to take a plurality of photos in advance and select the most suitable photo from them. On the other hand, according to the facial expression guidance device 100, a suitable group photo can be taken without causing such work.

The facial expression guidance device 100 has been described above.
In the present embodiment, it has been described that the facial expression guidance device 100 acquires a captured image when the facial expression of the subject and the suggested facial expression match. However, the present invention is not limited to this, and a predetermined process such as a computer game command may be executed on the condition that the facial expression of the subject and the suggested facial expression are matched. I will add a note about such an example.

  When the execution of the computer game A is started, the object selection unit 152 dynamically selects various guidance objects 180 at a predetermined timing. The user needs to make a facial expression on the guidance object 180 every time the guidance object 180 is switched. If the facial expression is suitable for the guidance object 180 within a predetermined time, for example, 3 seconds after the guidance object 180 is switched, one point is added as a matching point. One point is subtracted. The matching points start from 50 points, and when the score is 0, the game is over. The matching points may be used as game scores as they are. The object selection unit 152 performs control so that the difficulty of the game is gradually improved by increasing the switching frequency of the guidance object 180 or increasing the types of the guidance objects 180 as the game progresses. With such a processing method, it is possible to realize a game characteristic corresponding to the degree of matching between the facial expression of the subject and the suggested facial expression suggested by the guidance object 180.

  Various application scenes can be considered for the expression guidance function by the guidance object 180. For example, the guidance object 180 may be displayed on the entrance intercom and processing may be performed so that the key can be opened on condition that the visitor smiles. The effect of cooling down the emotions of visitors can be exhibited by authentication processing based on facial expression guidance rather than simple face authentication. In addition to the guiding object 180, the subject's facial expression may be guided by sound effects. Further, not only facial expressions but also gesture gestures may be targeted. For example, the emotion suggested by the guiding object 180 may be expressed by gestures such as nagging, raising the neck, stretching the chest, and so on.

  The subject of facial expression guidance may be all of the subject or a part thereof. As shown in the present embodiment, the guidance object 180 may be displayed so that the subject can recognize that he is the subject of facial expression guidance. The guiding object 180 may be displayed so as not to know that it is the target. For example, in FIG. 5A, the guiding object 180 is displayed at a predetermined position on the imaging screen, for example, at the upper right corner. The guidance object 180 is actually displayed for the subject A, but the subjects A to E do not know whether they are subject to facial expression guidance. The shutter is released when the facial expression similarity of the subject A exceeds a predetermined threshold. According to such a processing method, it is possible to realize a photographing apparatus having an element of play that does not know who has the authority to release the shutter.

  In the above, this invention was demonstrated based on the Example. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications can be made to the combination of each component and each processing process, and such modifications are also within the scope of the present invention. is there.

The functions of the guidance display unit and the user interface processing unit described in the claims are mainly realized by the guidance display unit 122 in this embodiment.
It should be understood by those skilled in the art that the functions to be fulfilled by the constituent elements described in the claims are realized by a single function block or a combination of the functional blocks shown in the present embodiment.

It is a hardware block diagram of a facial expression guidance apparatus. It is a functional block diagram of a facial expression guidance device. FIG. 3A is a screen diagram when the guidance object is displayed. FIG. 3B is a screen diagram when the photographed person changes his / her facial expression in accordance with the guidance object. It is an example of the flowchart which shows the process in which a to-be-photographed person's facial expression is induced | guided | derived and a picked-up image is acquired. FIG. 5A is a first screen diagram when a guiding object is displayed when there are a plurality of subjects. FIG. 5B is a first screen diagram when the photographed person changes the facial expression according to the guidance object. FIG. 6A is a second screen diagram when a guiding object is displayed when there are a plurality of subjects. FIG. 6B is a second screen diagram when the photographed person changes the facial expression according to the guidance object. It is another example of the flowchart which shows the process in which a to-be-photographed person's facial expression is induced | guided | derived and a picked-up image is acquired. It is a time chart for demonstrating the process of synthesize | combining one picked-up image from the several to-be-photographed person's face image.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 Facial expression guidance apparatus, 102 Camera, 104 Monitor, 106 User, 110 User interface processing part, 112 Input part, 114 Imaging part, 120 Display part, 122 Guidance display part, 124 Captured image display part, 130 Data processing part, 140 Facial expression Processing unit 142 head identification unit 144 expression determination unit 150 guidance processing unit 152 object selection unit 154 object adjustment unit 160 image recording unit 162 image quality control unit 164 composition unit 170 image holding unit 180 guidance Object, 182 Prompt object.

Claims (13)

An imaging unit that acquires a captured image of the user's face from the camera;
A user display unit that displays a captured image of the user on a screen that is visible to the user being imaged during imaging;
A guidance display unit for displaying a guidance object for guiding a user's facial expression to a desired facial expression on the captured image;
A facial expression detection unit for detecting the facial expression of the user;
A user interface processing unit for notifying on the screen the degree of matching between the detected facial expression of the user and the desired facial expression;
A facial expression guidance device comprising:   The facial expression guidance device according to claim 1, wherein the user interface processing unit notifies the fitness level by instructing the guidance display unit to change a display mode of the guidance object.   The guidance display unit selects a facial expression to be guided according to success or failure of a predetermined imaging condition and displays a guidance object corresponding to the selected facial expression on the captured image. The facial expression guidance device described. When capturing multiple users at the same time,
The facial expression guidance device according to claim 1, wherein the user interface processing unit notifies the degree of matching between a facial expression detected for a predetermined user among the plurality of users and the desired facial expression. When capturing multiple users at the same time,
The facial expression guidance device according to claim 1, wherein the user interface processing unit notifies the degree of matching between each of a plurality of types of facial expressions detected for the plurality of users and the desired facial expression. When capturing multiple users at the same time,
The user interface processing unit notifies a user whose degree of matching between the detected facial expression and the desired facial expression is a predetermined value or less on the screen among the plurality of users. Facial expression guidance device. As a photographing process, an image recording unit that deterministically records a user's captured image on a predetermined recording medium,
The facial expression according to claim 1, wherein the user interface processing unit determines a timing at which the detected facial expression of the user matches the desired facial expression as an execution timing of the photographing process. Guidance device.   The expression guide according to claim 7, wherein the image recording unit changes an image quality of a captured image in the photographing process according to a degree of matching between the detected facial expression of the user and the desired facial expression. apparatus. When capturing multiple users at the same time,
The user interface processing unit determines the execution timing of the photographing process for each user, with the timing at which the facial expression detected for each user is matched with the desired facial expression as the execution timing,
The facial expression guidance device according to claim 7, wherein the image recording unit combines and records each user's face image at each execution timing into a single captured image.   The facial expression guidance device according to claim 7, wherein the image recording unit records a captured image including a guidance object. A camera that images the user;
A display device that allows the user being imaged to visually recognize the screen during imaging;
A facial expression guidance device,
The facial expression guidance device includes:
An imaging unit that acquires a captured image of the user's face from the camera;
A user display unit that displays a captured image of the user on the display device during imaging;
A guidance display unit for displaying the derived object to induce the user's expression in the desired expression superimposed on the captured image of the display device,
A facial expression detection unit for detecting the facial expression of the user;
A user interface processing unit for notifying on the screen of the display device the degree of matching between the detected facial expression of the user and the desired facial expression;
A facial expression guidance system characterized by comprising: Displaying a guidance object for guiding a user's facial expression to a desired facial expression over the user's captured image on a screen that is visible to the user to be imaged;
Detecting a user's facial expression;
Deterministically recording a captured image at a timing at which the detected facial expression of the user and the desired facial expression are matched;
A facial expression guidance method characterized by executing A function of displaying a captured image of the user's face on a screen visible to the user being imaged during the period in which the user is imaged;
A function of displaying the derived object to induce the user's expression in the desired expression superimposed on the captured image,
A function to detect the user's facial expression;
A function for executing a predetermined process on condition that the detected facial expression of the user and the desired facial expression are matched;
A facial expression guidance program characterized by letting a computer demonstrate its function.

Images