JP2006331065A - Face information transmitting device, face information transmitting method and recording medium with its program recorded - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a face information transmitting device for generating a natural avatar composite image without misalignment in the direction of the face of a user even when the user faces sideways in avatar communication. <P>SOLUTION: This face information transmitting device is provided with a first face direction determination part and a second face direction determination part as two determination parts whose thresholds are different, and the direction of the face is shifted to a follow-up operation state that an avatar follows face characteristics and an autonomous operating state that the avatar moves by using already stored face movement according to those determination parts. The face direction threshold of the second face direction determination part changing from the autonomous operating state to the follow-up operation state is set so as to be narrower than the face direction threshold of the first face direction determination part changing from the follow-up operation state to the autonomous operation state so that even when the follow-up operation state is held in a long time, and the autonomous operation state is temporarily set, the movement of the face is shifted to the follow-up operation state in such configurations that the directions of the avatar face and the user face do not misaligned. Thus, it is possible to generate a natural avatar composite image without misalignment in the direction of the face of a user even when the user faces sideways. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a face information transmitting apparatus, a face information transmitting method, and a recording medium storing the program for transmitting an avatar image such as a character by CG to a partner instead of transmitting an actual face when making a videophone. It is.

  In recent years, with the development of communication tools such as mobile phones, camera mounting has become the standard, and in addition to transmitting voice information and text information, there are increasing cases of transmitting real-time image information such as videophones.

  In such real-time video communication, the user is hesitant to transmit his / her face image as it is, but there is a demand for feeling and intention to be communicated to the other party.

In order to satisfy this demand, in Patent Document 1, the position of a facial feature point is extracted from the acquired face image, and a character CG image (hereinafter referred to as an avatar image) reflecting this position information is generated and transmitted to the other party. is doing.
JP 2004-236186 A

  However, in Patent Document 1, it is premised that the feature points of the face can always follow, and when the face turns sideways, the feature points of the face cannot be taken and cannot follow. There is no description of how to deal with such cases, and if it is not possible to follow, normally, a place other than the face is taken as the face, so the movement of the face of the avatar image becomes indefinite and unnatural movement .

  Furthermore, once it becomes impossible to follow, there is a problem that it will be further away from the face while searching for feature points in the vicinity, and it will not be able to return to normal unless it is manually initialized and the face is searched again from the entire image. It was.

  SUMMARY OF THE INVENTION The present invention solves the above-described conventional problems, and an object thereof is to provide a face information transmitting apparatus capable of generating a natural avatar composite image that does not deviate in the face direction even when the user turns the face sideways.

  In the present invention, the face orientation is estimated, and transition is made to two states, a follow-up action state and an autonomous action state, depending on the face orientation. At this time, a first face direction determination unit for threshold determination processing based on a face direction for switching the state, an avatar operation correction unit for performing avatar composition processing by tracking operation by determination, an avatar composition processing by autonomous operation by determination, and an avatar operation generation unit It has.

  In this configuration, switching to two states of tracking and autonomous with the threshold of the face direction makes it switch autonomously even if the face is facing sideways and cannot follow the facial parts, and naturally follows if it is within the threshold. You can control your avatar.

  In the face information transmitting apparatus according to the second invention, there are two threshold values for switching between the follow-up operation state and the autonomous operation state, the first face direction determination unit and the second face direction determination unit, and which one is determined according to the operation state of the previous frame. A decision branch control unit that determines whether to use the unit.

  In this configuration, the threshold condition of the face direction that changes from tracking to autonomous and the threshold condition of the face direction that changes from autonomous to tracking are different, and in the case of one threshold, switching between tracking and autonomous is frequent when the face direction moves near the threshold. A natural state switch can be realized while it occurs.

  In the face information transmitting apparatus according to the third aspect of the invention, the face direction threshold value of the first face direction determination unit is set wider than the face direction threshold value of the second face direction determination unit.

  In this configuration, the threshold to change from tracking to autonomy is wide, and the threshold to change from tracking to autonomy is narrow, so that the tracking operation by the face part can be continued for a long time. Transition is possible with the face direction of the person to be photographed close.

  In the face information transmitting apparatus according to the fourth aspect of the present invention, the transition to the follow-up operation state is not made until the subject person becomes a front face in the autonomous operation state.

  In this configuration, the avatar face gradually returns to the front direction when it enters the autonomous operation state, and the tracking operation state is entered only when the avatar face and the face of the subject person coincide with each other in the front. A natural transition can be realized by matching the avatar face and the face direction of the subject person.

  In the face information transmitting apparatus according to the fifth aspect of the invention, the threshold value of the first face direction determining unit is not more than the detectable range in which the face part can be detected.

  In this configuration, face parts can always be detected in the tracking state, and stable tracking can be realized.

  In the face information transmitting apparatus according to the sixth aspect of the invention, even if the subject person does not become a front face when in the autonomous operation state, the face information transmission device transitions to the follow-up operation state if it matches the face direction of the avatar face at that time.

  In this configuration, if the face orientations match, it immediately transitions to the follow-up action state and can maintain the follow-up action state for a long time, so communication from nodding and swinging to facial expressions can be enhanced through the avatar face. .

  The present invention has the following effects. Even when the user turns sideways in the video communication, the avatar image autonomously moves without being interrupted, so that the natural movement of the avatar image can be shown to the other party.

  Embodiments of the present invention will be described below with reference to the drawings.

(Embodiment 1)
Next, embodiments of the present invention will be described with reference to the drawings. 1 is a functional block diagram of a face information transmitting apparatus according to Embodiment 1 of the present invention, FIG. 2 is a flowchart of the same face information transmitting apparatus, and FIG. 4 is a block diagram of the same face information transmitting apparatus.

  Before explaining the details of each part, the flow of processing of the image processing apparatus of this embodiment will be explained with reference to FIG. First, a face is found from the input image, and a feature amount of the face is extracted (step 1), and the face orientation is estimated from the feature amount (step 2). Then, it is determined whether the follow-up action or the autonomous action is based on whether the face orientation is within the threshold value or more (step 3).

  Here, the tracking operation is an operation that extracts the face position, face orientation, face tilt, face action, etc. from the extracted facial feature amount, and synthesizes and displays the avatar image in accordance with this. It is.

  Furthermore, the autonomous operation is an operation of synthesizing and moving an avatar image in accordance with an already stored facial motion or a previously stored facial motion, regardless of the extracted facial feature amount.

  In the case of the follow-up motion, an avatar motion correction process is performed in which only the difference in the position of the facial feature from the previous frame is corrected to determine the avatar motion (step 4).

  In the case of autonomous movement, if the face direction is other than front, avatar movement generation processing is performed (step 5).

  After performing the avatar motion correction process (step 4) and the avatar motion generation process (step 5), the created avatar image is transmitted to the other party by the avatar transmission process (step 6).

  The avatar image is transmitted while repeating the above steps 1 to 6 for each frame.

  Next, the configuration of the image processing apparatus of this embodiment will be described. FIG. 4 shows an example in which the elements of FIG. 1 are specifically configured. That is, in FIG. 4, a CPU (Central Processing Unit) 20 is stored in a ROM (Read Only Memory) 21 and executes an image processing program according to the flowchart of FIG. Control each element.

  In the RAM (random access memory) 22 and the non-volatile RAM 23, in addition to the areas for the respective storage units shown in FIG.

  Each processing unit shown in FIG. 1 is realized by the CPU 20 executing an image processing program stored in the ROM 21. Further, this program can be stored in a known storage medium such as the nonvolatile RAM 23 or an external storage memory.

  In the example of FIG. 4, a camera 25 is connected to the interface 24 so that an image that may include an object can be acquired in real time. The reason why the image is supposed to be included is that it is unknown whether the object is included before detection. Note that the camera 25 may be a CCD or CMOS module, or a camera attached to a mobile phone.

  Next, the overall configuration of the present invention will be described using the functional block configuration diagram of FIG.

  In FIG. 1, the face feature extraction unit 1 detects a face from input image data in the input image storage unit 9, extracts a face feature amount, and stores the face feature amount in the face feature storage unit 10. FIG. 6 shows the facial feature extraction process.

  The face feature amount is position data of feature points of each face part part of the face as shown in (g). The face feature amount may be any position, form, or feature amount as long as it represents the feature point of the face part part.

  First, an average face template (a) is prepared, pattern matching between the input image (b) and this template is performed, and the face position is specified as shown in (f) by the correlation value.

  In order to specify the face position, the face may always be searched from the entire input image, or after the face position is specified once, only the vicinity thereof may be searched using the face position of the previous frame. .

  After that, the position of each part is accurately detected by template matching using an average template of each part (for example, eyes (c), nose (d), mouth (e)). Calculate the coordinates.

  Further, the method for extracting the facial feature amount is not limited to this method, and other methods may be used as long as the method exhibits the same effect.

  The face direction estimation unit 2 estimates the face direction using the face feature amount in the face feature storage unit 10 as an input, and stores the estimated face direction and face angle in the face direction storage unit 11. FIG. 7 shows the relationship between the arrangement of face parts and the face orientation. (A) With reference to the front, (b) leftward / (c) rightward, the position of the nose is to the left / right. Further, (d) upward / (e) downward indicates that the distance between the eyes and the nose is short / long. The face orientation is estimated using the geometric information of the face part that changes depending on the face orientation.

  A specific method for calculating the face orientation will be described with reference to FIG. (A) is the schematic diagram seen from the head of a person. Assuming that the midpoint of both eyes is A, the head of the nose is D, and the points projected on the image plane are B and C, the face angle θ to the left and right is

Can be expressed as

  AD is the height of the nose, BC is the distance between the midpoint of both eyes and the head of the nose, and the left and right face orientation angle θ is determined by the ratio of these two numerical values. Since the height of the nose has little individual difference in terms of shaping the human face, the ratio of the height of the nose to the distance between both eyes is calculated by a statistical average and used. By using the above method, the left and right face angle is calculated.

  (B) and (c) are schematic diagrams showing a profile of a person. (B) is a front view, and (c) is a view in which the face is moved in the vertical direction. In (c), when the position of the middle point of both eyes is E, H is below the nose, and F / G is a point obtained by projecting E · H onto the image plane, the vertical face orientation angle θ is

Can be expressed as

  Since EH is the distance from the middle point of both eyes to the bottom of the nose, the ratio of the distance of EH to the distance of both eyes is calculated using a statistical average, or (b) the distance of EH at the front face By registering with the distance of both eyes. The FG can be calculated sequentially using the coordinates under both eyes and nose at the time of feature extraction. The face orientation angle in the vertical direction is calculated by the above method.

  Note that face orientation estimation is not limited to the above-described method, and other methods may be used as long as the method exhibits the same effect.

  Next, the 1st face direction determination part 4, the avatar action correction part 5, and the avatar action generation part 7 which are the points of this invention are demonstrated in detail.

  The first face orientation determination unit 4 determines which one of the avatar motion correction unit 5 and the avatar motion generation unit 7 is executed according to the face orientation stored in the face orientation storage unit 11. If it is within the face direction threshold, it is a follow-up motion state and the avatar motion correction unit is executed. If it is more than a face direction threshold value, it is an autonomous operation state and an avatar action generation part is performed.

  The avatar motion correction unit synthesizes an avatar image using information in the face feature storage unit and the character data storage unit. The character data storage unit contains reference face character data for the front face, and character data feature points corresponding to the feature points of the face feature amount are set. If the face feature amount changes, the character feature point is also changed accordingly, and an avatar image is generated. For example, if all the facial parts are face feature values from the right, the character also re-synthesizes the facial parts from the right to create an avatar image. By doing so, it is possible to generate an avatar image facing right slightly according to the feature amount of the face.

  The avatar motion generation unit synthesizes an avatar image using information in the face motion storage unit and the character data storage unit. The character data storage unit contains reference face character data for the front face, and character data feature points corresponding to the feature points of the face feature amount are set. In the face motion storage unit, variations of face feature amounts are stored in advance for several frames. If the stored face feature value changes, the feature point of the character is also changed accordingly, and an avatar image is generated. For example, when the face feature amount in the nodding motion is stored in the face motion storage unit for several frames, the character is re-synthesized so that it nods accordingly, and an avatar image is created. By doing this, an avatar image can be generated even when the facial feature amount cannot be detected.

  Note that the face motion information stored in the face motion storage unit may not store the previous face motion, but may store a face motion previously given before execution.

  As described above, it is possible to transmit the avatar image operation to the other party without performing an unnatural operation even in a situation where the face is facing sideways and the face feature amount cannot be extracted.

  Although FIG. 1 illustrates a stand-alone system, a server / client format may be used. In other words, in addition to the case where only one terminal includes all the elements appearing in this specification, one terminal is a client, and all or part of it on a server or network to which this terminal can be connected. The elements of can exist.

  Furthermore, most of the elements in FIG. 1 may be held on the server side, and on the client side, for example, only the WWW browser may be used. In this case, various types of information are usually on the server and are basically distributed to the client via the network, but when the necessary information is on the server, the storage device of the server When the “recording medium” here is on the client, the recording device of the client becomes the “recording medium” here.

  Furthermore, in addition to an application that has been compiled into machine language, this “program” exists as intermediate code that is interpreted by the processes or threads described above, or at least resources and source code are stored on the “recording medium”. In this case, the “recording medium” includes an interpreter that can store an intermediate code application.

(Embodiment 2)
Next, Embodiment 2 of the present invention will be described with reference to the drawings. FIG. 3 is a functional block diagram of the face information transmitting apparatus according to Embodiment 2 of the present invention, FIG. 4 is a block diagram of the face information transmitting apparatus, and FIG. 5 is a flowchart of the face information transmitting apparatus.

  Before explaining the details of each part, the flow of processing of the image processing apparatus of this embodiment will be explained with reference to FIG. First, a face is found from the input image, and a feature amount of the face is extracted (step 1), and the face orientation is estimated from the feature amount (step 2). Then, the decision unit is switched by decision branch control depending on whether the previous frame is a follow-up operation or an autonomous operation (step 3).

  In the case of the tracking operation in the decision branch control, a first face orientation determination process based on the face angle is performed (step 4). If it is within the first face direction threshold, it remains in the follow-up operation state, and if it is greater than or equal to the first face direction threshold, it changes to the autonomous operation state. Thereafter, an avatar motion correction process is performed in which only the difference in the position of the facial feature from the previous frame is corrected to determine the avatar motion (step 5).

  In the case of autonomous operation in the determination branch control, a second face orientation determination process is performed to determine whether the face orientation is the front or other than the front (step 6). If it is other than the front, it remains in the autonomous operation state, and if it is the front, it changes to the following operation state. Thereafter, an avatar motion generation process is performed for synthesis using face motion memory data stored in advance (step 7).

  After performing the avatar motion correction process (step 5) and the avatar motion generation process (step 7), the created avatar image is transmitted to the other party by the avatar transmission process (step 8).

  The avatar image is transmitted while repeating the above steps 1 to 8 for each frame.

  Next, the configuration of the image processing apparatus of this embodiment will be described. FIG. 4 shows an example in which the elements of FIG. 3 are specifically configured. That is, in FIG. 4, a CPU (Central Processing Unit) 20 is stored in a ROM (Read Only Memory) 21 and executes an image processing program according to the flowchart of FIG. Control each element.

  In the RAM (random access memory) 22 and the non-volatile RAM 23, in addition to the areas for the storage units 9, 10, 11, 12, 13, and 14 shown in FIG. A storage area is reserved.

  Note that the processing means 1, 2, 3, 4, 5, 6, 7, and 8 shown in FIG. 3 are realized by the CPU 20 executing an image processing program stored in the ROM 21. Further, this program can be stored in a known storage medium such as the nonvolatile RAM 23 or an external storage memory.

  In the example of FIG. 4, a camera 25 is connected to the interface 24 so that an image that may include an object can be acquired in real time. Note that the camera 25 may be a CCD or CMOS module, or a camera attached to a mobile phone.

  Next, the overall configuration of the present invention will be described using the functional block configuration diagram of FIG.

  In FIG. 3, the face feature extraction unit 1 detects a face from input image data in the input image storage unit 9, then extracts a face feature amount, and stores the face feature amount in the face feature storage unit 10. Details of the face feature extraction unit are the same as those in the first embodiment.

  The face direction estimation unit 2 estimates the face direction using the face feature amount in the face feature storage unit 10 as an input, and stores the estimated face direction and face angle in the face direction storage unit 11. The details of the face direction estimation unit are the same as those in the first embodiment.

  Next, the determination branch control unit 3, the first face direction determination unit 4, and the second face direction determination unit 6 which are the points of the present invention will be described in detail.

  The decision branch control unit 3 determines whether to execute the first face direction determination unit 4 or the second face direction determination unit 6 according to the two operation state flags of the previous frame stored in the state transition storage unit 13. .

  The first face orientation determination unit 4 uses the face orientation / face angle information in the face orientation storage unit 11 to determine whether or not the face of the current frame is within the first face orientation range, and out of the range. If there is, an autonomous operation state flag is set in the state transition storage unit 13. Thereafter, the process proceeds to the avatar operation correcting unit 5. The first face direction range is a threshold value that is wider than the front direction and is set on the inner side of the detectable range in which the face feature amount can be extracted. FIG. 9 shows an example of setting the first face direction range. Although only the face orientation with respect to the left and right is illustrated in the figure, the same applies to the upper and lower face orientations.

  The second face orientation determination unit 6 uses the face orientation / face angle information in the face orientation storage unit 11 to determine whether or not the face of the current frame is within the second face orientation range, and out of the range. If there is, the tracking operation state flag is set in the state transition storage unit 13. Thereafter, the process proceeds to the avatar motion generation unit 7. The second face direction range indicates a range other than the front face range. FIG. 9 shows an example of setting the second face orientation range. Conversely, the only face front is outside the second face orientation range. Although only the face orientation with respect to the left and right is illustrated in the figure, the same applies to the upper and lower face orientations.

  FIG. 10 shows a state transition diagram of the state transition storage unit 13. If the face orientation of the previous frame is within the first face orientation range at the first face orientation determination unit 4 or outside the second face orientation range at the second face orientation determination unit 6, the state transition storage unit 13 follows the state. The flag stands. In this state, the determination branch control unit 3 selects the first face orientation determination unit 4 in the current frame and proceeds to execution.

  While the first face orientation determination unit 4 determines that the face orientation is within the first face orientation range, it is always set to the following motion state, that is, the avatar motion modification unit 5 is executed even in the next frame, and the avatar becomes a face. Follows the movement. If the face orientation exceeds the first face orientation range, transition to the autonomous motion state, and in the next frame, the avatar motion generation unit 7 is executed, and the avatar uses the information in the face motion storage unit 12 in which the face motion is stored in advance. Be autonomous.

  If the face orientation of the previous frame is within the second face orientation range at the second face orientation determination unit 6 or outside the first face orientation range at the first face orientation determination unit 4, the state transition storage unit 13 is autonomously operated. A status flag is raised. In this state, the determination branch control unit 3 selects the second face orientation determination unit 6 in the current frame and moves to execution.

  While the second face direction determination unit 6 determines that the face direction is within the second face direction range, that is, not in front of the face, it is always set to the autonomous operation state, that is, the avatar operation generation unit 7 is executed also in the next frame. The avatar performs an autonomous operation using information stored in the face motion storage unit 12 in which the facial motion is stored in advance. If the face orientation is out of the second face orientation range, that is, the face is in front, a flag is set in the state transition storage unit 13 to transit again to the follow-up motion state, and the avatar motion correction unit 5 is executed in the next frame. The avatar moves following the face.

  Next, the movement of each block when the state transition storage unit 13 is in the following operation state will be described with reference to FIG. In the follow-up operation state, the determination branch control unit selects the first face orientation determination unit and shifts. Various face orientations are input to the first face orientation determination unit from the face orientation storage unit as shown in FIG. Among them, in the case of a face orientation that falls within the first face orientation range, the tracking operation state flag is set in the state transition storage unit, and the process proceeds to the avatar operation correction unit. In the case of a face orientation outside the first face orientation range, an autonomous operation state flag is set in the state transition storage unit, and the process proceeds to the avatar operation correction unit.

  The avatar motion correction unit synthesizes an avatar image using information in the face feature storage unit and the character data storage unit. The character data storage unit contains reference face character data for the front face, and character data feature points corresponding to the feature points of the face feature amount are set. If the face feature amount changes, the character feature point is also changed accordingly, and an avatar image is generated. For example, if all the facial parts are face feature values from the right, the character also re-synthesizes the facial parts from the right to create an avatar image. By doing so, it is possible to generate an avatar image facing right slightly according to the feature amount of the face.

  Next, the movement of each block when the state transition storage unit 13 is in the autonomous operation state will be described with reference to FIG. In the autonomous operation state, the determination branch control unit selects the second face orientation determination unit and shifts. Various face orientations are input to the second face orientation determination unit from the face orientation storage unit as shown in FIG. Among them, in the case of a face orientation that falls within the second face orientation range, that is, in the case of a face other than the front face, an autonomous operation state flag is set in the state transition storage unit and the process proceeds to the avatar operation generation unit. In the case of a face orientation outside the second face orientation range, that is, a front face, the tracking motion state flag is set in the state transition storage unit, and the process proceeds to the avatar motion generation unit.

  The avatar motion generation unit synthesizes an avatar image using information in the face motion storage unit and the character data storage unit. The character data storage unit contains reference face character data for the front face, and character data feature points corresponding to the feature points of the face feature amount are set. In the face motion storage unit, variations of face feature amounts are stored in advance for several frames. If the stored face feature value changes, the feature point of the character is also changed accordingly, and an avatar image is generated. For example, when the face feature amount in the nodding motion is stored in the face motion storage unit for several frames, the character is re-synthesized so that it nods accordingly, and an avatar image is created. By doing this, an avatar image can be generated even when the facial feature amount cannot be detected.

  Next, the relationship between the face orientation and the avatar face orientation will be described with reference to FIG. In the graph of FIG. 13, the vertical axis represents the face orientation θ from side to side, and the horizontal axis represents time t. As an example, how the avatar's face orientation changes according to the present invention with respect to the change in the face orientation θ over time will be described. In the figure, the first face direction range is within 45 °, but the value is not limited to this value, and any other value may be used as long as it is within the range described with reference to FIG.

  Even when the face direction moves to the left and right at the position (a), this is within the first face direction range in the follow-up operation state, so the same direction as the face is synthesized and displayed from the facial feature points. Further, the movement of the face at this time is sequentially stored in the face motion storage unit, and overwritten when a new face motion appears. In this way, recent face motions are always stored in the face motion storage unit.

  The synthesis of the avatar face following the movement of the face continues until the time of (b). Since the face direction exceeds the first face direction range from (b), the tracking operation state is shifted to the autonomous operation state. . When entering the autonomous operation state, the avatar image of the limit angle of the first face direction range first returns smoothly to the front. At this time, if the face orientation is within the second face orientation range, it operates autonomously regardless of the face motion.

  After the avatar image faces the front, an avatar image is generated using the immediately previous face motion information stored in the face motion storage unit. In the figure, the face motion stored at the position (a) is used at the position (c) and synthesized as an avatar face motion. Then, the face motion stored in the face motion storage unit is repeated until the face orientation is out of the second face orientation range, that is, a front face.

  Note that the face motion information stored in the face motion storage unit may not store the previous face motion, but may store a face motion previously given before execution.

  At the time of (d), the face direction is out of the second face direction range, that is, a front face, and at this point, the autonomous operation state shifts to the follow-up operation state. In this state, the avatar synthesis using the face feature amount is performed so that the face direction and the face direction of the avatar are synchronized again.

  As described above, it is possible to transmit the avatar image operation to the other party without performing an unnatural operation even in a situation where the face is facing sideways and the face feature amount cannot be extracted.

  Although the left and right face orientations have been described with reference to FIG. 13, the left and right face orientations are not limited to left and right, and avatar communication is performed without causing unnaturalness even in a situation in which face feature amounts cannot be extracted by up and down movements by processing in the same way up and down Can be done.

  The face information transmitting apparatus according to the present invention is useful as proxy TV phone communication using a CG character image in a mobile phone. It can also be applied to applications such as video conferencing using avatar images on a PC.

Functional block diagram of face information transmitting apparatus according to Embodiment 1 of the present invention Flow chart of the facial information transmitter Functional block diagram of face information transmitting apparatus according to Embodiment 2 of the present invention Block diagram of the facial information transmitter Flow chart of the facial information transmitter (A) Illustration of the same average face template (b) Illustration of the same input image (c) Illustration of the same eye template (d) Illustration of the same nose template (e) Illustration of the same mouth template ( f) Illustration of the same face detection result (g) Illustration of the same face part detection result (A) Explanatory diagram of relationship between facial parts and face orientation (front) (b) Explanatory diagram of relationships between facial components and face orientation (leftward) (c) Explanatory diagram of relationships between facial components and face orientation (rightward) ) Explanatory diagram of relationship between facial parts and face orientation (upward) (e) Explanatory diagram of relationships between facial parts and face orientation (downward) (A) Illustrated calculation method of the same face direction estimation unit (b) Illustrated calculation method example of the same face direction estimation unit (c) Illustrated illustration of calculation method of the same face direction estimation unit (A) Illustration of determination threshold values of first face direction determination unit and second face direction determination unit (b) Illustration of determination threshold values of first face direction determination unit and second face direction determination unit (c) Illustration of determination threshold values of face direction determination unit and second face direction determination unit (d) Illustration of determination threshold values of first face direction determination unit and second face direction determination unit State transition explanatory diagram of the state transition storage unit Operation explanatory diagram when the state transition storage unit is in the follow-up operation state Operation explanatory diagram when the state transition storage unit is in an autonomous operation state (A) Relationship explanatory diagram of the same face direction and avatar face direction (b) Relationship explanatory diagram of the same face direction and avatar face direction (c) Relationship explanatory diagram of the same face direction and avatar face direction (d) Same face direction and avatar Illustration of the relationship of face orientation

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Face feature extraction part 2 Face direction estimation part 3 Judgment branch control part 4 1st face direction determination part 5 Avatar action correction part 6 2nd face direction determination part 7 Avatar action generation part 8 Avatar transmission part 9 Input image memory | storage part 10 Face Feature storage unit 11 Face orientation storage unit 12 Face motion storage unit 13 State transition storage unit 14 Character data storage unit 15 Face orientation determination unit 19 Bus 20 CPU (central processing unit)
21 ROM (Read Only Memory)
22 RAM (random access memory)
23 Nonvolatile RAM
24 interface 25 camera

Claims (10)

A face feature extraction unit that extracts a face feature amount from an input image;
An avatar operation correction unit that synthesizes an avatar image according to the extracted facial feature amount;
An avatar transmitter that transmits the synthesized avatar image to the communication partner;
A face direction estimation unit that estimates a face direction based on a facial feature amount;
The face orientation information of each frame is determined by a threshold, and the following movement state where the avatar follows the movement of the facial feature and the autonomous movement state where the avatar repeats the given motion regardless of the movement of the facial feature. A face information transmitting apparatus comprising: a face orientation determining unit that identifies a face direction determining unit to identify which one is an avatar image composition method. The face orientation determination unit
A first face orientation determination unit that determines the face orientation information of each frame with a first threshold value and identifies whether the state is a follow-up action state or an autonomous action state;
A second face direction determination unit that determines the face direction by a second threshold value different from the first threshold value and identifies the state;
A state transition storage unit that stores a state specified by one of the two determination units;
The face information transmitting apparatus according to claim 1, further comprising: a determination branch control unit that determines which of the two determination units to use according to the state stored in the state transition storage unit of the previous frame. The face direction threshold value of the first face direction determination unit is wider than the face direction threshold value of the second face direction determination unit. The face information transmitting apparatus according to claim 2, wherein the face information transmitting apparatus is configured to transition to an operating state. The face information transmitting apparatus according to claim 2, wherein the second face direction determining unit is configured such that the face direction threshold value for determining and transitioning to the follow-up operation state is a front face. The face information transmission device according to claim 2, 3 or 4, wherein the face direction threshold value of the first face direction determination unit is configured to be equal to or less than a detectable range in which face parts can be detected. The face information transmitting device according to any one of claims 1 to 5, wherein the face information transmitting device is configured to transition to a follow-up operation state when a face direction of an avatar face that operates in an autonomous operation state coincides with a face direction to be photographed. A facial feature extraction step for extracting facial feature values from the input image;
An avatar operation correcting step for synthesizing an avatar image in accordance with the extracted face feature amount;
An avatar transmission step of transmitting the synthesized avatar image to the communication partner;
A face orientation estimating step for estimating the face orientation based on the facial feature amount;
The face orientation information of each frame is determined by a threshold, and the following movement state where the avatar follows the movement of the facial feature and the autonomous movement state where the avatar repeats the given motion regardless of the movement of the facial feature. A face information transmitting method comprising: a face orientation determining step that identifies a face direction and changes the avatar image composition method. The face orientation determination step includes
A first face orientation determination step for determining the face orientation information of each frame with a first threshold value and identifying whether the state is a follow-up action state or an autonomous action state;
A second face orientation determination step for determining a face orientation with a second threshold value different from the first threshold value and specifying a state;
A state transition storage step for storing a state specified by one of the two determination units;
The face information transmission method according to claim 7, further comprising: a decision branch control step for determining which of the two decision units to use according to the state stored in the state transition storage unit of the previous frame. On the computer,
A facial feature extraction step for extracting facial feature values from the input image;
An avatar operation correcting step for synthesizing an avatar image in accordance with the extracted face feature amount;
An avatar transmission step of transmitting the synthesized avatar image to the communication partner;
A face orientation estimating step for estimating the face orientation based on the facial feature amount;
The face orientation information of each frame is determined by a threshold, and the following movement state where the avatar follows the movement of the facial feature and the autonomous movement state where the avatar repeats the given motion regardless of the movement of the facial feature. A computer-readable recording medium on which is recorded a face information transmission program for executing a face orientation determination step of changing the avatar image composition method by specifying which one is. In the face information transmission program,
The face orientation determination step
A first face orientation determination step for determining the face orientation information of each frame with a first threshold value and identifying whether the state is a follow-up action state or an autonomous action state;
A second face orientation determination step for determining a face orientation with a second threshold value different from the first threshold value and specifying a state;
A state transition storage step for storing a state specified by one of the two determination units;
The recording medium according to claim 9, further comprising a determination branch control step for determining which of the two determination units to use depending on the state stored in the state transition storage unit of the previous frame.

Images