JP2006227838A - Image processor and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide high-amusement divination service practically using motion of a moving image. <P>SOLUTION: A moving image of a face photographed by a camera-equipped cellphone is sent to a server via a network. When the server receives the moving image of the face, the server reads data on a stereoscopic image such as a tear or a heart mark for showing a divination result from a storage device. The server creates the moving image such as the tear or the heart mark such that a direction, a position and a size follow motion of the face inside the moving image to change by use of the stereoscopic image, composes it with the moving image of the face, and sends it to the cellphone. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a technique for processing a face image.

  In recent years, the use of movie contents with a large amount of data is expected to increase with the flat rate system for data communication charges of mobile phone terminals. Development of attractive services using moving images is desired.

  Japanese Patent Application Laid-Open No. 2004-228561 is a decoration moving image creating apparatus in a client server system for viewing a decoration moving image stored in a material information database and created by a moving image selected by a client terminal and decoration material information. There is disclosed a decorative moving image creating apparatus comprising a decorative moving image creating means for creating a synchronous multimedia integrated language file.

Patent Document 2 discloses a three-dimensional object model composed of a three-dimensional shape of an object and a luminance value at each position on the surface of the object in a position and orientation recognition apparatus that recognizes the position and orientation of an object from an image to be recognized. A position / orientation recognition apparatus that generates and recognizes the position / orientation of the object in the image to be recognized based on the three-dimensional object model is disclosed.
Japanese Patent Laid-Open No. 2004-15752 JP 2003-58896 A

An object of the present invention is to provide an image processing apparatus and an image processing program for adding a decoration image to a face image that moves three-dimensionally.
Another object of the present invention is to provide an image processing apparatus and an image processing program for performing a fortune-telling service utilizing a three-dimensional motion of a moving image.

  In the following, means for solving the problem will be described using the numbers used in [Best Mode for Carrying Out the Invention] in parentheses. These numbers are added to clarify the correspondence between the description of [Claims] and [Best Mode for Carrying Out the Invention]. However, these numbers should not be used to interpret the technical scope of the invention described in [Claims].

  An image processing apparatus (150) according to the present invention includes an input unit (160) for acquiring face image data (178), a storage unit (154) for storing decorative image data (186) including information on a three-dimensional shape, An extraction unit (171) for extracting the orientation of the face from the face image data (178). The face orientation is specified by a parameter including a rotation angle around a rotation axis different from the normal line (x axis) of the display screen (110). The image processing device (150) further includes an image composition unit (172) that creates a decoration image from the decoration image data (186) so as to match the face direction and combines the face image data with the decoration image data (186). .

  According to such an image processing device, the angle also with respect to an image of a face that faces obliquely without facing the display screen (that is, a face whose line of sight is shifted from a direction parallel to the normal line of the display screen) A decoration image with a shape suitable for Since items can be pasted on images of people in various poses, the enjoyment of processing images is expanded.

  In the image processing apparatus (150) according to the present invention, the storage unit (154) stores standard face model data (162) including information related to a three-dimensional shape. The extraction unit (171) compares the face image data with each of a plurality of plane face model data obtained by obliquely projecting the standard face model data (162) in a plurality of oblique directions, and obtains the most approximate plane face model data. The face orientation is extracted from the oblique direction.

  In the image processing apparatus (150) according to the present invention, the face image data (178) is moving image data. The extraction unit (171) extracts parameters from each of a plurality of still images constituting the face image data (178) in time series. The image composition unit (172) creates a composite image from each of a plurality of still images, and creates composite moving image data configured to reproduce the created composite image in time series.

  In the image processing apparatus (150) according to the present invention, the storage unit (154) stores a plurality of decoration image data candidates. The image processing apparatus (150) further extracts a feature amount from the face image data (178), and uses the feature amount to select one of a plurality of decoration image data candidates as the decoration image data (186) ( 170).

  The extracted feature amounts are the brightness and darkness of the image, the color, the feature of the arrangement of the facial parts, and the like. Corresponding to these feature amounts, a decorative image symbolizing the fortune-telling result is pasted on the face.

  The image processing system according to the present invention includes a portable terminal (100) and an image processing device (150). The mobile terminal (100) includes a camera (108) that captures a moving image (178) and a communication unit (112) that transmits the moving image (178) via a communication network. The image processing device (150) extracts the orientation of the face imaged in the moving part (154) and the storage unit (154) that stores the decorative image data (186) including information about the three-dimensional shape in time series. And an extraction unit (171). The face orientation is specified by a parameter including a rotation angle around a rotation axis different from the normal line of the display screen (110). The image processing device (150) further creates a composite video by creating a plurality of decorative images in time series from the decorative image data (186) so as to match the face direction and combining them with the video. 172).

  The image processing program (168) according to the present invention includes a step (S203) of acquiring a moving image (178), a step (S205) of acquiring decorative image data (186) including information on a three-dimensional shape, and a moving image (178). (S206) for extracting the orientation of the face imaged on the screen in time series, and the orientation of the face is specified by a parameter including a rotation angle around a rotation axis different from the normal line of the display screen (110). In addition, there are steps (S207 and S208) of creating a plurality of decoration images in time series from the decoration image data (186) so as to match the face direction and synthesizing with a moving image (178) to create a composite image. Have the computer execute the method.

According to the present invention, an image processing apparatus and an image processing program for adding a decoration image to a face image that moves three-dimensionally are provided.
Furthermore, according to the present invention, there are provided an image processing apparatus and an image processing program for providing a fortune telling service utilizing a three-dimensional motion of a moving image.

  Hereinafter, an image processing apparatus and an image processing program according to the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows a configuration of an image processing system using an image processing apparatus according to the present embodiment. The image processing system includes a mobile terminal 100 and an image processing device 150. The mobile terminal 100 is a mobile phone, a PDA (Personal Digital Assistant) or the like. The portable terminal 100 includes a CPU 102, a storage device 104, an input device 106, a camera 108, a display screen 110, and a communication device 112. The mobile terminal 100 can capture a moving image using the camera 108. The image processing apparatus 150 includes a CPU 152, a storage device 154, an input device 156, an output device 158, and a communication device 160. The image processing apparatus 150 can be configured by a personal computer, a workstation, or the like. The communication device 112 of the portable terminal 100 and the communication device 160 of the image processing device 150 are connected to be communicable.

  FIG. 2 shows a configuration of data stored in the storage device 154 of the image processing apparatus 150. The storage device 154 stores a website 161, a 3D standard face model 162, a face moving image database 163, a 3D decorative image database 164, a fortune telling database 166, and a fortune telling program 168. The fortune telling program 168 includes a fortune teller 170, a posture extractor 171, a decorative face image creator 172, and a data transmitter / receiver 174.

  The three-dimensional standard face model 162 is a standard face model created by collecting three-dimensional face data from a large number of people and averaging the three-dimensional data. The face three-dimensional data is data including information on the three-dimensional shape of the face, and can generate image data of the face when viewed from a desired angle based on the data.

  FIG. 3 shows the configuration of the face animation database 163. The face moving image database 163 stores the reception number 176 and the face moving image data 178 in association with each other.

  FIG. 4 shows the configuration of the three-dimensional decoration image database 164. The three-dimensional decoration image database 164 stores the decoration image ID 184 and the three-dimensional decoration image data 186 in association with each other. The decoration image ID 184 is an identifier for specifying the three-dimensional decoration image data 186 individually. The three-dimensional decoration image data 186 is three-dimensional data of decoration images (for example, stars, heart marks, tears, flames, etc.) added to the face image.

  The three-dimensional decorative image data 186 includes information indicating the positional relationship spatially with respect to the three-dimensional standard face model 162. Thus, for example, if the decoration image is tears, it is in a position below the eyes, and if it is a head covering (an artificial article imitating an animal or the like that is worn on a person's head), the face is surrounded. It is specified that there is.

  FIG. 5 shows the configuration of the fortune telling database 166. Fortune-telling database 166 stores fortune-telling parameters 180, fortune-telling result text 182 and decorative image ID 184 in association with each other.

  The fortune-telling result text 182 and the three-dimensional decorative image data 186 correspond semantically or symbolically with the decorative image ID 184 as a medium term. For example, the fortune-telling result text 182 of “Love luck is the best!” Corresponds to the three-dimensional decorative image data 186 surrounded by a heart mark around the face.

  6 and 7 show the operation of the image processing system. In the following description, an operation that the CPU 152 reads a program stored in the storage device 154 and executes according to a procedure described in the program is described as an operation performed by the program itself.

  Step S <b> 200: The user of the mobile terminal 100 shoots a video of himself or another person using the camera 108. The captured moving image is stored in the storage device 104.

  Step S201: The user of the portable terminal 100 accesses the website 161 via the communication device 112.

  Step S202: The user of the portable terminal 100 uploads the captured moving image to the image processing apparatus 150 in accordance with the service reception screen display of the website 161 displayed on the display screen 110.

  Step S203: The data transmitting / receiving unit 174 of the fortune telling program 168 issues a receipt number 176 when a moving image (movie file) is uploaded. The fortune telling program 168 stores the reception number and the moving image as the reception number 176 and the face moving image data 178 in the face moving image database 163.

  Step S204: The fortune telling unit 170 of the fortune telling program 168 generates a fortune telling parameter 180 corresponding to the uploaded face animation. The generation of the fortune-telling parameter 180 is performed by a predetermined algorithm using, for example, the reception number 176 or using the feature amount (color, brightness, etc.) extracted from the face moving image data 178. Alternatively, the fortune-telling parameter 180 is generated using information such as a birthday and blood type collected from the user.

  Step S205: The fortune-telling unit 170 searches the fortune-telling database 166 using the generated fortune-telling parameter 180, and extracts the corresponding fortune-telling result text 182 and decoration image ID 184. The ornamental face image creation unit 172 of the fortune telling program 168 searches the three-dimensional ornamental image database 164 using the extracted ornamental image ID 184 as a search key, and extracts the corresponding three-dimensional ornamental image data 186.

  Step S206: The posture estimation unit 171 extracts a still image (corresponding to one frame of a moving image) from the face moving image data 178. The posture estimation unit 171 collates the extracted still image with the three-dimensional standard face model 162. For example, the posture estimation unit 171 generates planar face image data obtained by obliquely projecting the three-dimensional face standard model 162 at various angles, and compares the generated planar face image data with the extracted still image. Then, the angle of the bevel when the similarity between them is the largest is calculated, and the angle is stored as information indicating the posture of the face included in the still image.

  The face orientation to be determined includes the size of the face (or distance from the camera), the axis orthogonal to the display screen, the axis skewed to the display screen, or the rotation angle around the axis parallel to the display screen. .

  Step S207: The decorative face image creation unit 172 creates a decorative image that matches the posture of the face. That is, using the information indicating the facial posture stored in step S206, two-dimensional decorative image data matching the facial posture is created from the three-dimensional decorative image data 186 extracted in step S205.

  Step S208: The decorative face image creation unit 172 combines the still image extracted in step S206 and the decorative image created in step S207.

  Step S209: If the still image included in the face moving image data 178 has not been read to the end of the time series, the decorative face image creation unit 172 proceeds to step S206 and processes the next still image in time series. I do. If the still image has been read to the end of the time series, the process proceeds to step S210.

  Step S210: The decorative face image creation unit 172 integrates a group of still images in which the face image created in steps S206 to S209 and the decorative image are combined in a time series, and a decorative moving image file (composite movie file). Create

  Step S211: The data transmission / reception unit 174 transmits the decorative moving image file and the fortune telling result text 182 extracted in step S205 to the portable terminal 100.

  Step S212: When the user of the mobile terminal 100 performs a predetermined operation, the composite video file is played in one area of the display screen 110 by the video playback player provided in the mobile terminal 100, and the fortune-telling result text 182 is displayed in the other area. The

  FIG. 8A shows one frame of the synthesized moving image. A decoration image 192a corresponding to the fortune telling result is synthesized with the face image 190a. FIG. 8B shows another frame of the composite video created from the same face video data. The face image 190b has a different posture from the face image 190a. The decoration image 192b follows the change in the posture of the face image 190b and changes its posture as compared with the decoration image 190a. In the example of FIGS. 8A and 8B, the posture of the face is rotated around the z axis parallel to the display screen 110. In addition to this, the posture of the face may change in any direction. In particular, when the posture of the face rotates around an axis that is not parallel to the x-axis (axis perpendicular to the display screen 110), a stereoscopic effect including depth is felt, which is interesting.

  In this embodiment, the decoration image is pasted in accordance with the movement of the face. As another example, a configuration in which the image processing apparatus 150 detects a facial expression and pastes a decoration image corresponding to the facial expression in a database stored in the storage device 154 is possible. A configuration in which another face previously stored in the face portion of the moving image is pasted as a decorative image is also possible. Further, when the shadow information related to the shadow (light brightness distribution) is extracted from the face image data, and the decorative image is processed so that the shadow matches the face image data based on the shadow information, the decorative face image is processed. Is more realistic and preferable.

  Fortune-telling program 168 or a part thereof may be converted into a download file. In that case, the program is downloaded from the mobile terminal 100, and the composition of the moving image and the decoration image is performed by the processing function of the mobile terminal 100.

  It is also possible to apply a DRM (Digital Light Management) technique to the image processing system in the present embodiment. In this case, detailed setting items (for example, service use period and number of times) of the fortune-telling service can be set on the Web site 161, and the image processing apparatus 150 issues a use right. Thereby, it becomes possible to set the usage fee of the fortune-telling service in detail.

FIG. 1 shows the configuration of an image processing system. FIG. 2 shows the structure of data stored in the storage device. FIG. 3 shows the data structure of the face animation database. FIG. 4 shows the data structure of the three-dimensional decorative image database. FIG. 5 shows the data structure of the fortune-telling database. FIG. 6 is a flowchart showing the operation of the image processing system. FIG. 7 is a flowchart showing the operation of the image processing system. FIG. 8A shows an example of a synthetic face image. FIG. 8B shows an example of a synthesized face image.

Explanation of symbols

154 ... Storage device 163 ... Face animation database 164 ... 3D decoration image database 166 ... Fortune-telling database 190 ... Face image 192 ... Decoration image

Claims (6)

An input unit for acquiring face image data;
A storage unit for storing decorative image data including information on a three-dimensional shape;
An extraction unit for extracting a face direction from the face image data, and the face direction is specified by a parameter including a rotation angle around a rotation axis different from a normal line of the display screen;
An image processing apparatus comprising: an image composition unit configured to create a decoration image from the decoration image data so as to match the orientation of the face and combine the image with the face image data. An image processing apparatus according to claim 1,
The storage unit stores standard face model data including information on a three-dimensional shape,
The extraction unit compares each of a plurality of planar face model data obtained by obliquely projecting the standard face model data in a plurality of oblique directions and the face image data, and the obliqueness of the planar face model data that is most approximated. An image processing apparatus that extracts the orientation of the face from a direction. An image processing apparatus according to claim 1,
The face image data is video data,
The extraction unit extracts the parameter from each of a plurality of still images constituting the face image data in time series,
The image processing unit generates the composite image from each of the plurality of still images, and generates composite moving image data configured to reproduce the generated composite image in time series. An image processing apparatus according to claim 1,
The storage unit stores a plurality of decoration image data candidates,
An image processing apparatus further comprising: a fortune-telling unit that extracts a feature amount from the face image data and selects one of the plurality of decoration image data candidates as the decoration image data using the feature amount. A mobile device,
An image processing device,
The portable terminal is
A camera for recording videos,
A communication unit that transmits the moving image via a communication network,
The image processing apparatus includes:
A storage unit for storing decorative image data including information on a three-dimensional shape;
An extraction unit that extracts the orientation of the face shown in the video in time series, and the orientation of the face is specified by a parameter including a rotation angle around a rotation axis different from the normal line of the display screen,
An image processing system comprising: a moving image combining unit that generates a plurality of decorative images in time series from the decorative image data so as to match the face direction and combines the moving image with the moving image. A step to get a video,
Obtaining decorative image data including information on a three-dimensional shape;
Extracting the orientation of the face shown in the video in time series, and the orientation of the face is specified by a parameter including a rotation angle around a rotation axis different from the normal of the display screen;
Image processing that causes a computer to execute a method comprising: creating a plurality of decoration images from the decoration image data in time series so as to match the face orientation and synthesizing the decoration images with the moving image. program.

Images