JP2007026090A - Video preparation device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video preparation device capable of preparing a moving image in which a human being is converted to a character for instance and not only just hiding the face of a photographing object but also preparing interesting video images. <P>SOLUTION: In a video content preparation device 14, the face image data 26 of source information are inputted in an image input part 30, feature point data 42 which are deformation information are generated from the supplied face image data 40 in a face feature extraction part 34, character image data 24 to be a reference are deformed corresponding to the deformation information in a character deformation part 36, and composite image data 46 for which background image data 38 supplied from the image input part 28 and the deformed character image data 44 are combined are generated as video contents in an image combining part 32. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a video creation device. The video creation apparatus according to the present invention particularly relates to a system for creating video content by combining character images.

  With the spread of networks, services that distribute various contents are being implemented. Moving images are becoming common for content to be distributed due to improvements in communication technology. For example, educational content as in Patent Document 1 and news content including moving images as in Patent Document 2 are created or distributed.

JP 2004-207948 A JP 2002-244957 A Japanese Patent No. 3593067

  However, in order to reduce the cost of creating current moving image content, video resources are diverted like videos used in lectures used in the past. The video of commentators and performers is often created by combining still images such as slide images for explanation and moving images such as videos for explanation.

  When using such past video resources, there may be cases where the commentator / speaker himself / herself does not want to be able to see the video in an unspecified number. In addition, when creating an in-house educational video content and using an employee as a commentator, there is a problem of portrait rights that may not be used as is. In fact, even if a person who is explaining as it is, it is highly possible that the content will not be interesting.

  The present invention eliminates the disadvantages of the prior art, for example, creates a moving image in which a person is converted into a character, and can not only hide the face to be photographed but also create an interesting video. An object is to provide an apparatus.

  In order to solve the above-described problem, the present invention uses a first image input means for inputting a first image, and a facial part in a model image as a reference as a first feature point, and the movement of the first feature point Information input means for inputting information reflecting the information, and the position where each part of the first feature point has moved based on this information is analyzed as the second feature point, and the second feature point is generated as deformation information Analyzing processing means, model deforming means for creating a deformed model image by moving each first feature point for the model image according to the deformation information, and the first image supplied from the first image input means And image synthesis means for synthesizing the model image and creating a synthesized image.

  According to the video creation apparatus of the present invention, information is input by the information input means, deformation information is generated from the information supplied by the analysis processing means, and the model image used as a reference is deformed according to the deformation information by the model deformation means. Then, the first image and the deformed model image are synthesized by the image synthesizing unit, and the synthesized image is generated as video content, thereby generating an image that moves in synchronization with the photographed person, and viewing the generated image It is possible to easily create video content that does not give the user a sense of incongruity.

  Next, an embodiment of a video creation device according to the present invention will be described in detail with reference to the accompanying drawings.

  In this embodiment, the video creation apparatus of the present invention is applied to the video content creation system 10. The illustration and description of parts not directly related to the present invention are omitted. In the following description, the signal is indicated by the reference number of the connecting line in which it appears.

  As shown in FIG. 1, the video content creation system 10 includes an external information providing unit 12 and a video content creation device 14. The external information providing unit 12 includes storage units 16 and 18 and a digital camera 20. The storage unit 16 has a function of storing background image data such as a slide image or a moving image of one frame, and outputting it in response to reading. The storage unit 18 has a function of storing character image data and outputting it in response to reading. The storage units 16 and 18 are stored as a database in a set corresponding to a keyword that enables search for stored image data.

  The storage unit 16 outputs the background image data 22 to the video content creation device 14. The storage unit 18 also outputs the character image data 24 to the video content creation device 14. In particular, the digital camera 20 has a function of photographing a face as a subject and outputting the obtained face image data. The digital camera 20 outputs the captured face image data 26 to the video content creation device 14. The digital camera 20 is not limited to a still image, and may be a digital video camera capable of moving image shooting, that is, movie shooting.

  The video content creation device 14 includes image input units 28 and 30, an image composition unit 32, a face feature extraction unit 34, and a character transformation unit 36. The image input unit 28 has an input interface function for taking the supplied background image data 22 into the video content creation device 14. The image input unit 28 outputs the background image data 38 to the image composition unit 32. Further, the image input unit 30 has an input interface function for taking the supplied face image data 26 into the video content creation device 14. The image input unit 30 outputs the face image data 40 to the face feature extraction unit 34.

  The face feature extraction unit 34 has a function of detecting the position of a feature point provided in advance corresponding to the face part from the supplied face image data 40. The face feature extraction unit 34 detects, for example, the position of the feature point that has moved from the currently supplied information with respect to the feature point obtained based on the previous information. The face feature extraction unit 34 may obtain the position detection or may obtain the vector information having the distance and the direction. The face feature extraction unit 34 outputs the feature point data 42 obtained from the face image data 40 to the character deformation unit 36.

  The character transformation unit 36 has a function of replacing, for example, position data with the feature point data included in the character image data 24 read from the storage unit 18 in accordance with each feature point data 42 corresponding to the facial part. This feature point can be moved by this replacement function. This movement is to deform the face of the character image. The character deforming unit 36 outputs the deformed character image data 44 to the image composition unit 32.

  The image composition unit 32 has a function of rendering using character image data 44 so as to become a two-dimensional or three-dimensional character image. The image composition unit 32 also has a function of compositing the rendered character image and the background image. The image composition unit 32 outputs the synthesized image data 46 as created video content.

  Next, the operation of the video content creation system 10 will be described with reference to FIG. 2, FIG. 3, and FIG. As shown in FIG. 2, slide image data is read as an example of the background image data 22. The digital camera 20 captures and captures a face image of the subject 48. The image input unit 30 may be a face image taken by the camera 20 or an image recorded in advance.

  The face image data 40 output from the image input unit 30 is converted into feature point data based on, for example, the face orientation, the degree of opening of each part of the face such as eyes and mouth, the position, and the like by the face feature extraction unit 34. The converted feature point data 42 is supplied to the character deforming unit 36. The character conversion unit 36 deforms the character image so as to be synchronized with the supplied feature point data 42. The created character image data 44 is output to the image composition unit 32.

  Here, Patent Document 3 may be used as a method of creating feature point data from a face image.

  The image composition unit 32 performs rendering based on the character image data 44 created by the character transformation unit 36, and composes the generated character image and the background image data 38 supplied from the image input unit 28. As a result, one frame of composite image data 46 of the video content to be finally created is created.

  Here, the video content creation system 10 can change the size, position, and orientation of the character in the composite image, and can display the character image on the full screen for displaying the composite image. In addition, the video content creation system 10 has a function that allows the character position to be set to the center even if the shooting target does not exist at the center of the screen. By applying such a function, it is possible to easily create a composite image in which a character is displayed at an appropriate position even from a video of a workshop where the entire whiteboard and the performer are filmed.

  Furthermore, in order to simplify the creation of video content, it is convenient to use an editing screen 50 in the content editing tool as shown in FIG. The content editing tool is a tool that visualizes control in image input of the image input units 28 and 30. This control is controlled by a control unit (not shown). Specifically, in the content editing tool, an image supplied to the image input unit 28 is set as an image 1 as shown in FIG. 3, and an image supplied to the image input unit 30 is set as an image 2. The images supplied to the images 1 and 2 clearly indicate the target of the input image or moving image in the frames 52 and 54, respectively, and specify the input time from the beginning. The edit screen 50 may represent a time range corresponding to this time designation, for example, with an arrow. The content editing tool may control the images input to the images 1 and 2 according to the time designation.

  The character transformation unit 36 may give transparency information to the created character image. The transmission information is information indicating a range other than the character in the character image, for example, the range of the background region 56 as shown in FIG. 4 (a). When only the character image is synthesized with the background image input to the image input unit 28 while referring to the transmission information, the video content image shown in FIG. 4B can be easily created. If the character image is changed with the input information other than the character as it is, only the displayed character can be easily changed. For children, more suitable content can be created by setting characters according to the content target, such as anime characters.

  By repeating such processing, the composite image data 46 is created for each frame, a moving image is created from a plurality of composite images, or a composite image is delivered as a stream one by one, so that desired video content can be obtained. Can be easily created. That is, it is possible to process in real time the creation of video content by a character synchronized with a person instead of a photographed person. This makes it possible to immediately deliver video content with hidden faces.

  The deformation of the character in the video content creation system 10 is not limited to a face image, and sound or voice and specific data may be used. The video content creation system 10 described below replaces some components of the external information providing unit 12 and the video content creation device 14 with other components. The same common components are denoted by the reference symbols used in the previous embodiments. In order to avoid complications due to repetition of the same description, description of common components will be omitted.

  In the video content creation system 10 shown in FIG. 5, when the character is deformed based on sound, the external information providing unit 12 is provided with a microphone 58 together with common components. The microphone 58 is replaced with the camera 18. The microphone 58 has a function of converting the collected sound into an analog signal. The microphone 58 outputs the converted analog signal 60 to the video content creation device 14.

  The video content creation device 14 includes an audio input unit 62 and an audio recognition unit 64 along with common components. The audio input unit 62 has an interface for inputting the supplied analog signal 60 and has a function of converting the input analog signal 60 into a digital signal 66. In the present embodiment, the supplied analog signal 60 is an audio signal. The voice input unit 62 outputs a digital signal 66 to the voice recognition unit 64.

  The voice recognition unit 64 has a function of recognizing voice based on the supplied digital signal 66. The voice recognition unit 64 uses voice recognition, which is a known technique, and particularly recognizes vowels. The voice recognizing unit 64 further generates feature point data 68 for deforming the mouth shape of the recognized vowel using the target in the facial part as the mouth. The voice recognition unit 64 outputs the feature point data 68 to the character transformation unit 36.

  In this embodiment, the character can be deformed in synchronization with the voice, and in particular, a character image with a moving mouth can be synthesized. The voice recognizing unit 68 may output feature point data created in advance for a facial part other than the mouth to the character deforming unit 36. Examples of operations here include shaking the head and periodically blinking. By supplying this, the synthesized image can be moved more naturally.

  The video content creation system 10 of the present embodiment can create video content as if watching television even from content such as radio that has only audio. At this time, the image input to the image input unit 28 may be an ordinary background such as a landscape, but if used for English conversation, a video content that is easier to understand can be created by inputting a slide in which the text content is written.

  Also, by combining the previous embodiment, that is, the face image and the voice described above, the movement of the mouth of the character is voice, and other movements are created more realistically by creating an image based on the face image. Can be operated.

  In this case, video content in which the mouth of the character moves in synchronization with the sound can be created. Even without video, video content can be created with audio alone.

  In the video content creation system 10 shown in FIG. 6, when the character is deformed based on the specific data, the external information providing unit 12 is provided with the operation unit 70 together with the common components. The operation unit 70 is replaced with the camera 18. The operation unit 70 has a function of generating instruction data that designates an operation as specific data. The operation unit 70 may output the instruction data 72 assigned to each key or key combination to the video content creation device 14. The instruction data 72 of this embodiment is a serial number indicating the type of registered operation. The operation unit 70 outputs this serial number as instruction data 72.

  The video content creation device 14 includes an action creation unit 74 along with common components. The action creation unit 74 has a function of generating feature point data in accordance with the supplied instruction data 72. The action creation unit 74 inputs the feature point data 24 related to the action to be performed by the character from the storage unit 18 and stores or registers it. Further, the feature point data 24 relating to the motion is registered in advance in the storage unit 18 as part of the character data. The action creation unit 74 outputs feature point data 76 corresponding to the instruction data 72 from the stored feature point data to the character transformation unit 36. The feature point data 76 is data that causes the character to perform a predetermined action such as bowing or laughing. Corresponding to the case where the operation is not designated, it is desirable to provide the constituent elements as shown in the above-described embodiment. By creating feature point data in this way, character movement can be diversified.

  By the way, it is considered that the video content created may become monotonous only with the above-described embodiment. Therefore, in the present embodiment, for example, an operation in which the character walks around the slide, points to a location described by the character, or the character is happy is inserted. Thereby, the monotonousness of the created video content can be reduced.

  In this way, when the character is synthesized including the designated operation, it can be rendered more complicated and effective. For example, it is possible to create more effective video content in which characters are exaggeratedly expressed with emotions that cannot be expressed in the previous embodiment.

  Furthermore, specific data may be generated based on sound and image synthesis may be performed. In the video content creation system 10 shown in FIG. 7, the external information providing unit 12 includes components added by the microphone 58 and the storage unit 78. The storage unit 78 has a function of storing and reading words based on registered voice data such as words and idioms. The storage unit 78 outputs the stored audio data 80 to the video content creation device 14.

  The video content creation device 14 includes an audio input unit 62, an audio comparison unit 82, and an operation creation unit 74, along with common components. The voice comparison unit 82 has a function of comparing the digital signal 66 with the read data 80 and outputting instruction data corresponding to the result recognized by this comparison. The voice comparison unit 82 outputs the instruction data 84 to the action creation unit 74. The action creation unit 74 supplies feature point data 76 corresponding to the instruction data 84 to the character transformation unit 36.

  The operation will be briefly described. The voice data 66 input by the voice comparison unit 82 is compared with the registered voice data 80, and when it is recognized that the words are the same, the operation is automatically designated. As specific data to be registered, since the voice data “this” indicates an instruction, it is operated to point the finger toward the object. In addition, to register the operations such as to bow to the voice data of "Hello". This registration effectively produces the content while reducing the burden on the video content creator. Corresponding to the case where no operation is set or no operation is specified, it is desirable to provide the components as shown in the above-described embodiments. By creating feature point data in this way, character movement can be diversified.

  By comparing the registered audio data with the input audio data in this way, video content creators can easily create video content that synthesizes characters that operate properly without any operation from the operation unit. can do.

1 is a block diagram showing a schematic configuration of a video content creation system to which a video creation device according to the present invention is applied. It is a figure explaining the principle of operation of the video content creation system of FIG. It is an example of the content edit tool screen in the video content creation system of FIG. It is a figure explaining the principle which applied the transparent information in the video content creation system of FIG. It is a block diagram which shows other schematic structures in the video content creation system of FIG. It is a block diagram which shows other schematic structures in the video content creation system of FIG. It is a block diagram which shows other schematic structures in the video content creation system of FIG.

Explanation of symbols

10 Video content creation system
12 External information provider
14 Video content creation device
16, 18 Storage section
20 Digital camera
28, 30 Image input section
32 Image composition part
34 Face feature extraction unit
36 Character transformation

Claims (5)

First image input means for inputting a first image;
An information input means for inputting information reflecting the movement of the first feature point, with the face part in the model image as a reference as the first feature point;
Analysis processing means for analyzing the position where each part of the first feature point has moved based on the information as a second feature point and generating the second feature point as deformation information;
Model deformation means for creating a model image obtained by moving and deforming each first feature point for the model image according to the deformation information;
An image creation apparatus comprising: an image composition unit configured to compose a first image and the deformed model image to create a composite image. The apparatus according to claim 1, wherein the information input unit is a second image input unit that inputs a face image as a second image,
The video creation apparatus, wherein the analysis processing means is face feature extraction means for extracting a second feature point based on the face image. The apparatus according to claim 1 or 2, wherein the information input means is a sound input means for inputting a surrounding sound,
The video creation apparatus, wherein the analysis processing means is data creation means for recognizing an input sound and creating a second feature point according to the recognition.   4. The apparatus according to claim 1, wherein the apparatus receives specified information, registers a second feature point to be used corresponding to the information, and converts the registered second feature point into the model deformation. An image creating apparatus comprising an operation creating means for outputting to the means. The apparatus according to claim 1, wherein the apparatus compares a sound input means for inputting ambient sound to the information input means, a sound registered in advance and a sound output from the sound input means, and compares them. And means for outputting information corresponding to the matched sound,
Further, the apparatus includes a second feature point used corresponding to the information in advance, and an action creating unit that outputs the registered second feature point to the model deforming unit. Creation device.

Images