JP2010066844A - Method and device for processing video content, and program for processing video content - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately determine a feeling of a person appearing in video content and to process the video content based upon the determined feeling. <P>SOLUTION: A face detection unit 36 detects a face in each frame of a still image constituting the video content, and also detects time-series change and movement. An expression recognition unit 37 recognizes an expression of the detected face. A movement detection unit 38 detects movement of the person whose face is detected. A voice extraction unit 39 extracts a voice of the person whose face is detected. A voice feeling recognition unit 40 recognizes a voice feeling of the person whose face is detected. A feeling determination unit 41 determines a feeling of the person whose face is detected with reference to a feeling table 51 stored in a feeling DB 45. A decorative image acquisition unit 42 acquires a decorative image corresponding to the feeling determined by the feeling determination unit 41 from a decorative image DB 46. A composition processing unit 43 combines the decorative image acquired by the decorative image acquisition unit 42 with the video content. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a moving image content processing method and apparatus for processing moving image content based on the emotion of a photographed person, and a moving image content processing program.

  On the Internet, there are distribution sites such as YouTube (registered trademark), Nico Nico Douga (registered trademark), Yahoo! Recent video content distribution sites increase the added value of services by providing video content that combines user comments and decorative images.

As a moving image content processing technique, there is a technique for determining an emotion from the face of a person being photographed and synthesizing a decoration image corresponding to the determined emotion (for example, see Patent Documents 1 and 2).
JP 2008-084213 A JP 2006-185329 A

  By the way, as a whole, adults may endure feelings on their faces in public. For example, there is a case of laughing while suppressing anger, and it is difficult to judge an emotion from such a face (facial expression). That is, in the inventions of Patent Documents 1 and 2 that determine an emotion only from the face, the emotion cannot be accurately determined, and there is a possibility that a decorative image suitable for the moving image content cannot be synthesized.

  The present invention has been made in view of the above-described problems, and a moving image content processing method and apparatus that accurately determines the emotion of a person being photographed and processes the moving image content based on the determined emotion, and the moving image content The purpose is to provide a machining program.

  In order to achieve the above object, a moving image content processing apparatus according to the present invention includes a face detection unit that detects a face from moving image content, a facial expression recognition unit that recognizes a facial expression detected by the face detection unit, and a moving image. A motion detection unit that detects a motion of a person whose face is detected by the face detection unit from a content; a voice extraction unit that extracts a voice of a person whose face is detected by the face detection unit from video content; Emotion that determines the emotion of the person whose face is detected by the face detection unit from the facial expression recognized by the facial expression recognition unit, the movement of the person detected by the motion detection unit, and the voice extracted by the voice extraction unit A determination unit; and a processing unit that processes the moving image content based on the emotion determined by the emotion determination unit.

  In the invention according to claim 2, a voice emotion recognition unit for recognizing voice emotion from the voice extracted by the voice extraction unit, a combination of facial expression recognized from a face, movement of a person, and voice emotion, And an emotion database storing an emotion table stored in association with each other, the emotion determination unit using the emotion table, the facial expression recognized by the facial expression recognition unit, the person detected by the motion detection unit The emotion corresponding to the combination of the movement and the voice emotion recognized by the voice emotion recognition unit is determined as the emotion of the person whose face is detected by the face detection unit.

  According to a third aspect of the present invention, there is provided a decoration content database in which decoration content to be decorated to the moving image content is stored in correspondence with emotion, and the decoration content corresponding to the emotion determined by the emotion determination unit is stored in the decoration content database. The processing unit is a composition processing unit that combines the decoration content acquired by the decoration content acquisition unit with the moving image content.

  According to a fourth aspect of the present invention, there are a plurality of types of decorative content stored in the decorative content database in association with emotions, and the decorative content acquisition unit acquires the input type of decorative content.

  In the invention according to claim 5, a gender estimating unit that estimates the gender of the person whose face is detected by the face detecting unit from the face detected by the face detecting unit and the voice extracted by the voice extracting unit. The processing unit processes the moving image content based on the gender estimated by the gender determination unit.

  In the invention according to claim 6, an age estimation unit that estimates the age of a person whose face is detected by the face detection unit from the face detected by the face detection unit and the voice extracted by the voice extraction unit The processing unit processes video content based on the age estimated by the age determination unit.

  In the moving image content processing method of the present invention, in the face detection unit, a face detection step of detecting a face from the moving image content, a facial expression recognition step of recognizing a facial expression from the face detected in the face detection step, A motion detection step for detecting a motion of the person whose face is detected in the face detection step from the moving image content by a motion detection unit, and a voice extraction unit for the voice of the person whose face is detected in the face detection step from the video content The face is detected in the face detection step from the voice extraction step extracted in step S, the facial expression recognized in the facial expression recognition step, the movement of the person detected in the motion detection step, and the voice extracted in the voice extraction step. The emotion determination step of determining the emotion of the person who has been performed by the emotion determination unit, and processing the video content based on the emotion determined in the emotion determination step And a processing step of processing by the processing section.

  The moving image content processing program of the present invention includes a face detecting step for detecting a face from the moving image content, a facial expression recognition step for recognizing an expression from the face detected in the face detecting step, a moving image content, and the face detecting step. A motion detection step for detecting the motion of the person whose face is detected, a voice extraction step for extracting the voice of the person whose face is detected in the face detection step from the video content, and the facial expression recognized in the facial expression recognition step An emotion determination step for determining the emotion of the person whose face is detected in the face detection step from the movement of the person detected in the motion detection step and the voice extracted in the voice extraction step; and the emotion determination step The computer is caused to execute a processing step for processing the moving image content based on the emotion determined in (1).

  According to the moving image content processing method and apparatus and the moving image content processing program of the present invention, it is possible to accurately determine the emotion of the person in the image and process the moving image content based on the determined emotion.

[First Embodiment]
In FIG. 1, the moving image content processing apparatus according to the first embodiment is realized in a format constructed in the server 11 by installing a moving image content processing program 44 (see FIG. 3). The moving image content processing device processes moving image content based on the emotion of the person in the image. In this specification, the moving image content means a moving image with sound. In this embodiment, a case where moving image content is processed by combining a decorative image with a still image frame constituting the moving image content will be described as an example. However, moving image content is processed by combining audio and other content. You may make it do.

  The server 11 and the client terminal 13 connected via the Internet 12 constitute a network system 14. The client terminal 13 is, for example, a known personal computer or workstation, and includes a monitor 15 that displays various operation screens, and an operation unit 18 including a mouse 16 and a keyboard 17 that output operation signals.

  The client terminal 13 receives moving image content captured by the digital camera 19 or moving image content recorded on a recording medium 20 such as a memory card or a CD-R, or the moving image content via the Internet 12. The content is transferred.

  The digital camera 19 is connected to the client terminal 13 by a communication cable compliant with, for example, IEEE 1394, USB (Universal Serial Bus), a wireless LAN, or the like, and data communication with the client terminal 13 is possible. Similarly, the recording medium 20 can exchange data with the client terminal 13 via a dedicated driver.

  As shown in FIG. 2, the CPU 21 constituting the client terminal 13 comprehensively controls the entire client terminal 13 according to an operation signal input from the operation unit 18. In addition to the operation unit 18, a RAM 23, a hard disk drive (HDD) 24, a communication interface (communication I / F) 25, and a monitor 15 are connected to the CPU 21 via a data bus 22.

  The RAM 23 is a working memory for the CPU 21 to execute processing. In addition to storing various programs and data for operating the client terminal 13, the HDD 24 stores moving image content captured from the digital camera 19, the recording medium 20, or the Internet 12. The CPU 21 reads a program from the HDD 24 and develops it in the RAM 23, and sequentially processes the read program.

  The communication I / F 25 is, for example, a modem or a router, controls a communication protocol suitable for the Internet 12, and exchanges data via the Internet 12. The communication I / F 25 also performs data communication with external devices such as the digital camera 19 and the recording medium 20.

  As shown in FIG. 3, the CPU 31 configuring the server 11 comprehensively controls the entire server 11 according to an operation signal input from the client terminal 13 via the Internet 12. The CPU 31 has a RAM 33, a hard disk drive (HDD) 34, a communication interface (communication I / F) 35, a face detection unit 36, a facial expression recognition unit 37, a motion detection unit 38, a voice extraction unit 39, via a data bus 32. A voice emotion recognition unit 40, an emotion determination unit 41, a decorative image acquisition unit 42, and a synthesis processing unit (processing unit) 43 are connected.

  The RAM 33 is a working memory for the CPU 31 to execute processing. The HDD 34 stores various programs and data for operating the server 11. The HDD 34 stores a moving image content processing program 44. The CPU 31 reads a program from the HDD 34 and develops it in the RAM 33, and sequentially processes the read program.

  The HDD 34 is provided with an emotion database (emotion DB) 45 and a decoration image database (decoration image DB) 46. The emotion DB 45 stores an emotion table 51 shown in FIG.

  The emotion table 51 stores a combination of a facial expression, a person's movement, and a voice emotion in association with an emotion that is conceived from this combination. The voice emotion means an emotion recognized from the voice, but in the following, a case where the emotion is an element indicating the strength of the emotion will be described as an example. There are four facial expressions: “no expression”, “smile”, “anger face”, “sad face”, and the movement of the person is “None”, “Small”, “Medium”, “Large” If the voice emotions are five levels of “silence”, “none”, “small”, “medium”, “large”, emotions corresponding to 80 (= 4 × 4 × 5) combinations are provided. It will be remembered. For example, as shown in FIG. 4, “loving laughter” is stored as an emotion corresponding to a combination in which the facial expression is “smile”, the person's movement is “none”, and the voice emotion is “silent”. Has been.

  In the decoration image DB 46, a plurality of types of decoration images are stored in association with emotions. For example, as shown in FIG. 5, three types of decorative images are stored corresponding to the emotion “loving laughter”. In the first type, “I want to go home early” is stored as a balloon that shows the inside of the mind, in the second type, “Tara // sweat mark” is stored as an onomatopoeia and comics, and for the third type, “...” is stored as a speech balloon. A comic book is a code that visualizes emotions and feelings. By default, a balloon decoration image indicating the first type of heart is selected. Based on the operation instruction from the client terminal 13, the selected decorative image is switched. Specifically, a decorative image of the second type of onomatopoeia and comics, or a decorative image of the third type of speech, or a plurality of types of decorative images are selected or randomly selected. It switches as follows.

  The communication I / F 35 is, for example, a modem or a router, controls a communication protocol suitable for the Internet 12, and exchanges data via the Internet 12. Data input via the communication I / F 35 is temporarily stored in the RAM 33.

  When the moving image content is input to the server 11, the face detection unit 36 detects a face from each frame of a still image constituting the moving image content, or detects a time-series change or movement. For the detection of the face, a method using red-eye detection disclosed in Japanese Patent Laid-Open No. 2005-267512 is used. For details, refer to JP-A-2005-267512. For face detection, a technique using pattern matching, skin color detection, or the like may be used.

  The facial expression recognition unit 37 recognizes the facial expression detected by the face detection unit 36. For recognition of facial expressions, a technique using a hidden Markov model (HMM) disclosed in Japanese Patent Laid-Open No. 10-255043 is used. For details, refer to Japanese Patent Laid-Open No. 10-255043.

  The motion detector 38 detects the motion of the person whose face is detected by the face detector 36. For detecting the movement of a person, a technique using statistical data disclosed in Japanese Patent Laid-Open No. 09-251542 is used. For details, refer to Japanese Patent Application Laid-Open No. 09-251542. Further, the movement of a person may be detected by comparing the position of the face detected by the face detection unit 36 between frames.

  The voice extraction unit 39 extracts a person's voice from the voice constituting the moving image content input to the server 11. Then, it is determined whether or not the extracted voice is a voice of a person whose face is detected by the face detection unit 36. A band-pass filter or the like is used for voice extraction, and the technique disclosed in Japanese Patent Application Laid-Open No. 09-127975 or Japanese Patent Application Laid-Open No. 11-1119791 is used for the above determination. Refer to Japanese Patent Application Laid-Open No. 09-127975, Japanese Patent Application Laid-Open No. 11-1119791, and the like for a detailed description of the above determination. For the above-described determination, a technique for detecting a change in the mouth angle, that is, a movement of the mouth angle, may be used.

  The voice emotion recognition unit 40 recognizes the voice emotion of the person whose face is detected by the face detection unit 36 based on the voice extracted by the voice extraction unit 39. For voice emotion recognition, techniques disclosed in Japanese Patent Application Laid-Open No. 09-127975, Japanese Patent Application Laid-Open No. 10-049188, Japanese Patent Application Laid-Open No. 11-1119791, and the like are used. For detailed explanation, refer to JP-A 09-127975, JP-A 10-049188, JP-A 11-1119791, and the like.

  The emotion determination unit 41 accesses the emotion DB 45 and uses the emotion table 51 shown in FIG. 4 to express the emotion of the person whose face is detected by the face detection unit 36 every predetermined span (for example, a span of 5 seconds). judge. Specifically, for each person whose face is detected by the face detection unit 36, the facial expression recognized by the expression recognition unit 37, the movement of the person detected by the motion detection unit 38, and the voice emotion recognition unit 40. The emotion corresponding to the recognized voice emotion combination is determined to be the emotion of the person.

  For example, for a person whose face is detected by the face detection unit 36, the facial expression recognition unit 37 recognizes that the facial expression is “smile”, and the motion detection unit 38 detects that the person's movement is “none”. When the voice emotion recognition unit 40 recognizes that the voice emotion is “silence”, the emotion determination unit 41 determines that the emotion of the person is “loving laughter” (third row from the top in FIG. 4). Column). If the facial expression recognized by the facial expression recognition unit 37 or the voice emotion recognized by the voice emotion recognition unit 40 has changed within a predetermined span, the emotion determination unit 41 determines whether the facial expression is changed. The above determination is performed using the facial expression or voice emotion.

  The decoration image acquisition unit 42 accesses the decoration image DB 46 and acquires a decoration image corresponding to the emotion determined by the emotion determination unit 41. For example, when the emotion determination unit 41 determines that the emotion is “loving laughter”, by default, “I want to go home early”, which is a balloon decoration image showing the inside of the heart, is acquired. When there is an operation instruction from the client terminal 13, based on the operation instruction, “Tara // sweat mark” which is a decorative image of onomatopoeia and comics, or a decorative image of speech balloon “. . "

  The composition processing unit 43 synthesizes the decoration image acquired by the decoration image acquisition unit 42 with each frame of the still image constituting the moving image content. A case where three faces are detected by the face detection unit 36 will be described as an example with reference to FIG. A person in the upper frame where the facial expression is “smile”, the person ’s movement is “none”, and the voice emotion is “silence” is determined to be “loving laughter”. The acquisition unit 42 acquires “I want to go home early” which is a decorative image of a balloon showing the inside of the first kind of heart. The compositing processing unit 43 arranges the still image constituting the moving image content so that the decoration image acquired by the decoration image acquiring unit 42 “I want to go home early” is arranged around the person appearing above the frame. Composite to frame.

  A person who appears to the left of the frame with a facial expression of “smiling”, a person's movement of “small”, and a voice emotion of “small” is judged as “slightly joke” by default. The decoration image acquisition unit 42 has acquired “Ufufufu. Do you see me?”, Which is a decoration image of a balloon showing the inside of the first type of heart. The compositing processing unit 43 sets the still image that constitutes the moving image content so that the decoration image acquired by the decoration image acquisition unit 42 is “Ufufufu. Composite to the frame of the image.

  A person who appears to the right of the frame with a facial expression of “smiling”, a human movement of “large”, and a voice emotion of “large” is judged to be “very fun”. The image acquisition unit 42 has acquired “very fun ♪”, which is a decoration image of a balloon showing the inside of the first kind of mind. The compositing processing unit 43 arranges the frame of the still image that constitutes the moving image content so that the decoration image acquired by the decoration image acquisition unit 42 is arranged around the person in the right side of the frame. To synthesize.

  Also, as shown in FIG. 7, when the “Tara ~ / sweat mark”, which is a decorative image of the second type of onomatopoeia and comics, has been acquired for the person shown above the frame, the composition processing unit 43 Then, the acquired decoration image “Tara // sweat mark” is combined with the frame of the still image constituting the moving image content so as to be arranged around the person in the upper part of the frame.

  In the case where “Niyanya”, which is a decoration image of the second type of onomatopoeia and comics, has been acquired for the person appearing on the left side of the frame, the compositing processing unit 43 “Niyanya” which is the acquired decoration image. Are arranged in a frame of a still image constituting the moving image content so as to be arranged around a person appearing on the left side of the frame.

  When “Ahaha (laughs)”, which is a decorative image of the second type of onomatopoeia and comics, has been acquired for the person shown on the right side of the frame, the composition processing unit 43 is the acquired decorative image. “Ahaha (laughs)” is combined with a still image frame constituting the moving image content so as to be arranged around a person shown on the right side of the frame.

  Similarly, as illustrated in FIG. 8, when “..”, which is a third-type speech balloon decoration image, has been acquired for the person shown above the frame, the composition processing unit 34 acquires the image. The decorative image “..” is combined with a still image frame constituting the moving image content so as to be arranged around a person appearing above the frame.

  In the case where “Cash is a cash!”, Which is a decoration image of the third type of speech, is acquired for the person shown on the left side of the frame, the composition processing unit 43 is the acquired decoration image. “Cash?” Is combined with the still image frame that composes the video content so that it is placed around the person on the left side of the frame.

  For the person appearing on the right side of the frame, when the decoration image of the third type of speech is acquired, “How are you !!”, the composition processing unit 43 is the acquired decoration image. “I'm fine!” Is combined with each frame of the still image that composes the video content so that it is placed around the person in the right side of the frame.

  Next, a processing procedure when the server 11 (see FIGS. 1 and 3) configured as described above functions as a moving image content processing device will be described with reference to the flowchart of FIG. The user operates the operation unit 18 of the client terminal 13 to input moving image content to the server 11. The moving image content input to the server 11 is stored in the RAM 33.

  The moving image content input to the server 11 is processed in real time every predetermined span (for example, a span of 5 seconds (150 frames)). The moving image content is first read from the RAM 33 to the face detection unit 36, the motion detection unit 38, and the voice extraction unit 39, respectively. In the face detection unit 36, a face is detected from each frame of a still image constituting the moving image content, or a time-series change or motion is detected. The detected face and time-series changes and movements are stored in the RAM 33.

  The face detected by the face detection unit 36 and time-series changes and movements are read from the RAM 33 to the facial expression recognition unit 37. The facial expression recognition unit 37 recognizes the facial expression detected by the face detection unit 36. The recognized facial expression is stored in the RAM 33.

  The facial expression recognized by the facial expression recognition unit 37 is read from the RAM 33 to the motion detection unit 38. The motion detector 38 detects the motion of the person whose face is detected by the face detector 36. The detected movement of the person is stored in the RAM 33.

  In the voice extraction unit 39, the voice of the person whose face is detected by the face detection unit 36 is extracted from the sound constituting the moving image content. The extracted human voice is stored in the RAM 33.

  The voice of the person extracted by the voice extraction unit 39 is read from the RAM 33 to the voice emotion recognition unit 40. The voice emotion recognition unit 40 recognizes the voice emotion of the person whose face is detected by the face detection unit 36. The recognized voice emotion of the person is stored in the RAM 33.

  The facial expression recognized by the facial expression recognition unit 37, the movement of the person detected by the motion detection unit 38, and the voice emotion recognized by the voice emotion recognition unit 40 are read from the RAM 33 to the emotion determination unit 41. The emotion determination unit 41 determines the emotion of the person whose face is detected by the face detection unit 36 while referring to the emotion table 51 (see FIG. 4) stored in the emotion DB 45. The determined emotion is stored in the RAM 33.

  The emotion determined by the emotion determination unit 41 is read from the RAM 33 to the decoration image acquisition unit 42. In the decoration image acquisition unit 42, a decoration image (see FIG. 5) corresponding to the emotion determined by the emotion determination unit 41 is acquired from the decoration image DB 46. The acquired decoration image is stored in the RAM 33.

  The decoration image acquired by the decoration image acquisition unit 42 is read from the RAM 33 to the composition processing unit 43. In the composition processing unit 43, the decoration image acquired by the decoration image acquisition unit 42 is combined with a frame of a still image constituting the moving image content. The moving image content combined with the decoration image is stored in the RAM 33.

  The moving image content in which the decoration image is synthesized by the composition processing unit 43 is a series of processes of face detection → expression recognition → motion detection → voice extraction → speech emotion recognition → emotion determination → gender determination → age determination → decoration image acquisition → image composition. Is read from the RAM 33 at every predetermined span for which.

  The moving image content read from the RAM 33 is transmitted to the client terminal 13 via the Internet 12. The moving image content transmitted to the client terminal 13 is stored in the RAM 23.

  The moving image content transmitted to the client terminal 13 is read from the RAM 23 and displayed on the monitor 15 in real time.

  As described above, not only facial expressions but also human movements and voice emotions are used to comprehensively determine emotions, and a decorative image corresponding to the determined emotions is synthesized, so a decorative image suitable for video content Can be synthesized. In addition, by storing a decoration image that is not related to emotions in the decoration image DB 46 in correspondence with the emotions, it is possible to process video content having a surprising new story (see FIG. 8). .

[Second Embodiment]
In the moving image content processing apparatus according to the first embodiment, a decoration image that is not related to the gender / age of the person being photographed is synthesized. If the decorative image is synthesized, the girl child is uttering the words of the old father, which gives a strange feeling to the person who viewed the moving image content. Certainly, when a decorative image of a female speech balloon is combined with a video content showing a male, the male is speaking a female speech. It is also possible to entertain unexpectedness, such as making you feel that she is a shemale. However, synthesizing decorative images that are not related to gender and age for all persons only gives a sense of incongruity and is not preferable. Therefore, in the moving image content processing apparatus according to the second embodiment, which will be described next, a decorative image that is related to the gender and age of the person being photographed is synthesized so as not to give a sense of incongruity to the person who viewed the moving image content. To do.

  In FIG. 10, the moving image content processing apparatus according to the second embodiment is realized in a format constructed in the server 61 by installing the moving image content processing program 44. A sex estimation unit 62 and an age estimation unit 63 are connected to the CPU 31 via the data bus 32.

  In the decorative image DB 46, a plurality of types of decorative images are stored in association with combinations of emotion, sex, and age.

  The gender estimating unit 62 estimates the gender of the person whose face is detected by the face detecting unit 36. Specifically, the face detected by the face detection unit 36, the facial expression recognized by the expression recognition unit 37, the voice of a person extracted by the voice extraction unit 39, and the voice recognized by the voice emotion recognition unit 40 Estimate the gender of a person based on emotion. For gender estimation, a technique disclosed in Japanese Patent Application Laid-Open No. 2007-080057 is used. For details, refer to Japanese Patent Application Laid-Open No. 2007-080057.

  The age estimation unit 63 estimates the age of the person whose face is detected by the face detection unit 36. Specifically, the face detected by the face detection unit 36, the facial expression recognized by the expression recognition unit 37, the voice of a person extracted by the voice extraction unit 39, and the voice recognized by the voice emotion recognition unit 40 Estimate the age of a person based on emotion. The technique disclosed in Japanese Patent Application Laid-Open No. 2007-080057 is used for estimating the age. For details, refer to Japanese Patent Application Laid-Open No. 2007-080057.

  The decoration image acquisition unit 42 accesses the decoration image DB 46, and the decoration corresponding to the combination of the emotion determined by the emotion determination unit 41, the gender estimated by the gender estimation unit 62, and the age estimated by the age estimation unit 63. Get an image. Note that the description of the same configuration as in the first embodiment is omitted.

  Next, a processing procedure when the server 61 (see FIG. 10) configured as described above functions as a moving image content processing apparatus will be described with reference to the flowchart of FIG. Note that a description of processing procedures similar to those in the first embodiment is omitted.

  The face detected by the face detection unit 36, the facial expression recognized by the expression recognition unit 37, the voice of the person extracted by the voice extraction unit 39, and the voice emotion recognized by the voice emotion recognition unit 40 are read from the RAM 33. The data is read out to each of the sex estimation unit 62 and the age estimation unit 63. The gender estimating unit 62 estimates the gender of the person whose face is detected by the face detecting unit 36. The estimated gender is stored in the RAM 33. When the gender of the person whose face is detected by the face detection unit 36 is once estimated, the process by the gender estimation unit 62 may be omitted for the corresponding person.

  The age estimation unit 63 estimates the gender of the person whose face is detected by the face detection unit 36. The estimated age is stored in the RAM 33. When the age of the person whose face is detected by the face detection unit 36 is once estimated, the process by the age estimation unit 63 may be omitted for the corresponding person.

  The emotion determined by the emotion determination unit 41, the gender estimated by the gender estimation unit 62, and the age estimated by the age estimation unit 63 are read from the RAM 33 to the decoration image acquisition unit 42. In the decoration image acquisition unit 42, a decoration image corresponding to the combination of the emotion determined by the emotion determination unit 41, the gender estimated by the gender estimation unit 62, and the age estimated by the age estimation unit 63 is obtained from the decoration image DB 46. To be acquired.

  The moving image content in which the decoration image is synthesized by the synthesis processing unit 43 is a series of processes of face detection → expression recognition → motion detection → voice extraction → voice emotion recognition → emotion determination → sex estimation → age estimation → decoration image acquisition → image synthesis. Is read from the RAM 33 at every predetermined span for which.

  As explained above, since the decoration image that is related to the gender and age of the person in the picture is synthesized, for example, it will not become a video content that the girl is uttering the words of the old father, It does not give a sense of incongruity to those who have seen video content.

  In each of the above embodiments, a case has been described in which a moving image content processing apparatus is constructed in the server 11 connected to the Internet 12 and can be accessed by everyone. However, the present invention is not limited to this. For example, a moving image content processing apparatus may be constructed on a personal computer used by an individual. In addition, a digital camera capable of shooting a moving image may be provided with a moving image content processing function.

  Further, in each of the above embodiments, the processing is performed on all of the sections divided into the predetermined spans, but the processing may be performed only when the emotion changes. In this case, the decoration image acquisition unit 42 and the composition processing unit 43 execute the process only when the emotion determined by the emotion determination unit 41 changes.

  Further, in each of the above embodiments, the case of the decoration image of the balloon showing the inside of the heart, the onomatopoeia and the comics, and the balloon of the dialogue is described as an example, but the present invention is not limited to this case. In addition, decorative images corresponding to scenes such as travel, wedding ceremony, entrance ceremony, and birthday may be used.

  Further, in each of the above embodiments, the case where the type of decoration image selected by the user on the client terminal 13 is combined has been described as an example. However, the type of decoration image randomly selected by the moving image content processing apparatus is combined. You may do it.

  In each of the above-described embodiments, the case where a decorative image that is a still image is combined has been described as an example. However, a moving image, audio, or other content may be combined.

  Further, the moving image content processing apparatus shown in each of the above embodiments is merely an example, and can be appropriately changed to any mode without departing from the gist of the present invention.

It is the schematic which shows the structure of a network system. It is a block diagram which shows the internal structure of a client terminal. It is a block diagram which shows the internal structure of a server. It is explanatory drawing which shows the structure of an emotion table. It is a list of the decoration image memorize | stored in decoration image DB. It is a figure explaining the process which synthesize | combines the decoration image of the 1st type with the frame of the still image which comprises a moving image content. It is a figure explaining the process which synthesize | combines the 2nd kind of decoration image with the frame of the still image which comprises a moving image content. It is a figure explaining the process which synthesize | combines the 2nd kind of decoration image with the frame of the still image which comprises a moving image content. It is a flowchart explaining the processing procedure of a moving image content. It is a block diagram which shows the internal structure of the server of another form. It is a flowchart explaining the processing procedure of the moving image content of another form.

Explanation of symbols

11, 61 server (video content processing device)
36 face detection unit 37 facial expression recognition unit 38 motion detection unit 39 voice extraction unit 40 voice emotion recognition unit 41 emotion determination unit 42 decoration image acquisition unit 43 composition processing unit 44 video content processing program 45 emotion database (emotion DB)
46 Decorative Image Database (Decorative Image DB)
51 Emotion Table 62 Gender Estimation Unit 63 Age Estimation Unit

Claims (8)

A face detection unit that detects faces from video content;
A facial expression recognition unit that recognizes facial expressions detected by the face detection unit;
A motion detection unit that detects a motion of a person whose face is detected by the face detection unit from video content;
A voice extraction unit that extracts a voice of a person whose face is detected by the face detection unit from video content;
From the facial expression recognized by the facial expression recognition unit, the movement of the person detected by the motion detection unit, and the voice extracted by the voice extraction unit, the emotion of the person whose face is detected by the face detection unit An emotion determination unit for determining;
A video content processing apparatus comprising: a processing unit that processes video content based on the emotion determined by the emotion determination unit. A voice emotion recognition unit that recognizes voice emotions from the voice extracted by the voice extraction unit;
An emotion database storing an emotion table that stores a combination of facial expressions, human movements, and voice emotions, and the emotions stored in correspondence with each other;
The emotion determination unit uses the emotion table to determine the facial expression recognized by the facial expression recognition unit, the movement of the person detected by the motion detection unit, and the voice emotion recognized by the voice emotion recognition unit. The moving image content processing apparatus according to claim 1, wherein an emotion corresponding to the combination is determined to be an emotion of a person whose face is detected by the face detection unit. A decoration content database that stores decoration content to be decorated in video content in correspondence with emotions;
A decoration content acquisition unit that acquires the decoration content corresponding to the emotion determined by the emotion determination unit from the decoration content database;
The moving image content processing apparatus according to claim 1, wherein the processing unit is a composition processing unit that combines the decorative content acquired by the decorative content acquisition unit with the moving image content. The decorative content stored in the decorative content database corresponding to emotions is a plurality of types,
The video content processing apparatus according to claim 3, wherein the decoration content acquisition unit acquires the input type of the decoration content. A gender estimating unit that estimates the gender of the person whose face is detected by the face detecting unit from the face detected by the face detecting unit and the voice extracted by the voice extracting unit;
5. The moving image content processing apparatus according to claim 1, wherein the processing unit processes the moving image content based on the gender estimated by the gender determination unit. An age estimation unit for estimating the age of a person whose face is detected by the face detection unit from the face detected by the face detection unit and the voice extracted by the voice extraction unit;
The moving image content processing apparatus according to claim 1, wherein the processing unit processes the moving image content based on the age estimated by the age determination unit. A face detection step of detecting a face from the video content in the face detection unit;
A facial expression recognition step for recognizing the facial expression detected in the face detection step by a facial expression recognition unit;
A motion detection step in which a motion detection unit detects a motion of a person whose face is detected in the face detection step from video content;
A voice extraction step of extracting a voice of a person whose face is detected in the face detection step by a voice extraction unit from video content;
From the facial expression recognized in the facial expression recognition step, the movement of the person detected in the motion detection step, and the voice extracted in the voice extraction step, the emotion of the person whose face was detected in the face detection step is represented. An emotion determination step for determination by the emotion determination unit;
A processing method for moving image content, comprising: a processing step for processing the moving image content by a processing processing unit based on the emotion determined in the emotion determination step. A face detection step for detecting a face from video content;
A facial expression recognition step for recognizing facial expressions detected in the face detection step;
A motion detection step for detecting a motion of a person whose face is detected in the face detection step from video content;
A voice extraction step of extracting the voice of the person whose face was detected in the face detection step from the video content;
From the facial expression recognized in the facial expression recognition step, the movement of the person detected in the motion detection step, and the voice extracted in the voice extraction step, the emotion of the person whose face was detected in the face detection step is represented. An emotion determination step for determining;
A video content processing program that causes a computer to execute a processing step of processing video content based on the emotion determined in the emotion determination step.

Images