JP2011228936A - Moving image transmission system, transmitter, receiver, moving image management device, transmission program, reception program, and moving image management program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently transmit a moving image with added value.SOLUTION: In a moving image transmission system which transmits a moving image of a person between a transmitter and a receiver through a communication network, the transmitter comprises: transmission-side virtual makeup image generation means of generating a virtual makeup image by putting virtual makeup to an original image that includes the face of a person photographed by imaging means, at a predetermined position of the face; makeup information acquisition means of acquiring makeup information of the virtual makeup image; transmission means of transmitting the makeup information obtained by the makeup information acquisition means and an original image frame separately. The receiver comprises reception-side virtual makeup image generation means of acquiring the makeup information and the original image frame from the transmitter, and generates the virtual makeup image while reflecting the makeup information synchronized with the original image frame based upon identification information or time information added thereto respectively.

Description

  The present invention relates to a moving image transmission system, a transmitting device, a receiving device, a moving image management device, a transmission program, a receiving program, and a moving image management program, and in particular, a moving image transmission system for efficiently transmitting a moving image with added value, The present invention relates to a transmission device, a reception device, a moving image management device, a transmission program, a reception program, and a moving image management program.

  2. Description of the Related Art Conventionally, there are communication tools such as a video phone and a video conference that allow a user to talk while looking at the other person's face through a screen. However, it is difficult to say that videophones in particular are currently widely used in homes and offices.

  The reason for this is that, especially in the case of women, when the other party is not a close friend of a certain degree, there is resistance to appearing on the screen with a bare face at home, especially the video performance of recent videophones. Because of the high quality of the skin, the skin is clearly displayed, and this tendency is further spread.

Here, there exists a makeup simulation system that virtually applies makeup to a moving image obtained by photographing a user's face with respect to the user's face displayed on the screen (see, for example, Patent Document 1).
Such a system includes display means for displaying a captured user's moving image, recognizes the user's face from the moving image based on a predetermined tracking point, and further determines a predetermined face of the user based on the recognition result. A foundation, lipstick, or the like can be virtually applied to the location, and a predetermined makeup can be applied and output to the display means.

  In addition, a method of continuously displaying several still image frames per second is mainly used for moving images transmitted by a video phone or the like. For example, a mobile terminal having a videophone function can transmit and receive about 15 frames per second, and an IP telephone connected to a broadband line can transmit and receive about 15 to 30 frames per second. Also, in order to compress data and improve transmission efficiency during communication, instead of continuously sending images under the same conditions, only the difference is sent, the change in information is replaced with a numerical value, The data is sent after being compressed, such as changing the quality of the sent data depending on how busy the line is.

  In recent years, H.C. With the video compression standard of H.264, a compression format that has the same image quality as MPEG2 and requires about 1/2 or less data volume has been put into practical use, and is widely used for video images such as One-Seg broadcasting, Blu-ray DISC, PSP, iPod Yes. However, all of these are based on the premise that a moving image taken by a camera or the like is sent.

JP 2007-216000 A

  Here, in order to use the virtual makeup simulation image generated by the above-described system in a videophone or the like, it is preferable for the user that the face or makeup face can be freely selected by the talking partner, and how much makeup is applied. It is preferable to perform a charging process or the like on the user by judging whether the amount of data is high. In such a case, if the makeup image is simply combined with the makeup image and then transmitted, it is not possible to accurately acquire the switching between the makeup face and the makeup face, the amount of makeup data, the transmission efficiency, and the like.

  That is, the above-described conventional system configuration does not assume the above-described contents, and even if only a composite image subjected to makeup simulation is transmitted, the above-described problem cannot be dealt with.

  Further, in recent years, there is a so-called “Chara-den” mechanism that displays a favorite character instead of oneself to a person who does not want to show his / her face when using a videophone. However, “Chara-den” is not preferable from the viewpoint of the existence of a videophone because the person's face is not directly displayed.

  The present invention has been made in view of the above-described problems, and is a moving image transmission system, a transmission device, a reception device, a moving image management device, a transmission program, and a reception program for efficiently transmitting a value-added moving image. It is an object to provide a video management program.

  In order to solve the above-described problems, the present invention employs means for solving the problems having the following characteristics.

  According to a first aspect of the present invention, in the video transmission system for transmitting a moving image of a person between a transmission device and a reception device via a communication network, the transmission device is configured to transmit the person captured by an imaging unit. A transmission-side virtual makeup image generation unit that generates a virtual makeup image in which virtual makeup is applied to a predetermined position of the face with respect to an original image including a face; and a virtual makeup obtained by the transmission-side virtual makeup image generation unit Makeup information acquisition means for acquiring makeup information of the image, and transmission means for separately transmitting the makeup information obtained by the makeup information acquisition means and the original original image frame to the reception device, The makeup information and the original image frame are acquired from the transmission device, and the original image frame is obtained based on the identification information or time information added to each. And having a receiving-side virtual makeup image generating means for generating the virtual makeup image by reflecting the makeup information synchronized with the over arm.

  According to the first aspect of the present invention, since a virtual makeup image can be additionally generated and the information can be transmitted with a small amount of data, a value-added moving image can be efficiently transmitted. .

  The invention described in claim 2 includes a moving image transmission management device capable of transmitting and receiving data to and from the transmitting device and the receiving device via the communication network, and the moving image transmission management device receives the make-up from the transmitting device. Data amount acquisition means for acquiring information and acquiring the data amount of the acquired makeup information; and charging means for charging for transmission of the moving image based on the data amount obtained by the data amount acquisition means It is characterized by that.

  According to the second aspect of the present invention, the billing process at the time of moving picture transmission of the virtual makeup simulation can be performed efficiently and accurately by the moving picture transmission management apparatus.

  According to a third aspect of the present invention, the makeup information acquisition means includes vector data including facial part contour information and application areas from a virtual makeup image obtained by the transmission-side virtual makeup image generation means, and a makeup technique. And makeup information having parameter information including numerical information on the image.

  According to the third aspect of the present invention, since only makeup information with a small amount of information is transmitted, the amount of data can be reduced, the transmission efficiency can be improved, and the content of data being transmitted can be transferred to a third party. Even if it is seen, it is impossible to grasp the specific contents, so that confidentiality can be improved.

  In the invention described in claim 4, the moving picture transmission management device changes the content of the makeup information based on user information or transmission setting content preset for each receiving device, and the changed makeup information. Is transmitted.

  According to the fourth aspect of the present invention, it is possible to easily generate different virtual makeup images for a plurality of receiving apparatuses.

  According to a fifth aspect of the present invention, the moving picture transmission management device extracts user information of the receiving device, and based on the extracted user information, the make information is transmitted to the receiving device during a call. It is determined whether or not it is possible, and when the transmission is possible, the makeup information is transmitted.

  According to the fifth aspect of the present invention, it is possible to transmit appropriate information for each receiving device.

  According to a sixth aspect of the present invention, in a transmission device that transmits a moving image to a predetermined reception device via a communication network, the original image including the face of the person photographed by an imaging unit is Virtual makeup image generation means for generating a virtual makeup image obtained by performing virtual makeup on a predetermined position of the face; makeup information acquisition means for acquiring makeup information of a virtual makeup image obtained by the virtual makeup image generation means; Transmitter means for separately transmitting makeup information obtained by the makeup information acquisition means and the original original image frame to the receiving device.

  According to the sixth aspect of the present invention, since a virtual makeup image can be additionally generated and the information can be transmitted with a small amount of data, a value-added moving image can be efficiently transmitted. .

  According to a seventh aspect of the present invention, the makeup information acquisition means includes vector data including facial part contour information and application areas from the virtual makeup image obtained by the virtual makeup image generation means, and numerical values relating to makeup techniques. Makeup information having parameter information including information is acquired.

  According to the seventh aspect of the present invention, it is possible to generate a virtual makeup image for the original image even with the small amount of information described above.

  According to an eighth aspect of the present invention, in a receiving device that receives a moving image transmitted from a predetermined transmitting device via a communication network, an original image acquiring unit that acquires an original image frame photographed by the transmitting device. Makeup information obtained by generating a virtual makeup image for the original image frame processed by the transmission apparatus, makeup information obtaining means for obtaining, the original image frame obtained by the original image obtaining means and the makeup information Virtual makeup image generation means for generating the virtual makeup image by reflecting the makeup information synchronized with the original image frame based on identification information or time information added to makeup information obtained by the acquisition means. It is characterized by having.

  According to the eighth aspect of the invention, it is possible to additionally generate a virtual makeup image from the original image frame and the makeup information, thereby providing a service with added value.

  The invention described in claim 9 is a moving picture transmission management apparatus that manages a moving image of a person transmitted and received between a transmission apparatus and a reception apparatus via a communication network, and is photographed by an imaging unit in the transmission apparatus. A data amount acquisition means for acquiring a data amount of makeup information obtained when generating a virtual makeup image obtained by performing virtual makeup on a predetermined position of the face with respect to an original image including the face of the person; Based on the data amount obtained by the data amount acquisition means, charging means for charging for transmission of the moving image, and changing the contents of the makeup information in correspondence with the receiving device, and transmitting the changed makeup information Transmission setting means.

  According to the ninth aspect of the present invention, it is possible to perform appropriate charging setting based on the data amount of makeup information.

  The invention described in claim 10 extracts user information of the receiving device, and determines whether the makeup information can be transmitted to the receiving device during a call based on the extracted user information. And terminal management means for transmitting the makeup information when transmission is possible.

  According to the tenth aspect of the present invention, it is possible to transmit appropriate information for each receiving device.

  According to an eleventh aspect of the present invention, in a transmission program in a transmission device that transmits a moving image to a predetermined reception device via a communication network, the computer includes an original including the face of the person photographed by the imaging means. Virtual makeup image generation means for generating a virtual makeup image obtained by performing virtual makeup on a predetermined position of the face with respect to an image, makeup information for acquiring makeup information of a virtual makeup image obtained by the virtual makeup image generation means It functions as an acquisition means and a transmission means for separately transmitting the makeup information obtained by the makeup information acquisition means and the original original image frame to the receiving device.

  According to the eleventh aspect of the present invention, since a virtual makeup image can be additionally generated and the information can be transmitted with a small amount of data, a value-added moving image can be efficiently transmitted. . Further, by installing the program, the transmission processing in the present invention can be easily realized by a general-purpose personal computer or the like.

  In the invention described in claim 12, the makeup information acquisition means includes vector data including contour information and application areas of facial parts from the virtual makeup image obtained by the virtual makeup image generation means, and numerical values relating to makeup techniques. Makeup information having parameter information including information is acquired.

  According to the twelfth aspect of the present invention, it is possible to generate a virtual makeup image for the original image even with the small amount of information described above.

  According to a thirteenth aspect of the present invention, in a receiving program in a receiving device that receives a moving image transmitted from a predetermined transmitting device via a communication network, an original image frame captured by the transmitting device is received by a computer. Original image acquisition means to acquire, makeup information obtained by generating a virtual makeup image for the original image frame processed by the transmission device, makeup information acquisition means to acquire, and the original image obtained by the original image acquisition means The virtual makeup image is generated by reflecting the makeup information synchronized with the original image frame based on identification information or time information added to the image frame and the makeup information obtained by the makeup information acquisition unit, respectively. Function as virtual makeup image generation means.

  According to the thirteenth aspect of the present invention, it is possible to additionally generate a virtual makeup image from the original image frame and the makeup information, thereby providing a value-added service. Also, by installing the program, the reception processing in the present invention can be easily realized by a general-purpose personal computer or the like.

  According to a fourteenth aspect of the present invention, there is provided a video transmission management program in a video transmission management apparatus that manages a moving image of a person transmitted and received between a transmission device and a reception device via a communication network. In the apparatus, a data amount of makeup information obtained when a virtual makeup image is generated by performing virtual makeup on a predetermined position of the face with respect to an original image including the face of the person photographed by an imaging unit. Data amount acquisition means, charging means for charging for transmission of the moving image based on the data amount obtained by the data amount acquisition means, and changing the contents of the makeup information corresponding to the receiving device, It functions as a transmission setting means for transmitting the changed makeup information.

  According to the fourteenth aspect of the present invention, it is possible to perform appropriate billing settings based on the data amount of makeup information. Further, by installing the program, the moving image transmission process according to the present invention can be easily realized by a general-purpose personal computer or the like.

  The invention described in claim 15 extracts user information of the receiving device, and determines whether the makeup information can be transmitted to the receiving device during a call based on the extracted user information. And terminal management means for transmitting the makeup information when transmission is possible.

  According to the fifteenth aspect of the present invention, appropriate information can be transmitted for each receiving device.

  According to the present invention, a moving image with added value can be efficiently transmitted.

It is a figure for demonstrating an example of the system configuration | structure of the moving image transmission system in this embodiment. It is a figure which shows an example of a function structure of the transmitter in 1st Embodiment. It is a figure which shows an example of a function structure of the moving image management server in 1st Embodiment. It is a figure which shows an example of a function structure of the receiver in 1st Embodiment. It is a figure for demonstrating the mode of transmission of the image frame in this embodiment. It is a figure for demonstrating the production | generation process of the makeup information in this embodiment. 7 shows an example of a billing process in the moving image management server. It is a figure which shows an example of the parameter information in virtual makeup simulation. It is a figure for demonstrating the specific operation | movement of a moving image management server. It is a figure which shows an example of a function structure of the transmitter in 2nd Embodiment. It is a figure for demonstrating the Example of the separation process in 2nd Embodiment. It is a figure which shows an example of the hardware constitutions which can implement | achieve the moving image transmission process in this embodiment. It is a flowchart which shows an example of the transmission processing procedure in this embodiment. It is a flowchart which shows an example of the moving image management processing procedure in this embodiment. It is a flowchart which shows an example of the reception processing procedure in this embodiment.

<About the present invention>
The present invention can be obtained by virtual make-up on moving images taken on the transmission side by installing software on the transmission side and the reception side in each communication tool such as a mobile terminal such as a video phone, a video conference, and a mobile phone. The makeup information and the like are transmitted separately from the transmission of the original moving image information of the photographed user, and the virtual makeup image is synthesized on the receiving side or the moving image management server side and displayed on the receiving device.

  Specifically, in synchronization with the timing of transmitting a continuous image frame (raster information) as an original video stream signal from the transmission side, makeup information (for example, vector information or parameter information) corresponding to the image frame is transmitted. Send. In this way, by transmitting only makeup information separately from image frames, when performing charging processing at the time of transmission based on the amount of data or the contents of the data, it is possible to efficiently calculate based on that amount of information. Therefore, it is possible to perform an appropriate billing setting with high accuracy according to the usage contents for each user.

  Hereinafter, embodiments in which a moving image transmission system, a transmission device, a receiving device, a moving image management device, a transmission program, a reception program, and a moving image management program according to the present invention are suitably implemented will be described with reference to the drawings.

<Video transmission system: system configuration example>
First, a system configuration example according to this embodiment will be described with reference to the drawings. FIG. 1 is a diagram for explaining an example of a system configuration of a moving image transmission system according to the present embodiment.

  A moving picture transmission system 10 shown in FIG. 1 is configured to include a transmitting apparatus 11, a moving picture management server 12, and a receiving apparatus 13. Each component is controlled by a communication network 14 such as the Internet or a telephone line for data and control. Connected in a state where signals can be transmitted and received.

  In FIG. 1, the transmission device 11 may be a mobile terminal 11-1 with a camera (imaging means) capable of shooting a moving image, or may be a normal personal computer (PC) 11-2. Similarly, the receiving device 13 may be a camera-equipped mobile terminal 13-1 capable of shooting a moving image, or may be a PC 13-2. At this time, to the PCs 11-2 and 13-2, in order to transmit and receive the moving images of the users on the transmission side and the reception side, the imaging units 15-1 and 15-2 that respectively photograph the user's face. Is provided.

  The transmission device 11 and the reception device 13 are configured to have both functions of the transmission side and the reception side in order to realize bidirectional communication by the videophone function, but in the following description, for the sake of convenience, the transmission device 11 The function transmitted by the above and the function received by the receiving device 13 will be described separately.

  The transmission device 11 is a terminal for realizing a transmission-side function in a video phone or the like. Specifically, the transmission device 11 acquires a user's face image captured by the imaging device 15-1 as a moving image in real time. In addition, the transmission device 11 performs a virtual makeup image based on a virtual makeup simulation based on instruction information from a user, preset basic makeup information, and the like for an image constituting a moving image photographed by the imaging device 15-1. Is generated. In addition, about the virtual makeup simulation, the method etc. which are shown by the patent document 1 mentioned above can be used.

  Note that the virtual makeup performed in the present embodiment is usually makeup for the face (for example, all or part of the lips (mouth), eyes, eyebrows, eyelashes, cheeks, etc.). For example, hairstyles, hair colors, hats, glasses, earrings, necklaces, clothing, backgrounds, and the like can be simulated based on user settings or preset contents.

  The user of the transmission device 11 selects a favorite makeup while simulating virtual makeups of various preset patterns. A specific example of the virtual makeup simulation in the present embodiment will be described later. Thereafter, the transmission device 11 separately transmits the original image frame constituting the captured moving image and the makeup information obtained by the simulation to the reception side.

  The transmission device 11 transmits original moving image information (raster information) and makeup information (vector information, parameter information) in synchronization. Specifically, when transmitting the above-described two image frames, time information such as the same time stamp is added.

  Further, the transmission device 11 adds the same frame identification information for the image frame to the moving image information and the makeup information synthesized on the receiving side such as the moving image management server 12 or the receiving device 13. Thereby, the receiving apparatus 13 can synthesize | combine raster information and makeup information easily and correctly based on the time information and frame identification information mentioned above.

  As a transmission method in the present embodiment, for example, stream transmission can be performed by packet data or the like according to a general video compression standard, but the present invention is not particularly limited.

  Further, the transmission device 11 can transmit makeup information obtained by generating a simulation image instead of makeup information. Further, the transmission device 11 can store the set makeup information in a storage unit such as a memory, and can quickly read the makeup information from the storage unit and use it during a videophone call.

  Here, when transmitting image frames, makeup information, and the like, the transmission device 11 transmits the communication device 14 via the communication network 14 to the transmission device 13 that is the other party of the videophone that makes a call by calling processing or the like. At this time, the transmission device 11 transmits the original moving image information (original image frame) to the target transmission device 13 by a conventional communication method, and transmits the makeup information to the moving image management server 12. In this case, the moving image management server 12 can perform processing such as charging based on makeup information, the amount of makeup information, and the like. Note that the transmission device 11 may also transmit the original moving image information to the moving image management server 12.

  The moving image management server 12 acquires virtual makeup information transmitted from the reception device 11 to the transmission device 13, and performs a charging process based on the information amount of the acquired virtual makeup information, the contents of data, and the like.

  Further, the moving image management server 12 also acquires original moving image information as necessary, combines the original image frame with the makeup information transmitted in synchronism, generates a virtual makeup image, and generates the generated image. You may transmit to the transmission apparatus 13.

  In this case, software for synthesizing the original image frame and the makeup information is installed in the moving image management server 12 in advance, and the installed software is executed at the time of synthesis so that the makeup information is automatically added to the original image frame. It is synthesized and makeup information is generated.

  Further, the moving image management server 12 has a virtual makeup information DB in which makeup contents, images and the like corresponding to each user are stored based on makeup information, and from the virtual makeup information DB based on makeup information. By applying the extracted simulation contents to the original image, a virtual makeup image can be generated.

  Furthermore, the moving image management server 12 extracts user information of the transmission device 13, determines whether makeup information can be transmitted to the transmitting device 13 during a call based on the extracted user information, and can transmit the information. In case you can send makeup information.

  The receiving device 13 combines the original moving image information (original image frame) obtained from the transmitting device 11 and the makeup information obtained from the moving image management server 12 and displays the synthesized information on a display screen such as a display. In addition, when receiving the virtual makeup image already synthesized by the moving image management server 12, the receiving device 13 displays it on the display screen as it is.

  As a result, even if the user on the transmitting side is not wearing makeup, an image with virtual makeup can be displayed on the receiving side, so that the user on the transmitting side can make a videophone call with the other party without any hesitation I can talk.

  Note that since the receiving device 13 is in a two-way communication during a call from the transmitting device, for example, the receiving device 13-2 is also provided with an imaging device 15-2 that captures the face of the user.

  By the above-described processing, it is possible to efficiently transmit a value-added moving image by transmitting a makeup image or the like separately from the original image. Next, each functional configuration in the transmission device 11, the moving image management server 12, and the reception device 13 described above will be specifically described with reference to the drawings.

<First Embodiment>
<Transmitter 11: Functional Configuration Example>
An example of a functional configuration of the transmission device 11 according to the first embodiment will be described with reference to the drawings. FIG. 2 is a diagram illustrating an example of a functional configuration of the transmission device according to the first embodiment. 2 includes an input unit 21, an output unit 22, a storage unit 23, an imaging unit 24, a virtual makeup image generation unit 25, a makeup information acquisition unit 26, a screen generation unit 27, A transmission / reception unit 28 and a control unit 29 are included.

  The input unit 21 inputs a person image including the user's face taken by the imaging unit 24 or a person image including the user's face obtained from the external device by the transmission / reception unit 28. The input unit 21 receives input of various instructions such as a shooting instruction for shooting a moving image including the face of the transmitting user, a virtual makeup image generation instruction, a makeup information acquisition instruction, an image generation instruction, and a transmission / reception instruction. Note that the input unit 21 includes, for example, a keyboard and a pointing device such as a mouse.

  The output means 22 displays / outputs the contents input by the input means 21 and the contents of the makeup information executed based on the input contents. Note that the output means 22 includes a display, a speaker, and the like. Further, the output unit 22 may have a function such as a printer. In that case, the screen content generated by the screen generation unit 27 is printed on a print medium such as paper and presented to the user. You can also

  The storage means 23 is a variety of image information such as contrast information obtained by the screen information generating means 27, virtual makeup image, makeup information obtained by generating the original moving image information and virtual makeup image, and destination information (name, telephone). Various data such as numbers are stored.

  In addition, the storage unit 23 can also store the above-described various information acquired by the transmission / reception unit 28. Furthermore, the storage means 23 can read out various data stored as required.

  The imaging means 24 is composed of a CCD (Charge Coupled Device) camera or the like, and can capture a user's moving image used for a videophone or the like. Note that the imaging unit 24 is a built-in camera in the case of the transmission device 11-1, for example, and the imaging device 15-1 described above is connected in the case of the transmission device 11-2.

  The virtual makeup image generation means 25 performs a virtual makeup simulation on the image photographed by the imaging means 24 based on a makeup makeup method selected by the user, and displays a virtual makeup image on the image.

  Here, as described above, the virtual makeup image generation means 25 is set by the user such as the eyebrows, the eyes, the mouth, the cheek's face or all (including the local area) of the user's face, the hair, the clothes, and the like. The virtual makeup simulation is performed based on the stored contents or the contents set in advance to generate a virtual makeup image. In the present embodiment, as the makeup by the virtual makeup image generation means 25, for example, an image after makeup obtained by applying makeup such as foundation, eyebrows, shadow, lipstick, teak to the face image of the user is acquired.

  Here, the virtual makeup image generation means 25 specifically extracts the facial part contour information (vector data) from one frame (one frame) of the photographed face image, The extracted application area (vector data) for one frame is painted by controlling gradation, transparency, and the like based on the makeup method disclosed in, for example, Patent Document 1 (Japanese Patent Laid-Open No. 2007-216000) described above. . As a result of this painting, a composite image of the face with makeup for one frame is generated.

  Further, the virtual makeup image generation means 25 generates a makeup moving image by repeating the above-described method for each frame (15 to 30 or more per second). Therefore, by transmitting the data for each frame to the receiving side, it is displayed as a continuous moving image on the receiving side.

  The makeup information acquisition unit 26 acquires, as makeup information, information related to the virtual makeup simulation generated by the virtual makeup image generation unit 25 (information indicating which part (region) is applied with which color). . More specifically, the makeup information acquisition unit 26 obtains the vector data obtained in the generation of the virtual makeup image obtained by the virtual makeup image generation unit 25, and the vector coordinate value for associating the vector data with the face image. And makeup information consisting of parameter information such as numerical information representing the makeup technique and the like.

  The makeup information acquisition means 26 adds original moving image information (original image frame) and identification information for synchronizing the makeup information. For example, the above-described time information and image frame identification information (frame number) are added to the identification information. Thus, based on the makeup information acquired by the makeup information acquisition means 26, a charging process based on the data amount and data content can be performed.

  In the present embodiment, since only makeup information consisting only of text information obtained by the makeup information acquisition means 26 is acquired and transmitted, the amount of data can be reduced, transmission efficiency can be improved, and transmission in progress can be performed. Even if the contents of the data can be seen by a third party, the specific contents cannot be grasped, so that confidentiality can be improved.

  The screen generation unit 27 generates various screens such as data input necessary for execution by each functional configuration of the transmission device 11 in the present embodiment, output of execution results, and notification at the time of error, and the generated screens are generated. Output to output means 22 such as a display. Specifically, the screen generation unit 27 generates a screen for displaying at least one of the virtual makeup image, the original image frame, the makeup information, and the like obtained by the virtual makeup image generation unit 25 on the output unit 22. .

  The transmission / reception means 28 can input various necessary data in the present embodiment from an external device connected by the communication network 14 or the like, and can output various execution results executed by the transmission device 11 to the external device. Further, the transmission / reception means 28 outputs the frame of the original image and the makeup information to the receiving device 13 or the moving image management server 12 of the other party of the videophone based on a preset condition.

  The control unit 29 controls the entire components of the transmission device 11. Specifically, the control unit 29 installs an execution program for performing a virtual makeup simulation process, for example, to realize the videophone in the present embodiment, and executes the installed software, for example, Based on various instruction information from the input means 21 by the user, etc., each control such as a makeup image generation process, a makeup information acquisition process, a screen generation process, and a transmission / reception process is performed.

<Movie Management Server 12: Functional Configuration Example>
Next, a functional configuration example of the moving image management server 12 in the present embodiment will be described with reference to the drawings. FIG. 3 is a diagram illustrating an example of a functional configuration of the moving image management server according to the first embodiment. The moving image management server 12 shown in FIG. 3 includes an input unit 31, an output unit 32, a storage unit 33, a terminal management unit 34, a transmission setting unit 35, a virtual makeup information amount acquisition unit 36, and a charging unit 37. The virtual makeup image generation means 38, the transmission / reception means 39, and the control means 40 are included.

  The input unit 31 receives input of various instructions such as a terminal management instruction, a transmission setting instruction, a virtual makeup information amount acquisition instruction, a charging instruction, a virtual makeup image generation instruction, and a transmission / reception instruction from a user such as a server administrator. Note that the input unit 21 includes, for example, a keyboard and a pointing device such as a mouse.

  The output means 32 displays / outputs the contents input by the input means 31 and the contents of makeup information executed based on the input contents. Note that the output unit 32 includes a display, a speaker, and the like. Further, the output unit 32 may have a function of a printer or the like. In that case, the contents of the screen generated by the virtual makeup image generation unit 38 are printed on a print medium such as paper, etc. Can also be presented.

  The storage means 33 stores various data such as terminal management information, transmission setting information, virtual makeup information amount, charging contents, synthesized image contents, and images related to virtual makeup.

  The storage unit 33 also stores the above-described various information acquired by the transmission / reception unit 39. Further, the storage means 33 can read out various data stored as required.

  The terminal management unit 34 manages the communication contents of the transmission device 11 and the reception device 13 that have started the videophone. In other words, the terminal management unit 34 determines whether the original image from the transmission side is transmitted using the virtual makeup simulation tool, and to what level the application can be used. Judgment based on information. For example, if the level that can be used for a preset user is determined and the level is A, a composite image including the background is permitted, and if the level is B, the background cannot be combined and the face cannot be combined. Only the entire makeup can be simulated, and the processing content of the moving image can be changed according to the level.

  In addition, the terminal management unit 34 extracts user information of the transmission device 13 and determines whether makeup information can be transmitted to the transmission device 13 during a call based on the extracted user information. When a program for generating a virtual makeup image from the frame and makeup information has been installed and can be transmitted, the makeup information is transmitted. For example, when the program is not installed, the content instructed on the transmission device 12 side, When a virtual makeup image cannot be generated from the original image frame and makeup information on the receiving side based on the performance of the receiving device 13 that receives the virtual makeup image, the virtual makeup image reflecting the makeup information in the original image frame is displayed. And generating the generated virtual makeup image.

  Further, the transmission setting means 35 sets the image content when sending to the receiving side with respect to the original moving image information (original image frame) of the transmitting side user obtained from the transmitting device 11. In other words, the transmission setting unit 35 sets the setting information for each receiving device 13 obtained by the terminal management unit 34 (for example, the display of the pixel size and size, the transmission efficiency, the communication conditions, the processing performance of the receiving device 13, etc. ) To make image adjustments.

  Further, the transmission setting means 35, for example, when sending face images to a plurality of recipients simultaneously by streaming or the like, vector data transmitted from the moving image management server 12 and parameter information make-up information stored in advance as user information or Based on the transmission setting contents for each receiving device and the like, it can be changed corresponding to each receiving device, whereby different expressions can be given to each receiving device.

  The virtual makeup information amount acquisition unit 36 acquires makeup information transmitted from the transmission device 11, and acquires a data amount (information amount) corresponding to the virtual makeup simulation portion and / or contents of the data. For example, when the makeup identification information is makeup for the face and the content is lipstick, the basic data amount corresponding to the makeup identification information stored in the storage means 33 in advance is acquired.

  In addition, when there are a plurality of makeup ranges in one frame (for example, lips and eye shadows), the virtual makeup information amount acquisition unit 36 adds the data amounts obtained from each to calculate the total data amount. To do. The virtual makeup information amount acquisition means 36 acquires data contents such as the number of makeup ranges and the types of virtual makeup. Thereby, it is possible to easily and accurately charge according to the data amount and the data content.

  Further, the transmission setting unit 35 determines whether or not the usable time limit of the user of the transmission device 11 has expired, and if the time limit has expired, the virtual make-up in the present embodiment cannot be used, A notification for re-contracting can be generated and transmitted to the transmission device 11.

  The virtual makeup information amount acquisition unit 36 does not perform the data amount calculation process described above for each frame, for example, when the moving image transmission system in the present embodiment is executed, when the makeup location or makeup color is changed, in advance. The virtual makeup information amount described above is acquired at each set time interval or according to an instruction from an administrator (user) of the video management server 12 or the like.

  The charging unit 37 performs a charging process for the transmission device 11 based on the virtual makeup information amount obtained by the virtual makeup information amount acquisition unit 36. The billing unit 37 charges the billing timing when the virtual makeup information amount obtained by the virtual makeup information amount obtaining unit 36 is acquired, or by an instruction from an administrator or the like. In addition, the charging may be a quantitative charge setting, a pay-per-use charge setting, or a predetermined discount according to the number of uses or the use amount.

  In the present embodiment, as described above, charging is performed based on the virtual makeup information amount acquired by the moving image management server 12 from the transmission device 11 via the communication network 14, but the charging method in the present invention is limited to this. Not what will be done. For example, a billing mechanism similar to that used for actual cosmetics can be constructed.

  The above-described content will be described in detail. For example, when a user creates a virtual makeup image with virtual makeup using the transmission device 11, a foundation previously placed in a container such as a virtual bottle or a compact case. Each cosmetic such as lipstick, mascara, etc. is displayed on the screen, and a predetermined amount of each cosmetic used when performing the virtual makeup described above is consumed from each container. Also, when the cosmetic in the container is empty, the same cosmetic is purchased, or new cosmetic is purchased as necessary. Therefore, in the present invention, it is possible to charge at the time of purchase.

  In order to realize the above billing process, cosmetic data in the container is stored in a personal database for each user set in advance, and the amount of cosmetics decreases from the container each time it is used. On the other hand, if the purchase is made, the amount is managed to increase. The management information may be managed by both the transmission device 11 and the moving image management server 12 or may be managed by either one.

  Furthermore, with respect to the amount of makeup used in the above-described virtual makeup, for example, when using a touch panel or the like provided in the transmission device 11, an operation of scrubbing with the user's finger is performed on the cosmetic selected by the user. By performing with the touch panel, the area traced with the finger detected from the touch panel can be determined as an area where cosmetics are used, and the usage amount corresponding to the area can be acquired.

  Further, when the transmitting device 11 has a function of detecting finger force such as pressure sensing, information on the force (finger pressure) and softness and hardness preset for each cosmetic product are provided. Based on the component information or the like, the usage amount as a volume can be calculated together with the area information described above. In this case, for example, the volume is calculated by calculating “use amount = area × Z (depth by finger pressure) × coefficient (hardness of cosmetics, etc.)”. In this way, since the amount of use can be quantified from the calculated volume, the same amount of makeup as in the actual use of cosmetics in the virtual space can be displayed by displaying the amount of makeup for that amount from the container. Consumption and billing processing can be realized. In addition, the operation | movement of the finger | toe mentioned above is not limited to a finger | toe, For example, exclusive cosmetics, such as a brush, may be sufficient.

  A predetermined amount of cosmetics acquired from the cosmetic container designated by the user by the above-described processing or the like is reflected in the virtual makeup image corresponding to the acquired amount.

  When the transmission content obtained by the transmission setting unit 35 is “virtual makeup image”, the virtual makeup image generation unit 38 adds the original image frame and makeup information obtained from the transmission device 11 to the received stream signal or the like. A virtual makeup image is generated by synthesizing based on time information such as a frame identification signal and a time stamp.

  Note that the virtual makeup image generation means 38 generates a virtual makeup image by a method similar to the method in the transmission device 11 described above, for example, when performing a virtual makeup simulation on the original image frame based on the makeup information. can do. Further, in this case, the virtual makeup image generating means 38 is based on user information stored in the storage means 33 in advance, various setting information obtained by the terminal management means 34 and the transmission setting means 35, etc. ) Different virtual makeup images can be generated.

  Furthermore, as described above, the virtual makeup image generation unit 38 generates a virtual makeup image corresponding to the acquired amount using the predetermined amount of cosmetic acquired from the cosmetic container designated by the user. Can do. That is, in the present embodiment, the calculated usage amount can be used not only for billing but also as a physical amount (parameter) when performing virtual makeup simulation. Therefore, the virtual makeup image generation means 38 can generate a virtual makeup image by making the acquired physical quantity correspond to the darkness and thickness of the makeup color in the preset virtual makeup. In this case, instead of the cosmetic hardness, the cosmetic transparency or the like may be set as the coefficient used in the above-described usage amount calculation.

  Further, when the display screen of the transmission device 11 has a function of detecting finger force such as pressure sensing, for example, the user's finger or the like is used to trace the face image of himself or another person displayed on the screen. Thus, the virtual makeup image generation means 38 generates a virtual makeup image in which makeup corresponding to the finger pressure of the user and the physical quantity described above is applied to the traced area.

  The transmission / reception means 39 can input various necessary data in the present embodiment from an external device connected by the communication network 14 or the like, and can output various execution results executed by the transmission device 11 to the external device. Further, the transmission / reception means 28 receives the original image frame and the makeup information or the virtual makeup image frame generated by the virtual makeup image generation means 38 based on a preset condition as a stream signal from the other party of the videophone. Output to the device 13.

  The control means 40 controls the entire components of the moving image management server 12. Specifically, the control means 40 installs an execution program for performing virtual makeup processing, for example, in order to realize a videophone in the present embodiment, and executes the installed software, for example, a user Based on various instruction information from the input means 31 and the like, each control such as terminal management processing, transmission setting processing, virtual makeup information amount acquisition processing, charging processing, virtual makeup image generation processing, transmission / reception processing, and the like is performed.

<Receiving device 13: functional configuration example>
Next, a functional configuration example of the receiving device 13 in the present embodiment will be described with reference to the drawings. FIG. 4 is a diagram illustrating an example of a functional configuration of the receiving device according to the first embodiment. 4 includes an input unit 51, an output unit 52, a storage unit 53, an original image acquisition unit 54, a makeup information acquisition unit 55, a virtual makeup image generation unit 56, and a transmission / reception unit 57. , And control means 58.

  The input means 51 accepts input of various instructions such as an original image acquisition instruction, a virtual makeup information acquisition instruction, a virtual makeup image generation instruction, and a transmission / reception instruction from a receiving user or the like. Note that the input unit 51 includes, for example, a keyboard or a pointing device such as a mouse.

  The output means 52 displays / outputs the contents input by the input means 51 and the contents of the makeup / makeup image synthesized based on the input contents. Note that the output means 52 includes a display, a speaker, and the like. Furthermore, the output unit 52 may have a function of a printer or the like. In that case, the contents of the screen generated by the virtual makeup image generation unit 56 are printed on a print medium such as paper and the user or the like. Can also be presented.

  The storage unit 53 stores various data such as original image frames, makeup information, virtual makeup images, call history, address information for transmitting / receiving images and audio data to / from the transmission device 11 and the moving image management server 12. The storage unit 53 also stores the above-described various information acquired by the transmission / reception unit 57. Further, the storage means 53 can read out various data stored as required. Further, the storage means 53 has individual data for each user. When the virtual makeup is reflected in the original image frame based on the makeup information, for example, the numerical data included in the makeup information is included in the individual data. The makeup result can be displayed individually for each device by calculating using the numerical value or the like.

  The original image acquisition unit 54 acquires original image data that is original moving image information photographed on the transmission side from a stream signal received from the receiving device 11 or the moving image management server 12 via the communication network 14. At this time, since the acquired original image data is continuously acquired as a moving image stream, a moving image is formed by continuously displaying the original image frames included in the acquired stream signal. In the present embodiment, a still image may be displayed instead of a moving image.

  The makeup information acquisition means 55 acquires makeup information and the like transmitted from the preset transmission device 11 or the moving image management server 12.

  The virtual makeup image generation unit 56 generates a virtual makeup image based on the original image obtained from the original image acquisition unit 54 and the makeup information obtained from the makeup information acquisition unit 55. Specifically, when makeup information can be acquired, the virtual makeup image generation means 56 generates a frame based on identification information such as time information and frame identification information for the transmitted original image frame (raster data). A synchronization process is performed, and makeup information such as vector data and numerical information is reflected to generate a virtual makeup image and output it by the output means 52.

  Note that the virtual makeup image generation means 56 performs a predetermined calculation using numerical information included in the makeup information, or calculates using individual data such as numerical information stored in the storage means 53 in advance. It is also possible to generate a virtual makeup image unique to each receiving device by changing the contents of the individual makeup.

  When the virtual makeup image generation means 56 acquires makeup information other than numerical values such as makeup technique, for example, the virtual makeup image generation means 56 stores data corresponding to the makeup information in the storage means 53 or the storage means 33 of the moving image management server 12. A virtual makeup image is generated for the original image frame based on the makeup method, cosmetic information and the like extracted from the makeup information DB and included in the extracted data, and output by the output means 52.

  Here, when a virtual makeup image has already been generated in the moving image management server 12 and the information is received by the receiving device 13, the above-described original image acquisition unit 54, makeup information acquisition unit 55, and virtual makeup image The virtual makeup image obtained from the moving image management server 12 is output by the output unit 52 without performing the operation in the generation unit 56. Thereby, the processing burden of the receiving device 13 can be reduced.

  The transmission / reception means 57 can input various necessary data in the present embodiment from an external device connected by the communication network 14 or the like, and can output various execution results executed by the reception device 13 to the external device.

  The control unit 58 controls the entire components of the receiving device 13. Specifically, the control unit 58 installs an execution program for performing a virtual makeup process, for example, in order to realize a videophone in the present embodiment, and executes the installed software, for example, a user Based on various instruction information from the input means 51, etc., each control such as an original image acquisition process, a virtual makeup information acquisition process, a virtual makeup image generation process, and a transmission / reception process is performed.

  The transmission device 11, the moving image management server 12, and the reception device 13 in the present embodiment described above require raster data (an aggregate of image pixels) and vector data (closed loop coordinate data representing a region). Since it is composed of makeup information consisting of parameter information such as numerical information related to makeup techniques such as color and painting method, it is possible to efficiently transmit value-added moving images.

<Specific example of moving picture transmission in this embodiment>
Here, a specific example of moving image transmission in the present embodiment will be described with reference to the drawings. FIG. 5 is a diagram for explaining how image frames are transmitted in the present embodiment.

  In the present embodiment, with respect to moving image transmission, both the transmission and reception (playback) software are used, and the operation for the normal original image is used as it is, and the existing moving image transmission system 61 is used. The image transmission data is analyzed for transmission as the virtual makeup simulation moving image transmission system 62 for each of 15 to 30 frames / second (fps).

  The transmission apparatus 11 is preinstalled with transmission software and transmission software for acquiring makeup information (for example, vector information and parameter information) as mask data in the present embodiment. Is installed in advance with reception software for generating a virtual makeup image from the reception software and makeup information (mask) in the present embodiment.

  As shown in FIG. 5, the above-described makeup information is extracted as mask data from the virtual makeup image, and the corresponding frame is transmitted while being synchronized (synchronized) with each frame of the original image.

  In the example of the mask data shown in FIG. 5, mask data (make information) corresponding to the original image frame is transmitted corresponding to each frame, but the present invention is not limited to this. For example, one makeup information may be generated every several frames. When the movement is large according to the movement of the face of the previous and subsequent frames in the video, the mask frame interval is shortened, and when the movement is small Since there is no error even if the mask frame interval is increased, the frame interval can be adjusted according to the movement of the person being photographed. As for the motion of the photographed person, a general image analysis method or the like that extracts motion based on chromaticity information for each pixel of the previous and subsequent images can be used. In this case, a process of masking a plurality of original image frames using one mask frame can be performed.

  As described above, the make information (for example, vector information and parameter information) corresponding to the image frame is synchronized with the timing of transmitting a continuous image frame (raster information) as an original video stream signal from the transmission side. For example, when charging processing is performed, it is possible to efficiently calculate the amount of information at the time of transmission based on the amount of data and the content of the data from the makeup information, so depending on the usage details for each user Therefore, appropriate billing settings can be made with high accuracy.

<Generation of Make Information in the Present Embodiment>
Next, generation of makeup information in the present embodiment will be specifically described with reference to the drawings. FIG. 6 is a diagram for explaining a makeup information generation process according to the present embodiment. FIG. 6 shows a screen image representing processing performed by software such as virtual makeup simulation.

  FIG. 6A shows an example in which an image before virtual makeup (original image) actually taken by a camera or the like is displayed. FIG. 6B shows an example in which the contour information and tracking points obtained by the face recognition process are displayed superimposed on the pre-makeup image. FIG. 6C shows an example in which a post-makeup image (virtual makeup image) obtained by applying makeup to the user's face included in the pre-virtual makeup image based on the obtained information is displayed. .

  The transmission-side user confirms the image after the virtual makeup simulation shown in FIG. 6C, and when a favorite makeup image is obtained, based on the data, original moving image information (raster information) (FIG. 6D) and makeup information (FIG. 6E) are acquired.

  Here, the makeup information includes “contour information” and “make application area (for example, coordinates (x, y, z)) of face taken as described above, vector data such as color information, vector coordinate values, makeup information, and the like. Parameter information representing the technique is recorded together with “user information” and “frame information” added to the frame for synchronization with the original image.

  The makeup information is not limited to this in the present invention. For example, time information may be included instead of frame information, or other information may be included. As a result, the moving image management server 12 and the receiving device 13 perform the combining process with the above-described makeup information on the predetermined frame (raster information) of the original image based on the previously installed software based on these pieces of information. And a virtual makeup image can be generated. As a result, it is possible to realize a telephone call through video processing through billing processing or actual makeup information. Note that FIG. 6D described above can use the image of FIG. 6A as it is.

  FIG. 7 shows an example of billing processing in the moving image management server. In the example shown in FIG. 7, the video management server 12 includes the virtual makeup information DB 63 based on the original image shown in FIG. 6 (FIG. 6D) and the makeup information (FIG. 6E). A virtual makeup image can be generated by extracting various corresponding information for performing a virtual makeup simulation from existing data and performing a virtual makeup simulation on the original image. In addition, the charging process based on the makeup information described above is stored in, for example, a charging DB 64 provided in the moving image management server 12.

  FIG. 8 is a diagram showing an example of parameter information in the virtual makeup simulation. The parameter information shown in FIG. 8 is included in the virtual makeup information DB 63, and makeup parameters such as makeup method and color are set in the parameter file in association with the tracking points for the eyes, mouth, and cheeks, respectively. Thus, virtual makeup is performed. The parameter file described above is set for each makeup pattern (image), and the example of FIG. 8 represents an example of class elegance.

  In this embodiment, when virtual makeup is applied to an image before virtual makeup (original image), virtual makeup image generation means is obtained by obtaining contour information and application areas from the user's face included in the image before virtual makeup. In association with the contour information and the like, makeup techniques such as makeup method and color can be set as parameter information as shown in FIG. In the present embodiment, as an example of the makeup process, a virtual makeup process including a foundation process, an eyebrow process, a shadow process, a lipstick process, and a cheek process can be performed, and a combination with other processes is possible. Also good.

  In addition, as described above, the parameter information is not limited to the makeup for the face, and for example, a parameter file is generated for a hairstyle, hair color, hat, glasses, earrings, necklace, clothing, background, and the like. Thus, a variety of virtual makeup can be realized.

  As described above, according to the present embodiment, a moving image having added value can be efficiently transmitted.

<Operation of the video management server 12>
Here, a specific operation of the above-described moving image management server 12 will be described with reference to the drawings. FIG. 9 is a diagram for explaining a specific operation of the moving image management server. In FIG. 9A, a virtual makeup image is generated by the moving image management server 12 and transmitted to a plurality of receiving apparatuses (in the example of FIG. 9, receiving apparatuses 13-1 to 13-3 as an example) at a time. FIG. 9B shows an example in which the original moving image information and the makeup information are respectively transmitted. In the plurality of receiving apparatuses (in the example of FIG. 9, receiving apparatuses 13-1 to 13-3 as an example) An example of generating a virtual makeup image is shown. That is, FIG. 9A shows an example of generating and distributing moving images having different makeups to a plurality of distribution destinations on the moving image management server 12 side, and FIG. 9B shows moving image information and makeup information on the moving image management server 12 side. In this example, (vector + numerical information, etc.) is distributed in parallel, and a moving image is generated by performing different processes on each of the receiving devices 13-1 to 13-3.

  In the present embodiment, for example, when a face image is simultaneously sent to a plurality of recipients by streaming or the like, each vector data and parameter information transmitted from the video management server 12 is changed according to each recipient. It is also possible to make different expressions on the receiving device.

  For example, when performing a party where a plurality of people gather in a virtual space, in the present embodiment, it is possible to deliver a self-image with a different makeup according to the receiving party. In this case, when the existing technology such as streaming as shown in FIG. 9A is used, it is necessary to individually generate separate streaming videos (raster data). Since there is no load on the moving image management server 12, a large load is applied.

  Therefore, as shown in FIG. 9B, for example, raster data and make information such as vector data and numerical information are synchronized and processed on the receiving side using each individual data. The load on the moving image management server 12 that can be distributedly processed by the receiving devices 13-1 to 13-3 is reduced. In addition, the receiving apparatuses 13-1 to 13-3 perform a distributed process, so that the load is small.

  In addition, by using this embodiment, it is possible to express in accordance with the situation of the personal computer device on the receiving side. More specifically, for example, the color of a personal computer display is slightly different depending on the device. Therefore, by correcting the vector data and makeup information transmitted from the moving image management server 12 with the color information unified on the receiving device side, and reflecting the error in the color information of the received makeup information, the environment on the receiving side is corrected. Can realize accurate color development.

  In the example of FIG. 9, the number of the plurality of receiving apparatuses is three. However, the present invention is not limited to this, and may be two or four or more.

<Second Embodiment>
<Transmitter 11 ′: Functional Configuration Example>
As for the transmission device 11 in the first embodiment, the virtual makeup image generation unit 25 acquires makeup information obtained when a virtual makeup image is generated, and transmits the makeup information separately to the original moving image information to the reception device 13. However, the present invention is not limited to this. For example, information on only the virtual makeup portion is acquired as virtual makeup frame information from the virtual makeup image generation means 25, and the virtual makeup frame is converted into the original makeup frame. You may make it transmit separately from moving image information.

  Here, the above-described content is set as the second embodiment, and particularly, a different part from the first embodiment will be specifically described. FIG. 10 is a diagram illustrating an example of a functional configuration of the transmission device according to the second embodiment. The transmission apparatus 11 ′ shown in FIG. 10 is provided with a frame separation means 70 instead of the makeup information acquisition means 26, as compared with the transmission apparatus 11 in the first embodiment described above. 10, parts having the same functions as those of the transmission apparatus 11 in the first embodiment shown in FIG. 2 are denoted by the same reference numerals, and detailed description thereof is omitted here.

  The frame separation unit 70 separates the virtual makeup image into an original image frame constituting a moving image photographed by the imaging unit 24 and a virtual makeup frame obtained by extracting only the virtual makeup simulation portion.

  When performing the above-described separation, the frame separation unit 70 compares, for example, the original image and the virtual makeup image of the same frame, and the color difference is equal to or greater than a predetermined number based on chromaticity information for each pixel. Virtual makeup frame information can be extracted. Note that the frame separation means 70 gives the frame identification information such as the time information and the frame information described above so that the separated original image frame and the virtual make frame can be synchronized.

  By separating the original image frame and the virtual makeup frame by the frame separating means 70, it is possible to easily and accurately acquire only the data amount and the data content related to the virtual makeup. Based on this measurement result, the data amount and data Charge processing based on the contents can be performed.

  Also, the frame separation means 70 can reduce the amount of data by using only the image made up of only the virtual makeup simulation portion, improve the transmission efficiency, and make the contents of the virtual makeup frame being transmitted to a third party. Even if it is seen, it is impossible to grasp the specific contents, so that confidentiality can be improved.

  In addition to the functions of the first embodiment described above, the input unit 21 in the second embodiment inputs a frame separation instruction, and the control unit 29 performs various operations from the input unit 21 by a user or the like, for example. Each control such as separation processing is performed based on the instruction information and the like.

  In addition, the moving image management server 12 and the receiving device 13 in the second embodiment similarly calculate the charge and generate the virtual makeup image using the virtual makeup frame information instead of the makeup information in the first embodiment. .

  Here, FIG. 11 is a diagram for explaining an example of the separation processing in the second embodiment. In the embodiment of FIG. 11, FIGS. 11A to 11D are the same as FIGS. 6A to 6D described above, and a specific description thereof is omitted here.

  The transmission-side user confirms the image after the virtual makeup simulation shown in FIG. 11C, and when a favorite makeup image is obtained, the original original image frame (FIG. 11) is obtained based on the data. (D)) and a virtual makeup frame (FIG. 11 (e)). Then, identification information (for example, frame information) and time information to be synchronized on the receiving side are added to each frame, and transmitted to the moving image management server 12 or the receiving device 13 of the other party of the videophone or videoconference. . As a result, it is possible to realize a telephone call through billing processing or actually through a virtual make frame. Note that in FIG. 11D described above, the image of FIG. 11A can be used as it is.

<Example of hardware configuration in each device>
Here, in the transmission device 11 (11 ′), the video management server 12, and the reception device 13 in the video transmission system 10 described above, an execution program capable of causing a computer to execute each function is generated. By installing the execution program in a computer, a server, etc., the moving image transmission processing in the present invention can be realized. Here, a hardware configuration example of a computer capable of realizing the moving image transmission process in the present embodiment will be described with reference to the drawings.

  FIG. 12 is a diagram illustrating an example of a hardware configuration capable of realizing the moving image transmission process according to the present embodiment. 12 includes an input device 71, an output device 72, a drive device 73, an auxiliary storage device 74, a memory device 75, a CPU (Central Processing Unit) 76 that performs various controls, and a network connection device. 77, which are connected to each other by a system bus B.

  The input device 71 has a pointing device such as a keyboard and a mouse operated by a user or the like, and inputs various operation signals such as execution of a program from the user or the like. The input device 71 can also input various data such as an original image already captured by the external device or the like obtained from the external device connected to the network connection device 77 or the like via the communication network 14.

  The output device 72 has a display for displaying various windows, data, and the like necessary for operating the computer main body for performing processing in the present invention, and displays the program execution progress and results by the control program of the CPU 76. can do. Further, the output device 72 can print the above processing result or the like on a print medium such as paper and present it to the user or the like.

  Here, the execution program installed in the computer main body in the present invention is provided by, for example, a portable recording medium 78 such as a USB (Universal Serial Bus) memory, a CD-ROM, a DVD, or the like. The recording medium 78 on which the program is recorded can be set in the drive device 73, and the execution program included in the recording medium 78 is installed from the recording medium 78 to the auxiliary storage device 74 via the drive device 73.

  The auxiliary storage device 74 is a storage means such as a hard disk, and can store an execution program according to the present invention, a control program provided in a computer, etc., and perform input / output as necessary.

  The memory device 75 stores an execution program or the like read from the auxiliary storage device 74 by the CPU 76. The memory device 75 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

  The CPU 76 controls processing of the entire computer, such as various operations and input / output of data with each hardware component, based on a control program such as an OS (Operating System) and an execution program stored in the memory device 75. Thus, each processing can be realized. Various information necessary during the execution of the program can be acquired from the auxiliary storage device 74, and the execution result and the like can also be stored.

  The network connection device 77 obtains an execution program from another terminal connected to the communication network by connecting to a communication network or the like, or an execution result obtained by executing the program or an execution in the present invention The program itself can be provided to other terminals.

  With the hardware configuration as described above, the moving image transmission process in the present invention can be executed. Further, by installing the program, the moving image transmission process according to the present invention can be easily realized by a general-purpose personal computer or the like.

  Hereinafter, each processing procedure of transmission processing, moving image management processing, and reception processing in the moving image transmission processing of the present embodiment will be described with reference to flowcharts and the like. In the following processing procedure, the first embodiment will be mainly described. However, it will be described corresponding to the second embodiment as needed.

<Video transmission processing: transmission processing procedure>
A transmission processing procedure in this embodiment will be described. In the following transmission processing procedure, the video transmission system in a state where a videophone call has already been completed will be described.

  FIG. 13 is a flowchart illustrating an example of a transmission processing procedure in the present embodiment. In FIG. 13, the transmission device 11 first acquires a moving image including the user's face by using a photographing unit or the like (S01). Next, it is determined whether or not virtual makeup simulation is used (S02). When virtual makeup simulation is used (YES in S02), simulation is performed to generate a virtual makeup image (S03). The result is displayed on the screen (S04).

  Here, the displayed virtual makeup image is confirmed by the user, and it is determined whether or not an OK instruction has been input (S05). If not OK (NO in S05), the process returns to S03, and other parameters are used. Perform virtual makeup simulation.

  If the result is OK in S05 (YES in S05), the makeup information obtained by generating the virtual makeup image is acquired (S06), and the original image and the makeup information are separately sent to the receiving device based on the acquired information. Transmit (S07). In the process of S02, when the virtual makeup simulation is not used (NO in S02), the original image is transmitted as it is (S08).

  In addition, after the processing of S07 or S08 is completed, it is determined whether or not to end the videophone call based on the user's instruction (S09). I do.

  Further, in the process of S09, when the videophone is terminated (YES in S09), the moving image transmission process in the present embodiment is terminated.

  In the second embodiment, if it is OK in S05 (YES in S05), the original image frame and the virtual makeup frame are separated in the process of S06, and the original image is based on the information separated in the process of S07. And the virtual make frame are separately transmitted to the receiving device.

<Video transmission processing: Video management processing procedure>
Next, the moving image management processing procedure in this embodiment will be described. Note that the moving picture management processing procedure described below is for explaining the moving picture transmission system in a videophone call state.

  FIG. 14 is a flowchart illustrating an example of a moving image management processing procedure in the present embodiment. In FIG. 14, when the moving image management server 12 first acquires makeup information from the transmission device 11 (S11), it acquires the information amount of the data (S12). In the process of S12, as described above, the data amount, the data content, and the like for the makeup information are acquired. Next, a charging process is performed based on the content obtained in S12 (S13).

  Thereafter, an original image is acquired (S14), a virtual makeup image is generated from the original image based on the makeup information acquired in the process of S11 (S15), and the virtual makeup image is transmitted to the receiving device 13 (S16).

  In the process of S15, as described above, based on the frame identification information and time information previously added to the makeup information, the virtual image is generated by synchronizing the original image and the makeup information, Further, parameters such as numerical values included in the makeup information are calculated, or various parameters and image data are obtained by referring to the virtual makeup information DB 63 using the parameter information. Based on the obtained various data, the original image is obtained. A virtual makeup image is generated by performing virtual makeup.

  Here, it is determined whether or not to end the moving image management process (S17). If there is no instruction from the user and the process is not completed (NO in S17), the process returns to S11 to perform the subsequent process. In the process of S17, when an instruction to end the process is input (YES in S17), the moving image transmission process in the present embodiment is ended.

  In the process of S16, it is possible to output the original image and the makeup information for which only charging has been performed, without generating a virtual make-up image in the process of S15 based on the transmission setting contents set in advance. . In the second embodiment, a virtual make frame is acquired in the process of S11, and charging is performed according to the information amount of the acquired frame in the processes of S12 and S13. Further, in the process of S15, the original image frame and the virtual make frame are combined, and the virtual make image combined in the process of S16 is transmitted. At the time of composition, the composition is performed by superimposing the virtual makeup frame so that it is on the upper surface of the original image frame.

<Video transmission processing: reception processing procedure>
Next, the reception processing procedure in this embodiment will be described. In the following reception processing procedure, the moving picture transmission system in a videophone call is already described.

  FIG. 15 is a flowchart illustrating an example of a reception processing procedure in the present embodiment. In FIG. 15, the receiving device 13 acquires an image from the transmitting device 11 or the moving image transmission server 12 (S21). Next, it is determined whether or not the image is an original image (S22). If it is an original image (YES in S22), it is determined whether or not to use a virtual makeup simulation based on a preset condition (S23). If it is used (YES in S23), the transmission device 11 Alternatively, makeup information is acquired from the moving image transmission server 12 (S24), and a virtual makeup image is generated from the original image based on the acquired makeup information (S25). The virtual makeup image is generated by synchronizing the original image and the makeup information based on the frame identification information and the time information as described above.

  Here, after the process of S25 is completed, the virtual makeup image is displayed on the screen (S26). If it is not an original image in the process of S22 (NO in S22), since the virtual makeup image already generated in the moving image transmission management server 12 is transmitted, the received virtual makeup image is displayed on the screen as it is (S26).

  If the virtual makeup simulation is not used in the process of S23 (NO in S23), the original image is displayed on the screen as it is (S26). Note that in this case, since the transmission-side user's face is displayed, the image display can be rejected. In the present embodiment, the pre-virtual makeup image and the virtual after-makeup image can be switched and displayed.

  In addition, after the process of S26 is completed, it is determined from the user's instruction whether or not the videophone is to be terminated (S27). .

  If the videophone call is terminated in S27 (YES in S27), the reception process in this embodiment is terminated.

  In the second embodiment, a virtual make frame is acquired in the process of S24, the original image frame and the virtual make frame are combined in the process of S25, and the virtual make image combined in the process of S26 is displayed on the screen. To do. At the time of composition, the composition is performed by superimposing the virtual makeup frame so that it is on the upper surface of the original image frame.

  Here, in each of the embodiments described above, the contents of the virtual makeup on the face will be described, but in the present invention is not limited to this, for example, if the user is actually applying only the foundation and lipstick, Another makeup part (for example, mascara, eye shadow, cheek, etc.) may be virtually made and an image frame or the like of only that part may be transmitted.

  The data used in the virtual makeup simulation in each embodiment includes, for example, “a face image captured by a camera (frame)”, “a face part position, angle, etc. analyzed and extracted from the face image”. ”Geometry information”, “face part shape data extracted from the face image (vector data)”, “make-up application data (8-bit alpha channel data)”, and the like.

  Further, in the principle of virtual makeup simulation, among the above-mentioned data, the first face image (frame) is generated so as to correspond to the second geometry data, and the third and fourth positions are generated at that position. The second data is used to create makeup application data. That is, by combining and calculating data on the face image data (frame) (alpha channel), an image obtained by applying a makeup to the still image of the face image can be generated. In the case of moving image makeup, moving image makeup simulation can be performed by continuously displaying still images (frames) generated according to the above-described principle at 15 to 30 frames per second.

  As an image transmission method for realizing a virtual makeup simulation, for example, H.264 is used. It is possible to apply a compression method such as the H.264 method or a moving image transmission method using various devices such as CDMA-EVDO, and it is difficult to create a special moving image format or establish a new transmission method. Instead, it will disrupt the existing flow. Therefore, the existing moving image transmission method remains the same (as long as the frame method is used), and only the mask data is sent on the same line (Internet, CDMA, etc.) while being synchronized with the image data (frame), and is combined on the receiving side. This is realized.

  As a necessary measure for the existing moving picture transmission / reception system, it is necessary to add a mechanism for outputting bitmap data for each frame from the existing moving picture transmission system on the transmission side.

In addition, as shown in the embodiment of the present invention, a mechanism for inputting an additional frame number in a mechanism for outputting a frame number to an existing system, a mechanism for outputting bitmap data for each frame input from an existing video transmission system In addition, in the case of displaying a moving image, a preset interface is used, and the transmitting side analyzes and extracts the acquired bitmap data to create mask data. Furthermore, it sends while synchronizing every frame. On the receiving side, bitmap data of the received frame number is acquired, makeup is synthesized using mask data, and displayed on the screen.

  As described above, according to the present invention, a moving image having added value can be efficiently transmitted. Specifically, according to the present invention, digital cosmetics for performing virtual makeup can be generated not only for videophones on the Internet, but also for digital TV broadcasting, video on demand, etc. Realized by synchronizing (synchronizing) the mask frame and the original image layer without making any changes to solve the problems of current videophones, such as being unable to come out with real faces. Energize global communication.

  Further, the present invention is not limited to application to videophones and videoconferencing. For example, in the provision of TV programs by the development of digital TV, Internet TV, etc., the present invention is applied to one or more performers whose screens are displayed. By applying the invention and applying virtual makeup, low-cost program production can be realized.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications, within the scope of the gist of the present invention described in the claims, It can be changed.

DESCRIPTION OF SYMBOLS 10,62 Moving image transmission system 11, 11 'Transmitting device 12 Moving image management server 13 Receiving device 14 Communication network 15 Imaging device 21, 31, 51 Input means 22, 32, 52 Output means 23, 33, 53 Accumulating means 24 Imaging means 25 , 38, 56 Virtual makeup image generation means 26, 55 Make information acquisition means 27 Screen generation means 28, 39, 57 Transmission / reception means 29, 40, 58 Control means 34 Terminal management means 35 Transmission setting means 36 Virtual makeup information amount acquisition means 37 Billing means 54 Original image acquisition means 61 Existing video transmission system 63 Virtual makeup information DB
64 Billing DB
70 Frame separating means 71 Input device 72 Output device 73 Drive device 74 Auxiliary storage device 75 Memory device 76 CPU
77 Network connection device 78 Recording medium

Claims (15)

In a video transmission system for transmitting a moving image of a person between a transmission device and a reception device via a communication network,
The transmission device includes a transmission-side virtual makeup image generation unit that generates a virtual makeup image obtained by performing virtual makeup on a predetermined position of the face with respect to an original image including the face of the person photographed by an imaging unit. ,
Makeup information acquisition means for acquiring makeup information of a virtual makeup image obtained by the transmission-side virtual makeup image generation means;
Transmission means for separately transmitting the makeup information obtained by the makeup information acquisition means and the original original image frame to the reception device;
The receiving device acquires the makeup information and the original image frame from the transmitting device, and reflects the makeup information synchronized with the original image frame based on identification information or time information added to each. And a receiving-side virtual make-up image generating means for generating the virtual make-up image. A video transmission management device capable of transmitting and receiving data to and from the transmission device and the reception device via the communication network;
The moving image transmission management device acquires the makeup information from the transmitting device, acquires a data amount of the acquired makeup information, and a data amount obtained by the data amount acquisition unit. The moving image transmission system according to claim 1, further comprising charging means for charging for image transmission. The makeup information acquisition means
Acquiring from the virtual makeup image obtained by the transmitting-side virtual makeup image generating means, makeup information having vector data including facial part contour information and application area, and parameter information including numerical information related to a makeup technique; The moving image transmission system according to claim 1, wherein the moving image transmission system is a moving image transmission system. The video transmission management device is:
The moving image according to claim 2 or 3, wherein the content of the makeup information is changed based on user information or transmission setting content preset for each receiving device, and the changed makeup information is transmitted. Transmission system. The video transmission management device is:
User information of the receiving device is extracted, and based on the extracted user information, it is determined whether or not the makeup information can be transmitted to the receiving device during a call. The moving image transmission system according to any one of claims 2 to 4, wherein the moving image transmission system is transmitted. In a transmitting device that transmits a moving image to a predetermined receiving device via a communication network,
Virtual makeup image generation means for generating a virtual makeup image obtained by performing virtual makeup on a predetermined position of the face with respect to an original image including the face of the person photographed by the imaging means;
Makeup information acquisition means for acquiring makeup information of a virtual makeup image obtained by the virtual makeup image generation means;
A transmission apparatus comprising: transmission means for separately transmitting makeup information obtained by the makeup information acquisition means and an original original image frame to the reception apparatus. The makeup information acquisition means
Obtaining makeup information having vector data including contour information and application areas of facial parts and parameter information including numerical information related to a makeup technique from a virtual makeup image obtained by the virtual makeup image generation means, The transmission device according to claim 6. In a receiving apparatus that receives a moving image transmitted from a predetermined transmitting apparatus via a communication network,
Original image acquisition means for acquiring an original image frame photographed by the transmission device;
Makeup information obtained by generation of a virtual makeup image for the original image frame processed by the transmission device and makeup information acquisition means for acquiring;
The makeup information synchronized with the original image frame based on identification information or time information added to the original image frame obtained by the original image obtaining unit and the makeup information obtained by the makeup information obtaining unit, respectively. And a virtual make-up image generating means for generating the virtual make-up image by reflecting the image. In a video transmission management device that manages a moving image of a person transmitted and received between a transmission device and a reception device via a communication network,
Data amount of makeup information obtained when generating a virtual makeup image in which virtual makeup is applied to a predetermined position of the face with respect to an original image including the face of the person photographed by the imaging means in the transmission device A data amount acquisition means for acquiring
Charging means for charging the transmission of the moving image based on the data amount obtained by the data amount acquisition means;
A moving picture transmission management apparatus comprising: transmission setting means for changing contents of the makeup information in correspondence with the receiving apparatus and transmitting the changed makeup information.   User information of the receiving device is extracted, and based on the extracted user information, it is determined whether or not the makeup information can be transmitted to the receiving device during a call. 10. The moving image transmission management apparatus according to claim 9, further comprising terminal management means for transmitting. In a transmission program in a transmission device that transmits a moving image to a predetermined reception device via a communication network,
Computer
Virtual makeup image generation means for generating a virtual makeup image obtained by performing virtual makeup on a predetermined position of the face with respect to an original image including the face of the person photographed by the imaging means;
Makeup information acquisition means for acquiring makeup information of a virtual makeup image obtained by the virtual makeup image generation means; and
A transmission program for causing makeup information obtained by the makeup information acquisition means and the original original image frame to function as transmission means for separately transmitting to the reception device. The makeup information acquisition means
Obtaining makeup information having vector data including contour information and application areas of facial parts and parameter information including numerical information related to a makeup technique from a virtual makeup image obtained by the virtual makeup image generation means, The transmission program according to claim 11. In a receiving program in a receiving apparatus that receives a moving image transmitted from a predetermined transmitting apparatus via a communication network,
Computer
Original image acquisition means for acquiring an original image frame photographed by the transmission device;
Makeup information obtained by generation of a virtual makeup image for the original image frame processed by the transmission device, makeup information obtaining means for obtaining, and
The makeup information synchronized with the original image frame based on identification information or time information added to the original image frame obtained by the original image obtaining unit and the makeup information obtained by the makeup information obtaining unit, respectively. A receiving program for functioning as virtual make-up image generation means for generating the virtual make-up image while reflecting the above. In a video transmission management program in a video transmission management device that manages a moving image of a person transmitted and received between a transmission device and a reception device via a communication network,
Computer
Data amount of makeup information obtained when generating a virtual makeup image in which virtual makeup is applied to a predetermined position of the face with respect to an original image including the face of the person photographed by the imaging means in the transmission device Data amount acquisition means for acquiring,
Charging means for charging for transmission of the moving image based on the data amount obtained by the data amount acquiring means; and
A moving picture transmission management program for functioning as transmission setting means for changing the contents of the makeup information corresponding to the receiving device and transmitting the changed makeup information.   User information of the receiving device is extracted, and based on the extracted user information, it is determined whether or not the makeup information can be transmitted to the receiving device during a call. 15. The moving picture transmission management program according to claim 14, further comprising terminal management means for transmission.

Images