JP2005151231A - Video communication method, video communication apparatus, video creation program used for apparatus, and recording medium with program recorded thereon - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To surely protect the privacy of a speaker under communication by transmitting video signals of an optimal expression at all the times even to an unspecific speaker and regardless of an extent of fatigue without needing to prepare image data of the face at ordinary time. <P>SOLUTION: A video adjustment circuit 4 performs morphing of signals for the areas of specific parts indicating the expression of the speaker such as eyebrows, eyes, mouth and the like according to a preset expression adjustment ratio in video signals of the speaker which are imaged during a conversation, and video signals of a serious look that are stored when the communication starts, and video signals of the expression adjusted by the morphing, are transmitted to an apparatus of the communicating party. During a period of training that is performed when starting the communication, the image of the serious look of the speaker is fetched into the video adjustment circuit 4 and stored in a serious look information storage section 422. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a video communication method and a video communication apparatus used for, for example, a television telephone communication and a video conference communication, a video creation program used in the apparatus, and a recording medium for recording the program.

  In recent years, a television telephone communication system and a video conference communication system are frequently used as a communication system with a remote place. In this type of communication system, not only voice but also speakers can talk while looking at each other's facial expressions. Therefore, much more information can be transmitted than communication using only voice. This enables a more effective call or conference.

  However, in general, this type of communication system is configured to transmit and receive a speaker's face image captured by a camera as it is. For this reason, the speaker must be dressed before communication, and the other speaker will be informed of privacy information such as the speaker's emotions, fatigue, and tension from the transmitted face image. There was a problem called.

  Therefore, conventionally, image data representing the normal expression of the speaker is stored in advance. Then, the speaker's image data captured during communication is subjected to image processing to extract the speaker's fatigue level, and the image data of the captured facial expression is stored in advance in accordance with the extracted fatigue level. There has been proposed a technique in which it is replaced with stored image data representing a normal facial expression (see, for example, Patent Document 1).

JP-A-8-44861

  However, in the conventional proposed technique, it is necessary to register in advance image data representing the normal expression of the speaker. For this reason, for example, when a speaker is replaced by an unregistered speaker immediately before or during a call, or when an unregistered speaker joins in a jump, image data replacement processing is performed for these speakers. Thus, the face image data representing the expression during the call is still transmitted as it is. In addition, even for a registered speaker, for a speaker whose fatigue level is difficult to extract, face image data representing an expression during a call is transmitted as it is without performing image data replacement processing.

  The present invention has been made by paying attention to the above circumstances, and the purpose of the present invention is to prepare fatigue information for an unspecified speaker without preparing normal face image data in advance. Video communication method and video communication apparatus that can always transmit video signals with an optimal facial expression regardless of the degree, thereby enabling the privacy of the communicating speaker to be reliably protected, and to be used in this apparatus And a recording medium for recording the program.

  In order to achieve the above object, the present invention stores a video signal of a part including a face of a subject imaged during a non-conversation period during communication as a first video signal representing a true face of the subject. On the other hand, a first partial video signal in a specific part representing the facial expression of the subject is extracted from the second video signal of the part including the face of the subject imaged during the conversation period during communication. And a second partial video signal corresponding to the specific part in the stored first video signal in accordance with information representing a preset facial expression adjustment ratio to synthesize a third partial video Generate a signal. Then, the generated third partial video signal and the partial video signal other than the specific part in the second video signal are synthesized to generate a third video signal, and the generated third video signal is generated. The video signal is transmitted.

Therefore, according to the present invention, the video signal imaged during the non-conversation period during communication is stored as a true face image representing the normal expression of the subject, and the signal of the part representing the expression in the video signal imaged during the conversation And the stored true face image are synthesized in accordance with a preset facial expression adjustment ratio. This eliminates the need for the speaker to register in advance image data representing his / her normal facial expression. This allows the speaker to change to an unregistered speaker immediately before or during a call, for example. Even when registered speakers join in, it is possible to reliably protect the privacy of these unregistered speakers. That is, it becomes possible to protect the privacy of an unspecified number of speakers.
Also, facial expression adjustment processing is performed according to a preset facial expression adjustment ratio regardless of the speaker. For this reason, it is possible to adjust the facial expression to be the optimal facial expression for all speakers regardless of the facial expression fatigue level.

The present invention is also characterized by having the following functions.
The first function sets information representing the facial expression adjustment ratio of the subject in association with each of a plurality of assumed communication partners when setting the information representing the facial expression adjustment ratio, and the third partial video When generating a signal, select information representing the facial expression adjustment ratio corresponding to the communication partner from the multiple facial expression adjustment ratios set above based on the identification information of the communication partner used during communication Then, the first partial video signal and the second partial video signal are synthesized in accordance with the information representing the selected facial expression adjustment ratio.

  In this way, the expression adjustment ratio is selected according to the communication partner, and the expression during conversation and the expression of the true face stored during non-conversation are combined according to the selected table information adjustment ratio. For this reason, for example, when the communication partner is a family member or a close colleague who knows well, the facial expression of the speaker can be reduced by increasing the composition ratio of facial expressions during conversation and setting the composite ratio of true faces to zero or low. It can be transmitted to the communication partner as much as possible. On the other hand, if the communication partner is a business partner or the first meeting partner, the communicative partner is set as much as possible by setting the compositing ratio of facial expressions during conversation low and setting the compositing ratio of true faces high. You can avoid being transmitted to. In other words, it is possible to transmit a video signal having an optimal facial expression according to the communication partner.

A second function is a first video signal representing a true face obtained by imaging a part including a face of a subject during a training period for video communication when storing the first video signal. It is something to remember as.
In this way, a video signal representing the true face of the subject is stored in a preparation period from the start of communication to the actual start of conversation. For this reason, it is possible to reliably prepare a true-face video signal before the conversation is started.

The third function is to generate and store a first video signal representing a true face based on a plurality of video signals obtained during a non-conversation period when storing the first video signal.
In this way, a true face image is sequentially learned and corrected every non-conversation period. For this reason, it is possible to make a true face image as close to normal as possible.

In short, in the present invention, the video signal of the part including the face of the subject imaged during the non-conversation period during communication is stored as the first video signal representing the true face of the subject, and the subject imaged during the communication period during communication The first partial video signal in the specific part representing the expression of the subject is extracted from the second video signal of the part including the face, and the extracted first partial video signal and the stored first video are extracted. The second partial video signal corresponding to the specific part in the signal is synthesized in accordance with information representing a preset facial expression adjustment ratio, and the synthesized third partial video signal is synthesized with the background image and transmitted. Like that.
Therefore, according to the present invention, an image signal having an optimal facial expression is always transmitted to an unspecified speaker regardless of the degree of fatigue without preparing normal facial image data in advance. Video communication method and video communication apparatus capable of reliably protecting the privacy of a communicating speaker, a video creation program used in the apparatus, and a recording medium for recording the program Can be provided.

(First embodiment)
FIG. 1 is a principal block diagram showing a first embodiment of a video communication apparatus according to the present invention. In the figure, an analog video signal AS of a subject imaged by the camera 1 is converted into a digital video signal BS by an analog / digital conversion circuit (A / D) 2 and then input to a format conversion circuit 3. In the format conversion circuit 3, the input digital video signal BS is converted into a common intermediate format signal (CIF) CS used in a video encoding circuit of a television telephone device or a video conference device. The converted transmission CIF signal CS is input to the video adjustment circuit 4.

  The video adjustment circuit 4 performs a process for adjusting the facial expression of the subject with respect to the input transmission CIF signal CS. Then, the transmission CIF signal DS for which the adjustment processing has been completed is subjected to video encoding processing by the video encoding circuit 5 to become a transmission video encoded signal ES and then input to the demultiplexing circuit 6. The demultiplexing circuit 6 multiplexes the input transmission video encoded signal ES and the transmission audio encoded signal FS encoded by the audio encoding circuit (not shown), thereby performing transmission multiplexing of a predetermined transmission format. The transmission signal GS is generated, and the generated transmission multiplexed signal GS is transmitted to a transmission path (not shown).

  On the other hand, the multiplexed signal GS sent from the communication partner device via the transmission line is separated into the received video encoded signal HS and the received audio encoded signal FS by the demultiplexing circuit 6. Of these, the received video encoded signal HS is input to the video encoding circuit 6, and the received audio encoded signal FS is input to an audio encoding circuit (not shown). The video encoding circuit 6 converts the received video encoded signal HS into a received CIF signal IS by performing video decoding processing, and inputs the received CIF signal IS to the format inverse conversion circuit 7.

  The format reverse conversion circuit 7 reversely converts the received CIF signal IS into a digital video signal JS. The inversely converted digital video signal JS is converted into an analog video signal by a digital / analog conversion circuit (D / A) 8 and then supplied to a monitor 9 to be displayed as a received image. The received speech encoded signal FS is subjected to speech decoding by a speech encoding circuit, converted into an analog speech signal, and then output from a speaker (not shown).

By the way, the video adjustment circuit 4 is configured as follows. FIG. 2 is a block diagram showing the configuration of the video adjustment circuit 4. That is, the video adjustment circuit 4 includes a facial expression extraction unit 41, a facial expression synthesis unit 42, an adjustment ratio storage unit 43, and a video synthesis unit 44.
Among these, the adjustment ratio storage unit 43 stores information NS for designating a facial expression adjustment ratio in association with identification information of a communication partner, for example, a transmission ID or reception ID such as a subscriber telephone number or an IP address.

The facial expression extraction unit 41 is for extracting a partial video signal of a specific part representing the facial expression of the speaker's face as a subject from the transmission CIF signal CS. As shown in FIG. And a specific part cutout part 412.
Among these, the face area extraction unit 411 extracts a face area video signal PS corresponding to the face area and a background area video signal MS corresponding to the background area from the transmission CIF signal CS. As a method of cutting out the face area, for example, there is a skin color extraction method. In the skin color extraction method, a video signal defined by RGB values is expressed by an HSV display system, and a skin color region is extracted based on the hue (H) component of the HSV display system. In the HSV display system, since the hue (H) component and the lightness (V) component are independent, the skin color region can be separated and extracted without being affected by the brightness of the image.

Further, since there may be a skin color area in addition to the face area in the transmission CIF signal CS, a face area is further extracted from the extracted skin color area based on the size and aspect ratio of the face area. . In general, a subject image used in video communication has the largest area in the image, and the aspect ratio of the area in the vertical and horizontal directions is 0.5 to 2 times. Therefore, the face region can be extracted more accurately by specifying the skin color region that satisfies such conditions. Note that the aspect ratio of the vertical and horizontal directions can be arbitrarily set according to camera specifications, subject imaging conditions, and the like.
The face area extraction unit 411 corresponds the video signal PS corresponding to the face area extracted as described above to the specific part extraction unit 412 and the partial video signal of the area other than the extracted face area to the background area. The video signal MS to be input to the video composition unit 44.

  The specific part cutout unit 412 extracts a specific part representing the expression of the speaker, for example, an area representing the eyebrows, eyes, and mouth, from the input partial video signal PS of the face area. The extraction process of the area representing the specific part is performed based on the characteristics of the eyebrows, eyes, and mouth. For example, the eyebrows and eyes are areas darker than the luminance value of the skin color area, and the mouth is an area that is redr than the skin color area and has a shape that is elongated in the horizontal direction or close to an ellipse. Furthermore, the region of each part can be extracted by using the positional relationship between the eyebrows, the eyes and the mouth, for example, the vertical relationship between the eyebrows and the eyes, the symmetrical relationship between the eyebrows and the eyes, and the vertical relationship between the eyes and the mouth. .

  Further, the specific part cutout unit 412 identifies which part the extracted region corresponds to. For the identification, for example, a relaxation method is used. In the relaxation method, when there is ambiguity about which part the extracted region corresponds to, it is detected from the state in the vicinity of each part which part the extracted region is inconsistent. The ambiguity is reduced based on the detected value. Then, while repeating this process in parallel for the entire set of parts, the local contradiction is gradually eliminated, and finally, the unambiguous labeling is performed on the entire set of specific parts. Then, the partial video signal LS of each specific part extracted and specified as described above is input to the facial expression synthesis unit 42.

  The facial expression synthesizing unit 42, in accordance with the information NS for designating the facial expression adjustment ratio stored in the adjustment percentage storage unit 43, the specific part representing the facial expression of the speaker in conversation extracted by the specific part extraction unit 412 The morphing process of the partial video signal LS and the video signal QS of the true face of the corresponding speaker stored in advance is performed. Then, the partial video signal OS of the face area with the adjusted facial expression created by the morphing process is input to the video composition unit 44.

  FIG. 4 is a block diagram showing a configuration of the facial expression synthesis unit 42. The facial expression synthesis unit 42 includes a normalization unit 421, a true face information storage unit 422, and a morphing unit 423. The true face information storage unit 422 stores a true face video signal QS. As the true-face video signal QS, a normal face image of a speaker, for example, a video signal of a subject before the start of conversation captured during a training period executed at the initial stage of communication is used.

  The normalization unit 421 performs normalization processing of the partial video signal LS of the specific part extracted by the specific part cutting unit 412 as preprocessing of the morphing process. This normalization process is a process of matching the position and size of each part in the video signal LS of the specific part with the position and size of the specific part included in the true face video signal QS.

  The morphing unit 423 sets the coordinates of the shape of the specific part of the partial video signal RS and the true face video signal QS normalized by the normalization unit 421 to the facial expression adjustment ratio NS read from the adjustment ratio storage unit 43. In response, linear interpolation is performed, thereby calculating a new shape of a specific part. Similarly, the coordinates of the skin color region other than the specific part of the partial video signal RS and the true face video signal QS normalized by the normalization unit 421, for example, the coordinates of the shape of the cheek, forehead, nose, etc. Linear complementation is performed in accordance with the facial expression adjustment ratio NS read out from 43, thereby calculating a new shape of the skin color area other than the specific part. Through these processes, a partial video signal OS of the face area in which the opening of the mouth, the opening of the eyes, and the movement of the eyebrows are adjusted according to the adjustment ratio is created.

  The video synthesis unit 44 combines the partial video signal OS of the face area created by the morphing unit 423 and the video signal MS of the background area extracted by the facial expression detection unit 41 after aligning each other. Thus, a video signal DS having a facial expression adjusted, which is composed of a background area and a face area, is created.

Next, the operation of the video communication apparatus configured as described above will be described focusing on the operation of the video adjustment circuit 4.
Prior to communication, the speaker or the manager of the apparatus registers information NS for designating the facial expression adjustment ratio in the adjustment ratio storage unit 43. At this time, the value of the facial expression adjustment ratio is determined for each individual partner assumed as the communication destination, and the determined value is registered in association with the subscriber telephone number or IP address of the communication partner.

  In this state, it is assumed that videophone communication is started with a desired communication partner. At this time, in the video adjustment circuit 4, first, the video signal of the speaker before the start of conversation captured in the training period for performing video communication is taken in from the format conversion circuit 3. The captured video signal (CIF signal CS) is stored in the true face information storage unit 422 as a video signal representing the true face of the speaker. In other words, a true face image of a speaker is automatically obtained and stored during a training period performed at the start of communication.

  Subsequently, when the conversation is started, the video signal captured during the conversation period is input from the format conversion circuit 3 to the video adjustment circuit 4. The input video signal (CIF signal CS) is first separated into a partial video signal PS in the face area and a partial video signal MS in the other background area by the face area extraction unit 411, and among these, the partial video in the face area The signal PS is input to the specific part cutting unit 412.

  The specific part extraction unit 412 extracts a video signal LS of a specific part such as an eyebrow, an eye, and a mouth representing the expression of the speaker from the input partial video signal PS of the face region. Extraction of the video signal of the specific part is performed based on the color features of the eyebrows, eyes, and mouth and the mutual positional relationship as described above. When there is ambiguity in the extracted eyebrow, eye, and mouth video signals, the eyebrows, eyes, and mouth are identified by the relaxation method.

  When the video signal LS of the specific part is cut out, the video adjustment circuit 4 in the facial expression synthesis unit 42 then extracts the video signal LS of the specific part extracted in the specific part cut-out part 412 and the true face information storage unit first. Morphing processing with the true-face video signal QS stored in 422 is performed as follows.

  That is, first, the normalization unit 421 performs a normalization process for matching the position and size of the specific part with the position and size of the specific part included in the true face video signal QS. Subsequently, in the morphing unit 423, the coordinates of the shape of the partial video signal RS of the normalized specific part and the coordinates of the shape of the specific part of the true face video signal QS are read from the adjustment ratio storage unit 43. Processing for linear interpolation is performed in accordance with the facial expression adjustment ratio NS. Similarly, the coordinates of the shape of other skin color areas such as cheeks, foreheads and noses, and the skin color other than the specific parts in the true face video signal QS are normalized by the normalization unit 421. A process of linearly complementing the coordinates of the shape of the region in accordance with the facial expression adjustment ratio NS read from the adjustment ratio storage unit 43 is performed. Through the above linear interpolation processing, a new shape of the specific part and a new shape of the skin color region other than the specific part are respectively obtained, and accordingly, the opening of the mouth, the opening of the eyes, and the movement of the eyebrows are performed according to the expression adjustment ratio NS. A partial video signal OS of the adjusted face area is created.

  For example, suppose that the communication partner is now a business partner. Then, the corresponding facial expression adjustment ratio is read from the adjustment ratio storage unit 43 based on the telephone number or IP address of the business partner. Then, according to the value of the read facial expression adjustment ratio, the shape of the specific part and the shape of the specific part in the true face video signal QS are morphed. At this time, if the value of the facial expression adjustment ratio set in association with the business partner is, for example, “0” or a value close thereto, the video signal QS of the true face is synthesized as a main component, and thus the facial expression remains almost unchanged. A partial video signal of the face area is created.

  In contrast, the communication partner is a family. In this case, the corresponding facial expression adjustment ratio is read from the adjustment ratio storage unit 43 based on the family telephone number or IP address of the family. Then, according to the read facial expression adjustment ratio value, the shape of the specific part and the shape of the specific part in the true face video signal QS are morphed. At this time, if the value of the facial expression adjustment ratio set in association with the family is, for example, “1” or a value close thereto, the video signal captured during the conversation is synthesized as a main component, so that the facial expression during the conversation remains as it is. A partial video signal of the expressed face area is created.

  Similarly, if the communication partner is a boss or the like in the workplace, morphing is performed according to the value of the display adjustment ratio stored in association with the boss. For example, if the value of the facial expression adjustment ratio set in association with the boss of the workplace is “0.5”, for example, the video signal captured during the conversation and the video signal of the true face are combined at a ratio of half. Thus, a partial video signal of the face area in which the expression is expressed in an intermediate state is created.

  Next, the video adjustment circuit 4 performs a process of combining the partial video signal OS of the face area created by the morphing process and the partial video signal MS of the background area previously separated by the facial expression detection unit 41, As a result, a video signal DS whose facial expression is adjusted is obtained. The video signal (transmission CIF signal) DS whose facial expression has been adjusted is subjected to video coding processing by the video coding circuit 5, and then multiplexed with the audio coding signal FS by the demultiplexing circuit 6, and then the communication partner apparatus. Toward the transmission path (not shown).

As described above, in the first embodiment, in the video adjustment circuit 4, the face area image of the speaker captured during the conversation and the face area of the true face image captured and stored during the training period at the beginning of communication. Are morphed according to information NS designating a preset facial expression adjustment ratio, and a video signal whose facial expression is adjusted by this morphing process is transmitted to the communication partner apparatus. For this reason, the trouble that the other speaker is made aware of privacy-related information such as the speaker's emotions, fatigue level, and tension during conversation is reduced.
Moreover, since the speaker's true face image is automatically acquired and stored during the training period at the start of communication, it is not necessary to capture and store the speaker's true face image prior to communication. This makes it possible to handle a large number of unspecified speakers.

Further, in this embodiment, a facial expression adjustment ratio is set for each communication partner, and the set facial expression adjustment ratio is stored in the adjustment ratio storage unit 43 in association with the subscriber telephone number or IP address of the communication partner. Then, the facial expression adjustment ratio corresponding to the communication partner is read out from the adjustment ratio storage unit 43 based on the subscriber telephone number and IP address of the communication partner and used for the morphing process.
Therefore, for example, when the communication partner is an energetic partner such as a family, the facial expression of the speaker remains as much as possible by increasing the synthetic ratio of facial expressions during conversation and setting the synthetic ratio of true faces to zero or low. It can be communicated to the communication partner. On the other hand, if the communication partner is a business partner or the first meeting partner, the communicative partner is set as much as possible by setting the compositing ratio of facial expressions during conversation low and setting the compositing ratio of true faces high. You can avoid being transmitted to. That is, it is possible to transmit a video signal having an optimal facial expression according to the communication partner.

(Second Embodiment)
In the second embodiment of the present invention, a video is adjusted on a face image of a subject by executing a program in a central processing unit (CPU).

FIG. 5 is a block diagram showing a second embodiment of the video communication apparatus according to the present invention. In the figure, the same parts as those in FIG.
The video communication apparatus according to this embodiment includes a CPU 10 and a memory 11, and these constitute a video adjustment circuit. The memory 11 includes a non-volatile memory unit composed of a hard disk or a flash memory, and a volatile memory unit composed of a RAM, and a video adjustment processing program is stored in the non-volatile memory unit. The volatile memory unit stores information NS for designating a facial expression adjustment ratio and a true face video signal (true face image) as information necessary for the video adjustment process.
The CPU 10 follows the video adjustment processing program stored in the memory 11 and uses the information NS for designating the expression adjustment ratio and the video signal of the true face to generate a video for the face image of the subject captured during the conversation period. Execute the adjustment process.

Next, the operation of the apparatus configured as described above will be described according to the control procedure of the CPU 10. FIG. 6 is a flowchart showing the control procedure and control contents.
The CPU 10 monitors the input of the facial expression adjustment ratio in step 6a, and monitors the input of the video signal in step 6b. In this state, when the speaker or the administrator of the device performs an input operation of the information NS for designating the facial expression adjustment ratio on an input device (not shown), the process proceeds from step 6a to step 6c, and the inputted facial expression adjustment ratio is set. Information NS to be specified is stored in the memory 11. The information NS for designating the expression adjustment ratio is set for each communication partner, and the set information NS is stored in the memory 11 in association with identification information of the communication partner, for example, a subscriber telephone number or an IP address.

  When communication is started in this state and a video signal (CIF signal) CS is input from the format conversion circuit 3, the CPU 10 proceeds from step 6b to step 6d to determine whether or not a conversation is being performed. judge. Then, if the conversation is not started, for example, during a training period for video communication, the process proceeds to step 6e and the input video signal CS is stored in the memory 11 as a true face image QS. Thus, the true face image before the start of the conversation of the speaker is automatically stored.

  On the other hand, when a video signal (CIF signal) CS is input during a conversation, the CPU 10 proceeds from step 6d to step 6f, where first the face area and the background area are separately extracted from the input video signal CS. To do. Then, the extracted video signals of the face area and the background area are stored in the memory 11. Note that, as described in the first embodiment, the skin color extraction method and the method using the size and vertical / horizontal aspect ratio of the face region are used for extracting the face region.

  Next, in step 6g, the CPU 10 extracts an image of a specific part representing the speaker's table information, for example, an area representing the eyebrows, eyes, and mouth, from the extracted video signal of the face area. As described in the first embodiment, the extraction process of the region representing the specific part is also based on the shape and color characteristics of the eyebrows, eyes, and mouth, and the positional relationship between the eyebrows, eyes, and mouth. Done. Further, when the ambiguity exists in the extraction result, the ambiguity is reduced by using the relaxation method as in the first embodiment.

  When the region of the specific part is extracted, the CPU 10 subsequently, in step 6h, determines the position and size of the extracted specific part and the position of the specific part included in the true face image stored before the start of the conversation. Perform normalization to match the size. Then, in step 6i, the coordinates of the shape of the partial video signal of the normalized specific part and the coordinates of the shape of the partial video signal of the specific part in the true face image are read out from the memory 11. Performs linear interpolation according to the ratio. Similarly, the coordinates of the shape of other skin color areas other than the normalized specific part, such as cheeks, forehead, and nose, and the coordinates of the shape of the skin color area other than the specific part of the true face image are also obtained. Then, linear interpolation processing is performed in accordance with the expression adjustment ratio read from the memory 11. Thus, according to a preset facial expression adjustment ratio, a morphing process is performed between the face image captured during the conversation and the true face image captured and stored before the conversation.

When the morphing process ends, in step 6j, the CPU 10 performs a process of synthesizing the video signal of the face area created by the morphing process and the video signal of the background area previously separated in step 6f, As a result, a video signal DS whose facial expression is adjusted is obtained. Then, the video signal (transmission CIF signal) DS whose facial expression is adjusted is output to the video encoding circuit 5 in step 6k.
Thereafter, each time a video signal is input during a conversation, a series of processes for adjusting the facial expression of the face image is performed in steps 6b to 6k.

  As described above, according to the second embodiment, as in the first embodiment described above, the image of the face area of the speaker captured during the conversation and the training period during the initial communication are captured. The morphing process is performed on the image of the face area of the true face image stored in accordance with information NS designating a preset facial expression adjustment ratio, and a video signal whose facial expression is adjusted by the morphing process is transmitted to the communication partner apparatus. ing. For this reason, the trouble that the other speaker is made aware of privacy-related information such as the speaker's emotions, fatigue level, and tension during conversation is reduced. Moreover, since the speaker's true face image is automatically acquired and stored during the training period at the start of communication, it is not necessary to capture and store the speaker's true face image prior to communication. This makes it possible to handle a large number of unspecified speakers.

  Further, a facial expression adjustment ratio is set for each communication partner, the set facial expression adjustment ratio is stored in the memory 11 in association with the communication partner's subscriber telephone number or IP address, and the communication partner's subscriber telephone number or IP address is stored. Originally, the facial expression adjustment ratio corresponding to the communication partner is read from the memory 11 and is used for the morphing process. Therefore, it is possible to transmit a video signal having an optimal facial expression in accordance with the attributes of the communication partner, for example, an energetic partner such as a family member, a business partner or a first meeting partner.

(Other embodiments)
In each of the above-described embodiments, the video signal of the subject before the start of conversation captured during the training period executed at the initial stage of communication is stored as a true face video signal. However, the present invention is not limited thereto, and the video signal of the subject obtained in the first non-conversation period after the start of communication may be stored. Further, every time a video signal of a subject is obtained in a subsequent non-conversation period, the stored true video signal may be sequentially learned and corrected based on the obtained video signal. In this way, a true face image of the subject that is closer to the normal state can be obtained. The non-conversation period can be determined by monitoring the presence / absence of a transmission voice signal and a reception voice signal.

  In each of the above embodiments, the value of the facial expression adjustment ratio is determined for each partner assumed as a communication destination, and the determined value is stored in the adjustment ratio storage unit 43 in association with the subscriber's subscriber telephone number or IP address. I did it. However, the present invention is not limited to this, and possible communication partners are grouped, a facial expression adjustment ratio value is determined for each group, and the determined value is stored in the adjustment ratio storage unit 43 in association with the identification number of the group. You may do it. In this way, for example, the communication partner can be divided into a group of family members or relatives, a group of colleagues in the workplace, a group of business partners, etc., and a facial expression adjustment ratio can be set for each group. It is possible to simplify the setting management of the facial expression adjustment ratio compared to the case where the facial expression adjustment ratio is set in

In addition, the procedure and content of the video adjustment process, the facial expression adjustment ratio setting method, the true face image generation and storage method, and the like can of course be modified in various ways without departing from the scope of the present invention.
In short, the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, you may combine suitably the component covering different embodiment.

1 is a principal block diagram showing a first embodiment of a video communication apparatus according to the present invention. The block diagram which shows the structure of the image | video adjustment circuit in the video communication apparatus shown in FIG. The block diagram which shows the structure of the expression detection part in the image | video adjustment circuit shown in FIG. FIG. 3 is a block diagram showing a configuration of a facial expression synthesis unit in the video adjustment circuit shown in FIG. 2. The principal part block diagram which shows 2nd Embodiment of the video communication apparatus concerning this invention. The flowchart which shows the procedure and content of the image | video creation control performed in CPU of the video communication apparatus shown in FIG.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Camera, 2 ... Analog-digital conversion circuit (A / D), 3 ... Format conversion circuit, 4 ... Video adjustment circuit, 5 ... Video encoding circuit, 6 ... Demultiplexing circuit, 7 ... Format reverse conversion circuit, 8 ... Digital / analog conversion circuit (D / A), 9 ... monitor, 10 ... central processing unit (CPU), 11 ... memory, 41 ... facial expression extraction unit, 42 ... facial expression synthesis unit, 43 ... adjustment ratio storage unit, 44 ... Image synthesizing unit, 411... Face area extracting unit, 412... Specific part extracting unit, 421... Normalizing unit, 422.

Claims (16)

A process of setting information representing the expression adjustment ratio of the subject;
Imaging a part including the face of the subject during a non-conversational period during communication;
Storing the imaged video signal as a first video signal representing a true face;
Capturing a portion including the face of the subject during a conversation period during communication to obtain a second video signal;
Extracting a first partial video signal at a specific part representing the expression of the subject from the second video signal obtained during the conversation period;
The extracted first partial video signal and the second partial video signal corresponding to the specific part in the stored first video signal are combined according to the information representing the set facial expression adjustment ratio. And generating a third partial video signal,
Combining the generated third partial video signal and a partial video signal other than the specific part in the second video signal to generate a third video signal;
And a step of transmitting the generated third video signal. The process of setting information representing the facial expression adjustment ratio sets information representing the facial expression adjustment ratio of the subject in association with each of a plurality of assumed communication partners,
In the process of generating the third partial video signal, a facial expression corresponding to the communication partner is selected from the plurality of facial expression adjustment ratios set based on identification information of the communication partner used in communication. 2. The video communication according to claim 1, wherein information representing an adjustment ratio is selected, and the first partial video signal and the second partial video signal are synthesized according to the information representing the selected facial expression adjustment ratio. Method.   The process of storing the first video signal includes storing a video signal obtained by imaging a part including the face of the subject during a training period for video communication as a first video signal representing a true face. The video communication method according to claim 1, wherein:   2. The step of storing the first video signal includes generating and storing a first video signal representing a true face based on a plurality of video signals obtained during a non-conversation period. Video communication method. Memory means for storing information indicating the facial expression adjustment ratio of the subject;
Means for storing a video signal obtained by imaging a portion including the face of the subject during a non-conversation period during communication as a first video signal representing a true face;
Means for extracting a first partial video signal at a specific part representing the expression of the subject from a second video signal obtained by imaging a part including the face of the subject during a conversation period during communication;
The extracted first partial video signal and the second partial video signal corresponding to the specific part in the stored first video signal represent facial expression adjustment ratios stored in the memory means. Means for synthesizing according to the information to generate a third partial video signal;
Means for synthesizing the generated third partial video signal and a partial video signal other than the specific part in the second video signal to generate a third video signal;
And a means for transmitting the generated third video signal. The memory means stores information representing a facial expression adjustment ratio of a subject in association with each of a plurality of possible communication partners,
The means for generating the third partial video signal selectively reads out information representing a facial expression adjustment ratio corresponding to the communication partner from the memory means based on the identification information of the communication partner used in communication. 6. The video communication apparatus according to claim 5, wherein the first partial video signal and the second partial video signal are synthesized in accordance with the information indicating the read facial expression adjustment ratio.   The means for storing the first video signal stores a video signal obtained by imaging a part including the face of the subject during a training period for video communication as a first video signal representing a true face. The video communication apparatus according to claim 5.   6. The video according to claim 5, wherein the means for storing the first video signal generates and stores a first video signal representing a true face based on a plurality of video signals obtained during a non-conversation period. Communication device. A video creation program used in a video communication device for creating a transmission video signal by a computer based on a video signal obtained by imaging a subject and transmitting the created transmission video signal,
A process of storing a video signal obtained by imaging a portion including the face of the subject during a non-conversation period during communication as a first video signal representing a true face;
A process of extracting a first partial video signal at a specific part representing the expression of the subject from a second video signal obtained by imaging a part including the face of the subject during a conversation period during communication;
The extracted first partial video signal and the second partial video signal corresponding to the specific part in the stored first video signal are combined according to information representing a preset facial expression adjustment ratio. A process of generating a third partial video signal;
The generated third partial video signal and the partial video signal other than the specific part in the second video signal are combined to generate a third video signal, and the generated third video A video creation program that causes the computer to execute a process of outputting a signal as the transmission video signal.   The process of generating the third partial video signal is based on identification information of a communication partner used in communication, and a facial expression corresponding to the communication partner from among a plurality of preset facial expression adjustment ratios. 10. The video creation according to claim 9, wherein information representing an adjustment ratio is selected, and the first partial video signal and the second partial video signal are synthesized according to the information representing the selected facial expression adjustment ratio. program.   The process of storing the first video signal includes storing a video signal obtained by imaging a part including the face of the subject during a training period for video communication as a first video signal representing a true face. The video creation program according to claim 9, wherein:   10. The process of storing the first video signal generates and stores a first video signal representing a true face based on a plurality of video signals obtained during a non-conversation period. Video creation program. A recording medium on which a transmission video signal is created by a computer based on a video signal obtained by imaging a subject and a video creation program used in a video communication apparatus for transmitting the created transmission video signal is recorded. ,
A process of storing a video signal obtained by imaging a portion including the face of the subject during a non-conversation period during communication as a first video signal representing a true face;
A process of extracting a first partial video signal at a specific part representing the expression of the subject from a second video signal obtained by imaging a part including the face of the subject during a conversation period during communication;
The extracted first partial video signal and the second partial video signal corresponding to the specific part in the stored first video signal are combined according to information representing a preset facial expression adjustment ratio. A process of generating a third partial video signal;
The generated third partial video signal and the partial video signal other than the specific part in the second video signal are combined to generate a third video signal, and the generated third video The recording medium which recorded the video production program which makes the said computer perform the process which outputs a signal as the said transmission video signal.   The process of generating the third partial video signal is based on identification information of a communication partner used in communication, and a facial expression corresponding to the communication partner from among a plurality of preset facial expression adjustment ratios. 14. The video creation according to claim 13, wherein information representing an adjustment ratio is selected, and the first partial video signal and the second partial video signal are synthesized according to the information representing the selected facial expression adjustment ratio. A recording medium that records the program.   The process of storing the first video signal includes storing a video signal obtained by imaging a part including the face of the subject during a training period for video communication as a first video signal representing a true face. 14. A recording medium on which the video creation program according to claim 13 is recorded.   14. The processing for storing the first video signal generates and stores a first video signal representing a true face based on a plurality of video signals obtained during a non-conversation period. A recording medium that records a video creation program.

Images