JP6539624B2 - Gaze-matched face image synthesizing method, video conference system, and program - Google Patents

Description

  The present invention relates to a gaze-matched face image synthesis method, a video conference system, and a program.

  Conventionally, with regard to the gaze-matched face image synthesis method, it is premised that the camera position is fixed with respect to the display on the user terminal as an eye area conversion method for gaze matching between the communicators in the video conference. There is known a method of transforming the eye area (Non-patent Document 1).

Takuya Inoue, Tomokazu Takahashi, Takahiro Hirayama, Exit Oosuke, Ide Ichiro, Hiroshi Murase, Hiroyuki Kurozumi, Kunio Kanno. No. PRMU 2014-60, pp. 33-38, October 2014.

  However, the method of Non-Patent Document 1 described above has a drawback that the line of sight can not be determined with sufficient accuracy when the camera viewpoint position changes with respect to the display on the user terminal.

  The present invention has been made in view of the above circumstances, and even when the camera viewpoint position with respect to the display display changes, it is possible to accurately determine the line of sight coincidence and generate an image in which the line of sight is coincident. It is an object of the present invention to provide a gaze-matched face image synthesizing method, a video conference system, and a program.

  In order to achieve the above object, the gaze-matched face image synthesis method of the first invention is a gaze-matched face image synthesis method in a video conference system, and a feature extraction process of extracting features from a camera input image representing a communicator An eye open / close determination step of determining the open / close of the eye using the feature of the camera input image; The camera input image is based on whether a gaze at a dialogue partner has been learned in advance from a face image whose viewpoint position is known, and a identifier for identifying the camera viewpoint position, and a feature of the camera input image. Whether or not the communicator who is gazing at the conversation partner, the camera position dialogue partner gaze determination process for determining the camera viewpoint position, and the gaze at the conversation partner In the case that, in the interlocutor of the eye region in which the camera input image represents, including a synthesizing step of synthesizing the image when watching the camera.

  According to a second aspect of the present invention, a video conference system includes a feature extraction unit that extracts a feature from a camera input image representing a communicator, an eye open / close determination unit that determines the opening and closing of the eye using the feature of the camera input image, In the case where it is determined that the camera is open, whether or not it gazes at the conversation partner, whether or not it gazes at the conversation partner, whose camera viewpoint position with respect to the display display is learned in advance from known face images Whether the communicator represented by the camera input image is gazing at the other party of the conversation based on the identifier for identifying the viewpoint position and the feature of the camera input image, and the camera position for determining the camera viewpoint position When it is determined that the dialogue partner gaze determination unit and the dialogue partner are gazed, an image obtained when the camera gaze is gazed in the eye area of the communicator represented by the camera input image Is configured to include a combining unit that, the.

  A gaze-matched face image synthesis method according to a third invention is a gaze-matched face image synthesis method in a video conference system, comprising: a feature extraction step of extracting a feature from a camera input image representing a communicator; and a feature of the camera input image In the eye open / close determination process of determining the open / close of the eye using the above, and in the case where it is determined that the eye is open, the gaze point on the display on the interlocutor side A classifier that gazes at the position for each camera viewpoint position and each position pair on the display on the interlocutor side, which has been learned in advance, and identifies whether or not the camera viewpoint position, and the camera input image The gaze point determination process on the display of the interlocutor side on the basis of the features of A gaze partner determination step of determining whether a communicator represented by the camera input image gazes at the conversation partner based on a fixation point on the display of the speaker side, and it is determined that the conversation partner is gazed at And a combining step of combining an image obtained when the user gazes at a position corresponding to the conversation partner in the eye region of the dialog represented by the camera input image.

  A video conference system according to a fourth aspect of the invention includes a feature extraction unit that extracts a feature from a camera input image representing a communicator, an eye open / close determination unit that determines the opening and closing of the eye using the feature of the camera input image, When it is determined that the camera is open, the gaze point on the display on the interlocutor's display and the camera viewpoint position relative to the display are previously learned from the face image known, on each display on the interloper's display A gaze point on a display on the interlocutor side based on a discriminator that gazes at the position for each position pair and identifies whether or not the camera viewpoint position, and a feature of the camera input image, And a camera position display on the camera position display for determining the camera viewpoint position, and a pair represented by the camera input image based on the gaze point on the display on the interlocutor side. A gaze partner determination unit that determines whether a person gazes at a conversation partner, and the dialog in an eye area of a communicator represented by the camera input image when it is determined that the gaze at the conversation partner is determined And a combining unit configured to combine an image when the user gazes at a position corresponding to the other party.

  The program of the present invention is a program for causing a computer to execute each step of the gaze-matched face image synthesis method of the present invention.

  As described above, according to the gaze-matched face image synthesizing method, the video conference system, and the program of the present invention, whether or not the other party in the dialogue is gazed on the basis of the classifier learned in advance To determine the gaze point on the display of the camera and the camera viewpoint position with respect to the display display, and synthesize the image when the camera or the position corresponding to the conversation partner is gazed in the eye area of the interlocutor represented by the camera input image As a result, even when the camera viewpoint position with respect to the display display changes, it is possible to accurately determine the match of the sight line and to generate an image in which the sight line matches.

BRIEF DESCRIPTION OF THE DRAWINGS It is a block diagram which shows one structural example of the video conference system of the 1st Embodiment of this invention. It is a figure for demonstrating a camera viewpoint position and arrangement | positioning of a window. It is a figure for demonstrating a camera viewpoint position and arrangement | positioning of a window. It is a figure for demonstrating a camera viewpoint position and arrangement | positioning of a window. It is a figure for demonstrating a camera viewpoint position and arrangement | positioning of a window. It is a figure for demonstrating the method to extract eye region image. It is a figure which shows an example of an eye area | region image. FIG. 8 is a diagram showing the arrangement of points displayed on a display when capturing a learning camera input image. It is a flowchart which shows the content of the learning process routine in the video conference system of the 1st Embodiment of this invention. It is a flowchart which shows the content of the gaze matching face image synthetic | combination processing routine in the video conference system of the 1st Embodiment of this invention. It is a block diagram which shows one structural example of the video conference system of the 2nd Embodiment of this invention. It is a figure for demonstrating arrangement | positioning of a camera and a window. It is a flowchart which shows the content of the gaze matching face image synthetic | combination processing routine in the video conference system of the 2nd Embodiment of this invention. It is a block diagram which shows one structural example of the video conference system of the 3rd Embodiment of this invention. It is a figure for demonstrating arrangement | positioning of a camera and a window. It is a flowchart which shows the content of the image transmission side gaze-matched face image synthetic | combination processing routine in the video conference system of the 3rd Embodiment of this invention. It is a flowchart which shows the content of the image reception side gaze-matched face image synthetic | combination processing routine in the video conference system of the 3rd Embodiment of this invention.

<Overview of the embodiment of the present invention>
First, an outline of the embodiment of the present invention will be described. An embodiment of the present invention is to generate a face image that matches the line of sight in a video conference system. In the video conference of a plurality of interlocutors, even when the windows are arranged at various places, it is possible to show meeting participants in an easy-to-understand manner who is interacting with whom.

  The embodiment of the present invention is a camera position relative to the display on the user terminal, as compared with the eye area conversion method for eye-gaze matching between communicators in a video conference described in Non-Patent Document 1. The point is that it is possible to cope with the change in the orientation of the user terminal by making the determination. In particular, in the case where the display on the user terminal can be rotated according to the way of holding, as in the case of a tablet terminal, the camera viewpoint position and the gaze partner should be specified even when the camera viewpoint position changes with respect to the display on the user terminal. It is possible.

First Embodiment
<Configuration of Video Conference System>
Hereinafter, a first embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a video conference system 100 according to a first embodiment of this invention. The teleconferencing system 100 is configured using a computer including a CPU, a RAM, a learning processing routine, and a ROM storing a program for executing the gaze-matched face image synthesis processing routine, and the functions are as follows: It is configured as shown in.

  As shown in FIG. 1, the video conference system 100 according to the present embodiment includes a camera 10, an arithmetic unit 20, a communication unit 80, and a display 90.

  As shown in FIGS. 2A to 2D, the camera 10 captures the face of the communicator from any of a plurality of viewpoint positions for display on the display 90, and outputs a camera input image to the calculation unit 20. The display 90 displays a window 92 displaying an image of the other party, and the interlocutor interacts while looking at the window 92.

  The calculation unit 20 includes a learning image DB 22, eye region image extraction units 24, 32, feature extraction units 26, 34, a classifier construction unit 28, a classifier storage unit 30, eye open / close judgment unit 36, camera position dialogue partner gaze judgment A unit 38, a gaze image storage unit 40, a combining unit 42, and a screen output unit 44 are provided.

  The learning image DB 22 indicates the position of the camera 10 when the communicator is gazing at one point arranged at the position with respect to each viewpoint position of the camera 10 with respect to the display on the display 90 and each position on the display 90. A learning camera input image captured by the camera is stored. The display range of the window 92 on the display 90 is predetermined, and the communicator gazes at the other side of the dialog depending on whether the fixation point of each learning camera input image is within the display range of the window 92 or not. Information indicating whether or not the camera 10 has been given is given in advance, and furthermore, the information showing the viewpoint position of the camera 10 is given in advance. For example, the interlocutors move only their eyes while keeping their face fixed, and display the points as shown in FIG. 5 one by one on the display and have them gaze closely in sequence, and within the dotted rectangle The image that is watching the image is positive, and the other image is negative.

  The eye region image extraction unit 24 extracts eye region images from each of the learning camera input images as follows.

  For example, four feature points are extracted from the contour portions of the left and right eye regions as shown in FIG. Next, an eye area is extracted with a circumscribed rectangle of four feature points, and the left and right eyes are connected to form an eye area image. FIG. 4 shows an example of the eye area image extracted. At this time, the patterns of the eye region image are increased by adding perturbations such as parallel movement and scaling to the extracted feature points. This process is performed on all learning camera input images, and all eye region images are resized at the median value of the circumscribed rectangle size of all eye region images.

  The feature extraction unit 26 extracts feature amounts as follows for eye region images extracted from each of the learning camera input images.

  For example, an edge is detected by Canny from the extracted eye region image, and it is set as a feature amount whether or not there is an edge. The eye area image is then converted to grayscale and histogram equalization is performed. Thereafter, the luminance values are connected to be a feature amount. Finally, the luminance value and the edge feature amount are connected to make a final feature amount.

  The discriminator construction unit 28 is a feature amount extracted from each of the learning camera input images, and information indicating whether the communicator is gazing at the other party of conversation, which is given to each of the learning camera input images. And a discriminator for identifying whether or not the interlocutor is gazing at the other party of the dialogue and a pair of camera viewpoint positions based on the information indicating the viewpoint position of the camera 10. In this embodiment, an SVM (Suport Vector Machine) classifier that is generally considered to have high performance for the two-class classification problem is used as a classifier.

  The label of the class to be identified here is each pair of whether or not the interlocutor gazes at the other party of the dialogue and the viewpoint position of the camera 10. For example, the position of the window is four on the display: upper left, upper right, lower left, lower right, and 4 x 2 = 8 labels if the interlocutor gazes at the other party or not And learn the classifier.

  The classifier storage unit 30 stores the classifier learned by the classifier construction unit 28.

  The eye area image extraction unit 32 extracts an eye area image from the camera input image as in the case of the eye area image extraction unit 24.

  The feature extraction unit 34 extracts feature amounts from the eye region image extracted from the camera input image, as in the feature extraction unit 26.

  The eye opening / closing judgment unit 36 calculates the eye opening / closing degree based on the feature amount extracted for the camera input image and the feature amount extracted in advance from the reference image whose eye opening / closing degree is known, If the degree is greater than the threshold, it is determined that the eye is open.

  When it is determined that the eyes are open, the camera position dialogue partner gaze determination unit 38 determines, based on the classifier stored in the classifier storage unit 30 and the feature quantity extracted for the camera input image, It is determined whether the interlocutor gazes at the other party of the dialogue and the viewpoint position of the camera 10.

  The gaze image storage unit 40 stores an image representing an eye, which is captured in advance and extracted from a camera input image when the communicator gazes at the camera 10. In addition, you may prepare for each viewpoint position of the camera 10 the image showing the eyes extracted from the camera input image at the time of gazing at the camera 10 of the said viewpoint position.

  When it is determined that the communicator is gazing at the other party of the dialogue, the synthesizing unit 42 synthesizes an image representing the eye, which is stored in the gazing image storage unit 40, in the eye area of the camera input image. The composite image is generated and transmitted by the communication unit 80 to the teleconference system (not shown) on the other side of the dialog. When an image representing an eye is prepared for each viewpoint position of the camera 10, an image representing the eye corresponding to the viewpoint position of the camera 10 determined by the camera position dialogue partner gaze determination unit 38 is synthesized. do it.

  The screen output unit 44 displays the image received by the communication unit 80 from the teleconference system on the other side of the dialog on the window 92 of the display 90.

  The teleconference system on the other side of the conversation has the same configuration as the teleconferencing system 100 on the dialogue side, and the television conference system on the other side of the dialogue and the television on the dialogue side via a network such as the Internet. The conference system 100 is connected.

<Operation of the video conference system>
Next, the operation of the video conference system 100 according to the first embodiment will be described. First, as preprocessing, for each viewpoint position of the camera 10 and each position on the display 90, the camera 10 for learning by the camera 10 at the viewpoint position while the communicator is gazing at one point arranged at the position. An input image is captured, and each learning camera input image is stored in the learning image DB 22 together with information indicating whether the communicator gazes at the other party and information indicating the viewpoint position of the camera 10.

  Then, the video conference system 100 executes a learning process routine shown in FIG.

  First, in step S100, each learning camera input image is acquired from the learning image DB 22. In step S102, an eye area image is extracted from each learning camera input image.

  Then, in step S104, the feature amount is extracted from the eye area image of each learning camera input image. In step S106, the feature amount extracted for each learning camera input image in step S104, information indicating whether the communicator is gazing at the other party of the conversation, added to each learning camera input image, and Based on the information indicating the viewpoint position of the camera 10, a classifier is constructed and stored in the classifier storage unit 30, and the learning processing routine is ended.

  In addition, when a video conference with the conversation partner is started, and the image transmitted from the video conference system on the conversation partner side is displayed on the window 92 of the display 90, the camera input image is input from the camera 10. Each time the teleconference system 100 repeatedly executes the gaze-matched face image synthesis processing routine shown in FIG. 7.

  First, in step S110, an eye area image is extracted from the camera input image. In step S112, feature amounts are extracted from the eye region image.

  Then, in step S114, it is determined whether or not the eyes are open based on the feature quantity extracted in step S112. If it is determined that the eyes are open, the process proceeds to step S115. If it is determined that the eyes are not open, the process proceeds to step S122.

  In step S115, based on the feature quantity extracted in step S112 and the discriminator stored in the discriminator storage unit 30, whether or not the communicator is gazing at the dialogue partner, and the viewpoint position of the camera 10 Determine the pair of In step S116, based on the determination result in step S115, it is determined whether the communicator is gazing at the other party. If it is determined that the user is gazing at the dialogue partner, the process proceeds to step S118, and if it is determined that the dialogue partner is not gazed at, the process proceeds to step S122.

  In step S118, a composite image obtained by combining the image representing the eyes stored in the gaze image storage unit 40 is generated in the eye area of the camera input image, and in step S120, the composite image generated in step S118 is The communication unit 80 transmits it to the teleconference system on the other side of the dialog, and the gaze-matched face image synthesis processing routine is ended.

  In step S122, the camera input image is transmitted as it is to the teleconference system on the other side of the dialog by the communication unit 80, and the gaze-matched face image synthesis processing routine is ended.

  As described above, the video conference system according to the first embodiment determines whether the user is gazing at the other party of the conversation and the camera viewpoint position with respect to the display on the basis of the classifier learned in advance. Even when the camera viewpoint position with respect to the display display changes, by combining the image when gazing at the camera into the eye area of the communicator represented by the input image, the coincidence of the line of sight is accurately determined, Can generate a matched image.

Second Embodiment
<Configuration of Video Conference System>
Next, a video conference system according to the second embodiment will be described. In addition, about the part which becomes the structure similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  In the second embodiment, a point at which a plurality of windows representing a plurality of dialog partners are displayed on the display, and a dialog partner at which the interlocutor is gazing are determined to synthesize an image of the eye area Is different from the first embodiment.

  As shown in FIG. 8, the computing unit 220 of the video conference system 200 according to the second embodiment includes a learning image DB 22, eye area image extracting units 24 and 32, feature extracting units 26 and 34, and a classifier constructing unit 228. , Discriminator storage unit 230, eye open / close determination unit 36, camera position display upper fixation point determination unit 238, gaze partner determination unit 239, gaze image storage unit 240, gaze matching condition determination unit 241, combining unit 242, and screen output unit It has 44.

  As shown in FIG. 9, the camera 10 captures the face of the communicator from any of a plurality of viewpoint positions with respect to the display of the display 90, and outputs a camera input image to the computing unit 220. The display 90 displays a plurality of windows 92 in which images of a plurality of interaction partners are displayed, and the interlocutor interacts while viewing the plurality of windows 92.

  The discriminator construction unit 228 displays the interlocutor on the basis of the feature value extracted from each of the learning camera input images for each viewpoint position of the camera 10 with respect to the display of the display 90 and each pair of positions on the display 90. A gazer for the position on 90 is constructed, and a discriminator for identifying whether the position is the eye point position is constructed respectively.

  The label of the class to be identified here is each pair of the position on the display 90 and the viewpoint position of the camera 10. For example, the viewpoint positions of the camera 10 are four at the top, bottom, left, and right of the display 90, and the positions on the display 90 are sampled with skip widths preset in the vertical and horizontal directions on the display 90 And N positions. The classifier learning is performed using 4 × N = 4N labels in total. Further, the boundary for dividing whether or not the learning camera input image is watched is a rectangle whose center is the position.

  The classifier storage unit 230 stores classifiers learned for each viewpoint position of the camera 10 and each pair of positions on the display 90 by the classifier construction unit 28.

  When it is determined that the eyes are open, the camera position display upper fixation point determination unit 238 stores the pair of each viewpoint position of the camera 10 and each position on the display 90 in the identifier storage unit 230 Based on the classifier for the pair and the feature quantity extracted for the camera input image, it is determined whether the interlocutor gazes at the position on the display 90 and whether it is the viewpoint position or not.

  Based on the determination result of each viewpoint position of the camera 10 and the pair of each position on the display 90 by the gaze position determination unit 238 of the camera position display, the gaze target determination unit 239 gazes at which dialogue partner Determine if it exists. For example, for each of the positions on the display 90 determined to be gazed by the interlocutor, the interlocutor who is voting in the window 92 including the position and the window 92 with the highest number of votes is displayed is Determined as the other person in the dialogue that is watching. When there are a plurality of windows 92 having the largest number of votes, it may be determined that the average position of the central positions of the plurality of windows 92 having the largest number of votes is watched.

  The gaze image storage unit 240 represents an eye extracted from a camera input image captured in advance when a communicator gazes at each of a plurality of synthesis gaze points 290 corresponding to a plurality of windows 92. An image is stored (see FIG. 9 above).

  When it is determined that the communicator gazes at the dialogue partner, the synthesis unit 242 stores the eye region of the camera input image in the gaze image storage unit 240 corresponding to the dialogue partner gazed at The combined image is generated by combining the images representing the eyes, and the combined image is transmitted by the communication unit 80 to the video conference system on the image receiving side.

  The teleconferencing system on the image receiving side also has the same configuration as the teleconferencing system 200 on the communicator side, and the teleconferencing system on the image receiving side and the television on the communicator side via a network such as the Internet. The conference system 200 is connected.

<Operation of the video conference system>
Next, the operation of the video conference system 200 according to the second embodiment will be described. First, as preprocessing, for each viewpoint position of the camera 10 and each position on the display 90, the camera 10 for learning by the camera 10 at the viewpoint position while the communicator is gazing at one point arranged at the position. An input image is captured and stored in the learning image DB 22.

  Then, the video conference system 200 executes a learning processing routine as in the first embodiment. In the second embodiment, with respect to each viewpoint position of the camera 10 and each pair of positions on the display 90, the communicator is displayed on the display 90 based on the feature quantity extracted from each of the learning camera input images. A discriminator is constructed to gaze at the corresponding position of and to identify whether or not the camera 10 is the viewpoint position.

  Also, when a video conference with the conversation partner is started and the images transmitted from the plurality of video conference systems on the conversation partner side are respectively displayed on the plurality of windows 92 of the display 90, camera input from the camera 10 Every time an image is input, the teleconference system 200 repeatedly executes the gaze-matched face image synthesis processing routine shown in FIG. The same processing as that of the first embodiment is given the same reference numeral and the detailed description is omitted.

  First, in step S110, an eye area image is extracted from the camera input image. In step S112, feature amounts are extracted from the eye region image.

  Then, in step S114, it is determined whether or not the eyes are open based on the feature quantity extracted in step S112. If it is determined that the eyes are open, the process proceeds to step S200. If it is determined that the eyes are not open, the process proceeds to step S122.

  In step S200, based on the feature quantity extracted in step S112 and the classifier stored in the classifier storage unit 230, dialogue is performed for each viewpoint position of the camera 10 and each pair of positions on the display 90. It is determined whether a person gazes at the position and the camera 10 is at the viewpoint position. Then, in step S202, based on the determination result of the above-mentioned step S200, it is determined the conversation partner gazed by the communicator.

  Then, in step S204, it is determined based on the determination result in step S202 whether or not there is a dialogue partner gazed by the communicator. If it is determined that there is a dialogue partner who is gazing, the process proceeds to step S206. If it is determined that there is no dialogue partner who is gazing, the process proceeds to step S122.

  In step S206, based on the determination result in step S202, an image representing an eye stored in the gaze image storage unit 240 in the eye area of the camera input image corresponding to the other party of the dialogue being gazed by the communicator In step S120, the communication unit 80 transmits the composite image generated in step S206 to the video conference system on the image receiving side, and the gaze-matched face image composition processing routine is ended.

  In step S122, the camera input image is transmitted as it is to the teleconference system on the image receiving side by the communication unit 80, and the gaze-matched face image synthesis processing routine is ended.

  As described above, in the video conference system according to the second embodiment, the display on the interlocutor side is performed based on the classifiers learned in advance for each camera viewpoint position with respect to the display display and each pair of positions on the display. Determines the gaze point and camera viewpoint position above to determine the conversation partner who is gazing, and synthesizes an image when gazing at a position corresponding to the conversation partner in the eye area of the communicator represented by the camera input image Accordingly, even when the camera viewpoint position with respect to the display display changes, it is possible to accurately determine the line-of-sight match and generate an image with the line-of-sight match.

  Also, in a video conference of a plurality of interlocutors, even when a plurality of windows are arranged on the display, it is possible to show the meeting participants in an easy-to-understand manner who is interacting with whom.

Third Embodiment
<Configuration of Video Conference System>
Next, a video conference system according to the third embodiment will be described. In addition, about the part which becomes the structure similar to 1st Embodiment and 2nd Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The third embodiment is different from the second embodiment in that images are combined in a video conference system on the image receiving side.

  As shown in FIG. 11, in the third embodiment, a video conference system 300 on the image transmitting side and a video conference system 400 on the image receiving side are connected via the network 500.

  As shown in FIG. 12, in the image transmission side video conference system 300 and the image reception side video conference system 400, the display 90 displays a plurality of windows 92 on which images of a plurality of conversation partners are displayed, and a communicator Interact while viewing the plurality of windows 92. In the present embodiment, the arrangement of the plurality of windows 92 on the display 90 can be changed in each of the video conference system 300 on the image transmission side and the video conference system 400 on the image reception side. In the following, a case in which the arrangement of the plurality of windows 92 on the display 90 is different between the video conference system 300 on the image transmitting side and the video conference system 400 on the image receiving side will be described as an example.

  The operation unit 320 of the video conference system 300 on the image transmission side includes the learning image DB 22, the eye area image extraction units 24 and 32, the feature extraction units 26 and 34, the classifier construction unit 228, the classifier storage unit 230, and the eye opening / closing judgment The unit 36, the camera position display upper fixation point determination unit 238, the gaze partner determination unit 239, and the screen output unit 44 are provided.

  When the eye open / close determination unit 36 determines that the eyes are not open, the communication unit 80 of the video conference system 300 on the image transmission side determines the determination result by the eye open / close determination unit 36 and the camera input image It transmits to the teleconference system 400 on the receiving side. When the eye open / close determination unit 36 determines that the eyes are open, the determination result by the eye open / close determination unit 36, the determination result by the fixation partner determination unit 239, and the camera input image To the video conference system 400 of FIG.

  The calculation unit 420 of the video conference system 400 on the image reception side includes an eye open / close determination unit 422, a gaze target calculation unit 424, a gaze image storage unit 240, and a combination unit 242.

  The eye open / close determination unit 422 determines whether the eye of the communicator on the image transmission side is open based on the determination result of the eye open / close received by the communication unit 80.

  When it is determined that the eye of the interlocutor on the image transmission side is open, the gaze target calculation unit 424 determines the interlocutor on the image transmission side based on the information on the other party of the gaze at which the gaze is being received. It is determined whether the subject gazes at the other party. In addition, when it is determined that the communicator at the image transmission side is gazing at the conversation partner, the gazing target calculation unit 424 displays the display 90 based on the information on the gazed conversation partner received by the communication unit 80. It is calculated which of the windows 92 on the upper side is focused on. For example, among the plurality of windows 92 on the display 90, the window 92 corresponding to the information of the conversation partner being watched received by the communication unit 80 is identified, and the conversation partner displayed by the identified window 92 is selected. , It is calculated as the other side of the dialogue on which the interlocutor on the image sending side is watching.

  The gaze image storage unit 240 uses a camera input image captured in advance when a communicator gazes at each of a plurality of composition gaze points 290 corresponding to a plurality of windows 92 as shown in FIG. 12 described above. An image representing the extracted eye is stored.

  When it is determined that the communicator at the image transmission side is gazing at the conversation partner, the synthesis unit 242 receives the camera input received from the video conference system 300 at the image transmission side based on the calculation result by the gaze target calculation unit 424. In the eye area of the image, a composite image is generated by combining the image representing the eyes, which is stored in the gaze image storage unit 240 corresponding to the other party of the dialogue on the image transmission side, and the composite image is generated. , And displayed on the corresponding window 92 of the display 90.

  The video conference system 300 on the image transmitting side further has the same configuration as the video conference system 400 on the image receiving side, and the video conference system 400 on the image receiving side and the video conference system 300 on the image transmitting side. It further has the same configuration but is omitted for simplicity.

<Operation of the video conference system>
Next, the operation of the video conference system 300 on the image transmission side according to the third embodiment will be described. First, as preprocessing, with respect to each viewpoint position of the camera 10 with respect to the display on the display 90 and each position on the display 90, when the communicator is gazing at one point arranged at the position, the camera at the viewpoint position The camera input image for learning is captured by 10 and stored in the image for learning DB 22.

  Then, a learning processing routine is executed by the video conference system 300 on the image transmission side, as in the second embodiment.

  In addition, when a video conference with the other party of the dialogue is started and the images transmitted from each of the video conference systems 400 on the image receiving side are displayed on the plurality of windows 92 of the display 90, camera input from the camera 10 Every time an image is input, the video conference system 300 repeatedly executes the image transmission side gaze-matched face image synthesis processing routine shown in FIG. In addition, about the process similar to 1st Embodiment and 2nd Embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

  First, in step S110, an eye area image is extracted from the camera input image. In step S112, feature amounts are extracted from the eye region image.

  Then, in step S114, it is determined whether or not the eyes are open based on the feature quantity extracted in step S112. If it is determined that the eyes are open, the process proceeds to step S200. If it is determined that the eyes are not open, the process proceeds to step S300.

  In step S200, based on the feature quantity extracted in step S112 and the classifier stored in the classifier storage unit 230, dialogue is performed for each viewpoint position of the camera 10 and each pair of positions on the display 90. It is determined whether a person gazes at the position and the camera 10 is at the viewpoint position. Then, in step S202, based on the determination result of the above-mentioned step S200, it is determined the conversation partner gazed by the communicator.

  Then, in step S302, the communication unit 80 transmits the determination result in the above step S114, the determination result in the above step S202, and the camera input image to the video conference system 400 on the image receiving side, The image composition processing routine is ended.

  On the other hand, in step S300, the communication unit 80 transmits the determination result in step S114 and the camera input image to the teleconference system 400 on the image receiving side, and the image transmitting side gaze-matched face image synthesizing process routine is ended.

  Every time the various types of determination results and the camera input image are received from the video conference system 300 on the image transmission side, the video conference system 400 repeatedly executes the image reception side gaze-matched face image synthesis processing routine shown in FIG.

  First, in step S400, based on the determination result received from the video conference system 300 on the image transmission side, it is determined whether the eye of the interlocutor on the image transmission side is open. If it is determined that the eyes are open, the process proceeds to step S402. If it is determined that the eyes are not open, the process proceeds to step S404.

  In step S402, based on the determination result received from the video conference system 300 on the image transmission side, it is determined whether there is a conversation partner on whom the interlocutor on the image transmission side is gazing. If it is determined that there is a dialogue partner who is gazing, the process proceeds to step S406. If it is determined that there is no dialogue partner who is gazing, the process proceeds to step S404.

  In step S406, based on the determination result received from the image transmission side video conference system 300, it is calculated which window 92 on the display 90 the conversation party of the image transmission side gazes at.

  Then, in step S408, based on the calculation result in step S406, the eye region of the camera input image received from the teleconference system 300 on the image transmitting side corresponds to the other side of the conversation on which the communicator on the image transmitting side is gazing Then, a composite image obtained by combining the images representing the eyes, stored in the gaze image storage unit 240, is generated, and the composite image generated in step S408 is displayed in the corresponding window 92 of the display 90 in step S410. Then, the image reception side gaze-matched face image synthesis processing routine is ended.

  In step S404, the camera input image received from the video conference system 300 on the image transmission side is displayed as it is on the corresponding window 92 of the display 90, and the image reception side gaze-matched face image synthesis processing routine is ended.

  As described above, in the video conference system according to the third embodiment, the display on the image transmission side is based on the classifiers learned in advance for each camera viewpoint position for display display and each pair of positions on the display. When the gaze point and camera viewpoint position above are determined to determine the conversation partner who is gazing, and the image reception side gazes at a position corresponding to the conversation partner in the eye area of the communicator represented by the camera input image By combining the images of the above, even when the camera viewpoint position with respect to the display display changes, it is possible to accurately determine the line-of-sight match and generate the image with the line-of-sight match.

  In addition, in the video conference of multiple interlocutor, even when multiple windows are placed in various places for each video conference system, it is easy to understand who is interacting with who in the conference participant can do.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the scope of the present invention.

  For example, although the above-mentioned video conference system has a computer system inside, the "computer system" also includes a homepage providing environment (or display environment) if the WWW system is used.

  Furthermore, although the present invention has been described as an embodiment in which the program is installed in advance, it is also possible to provide the program by storing the program in a computer readable recording medium.

DESCRIPTION OF SYMBOLS 10 Cameras 20, 220, 320, 420 Arithmetic unit 24 Eye region image extraction unit 26 Feature extraction unit 28, 228 Discriminator construction unit 30, 230 Discriminator storage unit 32 Eye region image extraction unit 34 Feature extraction unit 36, 422 Eye opening and closing Judgment unit 38 Camera position dialogue other party gaze judgment unit 40, 240 Gaze image storage unit 42, 242 Synthesizing unit 44 Screen output unit 80 Communication unit 90 Display 92 Window 100, 200, 300, 400 Teleconference system 238 Camera position display on gaze point Judgment unit 239 gaze target determination unit 290 composite gaze point 424 gaze target calculation unit 500 network

Claims (6)

A gaze matching face image synthesizing method in a video conference system, comprising:
Feature extraction process for extracting features from camera input images representing interlocutor,
An eye open / close judging process of judging open / close of eyes using features of the camera input image;
In the case where it is determined that the eyes are open, whether or not they are gazing at the conversation partner, whether they are gazing at the conversation partner, in which the camera viewpoint position with respect to the display display is learned in advance from known face images Based on the identifier for identifying the camera viewpoint position and the feature of the camera input image, it is determined whether the communicator represented by the camera input image is gazing at the other party of the dialog and the camera viewpoint position Camera position dialogue partner gaze determination process,
Combining, when it is determined that the user is gazing at the conversation partner, an image when the user gazes at the camera in the eye area of the communicator represented by the camera input image;
A gaze-matched face image composition method including: A gaze matching face image synthesizing method in a video conference system, comprising:
Feature extraction process for extracting features from camera input images representing interlocutor,
An eye open / close judging process of judging open / close of eyes using features of the camera input image;
When it is determined that the eyes are open, each camera viewpoint position and the display on the interlocutor side in which the gaze point on the display on the interlocutor's display and the camera viewpoint position relative to the display are previously learned from known face images. Note on the display on the interlocutor's side based on a discriminator that looks at the position for each position pair on the top and identifies whether it is the camera viewpoint position and the features of the camera input image A gaze point determination process on a camera position display for determining a viewpoint and the camera viewpoint position;
A gaze partner determination step of determining whether the dialog person represented by the camera input image is gazing at the conversation partner based on the fixation point on the display of the dialogue partner side;
Combining the images when the user gazes at a position corresponding to the conversation partner in the eye area of the communicator represented by the camera input image when it is determined that the user gazes at the conversation partner;
A gaze-matched face image composition method including:   In the combining process, when it is determined that the user gazes at the conversation partner, the combining unit on the image receiving side combines the image when the user gazes in the eye area of the communicator represented by the camera input image. The gaze-matched face image synthesizing method according to claim 1 or 2. A feature extraction unit for extracting features from a camera input image representing a communicator;
An eye open / close judgment unit which judges the open / close of the eye using the feature of the camera input image;
In the case where it is determined that the eyes are open, whether or not they are gazing at the conversation partner, whether they are gazing at the conversation partner, in which the camera viewpoint position with respect to the display display is learned in advance from known face images Based on the identifier for identifying the camera viewpoint position and the feature of the camera input image, it is determined whether the communicator represented by the camera input image is gazing at the other party of the dialog and the camera viewpoint position Camera position dialogue partner gaze determination unit;
A combining unit configured to combine an image obtained when the user gazes at the camera into the eye area of the communicator represented by the camera input image when it is determined that the user is gazing at the conversation partner;
Video conferencing system including: A feature extraction unit for extracting features from a camera input image representing a communicator;
An eye open / close judgment unit which judges the open / close of the eye using the feature of the camera input image;
When it is determined that the eyes are open, each camera viewpoint position and the display on the interlocutor side in which the gaze point on the display on the interlocutor's display and the camera viewpoint position relative to the display are previously learned from known face images. Note on the display on the interlocutor's side based on a discriminator that looks at the position for each position pair on the top and identifies whether it is the camera viewpoint position and the features of the camera input image A gaze point determination unit on a camera position display that determines a viewpoint and the camera viewpoint position;
A gaze partner determination unit that determines whether a dialog represented by the camera input image is gazing at a conversation partner based on a gaze point on a display on the dialog person side;
A synthesizing unit that synthesizes an image obtained when gazing at a position corresponding to the conversation partner into the eye area of the communicator represented by the camera input image when it is determined that the user gazes at the conversation partner;
Video conferencing system including: The program for making a computer perform each process of the gaze matching face image synthetic | combination method in any one of Claims 1-3.