JP2010239499A - Communication terminal unit, communication control unit, method of controlling communication of communication terminal unit, and communication control program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a communication terminal unit capable of changing a user image transmitted according to user's movement, and to provide a communication control unit, a method of controlling a communication terminal unit, and a communication control program. <P>SOLUTION: It is detected whether or not the user of the terminal unit is in making a gesture during teleconferencing (S4). When the user is not in making a gesture (S5:NO), the user's face image is transmitted to the terminal unit of an opposite user (S3). When the user is in making a gesture (S5:YES), the user's upper-body image is transmitted to the terminal unit of the opposite user (S7). When the user tries to give an expression to his/her feeling, the opposite user can check the user's gesture. When the user does not make a gesture, the opposite user can check the user's facial expression. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a communication terminal device, a communication control device, a communication control method for the communication terminal device, and a communication control program that can bidirectionally transmit and receive images and sound between terminals at other sites.

  2. Description of the Related Art Conventionally, there has been known a video conference system in which a plurality of terminals are connected via a network, and an image and a sound are bidirectionally transmitted and received to realize a conference between persons in remote locations. In this system, a conference participant has a conference with the other party displayed on the display screen. Since information can be transmitted via the image, the conference participant can read the other party's facial expression from the other party's image displayed on the display screen and guess the other party's emotion. For example, a camera control method and apparatus and a storage medium that can estimate a speaker from input speech and automatically perform close-up photography have been proposed (see, for example, Patent Document 1). In this control method, since the speaker is photographed in close-up, the expression of the speaker can be reliably displayed on the display screen of the partner terminal.

JP-A-11-341334

  However, a conference participant may use gestures such as gestures when expressing emotions. In the camera control method described in Patent Document 1, since only the face image of the conference participant is displayed on the display screen, the gesture does not appear on the display screen even if the conference participant tries to express the emotion with the gesture. was there. If the shooting range is expanded to show the gesture on the display screen, the face images of the conference participants become relatively small, and the expression cannot be confirmed.

  The present invention has been made to solve the above-described problem, and provides a communication terminal device, a communication control device, a communication control method for a communication terminal device, and a communication control program capable of changing a user image to be transmitted according to a user's operation. The purpose is to provide.

  In order to achieve the above object, a communication terminal device according to a first aspect of the present invention is a communication terminal device that communicates with other communication terminal devices connected via a network via images and sounds, and is a user Photographing means for photographing the image, movement detection means for detecting a state in which at least one of the movements of the palm and arm of the user is greater than or equal to a predetermined amount from the photographed image photographed by the photographing means, When the motion is detected by the motion detection means, the first image shows a range of the upper body area image including the face area, the palm area, and the arm area of the user among the captured images captured by the imaging means. A first image range determining unit that determines the range; and when the motion is not detected by the motion detection unit, the face area of the user is selected from the captured image. A second image range determining means for determining the image range as the second image range, and an image of the first image range determined by the first image range determining means, or determined by the second image range determining means An image transmitting means for transmitting an image in the second image range to the other communication terminal apparatus, an image receiving means for receiving the image transmitted from the other communication terminal apparatus, and the image received by the image receiving means. Image display control means for displaying the image on the display screen when the image is received.

  According to a second aspect of the present invention, in addition to the configuration of the first aspect of the invention, the motion detecting means is configured to detect a state where both the palm and the arm have a predetermined amount of movement. It is detected as an operation.

  According to a third aspect of the present invention, there is provided the communication terminal device according to the first or second aspect, wherein the motion detecting means indicates that the palm shape has a change of a predetermined amount or more. It is characterized by detecting as a movement.

  According to a fourth aspect of the present invention, there is provided the communication terminal device according to any one of the first to third aspects, wherein the motion detecting means indicates that the arm position has changed by a predetermined amount or more. It is detected as a movement of the arm.

  In addition to the configuration of the invention according to any one of claims 1 to 4, the communication terminal device according to the invention according to claim 5 further includes a nose position detection means for detecting the nose position of the target user from the captured image. And the first image range determining means determines the first image range so that the nose position detected by the nose position detecting means is a central point in the horizontal direction of the first image range. It is characterized by.

  According to a sixth aspect of the present invention, there is provided a communication terminal device according to any one of the first to fifth aspects, further comprising a speaker specifying unit that specifies a speaker from a plurality of users. The motion detection means detects a state in which at least one of the movements of the palm and the arm of the speaker specified by the speaker specifying means is greater than or equal to the predetermined amount, and determines the first image range. The means determines an upper body image range including the speaker's face area, palm area, and arm area as the first image area, and the second image range determining means includes the face area of the speaker. The range of the face image is determined as the second image range.

  According to a seventh aspect of the present invention, there is provided a communication terminal apparatus according to the sixth aspect, in addition to the configuration of the sixth aspect of the invention, a person recognizing unit for recognizing a person from the photographed image photographed by the photographing unit; Mouth shape detecting means for detecting a change in the mouth shape of the person recognized by the speaker, wherein the speaker specifying means specifies a person whose mouth shape change is detected by a predetermined amount or more by the mouth shape detecting means as the speaker. It is characterized by doing.

  In addition to the configuration of the invention of claim 6 or 7, the communication terminal device of the invention according to claim 8 includes voice detection means for detecting the voice of the user and detecting the direction of the voice, The speaker specifying means specifies a person in the direction detected by the voice detecting means as the speaker.

  A communication control device according to an embodiment of the present invention is a communication control device that is connected to a plurality of communication terminal devices via a network and controls communication performed between the communication terminal devices, wherein the communication terminal device Based on the captured image received by the captured image receiving means and at least one of the palm and arm of the user, the captured image receiving means for receiving the captured image transmitted by the communication terminal device Motion detection means for detecting a state of such movement as the user's motion, and when the motion is detected by the motion detection means, the user's face area, palm area, and arm from the captured image A first image range determining means for determining an image range of the upper body area including the area as the first image range, and the motion detection means does not detect the motion. The second image range determining means for determining a range of the face image including the face area of the user as the second image range from the photographed image, and the first image range determining means. And an image transmitting means for transmitting the image in the first image range or the image in the second image range determined by the second image range determining means to the communication terminal device.

  A communication control method for a communication terminal device according to a tenth aspect of the invention is a communication control method for a communication terminal device that communicates with other communication terminal devices connected via a network via images and sounds. An operation detecting step for detecting a movement of at least one of the user's palm and arm as the user's operation, and an imaging unit for capturing the user when the operation is detected in the operation detecting step. In the first image range determination step for determining an image range of the upper body region including the user's face region, palm region, and arm region as the first image range from the captured images, and in the motion detection step A second image that determines, as a second image range, an image range of the face area of the user from the captured image when the operation is not detected. An image of the first image range determined in the range determining step and the first image range determining step, or an image of the second image range determined in the second image range determining means is used as the other communication terminal. An image transmitting step for transmitting to the device, an image receiving step for receiving the image transmitted from the other communication terminal device, and the image displayed on the display screen when the image is received in the image receiving step. An image display control step.

  According to an eleventh aspect of the present invention, a communication control program causes a computer to function as various processing means of the communication terminal device according to any one of the first to eighth aspects.

  In the communication terminal apparatus according to the first aspect of the present invention, communication is performed with other communication terminal apparatuses connected via a network via images and sounds. The photographing means photographs the user. The motion detection means detects a state in which at least one of the movements of the user's palm and arm is greater than or equal to a predetermined amount from the photographed image photographed by the photographing means. The first image range determining unit is configured to detect an image of an upper body region including a user's face region, palm region, and arm region from among the captured images captured by the capturing unit when the motion is detected by the motion detecting unit. The range is determined as the first image range. The second image range determining means determines the image range of the user's face area as the second image range from the captured images when no motion is detected by the motion detecting means. The image transmission unit transmits the image of the first image range determined by the first image range determination unit or the image of the second image range determined by the second image range determination unit to another communication terminal device. The image receiving means receives an image transmitted from another communication terminal device. The image display control means displays the image on the display screen when the image receiving means receives the image. As described above, when the motion is not detected by the motion detection means, the image of the user's face area is transmitted to another communication terminal device, and when the motion is detected, the image of the upper body area is transmitted to the other communication terminal. Sent to the device. Therefore, in another communication terminal device, when the user is making a gesture, an image of the upper body area of the user is displayed on the display screen, and when the user is not making a gesture, an image of the user's face area is displayed. Is displayed. Therefore, when a user tries to express an emotion with a gesture, the other user can confirm the user's gesture. Further, when the user does not perform a gesture, the partner user can check the facial expression of the user. Therefore, it is possible to take good communication between conference participants at different bases. Further, when the user does not perform a gesture, a face image having a data amount smaller than that of the upper body image is transmitted, so that the communication load can be reduced.

  In addition, in the communication terminal device according to the second aspect of the invention, in addition to the effect of the first aspect of the invention, the motion detection means detects a state where both the palm and the arm have a predetermined amount of movement as a motion. . That is, it is possible to detect only a large motion that is greater than or equal to a predetermined amount of the user without detecting it as a motion such as a small shaking of the upper body of the user.

  In addition, in the communication terminal device of the invention according to claim 3, in addition to the effect of the invention according to claim 1 or 2, the motion detection means uses a state in which the shape of the palm changes more than a predetermined amount as the movement of the palm. To detect. Thereby, when the user moves his / her palm and expresses emotions, the upper body image including the palm image can be displayed on the display screen. Furthermore, since detection is performed only when there is a change in the palm shape by a predetermined amount or more, if there is a small change in the palm shape, it is not detected as movement of the palm.

  Further, in the communication terminal device according to a fourth aspect of the invention, in addition to the effect of the invention according to any one of the first to third aspects, the motion detecting means detects the movement of the arm when the position of the arm has changed by a predetermined amount or more. Detect as. Thereby, when the user moves his / her arm and expresses emotion, the upper body image including the image of the arm can be displayed on the display screen. Further, since the detection is performed only when the arm position has changed by a predetermined amount or more, the arm movement is not detected when there is a small change in the arm position.

  In the communication terminal device according to the fifth aspect of the present invention, in addition to the effect of the first aspect of the present invention, the first image range determining means further includes a nose position detecting means. The nose position detection means detects the target user's nose position from the captured image. The first image range determination unit determines the first image range so that the nose position detected by the nose position detection unit is a center point in the horizontal direction of the first image range. Thereby, a user's face image can always be located in the center of a display screen.

  Moreover, in the communication terminal device of the invention according to claim 6, in addition to the effect of the invention according to any one of claims 1 to 5, the speaker specifying means specifies a speaker from a plurality of users. The motion detection means detects a state where the movement of at least one of the palm and the arm of the speaker specified by the speaker specifying means is greater than or equal to a predetermined amount as the motion of the speaker. The first image range determining means determines a range of the upper body image including the speaker's face region, palm region, and arm region as the first image range. The second image range determining means determines the range of the face image including the speaker's face area as the second image range. Therefore, when a plurality of users are at one base, the first image range or the second image range can be designated for the speaker within the user.

In the communication terminal device according to the seventh aspect of the invention, in addition to the effect of the invention according to the sixth aspect, the person recognition means recognizes a person from the photographed image photographed by the photographing means. The mouth shape detection means detects a change in the mouth shape of the person recognized by the person recognition means. The speaker identification means is
A person whose mouth shape change is detected by the mouth shape detecting means by a predetermined amount or more is specified as a speaker. Thereby, even when there are a plurality of users at one base, it is possible to accurately identify the speaker from among them.

  In the communication terminal device according to the eighth aspect of the invention, in addition to the effect of the sixth aspect, the voice detecting means detects the voice of the user and detects the direction of the voice. The speaker specifying unit specifies a person in the direction detected by the voice detection unit as a speaker. Thereby, even when there are a plurality of users at one base, it is possible to accurately identify the speaker from among them.

  In the communication control device according to the ninth aspect of the present invention, the communication control device is connected to a plurality of communication terminal devices via a network and controls communication performed between the communication terminal devices. The photographed image receiving means receives a photographed image that is photographed by the photographing means of the communication terminal apparatus and transmitted from the communication terminal apparatus. The motion detection means detects a state in which at least one of the user's palm and arm moves as a user motion based on the captured image received by the captured image receiving means. The first image range determining means determines the range of the upper body area image including the user's face area, palm area, and arm area from the captured image when the motion is detected by the motion detecting means. Determine as. The second image range determining means determines, as the second image range, a face image range including the user's face area from the photographed images when no action is detected by the action detecting means. The image transmission unit transmits the image of the first image range determined by the first image range determination unit or the image of the second image range determined by the second image range determination unit to the communication terminal device. As a result, if no motion is detected by the motion detection means, an image of the user's face area is displayed on the display screen of the communication terminal device. If a motion is detected, the user's face area, palm area, and arm An image of the upper body area including the area can be displayed on the display screen. Since the user's image size can be adjusted according to the user's action, the facial expression and action can be reliably displayed on the display screen of the communication terminal device according to the user's action. Therefore, since the user's emotions can be expressed richly, good communication with the other party can be achieved. In addition, when the user does not operate, an image in the second image range having a smaller data amount than the image in the first image range is transmitted, so that the communication load can be reduced.

  In the communication control method for the communication terminal device according to the tenth aspect of the present invention, in the motion detection step, a state in which at least one of the user's palm and arm moves is detected as the user motion. When a motion is detected in the motion detection step in the first image range determination step, the user's face region, palm region, and arm region are included from the captured images captured by the capturing unit that captures the user. The range of the upper body region image is determined as the first image range. In the second image range determination step, when no motion is detected in the motion detection step, the range of the image of the user's face area is determined as the second image range from the captured images. In the image transmission step, the image in the first image range determined in the first image range determination step or the image in the second image range determined by the second image range determination means is transmitted to another communication terminal device. . In the image receiving step, an image transmitted from another communication terminal device is received. In the image display control step, when an image is received in the image reception step, the image is displayed on the display screen. As described above, when no motion is detected by the motion detection means, an image of the user's face area is displayed on the display screen, and when a motion is detected, the user's face area, palm area, and arm area are displayed. An image of the upper body area including the image is displayed on the display screen. Thereby, since a user's image size can be adjusted according to a user's operation | movement, an expression and an operation | movement can be reliably displayed on a display screen according to a user's operation | movement. Therefore, since the user's emotions can be expressed richly, good communication with the other party can be achieved. In addition, when the user does not operate, an image in the first image range having a smaller data amount than the image in the second image range is transmitted, so the communication load can be reduced.

  In the communication control program of the invention according to claim 11, since the computer functions as various processing means of the communication terminal device according to claim 1, the computer program according to claim 1. An effect can be obtained.

1 is a block diagram showing a configuration of a video conference system 1. FIG. 3 is a block diagram showing an electrical configuration of a terminal device 3. FIG. 3 is a conceptual diagram showing various storage areas of an HDD 31. FIG. 3 is a conceptual diagram showing various storage areas of a RAM 22. FIG. It is a figure which shows the one display aspect in the display (when the other party user is not gesturing). It is a figure which shows the one display mode in the display 28 (when the other party user is performing gesture). It is a figure which shows the range of the face image 51 in the picked-up image 50. FIG. FIG. 5 is a diagram illustrating a range of an upper body image 52 in a captured image 50. It is a flowchart of the image transmission process by CPU20. It is a flowchart of the face image range determination process by CPU20. It is a flowchart of the operation | movement detection process by CPU20. It is a flowchart of the palm movement detection process by CPU20. It is a flowchart of the arm motion detection process by CPU20. It is a flowchart of the upper body image range determination process by CPU20. It is a flowchart of the image reception process by CPU20. It is a block diagram which shows the electric constitution of the terminal device 130 of 2nd embodiment. 3 is a conceptual diagram showing various storage areas of a RAM 122. FIG. 3 is a flowchart of image transmission processing by a CPU 120.

  Hereinafter, the terminal device 3 which is 1st embodiment of this invention is demonstrated with reference to drawings. First, the configuration of the video conference system 1 including the terminal device 3 as a component will be described with reference to FIG.

  The video conference system 1 includes terminal devices 3 and 4 connected to each other via a network 2. The terminal devices 3 and 4 are provided at different bases. In this video conference system 1, a remote conference between users at different bases is performed by transmitting and receiving images and sounds between the terminal devices 3 and 4 via the network 2. In the present embodiment, the base where the terminal device 3 is provided will be described as its own base, and the base where the terminal device 4 is provided will be described as another base.

  In the present embodiment, when the user is talking while gesturing in the terminal device 3, the upper body image of the user of the terminal device 3 is transmitted to the terminal device 4, and the user talks without gesturing. If it is, the user's face image is transmitted to the terminal device 4.

  The electrical configuration of the terminal device 3 will be described with reference to FIG. FIG. 2 is a block diagram showing an electrical configuration of the terminal device 3. Since the terminal device 3 and the terminal device 4 all have the same configuration, only the configuration of the terminal device 3 will be described here, and the description of the terminal device 4 will be omitted.

  The terminal device 3 is provided with a CPU 20 as a controller that controls the terminal device 3. Connected to the CPU 20 are a ROM 21 that stores BIOS, a RAM 22 that temporarily stores various data, and an I / O interface 30 that mediates data transfer. The I / O interface 30 is connected to a hard disk drive 31 (hereinafter referred to as HDD 31) having various storage areas.

  The I / O interface 30 captures a communication device 25 for communicating with the network 2, a mouse 27, a video controller 23, a key controller 24, a camera 34 for photographing a user, and a user's voice. A microphone 35 and a CD-ROM drive 26 are connected to each other. The video controller 23 is connected to a display 28 that displays a partner user who uses the terminal device 4. A keyboard 29 is connected to the key controller 24.

  The CD-ROM 114 inserted into the CD-ROM drive 26 stores the main program of the terminal device 3, the communication control program of the present invention, and the like. When the CD-ROM 114 is introduced, these various programs are set up from the CD-ROM 114 to the HDD 31 and stored in a program storage area 313 (see FIG. 3) described later.

  Next, various storage areas of the HDD 31 will be described with reference to FIG. The HDD 31 has a captured image data storage area 311 for storing a captured image 50 (see FIGS. 7 and 8) captured by the camera 34 and display screen data for storing screen data displayed on the display 28 of the terminal device 3. At least a storage area 312, a program storage area 313 that stores various programs, a predetermined value storage area 314 that stores predetermined values necessary for executing the program, and other information storage areas 315 are provided.

  The program storage area 313 stores a main program of the terminal device 3, a conference support program for executing a remote conference with the terminal device 4, a communication control program of the present invention relating to image display, and the like. In the other information storage area 315, other information used in the terminal device 3 is stored. When the terminal device 3 is a dedicated machine that does not include the HDD 31, various programs are stored in the ROM 21.

  Next, various storage areas of the RAM 22 will be described with reference to FIG. The RAM 22 includes at least an image range storage area 221, a palm motion storage area 222, an arm motion storage area 223, a motion detection storage area 224, a connection terminal storage area 225, and a processed image storage area 226. . The image range storage area 221 stores the image range of the transmission image in the captured image 50. The palm motion storage area 222 stores a palm area storage area 2221 that stores the palm area of the user in the captured image 50 and a palm change storage that stores the presence or absence of a change in the shape of the user's palm detected based on the captured image 50. An area 2222 is provided. The arm motion storage area 223 is provided with a contour data storage area 2231 for storing the contour data of the user's arm and an arm change storage area 2232 for storing whether or not the position of the user's arm has changed. Yes. The motion detection storage area 224 stores whether or not a user motion has been detected. In the connection terminal storage area 225, the terminal ID of the connection terminal currently connected via the network 2 is stored. The processed image storage area 226 stores image data for performing image processing.

  Next, the screen displayed on the display 28 of the terminal device 3 is demonstrated with reference to FIG. 5 and FIG. On the display 28 of the terminal device 3, an image of the partner user transmitted from the terminal device 4 is displayed. When the other user is not making a gesture, an image of the other user's face is displayed on the display 28 as shown in FIG. When the other user is making a gesture, as shown in FIG. 6, an image of the upper body of the other user is displayed on the display 28.

  Next, a method for detecting a user's gesture will be described. In this embodiment, the state in which the user moves both the palm and the arm is detected as the state in which the user is making a gesture.

  First, a method for detecting a user's palm movement will be described. The palm movement is detected based on a user-captured image 50 (see FIG. 7) captured by the camera 34. First, the user's palm area is extracted from the user's captured image 50. If there is a certain change in the area of the extracted palm region, it is detected that there is a palm movement.

Various well-known methods can be applied to the palm region extraction method. For example, the method described in Japanese Patent Application Laid-Open No. 2003-346162 is applicable. First, the captured image 50 displayed in the RGB color system is converted into the HSV color system. The HSV color system includes three elements: a hue H (hue) representing the type of color, a saturation S (saturation) representing the vividness of the color, and a lightness V (value) representing the degree of brightness. A conversion method from the RGB color system to the HSV color system is described, for example, in “Image Analysis Handbook” (published by the University of Tokyo Press, pp. 485-491, 1991), supervised by Takagi and Shimoda. In addition, the range of the value of H, S, and V is as follows.
・ 0 ≦ H ≦ 2π
・ 0 ≦ S ≦ 1
・ 0 ≦ V ≦ 1

Next, in order to extract a place which is a hand region in the image, skin color extraction is performed. In the present embodiment, the threshold value of the skin color area is set as follows.
・ 0.11 <H <0.22,
・ 0.2 <S <0.5
Pixels whose hue H and saturation S are within the above threshold are extracted as skin color pixels.

  In order to separate the hand area and the background, binarization is performed on skin color pixels and non-skin color pixels. Then, in the obtained binary image, a skin color pixel portion having an area within a predetermined range is extracted as a palm region. The palm area is extracted every 1/30 seconds.

  If there is a change in the extracted palm area over a predetermined value, it is detected that there is a change in the palm shape. If there is no change over the predetermined value in the area, there is no change in the palm shape. Detected.

  Next, a method for detecting the movement of the user's arm will be described. The movement of the arm is detected based on the user's captured image 50 (see FIG. 7) captured by the camera 34. Detection of arm movement is performed by a well-known method, for example, “Extraction of human arm region and estimation of motion parameter under complex background using optical flow” (The Institute of Electrical Engineers of Japan C Volume, Vol. 120-C No. 12, pp. 1801-1808 (2000 12)), a method of extracting contour data from the user's captured image 50 and detecting it based on the contour data, or a user from the captured image 50 It is possible to apply a method of detecting the palm of the hand and detecting it based on the change in the position of the palm. In the present embodiment, a method is used in which contour data of the user's arm is extracted from the captured image 50 of the user, and when there is a certain change in the position of the extracted contour data, it is detected that there is a movement of the arm. To do.

  In this method, first, an extraction region for extracting contour data of the user's arm is specified from the captured image 50. Here, a skin color portion having a predetermined area or more existing in the photographed image 50 of the user is extracted as a face region, a skin color portion existing below the face region is extracted as a palm region, and the face region and palm region in the vertical direction are extracted. The area between the two is designated as an extraction area for extracting the contour data of the arm. The skin color portion extraction method is as described above.

  Next, the captured image 50 is converted into a gray scale, and contour data is extracted. When extracting contour data, a well-known first-order differential method is used. In the first-order differential contour extraction, the strength and direction of the contour are calculated by obtaining the density gradient in each pixel, and the portion where the density value changes rapidly is extracted as the contour data. The designation of the extraction area and the extraction of the contour data are performed every 1/30 seconds.

  If there is a change in the extracted contour data, it is detected that there is a movement of the arm, and if there is no change in the contour data over a predetermined value, it is detected that there is no movement of the arm.

  Next, a method of determining the range of the user's face area image (face image 51) from the captured image 50 will be described with reference to FIG. The method for determining the face image range is a well-known method, and for example, the method described in JP-A-10-334213 is applicable.

  First, the face area of the user is extracted from the captured image 50 in which the upper body of the user is captured. When extracting a user's face area from the captured image 50, first, the captured image 50 displayed in the RGB color system is converted into the HSV color system. Then, pixels whose hue H and saturation S are within the above-described threshold are extracted as skin color pixels from the converted image converted into the HSV color system. In order to separate the face area from the background, binarization is performed on skin color pixels and non-skin color pixels. The conversion method to the HSV color system, the skin color pixel extraction method, and the binarization method are the same as those described in the palm region extraction method. Then, in the obtained binary image, the skin color pixel portion existing in the upper half of the image is extracted as a face region.

  Next, a face image range for cutting out the face image 51 is determined. When determining the face image range, first, the user's nose position is specified from the captured image 50. The nose position is specified by detecting two adjacent nostril portions in the center of the face area. Since the nostril portion is not irradiated with light, it is photographed darkly. In the photographed image, a darkly photographed portion at the center of the face area is detected as a nostril, and the center position of the detected nostril is specified as the nose position. Then, the XY coordinates (x3, y3) in the captured image 50 at the nose position are specified with the lower left portion of the image area of the captured image 50 as the origin of the XY coordinates in the captured image 50.

Next, the maximum value and minimum value of the X coordinate and the maximum value and minimum value of the Y coordinate of the extracted face area are detected, and the difference from the nose position (x3, y3) is calculated for each value. The largest value among the calculated differences is defined as a first difference α, and a first enlarged value N is calculated by adding a predetermined value to the first difference α. Using the first enlargement value N, the range of the face image 51 is determined such that the nose position (x3, y3) is the center of the face image 51. Specifically, the coordinates of the four corners of the face image 51 are determined as follows.
・ (X3 + N, y3 + N)
・ (X3 + N, y3-N)
・ (X3-N, y3 + N)
・ (X3-N, y3-N)

  Next, a method for determining the range of the upper body region image (upper body image 52) of the user from the captured image 50 will be described with reference to FIG. First, from the captured image 50, the color, geometric shape, shading pattern, movement, and the like of the captured image are extracted as parameters. Next, a database of human image parameters stored in advance in the other information storage area 315 of the HDD 31 is referred to, and matching processing between the extracted parameters described above and the human image parameters stored in the database is performed. Executed. Then, the portion of the photographed image that matches well with the parameters of the person image stored in the database is specified. The image of the specified part is specified as the user area. Next, the maximum value and the minimum value of the X coordinate of the user area are detected using the lower left portion of the image area of the captured image 50 as the origin of the XY coordinates in the captured image 50.

  Next, the user's nose position is detected from the captured image 50. The method for specifying the nose position is as described above. Then, the difference between the detected X coordinate x3 of the nose position and the maximum value and the minimum value of the detected X coordinate of the user area is calculated. A larger value of the differences is set as the second difference β, and a second enlarged value M that is a value obtained by adding the second difference β and the predetermined value U is calculated.

Then, the range of the upper body image 52 is determined using the nose position (x3, y3) and the second enlarged value M as parameters. In the present embodiment, the nose position (x3, y3) is positioned at the center in the left-right direction (X direction) of the upper body image 52 and is positioned at 2/3 from the lower end in the vertical direction (Y direction). The upper body image 52 is determined. Specifically, the coordinates of the four corners of the upper body image 52 are expressed as follows.
(X3 + M, y3 + (M × (2/3)))
(X3 + M, y3- (M × (4/3)))
(X3-M, y3 + (M × (2/3)))
(X3-M, y3- (M × (4/3)))

  A communication control process in consideration of the user's gesture in the video conference system 1 will be described with reference to the flowcharts of FIGS. In this description, it is assumed that the terminal device 3 on the local site side and the terminal device 4 on the other site side hold a conference. In the terminal devices 3 and 4, an “image transmission process” in which the user's face image 51 or upper body image 52 is cut out from the captured image 50 captured by the camera 34 and transmitted to another terminal device; Both “image reception processing” in which an image transmitted by the apparatus is received and displayed is performed. Therefore, for convenience of explanation, an example will be described in which “image transmission processing” is executed in the terminal device 3 on the local site side and “image reception processing” is executed on the terminal device 4 on the other site side.

  First, image transmission processing executed by the CPU 20 of the terminal device 3 on the local site side will be described. When the terminal device 3 and the terminal device 4 are connected to the network and start communication with each other, the image transmission process shown in FIG. 9 is started. When the image transmission process is started, first, the camera 34 is driven, and acquisition of the captured image 50 captured by the camera 34 is started (S1). Image data of the photographed image 50 photographed by the camera 34 is stored in the photographed image data storage area 311.

  When acquisition of a captured image is started (S1), face image range determination processing is performed (S2). The face image range determination process will be described with reference to FIG. This face image range determination process is a subroutine executed in S2 of the image transmission process of FIG.

  When the face image range determination process is started, first, the captured image 50 that is the RGB color system is converted into the HSV color system, and is stored as a converted image in the processed image storage area 226 (S151). The conversion method to the HSV color system is as described above. Then, from the converted image stored in the processed image storage area 226, pixels whose hue H and saturation S are within the above threshold are extracted as skin color pixels (S152). The skin color pixel extraction method is as described above. In order to separate the face area from the background, the skin color pixels and the non-skin color pixels are binarized (S153). The skin color pixel portion present in the upper half of the image is specified as the face region.

  Next, the processed image storage area 226 is referred to, and the user's nose position is specified (S154). The nose position is specified by detecting two nostril portions adjacent to each other in the center of the face area. Since the nostril portion is not irradiated with light, it is photographed darkly. A darkly photographed portion at the center of the face area is detected as a nostril, and the center position of the detected nostril is specified as the nose position. Then, the XY coordinates (x3, y3) in the photographed image 50 at the nose position are specified with the lower left portion of the image area of the photographed image 50 as the origin of the XY coordinates in the photographed image 50, and the nose position storage area in the image range storage area 221 (Not shown).

Next, for the face area, the maximum value x1 and minimum value x2 in the X coordinate, and the maximum value y1 and minimum value y2 in the Y coordinate are detected (S155). The detected x1, x2, y1, and y2 are stored in the face area storage area (not shown) of the image range storage area 221. Then, referring to the face area storage area and the nose position storage area, the difference a between x1 and x3, the difference b between x3 and x2, the difference c between y1 and y3, and the difference d between y3 and y2 are calculated. (S156). Differences a, b, c, and d can be expressed by the following equations.
A = x1-x3
B = x3-x2
C = y1-y3
D = y3-y2
The largest value among the four calculated differences a, b, c, and d is set as the first difference α and stored in the first difference storage area (not shown) of the image range storage area 221.

Then, a first enlarged value N that is a value obtained by adding the first difference α and the predetermined value K stored in the predetermined value storage area 314 is calculated (S157). The first enlarged value N is represented by the following formula.
・ N = α + K
The calculated first enlarged value is stored in a first enlarged value storage area (not shown) of the image range storage area 221.

Next, with reference to the first enlarged value storage area and the nose position storage area, an N large X coordinate (x3 + N) and an N small X coordinate (x3−N) are calculated from the X coordinate x3 of the nose position. Is done. From the Y coordinate y3 of the nose position, an N large Y coordinate (y3 + N) and an N small Y coordinate (y3-N) are calculated. (X3 + N) is the maximum value of the X coordinate in the face image 51, and (x3-N) is the minimum value of the X coordinate in the face image 51. (Y3 + N) is the maximum value of the Y coordinate in the face image 51, and (y3-N) is the minimum value of the Y coordinate in the face image 51. Then, the coordinates of four points made up of the combination of the X coordinate and the Y coordinate are calculated. The coordinates of the four points are expressed as follows.
・ (X3 + N, y3 + N)
・ (X3 + N, y3-N)
・ (X3-N, y3 + N)
・ (X3-N, y3-N)

  The coordinates of the four points are stored in the image range storage area 221 as information indicating the range of the face image 51 (S158). Then, the face image range determination process ends, and the process returns to the image transmission process (see FIG. 9).

  When the face image range determination process (S2) is completed, the captured image data storage area 311 and the image range storage area 221 are referred to, and the image data corresponding to the user's face image 51 is transferred to the terminal device 4 used by the partner user. It is transmitted (S3). When image data corresponding to the face image 51 is transmitted to the terminal device 4 (S3), an operation detection process for detecting the presence or absence of the user's operation is performed (S4).

  The operation detection process will be described with reference to FIG. The motion detection process is a subroutine executed in S4 of the image transmission process in FIG. When the motion detection process is started, first, the palm motion detection process is started (S51). The palm motion detection process is a subroutine executed in S51 of the motion detection process shown in FIG.

  The palm motion detection process will be described with reference to FIG. When the palm motion detection process is started, first, the captured image 50 which is the RGB color system is converted into the HSV color system and stored as a converted image in the processed image storage area 226 (S101). Next, the processed image storage area 226 is referred to, and skin color extraction is performed from the converted image (S102). Then, the skin color pixel and the non-skin color pixel are binarized, and the obtained binary image is stored in the processed image storage area 226 (S103). Then, the predetermined value storage area 314 and the processed image storage area 226 are referred to, and a skin color pixel portion having an area in a predetermined range is extracted as a palm region in the binary image (S104). The palm area is extracted every 1/30 seconds.

  Then, the area of the extracted palm region is calculated and stored in the palm area storage area 2221 of the palm motion storage area 222 (S105). For example, 60 storage areas are provided in the palm area storage area 2221, and the area (palm area) of the palm region is stored in each storage area every 1/30 seconds. When the palm area is stored in the 60th storage area, the latest palm area is overwritten in the first storage area. Then, when a predetermined amount of data is accumulated, the palm area storage area 2221 is referred to and, as an example, whether or not the ratio of the minimum palm area to the stored maximum palm area is less than 3/4. Determination is made (S106). If the ratio of the minimum palm area to the stored maximum palm area is less than 3/4 as an example, it is assumed that there is a change in the shape of the palm (the movement of the palm is detected). Is stored (S107). On the other hand, if the ratio of the minimum palm area to the stored maximum palm area is 3/4 or more, the palm shape is not changed (the palm movement is not detected), and the palm change storage area 2222 is detected. "0" is stored in (S108). Then, the palm motion detection process is terminated, and the process returns to the motion detection process (see FIG. 11).

  When the palm motion detection process (S51) ends, the palm change storage area 2222 is referred to and it is determined whether or not a palm movement is detected (S52). If “0” is stored in the palm change storage area 2222 and it is determined that the movement of the palm is not detected (S52: NO), it is determined that the user's movement is not detected, and the movement detection storage area “0” is stored in 224 (S56). When “1” is stored in the palm change storage area 2222 and it is determined that the movement of the palm is detected (S52: YES), the arm motion detection process is subsequently performed (S53).

  The arm motion detection process will be described with reference to FIG. The arm motion detection process is a subroutine executed in S53 of the motion detection process of FIG. When the motion detection process is started, first, an extraction area for extracting contour data of the user's arm is designated (S131). An area between the face area and the palm area in the vertical direction is extracted as a face area, and a skin color area existing below the face area is extracted as a palm area. Is designated as an extraction region for extracting arm contour data. The skin color portion extraction method is as described above.

  Next, the captured image 50 is converted into a gray scale, and contour data is extracted (S132). When extracting contour data, a well-known first-order differential method is used. In the first-order differential contour extraction, the strength and direction of the contour are calculated by obtaining the density gradient in each pixel, and the portion where the density value changes rapidly is extracted as the contour data. The designation of the extraction area and the extraction of the contour data are performed every 1/30 seconds.

  The extracted contour data is stored in the contour data storage area 2231 of the arm motion storage area 223 of the RAM 22 (S133). In the contour data storage area 2231, for example, 60 storage areas are provided, and the contour data is stored in each storage area every 1/30 seconds. When the contour data is stored in the 60th storage area, the latest contour data is overwritten in the first storage area. When a predetermined amount of data is accumulated in the contour data storage area 2231, 60 storage areas are referred to, and the difference ΔX1 between the maximum value and the minimum value in the X coordinate of the stored plurality of contour data, A difference ΔY1 between the maximum value and the minimum value of the Y coordinate is calculated (S134).

  Then, it is determined whether either the difference ΔX1 or the difference ΔY1 is larger than the predetermined value stored in the predetermined value storage area 314 of the HDD 31 (S135). If either the difference ΔX1 or the difference ΔY1 is larger than the predetermined value stored in the predetermined value storage area 314 of the HDD 31 (S135: YES), there is a change in the arm position (the movement of the arm is detected). "1" is stored in the arm change storage area 2232 (S136). If either the difference ΔX1 or the difference ΔY1 is smaller than the predetermined value stored in the predetermined value storage area 314 of the HDD 31 (S135: NO), there is no change in the arm position (the movement of the arm is not detected). "0" is stored in the arm change storage area 2232 (S137). Then, the arm motion detection process ends, and the process returns to the motion detection process (see FIG. 11).

  In the motion detection process, when the arm motion detection process is completed (S53), the arm change storage area 2232 is referred to and it is determined whether or not an arm movement is detected (S54). When “0” is stored in the arm change storage area 2232 and no movement of the arm is detected (S54: NO), it is determined that the user's movement is not detected, and “0” is stored in the movement detection storage area 224. Is stored (S56). When “1” is stored in the arm change storage area 2232 and it is determined that the movement of the arm is detected (S54: YES), it is determined that the user's motion is detected, and “ 1 "is stored (S55). Then, the motion detection process is terminated, and the process returns to the image transmission process (see FIG. 9).

  Returning to FIG. 9, when the motion detection process (S4) ends, the motion detection storage area 224 is referred to and it is determined whether or not a user motion has been detected (S5). When “0” is stored in the motion detection storage area 224, it is determined that the motion has not been detected (S5: NO), and it is determined whether the conference is over (S12). The determination as to whether or not the conference is ended is made by determining whether or not there are one or more terminal devices connected to the network 2 other than itself.

  When the terminal devices 3 and 4 are connected to the network 2, a connection signal indicating connection to the network 2 is transmitted to the counterpart terminal device. When a connection signal is received from another terminal device, the terminal ID of the terminal device that has transmitted the connection signal is stored in the connection terminal storage area 225 of the RAM 22. On the other hand, when the connection to the network 2 is disconnected, a disconnection signal indicating that the connection to the network is disconnected is transmitted to the partner terminal device. If the terminal device connected to the network 2 is 0 in addition to itself (S12: YES), the process is terminated. On the other hand, when one or more terminal devices are connected to the network 2 (S12: NO), the processes of S1 to S5 are repeated. That is, when the user's gesture is not detected in the terminal device 3 (S5: NO), the user's face image 51 is continuously transmitted to the terminal device 4 (S3).

  When “1” is stored in the motion detection storage area 224, it is determined that a motion has been detected (S5: YES), and the user's upper body image 52 is determined from the captured image 50 stored in the captured image data storage area 311. The upper-body image range determination process is performed in which the range of S is determined (S6).

  The upper body image range determination process will be described with reference to FIG. The upper body image range determination process is a subroutine executed in S6 of the image transmission process of FIG. As shown in FIG. 14, when the upper body image range determination process is started, first, the user area in the captured image is specified (S171). Specifically, the captured image data storage area 311 is referred to, and the color, geometric shape, shading pattern, movement, etc. of the captured image are extracted as parameters. A database of human image parameters stored in advance in the other information storage area 315 of the HDD 31 is referred to, and the above-described extracted parameters are matched with the parameters of the human image stored in the database. The Then, the portion of the photographed image that matches well with the parameters of the person image stored in the database is specified. The image of the specified part is specified as the user area. Next, the maximum value x4 and the minimum value x5 in the X coordinate of the user area are detected using the lower left portion of the image area of the captured image 50 as the origin of the XY coordinates in the captured image 50 (S172). The detected x4 and x5 are stored in the user area storage area (not shown) of the image range storage area 221.

Next, the user's nose position is specified from the captured image 50 (S173). The method for specifying the nose position is as described above. The detected X coordinate x3 of the nose position is stored in the nose position storage area (not shown) of the image range storage area 221. Then, the nose position storage area and the user area storage area are referred to, and the difference e between x4 and x3 and the difference f between x3 and x5 are calculated from x3, x4, and x5 (S174). The differences e and f can be expressed by the following equations.
E = x4-x3
・ F = x3-x5

The larger value of the calculated two differences e and f is set as the second difference β and stored in the second difference storage area (not shown) of the image range storage area 221. A second enlarged value M, which is a value obtained by adding the second difference β and the predetermined value U stored in the predetermined value storage area 314, is calculated (S175). The second enlarged value M is represented by the following formula.
・ M = β + U
The calculated second enlarged value M is stored in a second enlarged value storage area (not shown) of the image range storage area 221.

Next, with reference to the second enlarged value storage area and the nose position storage area, an M large X coordinate (x3 + M) and an M small X coordinate (x3-M) are calculated from the X coordinate x3 of the nose position. Is done. From the Y coordinate y3 of the nose position, (M × (2/3)) large Y coordinate (y3 + (M × (2/3))) and (M × (4/3)) small Y coordinate (y3− ( M × (4/3))) is calculated. (X3 + M) is the maximum value of the X coordinate in the upper body image 52, and (x3-M) is the minimum value of the X coordinate in the upper body image 52. (Y3 + (M × (2/3))) is the maximum value of the Y coordinate in the upper body image 52, and (y3− (M × (4/3))) is the minimum value of the Y coordinate in the upper body image 52. . Then, the coordinates of four points made up of the combination of the X coordinate and the Y coordinate are calculated. The coordinates of the four points are expressed as follows.
(X3 + M, y3 + (M × (2/3)))
(X3 + M, y3- (M × (4/3)))
(X3-M, y3 + (M × (2/3)))
(X3-M, y3- (M × (4/3)))

  The coordinates of the four points are stored in the image range storage area 221 as information indicating the range of the upper body image 52 (S176). Then, the upper body image range determination process ends, and the process returns to the image transmission process (see FIG. 9).

  Returning to FIG. 9, when the upper body image range determination process (S6) is completed, the captured image data storage area 311 and the image range storage area 221 are referred to, and the range of the upper body image 52 is cut out from the captured image 50. 4 (S7). That is, when it is determined that the user of the terminal device 3 is making a gesture (S5: YES), an image of the upper body of the user is transmitted to the terminal device 4 (S7).

  When the user's upper body image 52 is transmitted to the terminal device 4 (S7), the connected terminal storage area 225 is referred to, and whether or not there are one or more terminal devices connected to the network 2 other than itself. Is determined (S8). If the terminal device connected to the network 2 is 0 in addition to itself, it is determined that the conference has ended (S8: YES), and the process ends. On the other hand, when there are one or more terminal devices connected to the network 2, the operation detection process for detecting the presence / absence of the user's operation is performed again, assuming that the conference has not ended (S8: NO). (S9). Since the process of S9 is the same as S4, description is abbreviate | omitted.

  When the motion detection process (S9) ends, the motion detection storage area 224 is referred to and it is determined whether or not a user motion has been detected (S10). If “1” is stored in the motion detection storage area 224, it is determined that motion has been detected (S10: YES), and the user's face area and palm area are included in the range of the cut out upper body image 52. (S11) Specifically, first, the captured image data storage area 311 is referred to, and the face area and palm area of the user are extracted from the captured image 50. The method for extracting the face area and the palm area is as described above. Then, the maximum value and the minimum value of the X and Y coordinates of the extracted face area and palm area are detected. It is determined whether or not the detected maximum value and minimum value are within the image range stored in the image range storage area 221.

  When it is determined whether or not the detected palm area and face area are both within the image range stored in the image range storage area 221 (S11: YES), the process returns to S8 to determine whether the meeting is over. Is judged. If it is determined that either the face area or the palm area is outside the image range stored in the image range storage area 221 (S11: NO), the captured image stored in the captured image data storage area 311 is again displayed. The range of the upper body image 52 of the user from 50 is determined (S6). The determined range of the upper body image 52 is overwritten in the image range storage area 221 as the latest image range.

  On the other hand, if “0” is stored in the motion detection storage area 224 in S10, it is determined that no user motion has been detected (S10: NO), and it is determined whether or not the conference has ended. (S12). If the terminal device connected to the network 2 is 0 in addition to itself, it is determined that the conference is over (S12: YES), and the process is terminated. If there are one or more terminal devices connected to the network 2, the conference is not finished (S12: NO), and the process returns to S1. That is, in the terminal device 3, when the user's operation is not detected (S 10: NO) and the conference is not ended (S 12: NO), the user's face image 51 instead of the user's upper body image 52 is displayed on the terminal device 4. It is transmitted (S3).

  Next, image reception processing executed by the CPU 20 of the terminal device 4 on the other base side will be described with reference to the flowchart of FIG. When the terminal device 3 and the terminal device 4 are connected to the network and start communication with each other, the image reception process shown in FIG. 15 is started. When the image reception process is started, it is determined whether an image transmitted from the terminal device 3 has been received (S31). When the image is not received (S31: NO), the process of S31 is repeated until the image is received. When an image is received (S31: YES), the received image is displayed on the display 28 by the video controller 23 (S32).

  As described above, when the user of the terminal device 3 is making a gesture, the terminal device 3 transmits image data corresponding to the upper body image 52 of the user. On the other hand, when the user of the terminal device 3 is not making a gesture, the terminal device 3 transmits image data corresponding to the face image 51 of the user. Therefore, when the user of the terminal device 3 is making a gesture, an image of the upper body of the user is displayed on the display 28 of the terminal device 4 (see FIG. 5). Is displayed (see FIG. 6).

  Then, the connected terminal storage area 225 is referred to, and it is determined whether there are one or more terminal devices connected to the network 2 other than itself (S33). When the terminal device connected to the network 2 is 0 in addition to itself (S33: YES), the process is terminated. On the other hand, when there are one or more terminal devices connected to the network 2 (S33: NO), the process returns to S31, and the processes of S31 to S33 are repeated.

  As described above, the terminal device 3 according to the first embodiment is connected to other terminal devices 4 via the network 2. A video conference system 1 that implements a remote conference by transmitting and receiving images and sounds between these terminal devices is configured. In the video conference system 1, the user's operation (gesture) of the terminal device 3 (or 4) is detected during the remote conference. And when the user is not gesturing, the data corresponding to the face image 51 of the user is transmitted to the terminal device 4 (or 3) of the other user. When the user is making a gesture, data corresponding to the upper body image 52 of the user is transmitted to the terminal device 4 (or 3) of the other user.

  Therefore, when the user of the terminal device 3 is making a gesture, an image of the upper body of the user is displayed on the display 28 of the terminal device 4 (see FIG. 5). A face image is displayed (see FIG. 6). Therefore, when a user tries to express an emotion with a gesture, the other user can confirm the user's gesture. Further, when the user does not perform a gesture, the partner user can check the facial expression of the user. Therefore, it is possible to take good communication between conference participants at different bases. Further, when the user does not perform a gesture, the image data of the face image 51 having a data amount smaller than that of the upper body image 52 is transmitted, so that the communication load can be reduced.

  In the above description, the camera 34 shown in FIG. 2 corresponds to the “user photographing unit” of the present invention. The display 28 shown in FIG. 2 corresponds to the “display screen” of the present invention. The CPU 20 that executes the face image range determining process shown in FIG. 10 corresponds to the “second image range determining means” of the present invention. The CPU 20 that executes the motion detection process shown in FIG. 11 corresponds to the “motion detection means” of the present invention. The CPU 20 that executes the upper body image range determination process shown in FIG. 14 corresponds to the “first image range determination means” of the present invention. The CPU 20 that executes the processes of S3 and S7 shown in FIG. 9 corresponds to the “image transmission means” of the present invention. The CPU 20 that executes the process of S172 shown in FIG. 14 and the CPU 20 that executes the process of S154 shown in FIG. 10 correspond to the “nasal position detecting means” of the present invention. The CPU 20 that executes the process of S31 shown in FIG. 15 corresponds to the “image receiving means” of the present invention. The CPU 20 that executes the process of S32 shown in FIG. 15 corresponds to the “display control means” of the present invention.

  Next, the terminal device 130 which is 2nd embodiment of this invention is demonstrated. In the first embodiment, it is assumed that the terminal devices 3 and 4 are conferences that are performed under the condition of one user. The second embodiment is different from the first embodiment in that when there are a plurality of users in the terminal devices 3 and 4 at each base, a speaker is specified among them and the speaker is set as an imaging target of the camera 34. Different. In addition, the terminal device 130 of 2nd embodiment comprises the video conference system 1 shown in FIG. 1 similarly to the terminal device 3 of 1st embodiment.

  First, the electrical configuration of the terminal device 130 will be described with reference to FIG. The terminal device 130 is provided with a CPU 120 as a controller that controls the terminal device 130. Connected to the CPU 120 are a ROM 121 that stores BIOS, a RAM 122 that temporarily stores various data, and an I / O interface 30 that mediates data transfer. Connected to the I / O interface 30 are a hard disk drive 131 (hereinafter referred to as HDD 131) having various storage areas, an audio direction detection device 36, and a drive circuit 37. A microphone 35 to which a user's voice is input is connected to the voice direction detection device 36. The sound direction detection device 36 detects the direction of the sound source from which the sound is emitted based on the phase difference of the sound input to the microphone 35. A camera moving device 38 that rotates the camera 34 is connected to the drive circuit 37.

  As shown in FIG. 17, the RAM 122 is provided with a sound source direction storage area 227 for detecting the direction of the sound source in addition to various storage areas similar to the RAM 22 of the first embodiment (see FIG. 4). The other electrical configuration of the terminal device 130 has the same configuration as the terminal device 3 (see FIG. 2) of the first embodiment.

  The speaker identification method will be described. Various known methods can be applied as the speaker identification method. For example, the method described in JP-A-11-341334 and JP-A-2001-339703 can be applied. In this modification, first, the direction of the sound source from which the sound is emitted is detected by the sound direction detection device 36 based on the phase difference of the sound input to the microphone 35. Then, the camera moving device 38 takes a picture so as to photograph the direction of the detected sound source. Next, the shooting range of the camera 34 is narrowed.

  Then, the color, geometric shape, shading pattern, movement, etc. of the captured image are extracted as parameters. Next, a database of human image parameters stored in advance in the other information storage area 315 of the HDD 31 is referred to, and matching processing between the extracted parameters described above and the human image parameters stored in the database is performed. Executed. Then, the portion of the photographed image that matches well with the parameters of the person image stored in the database is specified. The identified part image is identified as the image of the speaker.

  Next, image transmission processing by the CPU 120 will be described with reference to the flowchart of FIG. Also in the present embodiment, the terminal device 130 and the terminal device 4 perform both “image transmission processing” for transmitting an image and “image reception processing” for receiving an image. The “image reception process” is the same as that in the first embodiment, and thus description thereof is omitted.

  When the terminal device 130 and the terminal device 4 are connected to the network and start communication with each other, the image transmission process shown in FIG. 18 is started. When the image transmission process is started, first, the camera 34 is driven, and acquisition of a captured image captured by the camera 34 is started (S71). The captured image captured by the camera 34 is stored in the captured image data storage area 311.

  Next, it is determined whether or not sound is input from the microphone 35 (S84). When no sound is input from the microphone 35, it is determined that there is no speaker on the local site side (S84: NO), and the processes of S71 and S84 are repeated.

  On the other hand, when the voice is input from the microphone 35, it is determined that there is a speaker on the local site side (S84: YES), and the voice direction detection device 36 determines the phase difference of the voice input from the microphone 35. The direction of the sound source from which the sound is emitted is detected. The direction of the sound source is stored in the sound source direction storage area 227. Then, the sound source direction storage area 227 is referred to, the camera moving device 38 is driven by the drive circuit 37, and the shooting direction of the camera 34 is directed to the direction of the sound source. Then, the image of the speaker is specified by the matching process (S85). The identified speaker image is stored in the captured image data storage area 311.

  Next, a face image range determination process is performed to determine the range of the speaker's face image from the captured image stored in the captured image data storage area 311 (S72). Since the face image range determination process is the same as that of the first embodiment, the description thereof is omitted. When the face image range determination processing is completed (S72), the photographed image data storage area 311 and the image range storage area 221 are referred to, and the speaker's face image is cut out from the photographed image as a transmission image and is sent to the terminal device 4. It is transmitted (S73).

  When the face image of the speaker is transmitted to the terminal device 4 (S73), an action detection process for detecting the presence or absence of the action of the speaker is performed (S74). Since the operation detection process is the same as that of the first embodiment, the description thereof is omitted. When the motion detection process (S74) ends, the motion detection storage area 224 is referred to, and it is determined whether or not the speaker's motion is detected based on the photographed image (S75). If “0” is stored in the motion detection storage area 224, it is determined that no motion has been detected (S75: NO), and it is determined whether the conference is over (S82).

  If the conference is over (S82: YES), the process is terminated. On the other hand, when there are one or more terminals connected to the network 2 (S82: NO), the processes of S71 to S75 are repeated. That is, in the terminal device 130, when the operation of the speaker is not detected (S75: NO) and the conference is not ended (S82: NO), the speaker's face image is continuously transmitted to the terminal device 4 as a transmission image ( S73).

  If “1” is stored in the motion detection storage area 224, it is determined that a motion has been detected (S75: YES), and the upper body image of the speaker is extracted from the captured image stored in the captured image data storage area 311. Upper body image range determination processing for determining the range is performed (S76). The upper body image range determination process is the same as that in the first embodiment, and a description thereof will be omitted. When the upper body image range determination process is completed, the photographed image data storage area 311 and the image range storage area 221 are referred to, and the upper body image of the speaker is cut out from the photographed image as a transmission image and transmitted to the terminal device 4. (S77). That is, when the operation of the speaker is detected in the terminal device 130 (S75: NO), the upper body image of the speaker is transmitted to the terminal device 4 (S77).

  When the upper body image of the speaker is transmitted to the terminal device 4 (S77), the connected terminal storage area 225 is referred to and whether or not there are one or more terminals connected to the network 2 other than the terminal itself. Determination is made (S78). If the terminal connected to the network 2 is 0 in addition to itself, it is determined that the conference has ended (S78: YES), and the process ends. On the other hand, when there are one or more terminals connected to the network 2, it is assumed that the conference has not ended (S 78: NO), and the operation detection process for detecting the presence or absence of the speaker's operation is performed again. (S79).

  When the motion detection process (S79) ends, the motion detection storage area 224 is referred to and it is determined whether or not a speaker's motion has been detected (S80). If “1” is stored in the motion detection storage area 224, it is determined that motion has been detected (S80: YES). If it is determined that an action has been detected (S80: YES), it is determined by the same method as in the first embodiment whether or not the speaker's face area and palm area are included in the clipped upper body image range. (S81).

  When it is determined whether or not the maximum value and minimum value of the detected palm area and face area are all within the image range stored in the image range storage area 221 (S81: YES), the process returns to S78. It is determined whether the conference is over. If it is determined that either the face region or the palm region is outside the image range stored in the image range storage area 221 (S81: NO), the process returns to S71 and the process is repeated.

  In S80, when “0” is stored in the motion detection storage area 224, it is determined that the speaker's motion has not been detected (S80: NO), and it is determined whether or not the conference has ended (S82). ). If there is no terminal connected to the network 2 other than the terminal device at the local site, it is determined that the conference is over (S82: YES), and the process is ended. On the other hand, if there are one or more terminals connected to the network 2, it is determined that the conference has not ended (S82: NO), and the process returns to S71.

  As described above, when there are a plurality of users at one site, the terminal device 130 according to the second embodiment identifies the speaker from among the users, and uses the other person's face image or upper body image as the other terminal device. Send to. Therefore, when a speaker tries to express his / her emotions with gestures, a user at another base can check the speaker's gestures. Further, when the speaker does not perform a gesture, the other user can confirm the facial expression of the speaker. Therefore, favorable communication can be taken between conference participants at different bases.

  The CPU 120 that performs the process of S85 shown in FIG. 18 corresponds to the “speaker specifying means” of the present invention. In the process of S85 shown in FIG. 18, the CPU 120 that recognizes a person image from a captured image corresponds to the “person recognition unit” of the present invention. The voice direction detection device 36 shown in FIG. 16 corresponds to “voice detection means” of the present invention.

  The present invention is not limited to the first and second embodiments described above, and various modifications can be made. In the above-described embodiment, the terminal device 3 performs the image transmission process in which a face image or an upper body image is cut out from the acquired captured image and the cut-out image is transmitted to another terminal device 4. Further, the terminal device 4 has received a captured image transmitted from another terminal device 3. However, the present invention is not limited to this configuration and may have other configurations.

  For example, when an MCU (Multipoint Control Unit) that controls the entire video conference is connected to the network 2, the following processing may be performed by the terminal device 3, the terminal device 4, and the MCU. The terminal device 3 performs processing for transmitting a captured image to the MCU. The MCU performs a process of cutting out the face image or the upper body image from the captured image transmitted from the terminal device 3 by the above-described method and transmitting the cut-out image to the terminal device 4. The terminal device 4 performs a process of receiving a captured image transmitted from the MCU and displaying it on the display 28. For convenience of explanation, the case where an image is transmitted in the terminal device 3 and an image is received in the terminal device 4 has been described as an example. However, in the terminal devices 3 and 4, a process of transmitting a captured image to the MCU Both image reception processing for receiving an image transmitted from the MCU is performed.

  In the first and second embodiments described above, for convenience of explanation, a video conference system including two terminal devices 3 and 4 has been described as an example. However, a video conference including two or more terminal devices is described. It is also applicable to the system.

  In the second embodiment, as a method for identifying a speaker from a plurality of users participating in the conference, the direction of the voice input to the microphone 35 is detected, and it is estimated that the speaker exists in the detected direction. The method used was used. The speaker specifying method is not limited to this, and for example, a change in the lip shape of a plurality of users may be detected from the captured image of the camera, and a user having a change in lip shape may be specified as the speaker.

  Therefore, a method for identifying a speaker from a change in lip shape will be described. In this method, first, from the captured image of the camera 34, the color, geometric shape, shading pattern, movement, etc. of the captured image are extracted as parameters. Next, a database of human image parameters stored in advance in the other information storage area 315 of the HDD 31 is referred to, and matching processing between the extracted parameters described above and the human image parameters stored in the database is performed. Executed. Then, the portion of the photographed image that matches well with the parameters of the person image stored in the database is specified. The image of the specified part is specified as a person image.

  Next, face areas are respectively detected from the images of a plurality of specified persons. Then, the contour data of the specified person image is extracted, and if there is a change in the extracted contour data in the lower half of the face area, it is detected that there is a change in the lip shape. The face area detection method and the contour data extraction method are as described above. Then, an image of a person whose lip shape is changed is specified as an image of a speaker. In this way, a speaker can be identified from a plurality of conference participants.

  In the first and second embodiments, the user's movement is detected when movement of both the palm and the arm is detected. However, at least one movement of the palm and the arm is detected. If it is detected, it may be that the user's action is detected. Specifically, in the motion detection process (see FIG. 11) of the first and second embodiments, when the movement of the palm is detected (S52: YES) and the movement of the arm is detected (S54: YES). It is assumed that the user's action has been detected (S55). However, the processes of S53 and S54 are not performed, and if the movement of the palm is detected (S52: YES), the user's movement is detected (S55), and the movement of the palm is not detected (S52: NO), the user's action may not be detected (S56). If the process of S51 and S52 is not performed and the movement of the arm is detected (S54: YES), the user's movement is detected (S55), and the movement of the arm is not detected (S54: NO). The user's action may not be detected (S56). If a palm movement is detected (S52: YES) or an arm movement is detected (S54: YES), a user action may be detected (S55).

1 video conference system 2 network 3 terminal device 4 terminal device 20 CPU
23 Video Controller 25 Communication Device 28 Display 30 Interface 31 Hard Disk Drive 34 Camera 35 Microphone 36 Voice Direction Detection Device 221 Image Range Storage Area 222 Palm Motion Storage Area 223 Arm Motion Storage Area 224 Motion Detection Storage Area 311 Captured Image Data Storage Area 312 Display screen data storage area

Claims (11)

A communication terminal device that communicates with other communication terminal devices connected via a network via images and sounds,
Photographing means for photographing the user;
Motion detection means for detecting, as a user motion, a state in which at least one of the user's palm and arm movements is a predetermined amount or more from a captured image captured by the imaging means;
When the motion is detected by the motion detection means, the first range of the upper body area image including the user's face area, palm area, and arm area is selected from the captured images captured by the imaging means. First image range determining means for determining the image range;
Second image range determining means for determining, as the second image range, an image range of the face area of the user from the captured image when the motion is not detected by the motion detecting means;
An image for transmitting the image of the first image range determined by the first image range determination unit or the image of the second image range determined by the second image range determination unit to the other communication terminal device A transmission means;
Image receiving means for receiving the image transmitted from the other communication terminal device;
An image display control means for displaying the image on a display screen when the image is received by the image receiving means.   The communication terminal apparatus according to claim 1, wherein the motion detection unit detects a state where both the palm and the arm have a predetermined amount or more of movement as the motion.   The communication terminal apparatus according to claim 1, wherein the motion detection unit detects a state in which the shape of the palm has a change of a predetermined amount or more as the movement of the palm.   The communication terminal apparatus according to claim 1, wherein the motion detection unit detects a state in which the position of the arm has changed by a predetermined amount or more as a movement of the arm. Further comprising nose position detection means for detecting the nose position of the target user from the captured image,
The first image range determining means includes
5. The first image range is determined such that the nose position detected by the nose position detection unit is a center point in the horizontal direction of the first image range. The communication terminal device according to claim 1. A speaker identification means for identifying a speaker from a plurality of users is provided.
The motion detection means detects a state where the movement of at least one of the palm and arm of the speaker specified by the speaker specifying means is greater than or equal to the predetermined amount, as the action of the speaker.
The first image range determining means determines an upper body image range including the speaker's face region, palm region, and arm region as the first image range,
The communication terminal apparatus according to claim 1, wherein the second image range determining unit determines a range of a face image including the face area of the speaker as the second image range. . Person recognizing means for recognizing a person from the photographed image photographed by the photographing means;
A mouth shape detecting means for detecting a change in the mouth shape of the person recognized by the person recognizing means,
The speaker specifying means includes:
The communication terminal apparatus according to claim 6, wherein a person whose change in mouth shape is detected by a predetermined amount or more by the mouth shape detecting unit is specified as the speaker. A voice detecting means for detecting the voice of the user and detecting the direction of the voice;
The speaker specifying means includes:
8. The communication terminal apparatus according to claim 6, wherein a person in the direction detected by the voice detection unit is specified as the speaker. A communication control device that is connected to a plurality of communication terminal devices via a network and controls communication performed between the communication terminal devices,
Photographed image receiving means for receiving a photographed image that is photographed by the photographing means of the communication terminal device and transmitted from the communication terminal device;
Based on the captured image received by the captured image receiving means, an action detecting means for detecting a state of movement of at least one of the user's palm and arm as the user's action;
When the motion is detected by the motion detection means, a first image range is determined as an image range of the upper body area including the user's face area, palm area, and arm area from the captured image. One image range determining means;
A second image range determining unit that determines a range of a face image including the face region of the user as a second image range from the captured image when the motion is not detected by the motion detection unit;
Image transmitting means for transmitting the image of the first image range determined by the first image range determining means or the image of the second image range determined by the second image range determining means to the communication terminal device And a communication control device. A communication control method for a communication terminal device that communicates with another communication terminal device connected via a network via an image and sound,
An operation detecting step of detecting a state in which at least one of the user's palm and arm is in motion as the user's operation;
When the motion is detected in the motion detection step, the range of the upper body region image including the user's face region, palm region, and arm region from among the captured images captured by the capturing unit that captures the user A first image range determination step for determining as a first image range;
A second image range determining step for determining, as a second image range, an image range of the face area of the user from the captured image when the motion is not detected in the motion detection step;
An image for transmitting the image of the first image range determined in the first image range determination step or the image of the second image range determined by the second image range determination means to the other communication terminal device Sending step;
An image receiving step of receiving the image transmitted from the other communication terminal device;
A communication control method for a communication terminal device, comprising: an image display control step for displaying the image on a display screen when the image is received in the image reception step.   A communication control program for causing a computer to function as various processing means of the communication terminal device according to claim 1.

Images