WO2010070820A1

WO2010070820A1 - Image communication device and image communication method

Info

Publication number: WO2010070820A1
Application number: PCT/JP2009/006382
Authority: WO
Inventors: 本田義雅; 岡田晋; 井藤好克
Original assignee: パナソニック株式会社
Priority date: 2008-12-17
Filing date: 2009-11-26
Publication date: 2010-06-24

Abstract

The objective is to prevent the obstruction of communication due to a warning tone or the display of one's own image, and to show a speaker the degree to which the speaker is shifted with respect to a reference point. Disclosed is an image communication device (100) which communicates image data with another device via a network (111), and which is equipped with: an image input unit (102) that acquires a photographic image picked up with a camera (101); a face detection unit (103) that detects the face region in the photographic image; an image transmission unit (105) that transmits first image data containing the photographic image; a image reception unit (106) that receives second image data containing the other party's image transmitted from another device; an image processing unit (108) that generates a processed image by processing the other party's image contained in the second image data; and an image output unit (109) that outputs the processed image to a monitor (110). The image processing unit (108) processes the other party's image such that there is a greater difference between the other party's image and the processed image as the difference between the position or size of the face region and a preset reference increases.

Description

Image communication apparatus and image communication method

The present invention relates to an image communication apparatus for conducting a TV (television) conference on a large screen, and more particularly to an image communication apparatus that allows a speaker to recognize his / her position.

In recent years, ADSL (Asymmetric Digital Subscriber Line) and optical fiber networks have spread rapidly, and low-speed and high-speed Internet connection has become available. Also, it has become possible to easily construct a TV conference system by bidirectionally transmitting video / audio data between remote bases using such a low-cost high-speed Internet. In addition, with the advent of cameras capable of capturing images with HD (High Definition) resolution and the increase in the size of displays typified by PDP (Plasma Display Panel), it is possible to display life-size people on a large screen display. As a result, a TV conference system with a sense of reality and high presence has appeared.

In a large-screen TV conference system with a high sense of realism, the viewing angle of the camera tends to be narrow in order to display a person in real size, and if the speaker moves greatly, the camera will move out of the imaging area, An image that does not include the speaker's face is transmitted. In a conventional TV conference system, an image captured by a camera is displayed as a self-portrait on a display, and the user moves to an image capturing area while confirming the self-image.

However, in a large-screen TV conference system with a face-to-face feeling, if the self-portrait is displayed, the area that can be displayed on the other party's image is reduced as the screen area, or the presence of face-to-face is reduced by displaying the self-portrait. There is a problem of doing it. Furthermore, there is also a problem that if the self-portrait is not always displayed, it cannot be noticed that it has moved out of the camera imaging area.

As a conventional method for solving this problem, Patent Document 1 discloses a technique for notifying an outside of an imaging area by generating a warning sound when a subject is outside the imaging area in a camera photographing apparatus. .

JP 2000-201348 A

However, in the above prior art, it is not clear how much the subject that is the speaker is deviated from the reference position such as the center of the imaging area of the camera. Since communication is hindered by sound reproduction, it is not suitable for communication applications represented by a TV conference with a large screen and high presence.

Therefore, the present invention prevents the communication from being interrupted due to the generation of a warning sound or the display of the self-portrait on the display, and visually indicates to the speaker how far the reference position is deviated. An object of the present invention is to provide an image communication apparatus that can be shown.

In order to solve the above-described conventional problems, an image communication apparatus according to the present invention is an image communication apparatus that communicates image data with another image communication apparatus via a network. An image input unit to be acquired, an image transmission unit that transmits first image data including a captured image acquired by the image input unit, and a face detection unit that detects a face region from the captured image acquired by the image input unit An image receiving unit that receives the second image data including the partner image transmitted from the other image communication device, and processes the partner image included in the second image data received by the image receiving unit. An image processing unit that generates a processed image, and an image output unit that outputs the processed image generated by the image processing unit to a display device, and the image processing unit is operated by the face detection unit. The difference between at least one of the position and size of the detected face area and a predetermined reference is calculated, and the partner image is processed so that the partner image and the processed image differ greatly as the calculated difference increases. To do.

According to this configuration, the larger the difference between the position or size of the face area and the predetermined reference, the larger the partner image displayed on the display device, so that the self-image is not displayed on the display device. By looking at the image of the other party, the speaker can check how much his / her position deviates from the reference position. Thereby, communication is not hindered by the display of a warning sound or a self-portrait, and a TV conference with a higher sense of reality can be performed.

Further, the face detection unit detects face region coordinates indicating the position of the face region in the captured image, and the image processing unit calculates an absolute value of a difference between the face region coordinates and a predetermined reference coordinate. When the calculated absolute difference value is greater than a predetermined threshold, the processed image is processed so that the larger the absolute difference value, the larger the counterpart image differs from the processed image. An image may be generated.

According to this configuration, since the other party image is processed based on the position of the face area in the captured image, the speaker can see how much he / she deviates from the reference position by looking at the other party's image displayed on the display device. It can be confirmed. At this time, if the amount of deviation from the reference position (difference absolute value) is less than or equal to a predetermined threshold, the image is not processed. If the user is within a predetermined range from the reference position, the image of the other party is displayed on the display device. Can be made.

Further, the image processing unit superimposes the superimposed image on the counterpart image such that the larger the absolute value of the difference between the face area coordinates and the reference coordinates, the larger the area of the predetermined superimposed image becomes. The processed image may be generated.

According to this configuration, since the processed image displayed on the display device is an image in which the superimposed image is superimposed on the partner image, the speaker can see the area of the superimposed image displayed on the display device by himself / herself. It can be confirmed how much is deviated from the reference position.

In addition, the image processing unit, when the face area coordinates are on the right side of the reference coordinates, from the left end of the counterpart image, or when the face area coordinates are on the left side of the reference coordinates, From the above, the superimposed image may be superimposed on the counterpart image so that the larger the absolute difference in the horizontal direction between the face region coordinates and the reference coordinates, the larger the area of the superimposed image.

According to this configuration, in order to control the area of the superimposed image to be superimposed on the partner image according to the horizontal shift direction and the distance from the reference coordinate, the speaker can position the superimposed image displayed on the display device. It is possible to confirm how much they are displaced in the horizontal direction by looking at the area. For example, the larger the distance from the reference coordinate, the larger the area of the superimposed image. Therefore, the larger the area of the superimposed image, the greater the deviation of the speaker from the reference position. In addition, when the face region coordinates are on the right side of the reference coordinates in the captured image, the superimposed image is superimposed from the left end of the partner image, so that the speaker can see the reference image by looking at the superimposed image displayed on the left side. You can recognize that you are on the left side of the position. Conversely, when the face area coordinates are on the left side of the reference coordinates in the captured image, the superimposed image is superimposed from the right end of the partner image, so that the speaker can see himself / herself by looking at the superimposed image displayed on the right side. It can be recognized that the user is on the right side of the reference position.

In addition, the image processing unit, when the face area coordinates are above the reference coordinates, from the upper end of the counterpart image, or when the face area coordinates are below the reference coordinates, From the lower end, the superimposed image may be superimposed on the counterpart image such that the larger the absolute value of the vertical difference between the face area coordinates and the reference coordinates, the larger the area of the superimposed image.

According to this configuration, in order to control the area of the superimposed image to be superimposed on the partner image according to the vertical deviation direction and distance from the reference coordinate, the speaker can position the superimposed image displayed on the display device. And the area, it is possible to confirm how much they are displaced in the vertical direction. For example, when the face area coordinates are above the reference coordinates in the captured image, the superimposed image is superimposed from the upper end of the partner image, so that the speaker can see the reference image by looking at the superimposed image displayed on the upper side. You can recognize that you are above the position. Conversely, if the face area coordinates are below the reference coordinates in the captured image, the superimposed image is superimposed from the lower end of the partner image, so the speaker can see the superimposed image displayed on the lower side, You can recognize that you are below the reference position.

Further, the face detection unit further determines whether or not a face region is in the captured image, generates a flag indicating the presence or absence of the face region, and the image processing unit determines that there is no face region. When the flag indicates, the superimposed image having a predetermined area may be superimposed on a predetermined region of the counterpart image.

According to this configuration, even if a face area is not detected, a superimposed image of a certain area is superimposed on the partner image, so that the speaker can see himself / herself by looking at the area of the superimposed image displayed on the display device. It can be confirmed that the user is outside the imaging area.

Further, the image communication apparatus further includes a buffer for storing the face area coordinates detected by the face detection unit, and the image processing unit has the flag indicating that the face area is not in the captured image. , When the face area coordinates stored in the buffer are on the right side of the reference coordinates, from the left end of the counterpart image, or when the face area coordinates are on the left side of the reference coordinates, from the right end of the counterpart image, and The face area coordinates stored in the buffer are predetermined from the upper end of the counterpart image when they are above the reference coordinates, or from the lower end of the counterpart image when they are below the reference coordinates. The superimposed image of the area may be superimposed on the counterpart image.

According to this configuration, even when a face area is not detected, a superimposed image of a certain area is superimposed on the partner image in accordance with the position of the face area detected last, so that the speaker can display on the display device. By looking at the position and area of the superimposed image, it is possible to confirm whether the user is outside the imaging area in the up, down, left, or right direction.

Further, the image processing unit generates the processed image by projective transforming the counterpart image so that the gradient of the projective transformation increases as the absolute difference between the face region coordinates and the reference coordinates increases. Also good.

According to this configuration, since the processed image displayed on the display device is an image obtained by projective transformation of the partner image, by looking at the inclination of the projective transformation of the processed image displayed on the display device, the speaker can You can check how far you are from the reference position.

In addition, the image processing unit may enlarge the counterpart image with a larger enlargement ratio as the difference absolute value between the face area coordinates and the reference coordinates is larger.

According to this configuration, since the processed image displayed on the display device is an image obtained by enlarging a part of the partner image, the speaker can see the degree of enlargement of the processed image displayed on the display device. It is possible to check how much the user is deviated from the reference position.

Further, the image communication apparatus further detects a face area from the counterpart image included in the second image data received by the image receiving unit, and uses the face area coordinates indicating the position of the detected face area as the reference coordinates. You may provide the reference coordinate setting part to set.

According to this configuration, image processing is performed using the reference coordinates as the coordinates of the face area of the partner image, and the speaker reference position is not fixed and can be set according to the partner position. That is, when the other party moves, the reference position of the speaker can be changed accordingly.

Further, the face detection unit detects the size of the face region, the image processing unit calculates a difference absolute value between the size of the face region and a predetermined reference size, and the calculated difference absolute value is When the threshold value is larger than a predetermined threshold, the processed image may be generated by processing the counterpart image so that the larger the difference absolute value is, the larger the counterpart image and the processed image are.

According to this configuration, since the partner image is processed based on the size of the face area in the captured image, the speaker can see how much he / she is out of the reference position by looking at the partner image displayed on the display device. It can be confirmed. At this time, if the amount of deviation from the reference position (difference absolute value) is less than or equal to a predetermined threshold, the image is not processed. If the user is within a predetermined range from the reference position, the image of the other party is displayed on the display device. Can be made.

Further, the image processing unit may generate the processed image by blurring the counterpart image so that the blur amount becomes larger as the difference absolute value between the size of the face area and the reference size is larger. .

According to this configuration, the blurring amount of the partner image is increased as the size of the face region is larger or smaller. Therefore, the speaker can see the reference level by looking at the degree of blur of the processed image displayed on the display device. It can be confirmed how much the position is deviated from the position. Specifically, the speaker can recognize that he is too close to the camera or too far away.

The image processing unit enlarges the counterpart image when the size of the face area is larger than the reference size, and reduces the counterpart image when the size of the face area is smaller than the reference size. Thus, the processed image may be generated, and the enlargement ratio or reduction ratio may be larger as the difference absolute value between the size of the face area and the reference size is larger.

According to this configuration, the larger the face area size is, the larger the partner image is, and the smaller the face area size is, the smaller the partner image is, so that the speaker can adjust the size of the face of the processed image displayed on the display device. By seeing you can see if you are too close or too far away from the camera.

Note that the present invention can be realized not only as an image communication apparatus but also as a method using a processing unit constituting the image communication apparatus as a step. Moreover, you may implement | achieve as a program which makes a computer perform these steps. Furthermore, it may be realized as a recording medium such as a computer-readable CD-ROM (Compact Disc-Read Only Memory) in which the program is recorded, and information, data, or a signal indicating the program. These programs, information, data, and signals may be distributed via a communication network such as the Internet.

In addition, some or all of the constituent elements constituting each of the image communication apparatuses described above may be configured by one system LSI (Large Scale Integration). The system LSI is an ultra-multifunctional LSI manufactured by integrating a plurality of components on a single chip, and specifically includes a microprocessor, ROM, RAM (Random Access Memory), and the like. Computer system.

According to the present invention, it is possible to prevent an alarm sound from being generated or to prevent communication from being hindered by displaying a self-portrait on a display, and to determine how far the camera is out of the imaging area of the camera. Can be shown.

FIG. 1 is a block diagram illustrating a configuration of the image communication apparatus according to the first embodiment. FIG. 2 is a flowchart illustrating transmission processing of the image communication apparatus according to the first embodiment. FIG. 3 is a flowchart illustrating a reception process of the image communication apparatus according to the first embodiment. FIG. 4 is a diagram illustrating a positional relationship among the speaker, the camera, and the monitor according to the first embodiment. FIG. 5 is a diagram illustrating a positional relationship between speakers in an input image captured by the camera according to the first embodiment. FIG. 6 is a diagram illustrating an example of a counterpart image before execution of the image superimposing process of Embodiment 1 and a processed image after execution. FIG. 7A is a diagram illustrating an example of a graph showing the relationship between the horizontal distance (dx) and the horizontal overlap size (W (dx)). FIG. 7B is a diagram illustrating an example of a graph showing the relationship between the vertical distance (dy) and the vertical superimposition size (H (dy)). FIG. 8 is a diagram illustrating an example of a processed image when the face area according to the first embodiment is not detected. FIG. 9 is a schematic diagram showing how the partner image changes when the speaker according to Embodiment 1 moves. FIG. 10 is a block diagram illustrating a configuration of the image communication apparatus according to the second embodiment. FIG. 11 is a flowchart illustrating a reception process of the image communication apparatus according to the second embodiment. FIG. 12 is a diagram illustrating an example of a partner image before execution of projective transformation according to Embodiment 2 and a processed image after execution. FIG. 13 is a diagram illustrating an example of a graph for calculating image processing parameters for projective conversion according to the second embodiment. FIG. 14 is a schematic diagram showing how the partner image changes when the speaker according to Embodiment 2 moves. FIG. 15 is a block diagram illustrating a configuration of the image communication apparatus according to the third embodiment. FIG. 16 is a flowchart illustrating a reception process of the image communication apparatus according to the third embodiment. FIG. 17 is a diagram illustrating an example of a graph for calculating the blur parameter according to the third embodiment. FIG. 18 is a diagram illustrating an example of an input image, a partner image, and a processed image according to the third embodiment. FIG. 19 is a block diagram illustrating a configuration of the image communication apparatus according to the fourth embodiment. FIG. 20 is a flowchart illustrating a reception process of the image communication apparatus according to the fourth embodiment. FIG. 21 is a diagram illustrating an example of a graph for calculating an enlargement / reduction parameter according to the fourth embodiment. FIG. 22 is a diagram illustrating an example of an input image, a partner image, and a processed image according to the fourth embodiment. FIG. 23 is a block diagram illustrating a configuration of the image communication apparatus according to the fifth embodiment. FIG. 24 is a flowchart illustrating a reception process of the image communication apparatus according to the fifth embodiment. FIG. 25A is a diagram illustrating an example of a partner image according to the fifth embodiment. FIG. 25B is a diagram illustrating an example of an input image according to the fifth embodiment. FIG. 26 is a block diagram illustrating a configuration of the image communication apparatus according to the sixth embodiment. FIG. 27 is a flowchart illustrating transmission processing of the image communication apparatus according to the sixth embodiment. FIG. 28 is a diagram illustrating an example of an input image and a transmission image according to the sixth embodiment. FIG. 29 is a block diagram illustrating a configuration of the image communication apparatus according to the seventh embodiment. FIG. 30 is a diagram illustrating a positional relationship between the two cameras and the speaker according to the seventh embodiment. FIG. 31 is a flowchart illustrating a transmission process of the image communication apparatus according to the seventh embodiment. FIG. 32 is a diagram illustrating an example of a first input image and a second input image according to the seventh embodiment. FIG. 33 is a diagram illustrating an example of a partner image and a processed image. FIG. 34 is a block diagram showing an example of a different form of the image communication apparatus according to the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(Embodiment 1)
The image communication apparatus according to Embodiment 1 is an apparatus that superimposes and displays a predetermined image on a partner image received from another image communication apparatus based on the position of the speaker's face in a captured image acquired from a camera or the like. It is.

FIG. 1 is a block diagram illustrating a configuration of the image communication apparatus 100 according to the first embodiment. The image communication apparatus 100 illustrated in FIG. 1 transmits image data including a captured image captured by the camera 101 to another image communication apparatus via the network 111, and from the other image communication apparatus via the network 111. This is a device for displaying a partner image included in the received image data on the monitor 110. As illustrated in FIG. 1, the image communication apparatus 100 includes an image input unit 102, a face detection unit 103, an image encoding unit 104, an image transmission unit 105, an image reception unit 106, and an image decoding unit 107. The image processing unit 108 and the image output unit 109 are provided.

The image input unit 102 is an interface connected to the camera 101 that captures an image, and acquires a captured image captured by the camera 101. The image input unit 102 outputs the acquired captured image to the face detection unit 103 as an input image.

The face detection unit 103 detects a face area from the input image acquired by the image input unit 102. For example, the face detection unit 103 detects a face area, thereby detecting face area coordinates indicating the position of the face area and the size of the face area. The face area is a circular area including the speaker's face in the input image.

Specifically, the face detection unit 103 performs face area detection processing using a technique such as template matching or ellipse detection, and determines the face detection flag indicating the presence or absence of a face area, and the center position of the face area as an example of face area coordinates. Information such as the center coordinates to be shown and the radius of the face area is output to the image processing unit 108. Further, the face detection unit 103 outputs the input image to the image encoding unit 104. The face area may be elliptical. In this case, the center position of the face area is the midpoint between the two focal points, and the radius of the face area is an average value of the major axis and the minor axis. In the present embodiment, the radius information of the face area need not be output.

The image encoding unit 104 is H.264. By compressing and encoding the input image using a compression encoding method such as H.264, compressed image data is generated. That is, the generated image data includes a captured image acquired by the image input unit 102. The image encoding unit 104 outputs the generated image data to the image transmission unit 105.

The image transmission unit 105 transmits the image data compressed by the image encoding unit 104 to another image communication apparatus or the like via the network 111. For example, the image transmission unit 105 packetizes the image data according to a packet transmission method such as RTP (Real-time Transport Protocol), and outputs the RTP packet generated by packetization to the network 111.

The image receiving unit 106 receives image data including a partner image from another image communication apparatus via the network 111. For example, the image receiving unit 106 receives an RTP packet from the network 111 and acquires compressed image data by removing the RTP header. The image receiving unit 106 outputs the acquired compressed image data to the image decoding unit 107.

The image decoding unit 107 generates a partner image by decoding the compressed image data received by the image receiving unit 106. The image decoding unit 107 outputs the generated partner image to the image processing unit 108. The partner image is an image showing a communication partner using another image communication apparatus, and is an image acquired by a camera or the like connected to the other image communication apparatus.

The image processing unit 108 is an example of an image processing unit that generates a processed image by processing a counterpart image. “Processing” here does not include superimposing a captured image, which is a self-portrait, on the partner image. For example, the image processing unit 108 calculates the difference between the position of the face area detected by the face detection unit 103 and a predetermined threshold, and the larger the calculated difference, the larger the partner image and the processed image differ. Process the other party's image. Specifically, the image processing unit 108 calculates image processing parameters according to the position of the face area detected by the face detection unit 103, and processes the partner image using the calculated image processing parameters. As illustrated in FIG. 1, the image processing unit 108 includes a determination unit 121, a buffer 122, a parameter calculation unit 123, and an image superimposition unit 124.

The determination unit 121 determines whether a face area exists in the input image using the face detection flag input from the face detection unit 103. When the face area exists, the determination unit 121 outputs the center coordinates and the radius of the face area input from the face detection unit 103 to the parameter calculation unit 123 and causes the buffer 122 to store the center coordinates and the radius. When the face area does not exist, the determination unit 121 reads the center coordinates from the buffer 122 and outputs the read center coordinates to the parameter calculation unit 123.

The buffer 122 is a memory for storing the center coordinates of the face area and the radius of the face area. The buffer 122 may always store only the latest center coordinates and radius, that is, only the center coordinates of the face area when the face area was last detected. Alternatively, the buffer 122 may store a plurality of center coordinates and a plurality of radii in association with the time when the face area is detected.

The parameter calculation unit 123 calculates a difference between the center coordinates input from the determination unit 121 and a predetermined reference coordinate, and calculates an image processing parameter using the calculated difference. Specifically, the parameter calculation unit 123 calculates an image processing parameter for superimposing an image based on the calculated positive / negative of the difference and the absolute value of the difference. The parameter calculation unit 123 outputs the calculated image processing parameters to the image superimposing unit 124. The calculation processing of image processing parameters for image superimposition will be described later.

The image superimposing unit 124 generates a processed image by superimposing a predetermined superimposed image on the counterpart image generated by the image decoding unit 107 using the image processing parameter. The superimposed image may be any image as long as it is different from the original counterpart image and the captured image that is the self-portrait. However, the superimposed image is preferably a single color image or a semi-transparent image, for example, an image that allows the speaker to easily determine that it is superimposed and its position. The image superimposing unit 124 outputs the generated processed image to the image output unit 109.

As described above, the image processing unit 108 calculates the absolute value of the difference between the center coordinates, which are an example of face area coordinates, and the reference coordinates. Then, when the calculated difference absolute value is larger than a predetermined threshold, the image processing unit 108 processes the partner image so that the partner image and the processed image are greatly different as the difference absolute value is larger. Generate an image. Specifically, the processed image is generated by superimposing the superimposed image on the counterpart image so that the larger the difference absolute value is, the larger the area of the predetermined superimposed image is.

The image output unit 109 outputs the processed image generated by the image processing unit 108 to the monitor 110 which is an example of a display device. Specifically, the image output unit 109 is an interface connected to a monitor 110 that displays an image, and displays a processed image on the monitor 110.

Next, the operation of the image communication apparatus 100 having the above configuration will be described using the flowcharts shown in FIGS. 2 and 3 is stored as a control program in a storage device (not shown) such as a ROM or flash memory, and is controlled by a CPU (Central Processing Unit) (not shown). The

First, transmission processing of the image communication apparatus 100 will be described with reference to FIG. FIG. 2 is a flowchart showing transmission processing of the image communication apparatus 100 according to the first embodiment.

The image input unit 102 acquires an uncompressed captured image from the camera 101 in units of frames, and outputs the acquired image to the face detection unit 103 (S101).

The face detection unit 103 detects a face area from a non-compressed captured image input from the image input unit 102 by a method such as template matching or ellipse detection (S102). Then, the face detection unit 103 calculates information such as a face detection flag, the center coordinates of the face region, and the radius of the face region, and outputs the calculated information to the image processing unit 108 and also converts the input image into an image code. To the conversion unit 104.

The image encoding unit 104 is H.264. The input image input from the face detection unit 103 is encoded using the H.264 compression encoding method, and the compression-encoded image data is output to the image transmission unit 105. The image transmitting unit 105 converts the image data input from the image encoding unit 104 into RTP packets according to a packet transmission method such as RTP, and outputs the packet to the network 111 (S103).

Note that the compression encoding method is H.264. H.264 is not limited to MPEG-2, MPEG-4, H.264. 261, H.H. Any compression coding scheme such as H.263 can be used. The packet transmission method is not limited to RTP, and any transmission method such as RTSP (Real Time Streaming Protocol) can be used.

Hereinafter, a specific example of the face detection process performed by the face detection unit 103 will be described.

FIG. 4 is a diagram illustrating a positional relationship among the speaker 201, the camera 101, and the monitor 110 according to the first embodiment. 4A is a view of the speaker 201, the camera 101, and the monitor 110 as viewed from above. FIG. 4B is a view of the speaker 201, the camera 101, and the monitor 110 as viewed from the front. is there.

As shown in FIG. 4, the camera 101 is installed in the upper center of the monitor 110, and an area that can be imaged by the camera 101 is an imaging area 202. Further, only the partner image is displayed on the monitor 110, and the self-portrait (speaker 201) is not displayed. In the present invention, even when the speaker 201 does not display the self-portrait on the monitor 110, the speaker 201 confirms where the speaker 201 is in the imaging area 202 with the image of the other party displayed on the monitor 110. The purpose is to do.

FIG. 5 is a diagram showing the positional relationship of the speaker 201 in the input image 301 captured by the camera 101 in FIG. As shown in FIG. 4, since the camera 101 is installed in the upper center of the monitor 110, when the speaker 201 is positioned on the left side of the monitor 110, the right side in the input image 301 captured by the camera 101 is displayed. The speaker image 302 exists. When the speaker 201 is positioned on the right side of the monitor 110, the speaker image exists on the left side in the input image captured by the camera 101. Note that the speaker image 302 is the speaker 201 shown in the input image 301.

In FIG. 5, the face detection unit 103 detects the face area 303 of the speaker image 302 from the input image 301 and calculates center coordinates (x1, y1) and a radius (R) that are the centers of the face areas. . Note that the reference coordinates (x0, y0) are coordinates that make a preset speaker position appropriate, and are used when detecting the amount of deviation of face coordinates. For example, the reference coordinate is the center of the imaging area 202 of the camera 101.

Next, the reception process of the image communication apparatus 100 will be described with reference to FIG.

FIG. 3 is a flowchart showing the reception process of the image communication apparatus 100 according to the first embodiment.

The image receiving unit 106 receives the RTP packet via the network 111, acquires the compressed image data by removing the RTP header, and outputs the acquired compressed image data to the image decoding unit 107. The image decoding unit 107 converts the image data input from the image receiving unit 106 into the H.264 format. The non-compressed counterpart image is generated by decoding based on H.264, and the generated counterpart image is output to the image processing unit 108 (S201).

The determination unit 121 determines whether a face area exists in the input image using the face detection flag input from the face detection unit 103 (S202).

When the face area exists (Yes in S202), the determination unit 121 outputs the center coordinates (x1, y1) input from the face detection unit 103 to the parameter calculation unit 123. At this time, the center coordinates (x1, y1) are stored in the buffer 122. The parameter calculation unit 123 calculates the difference between the center coordinates (x1, y1) input from the determination unit 121 and the reference coordinates (x0, y0) (S203).

Next, when the absolute value of the calculated difference is larger than the predetermined threshold (Yes in S204), the parameter calculation unit 123 calculates an image processing parameter based on the positive / negative and absolute value of the calculated difference (S207).

When the face area does not exist (No in S202), the determination unit 121 reads the past center coordinates from the buffer 122 (S205), and outputs the read center coordinates (x1, y1) to the parameter calculation unit 123. When a plurality of center coordinates are held in the buffer 122, the determination unit 121 reads the latest center coordinates, that is, the center coordinates when the face area was last detected.

Next, the parameter calculation unit 123 calculates the difference between the center coordinates (x1, y1) and the reference coordinates (x0, y0) (S206). Then, the parameter calculation unit 123 calculates an image processing parameter based on the sign of the calculated difference (S207). A specific example of the image processing parameter calculation process will be described later.

Next, the image superimposing unit 124 superimposes a predetermined superimposed image on the counterpart image using the image processing parameter (S208). Then, the image superimposing unit 124 outputs the processed image generated by the superimposition to the image output unit 109.

When the calculated difference is equal to or less than the threshold (No in S204), the image superimposing unit 124 outputs the partner image as it is to the image output unit 109 without executing the superimposing process on the input partner image.

Finally, the image output unit 109 outputs the processed image (or the partner image) input from the image superimposing unit 124 to the monitor 110 and displays it on the monitor 110 (S209).

Subsequently, details of the image processing parameter calculation processing performed by the image processing unit 108 and the superimposition processing will be described.

The image processing unit 108 calculates an image processing parameter for superimposition using the face detection flag input from the face detection unit 103 and the center coordinates of the face area.

First, when a face area exists using the face detection flag, the parameter calculation unit 123 calculates the reference coordinates (x0, y0) and the center coordinates (x1, y1) of the face area according to Expression 1 and Expression 2. The difference, that is, the horizontal distance (dx) and the vertical distance (dy) are calculated.

(Formula 1) dx = x1-x0
(Formula 2) dy = y1-y0

Next, the parameter calculation unit 123 performs threshold determination on the horizontal distance (dx) and the vertical distance (dy), respectively, and when the horizontal distance (dx) and the vertical distance (dy) that are the differences are larger than the threshold, Image processing parameters for image processing (image superimposition) are calculated.

FIG. 6 is a diagram illustrating an example of the partner image 401 before the execution of the image superimposing process according to the first embodiment and the processed image 402 after the execution. FIG. 6B shows a processed image 402 generated by superimposing the superimposed image 403 on the counterpart image 401 shown in FIG.

7A and 7B are diagrams illustrating an example of a graph for calculating an image processing parameter for superimposition. FIG. 7A is a diagram illustrating an example of a graph showing the relationship between the horizontal distance (dx) and the horizontal overlap size (W (dx)). FIG. 7B is a diagram illustrating an example of a graph showing the relationship between the vertical distance (dy) and the vertical superimposition size (H (dy)).

The image processing parameter for superimposition is a parameter indicating which size of the superimposed image 403 is to be superimposed at which position of the counterpart image 401, and is, for example, the following four points.

Image processing parameters for superimposition Horizontal superimposition start position: Left end or right end of partner image 401 Horizontal overlap size ... W (dx)
3. Vertical superimposition start position: upper end or lower end of the partner image 401 Vertical overlap size ... H (dy)

The parameter calculation unit 123 first sets and calculates horizontal image processing parameters according to the following.

1. Determination of horizontal superimposition start position (left end or right end of counterpart image 401) The parameter calculation unit 123 determines the horizontal superimposition start position according to the sign of dx. Specifically, the parameter calculation unit 123 determines the left end when the sign of dx is positive and the right end when the sign is negative.

As shown in FIG. 5, the fact that dx is positive means that the face region 303 exists on the right side in the input image 301, that is, as shown in FIG. That is, the speaker 201 exists on the right side as viewed from the right. In this case, since the partner is viewed from the left side, the image superimposing unit 124 superimposes the superimposed image 403 from the left end of the partner image 401. The opposite is true when dx is negative.

2. Calculation of Horizontal Superimposition Size W (dx) As shown in FIG. 7A, when | dx | is larger than a threshold value (th_x), the parameter calculation unit 123 has W (dx) having a size proportional to | dx | Is calculated. The parameter calculation unit 123 sets W (dx) = 0 when | dx | is less than the threshold value (th_x).

Next, the parameter calculation unit 123 calculates vertical image processing parameters according to the following.

3. Determination of vertical superimposition start position (upper end or lower end of counterpart image 401) The parameter calculation unit 123 determines the vertical superimposition start position according to the sign of dy. Specifically, the parameter calculation unit 123 determines the upper end when the sign of dy is positive and the lower end when it is negative.

As shown in FIG. 5, dy being positive means that the face region 303 exists on the upper side in the input image 301. In this case, since the partner is viewed from the upper side, the image superimposing unit 124 superimposes the superimposed image 403 from the upper end of the partner image 401. The opposite is true when dy is negative.

4). Calculation of Vertical Superimposition Size H (dy) As shown in FIG. 7B, when | dy | is larger than a threshold value (th_y), the parameter calculation unit 123 sets H (dy) having a size proportional to | dy | Is calculated. The parameter calculation unit 123 sets H (dy) = 0 when | dy | is less than the threshold value (th_y).

The calculation method is not limited to this example, and any method can be used as long as the value of the screen processing parameter increases as the horizontal distance or the vertical distance increases. That is, the horizontal distance | dx | and the horizontal superimposition size W (dx) or the vertical distance | dy | and the vertical superimposition size H (dy) need only have a positive correlation. Further, the thresholds th_x and th_y may be zero.

As described above, when the face area coordinates are on the right side of the reference coordinates in the input image, the image processing unit 108 superimposes the superimposed image from the left end of the counterpart image, and the face area coordinates are on the left side of the reference coordinates in the input image. If it is, the superimposed image is superimposed from the right end of the partner image. At this time, the image processing unit 108 superimposes the superimposed image on the partner image so that the larger the horizontal difference absolute value between the face region coordinates and the reference coordinates, the larger the area of the superimposed image.

Similarly, when the face area coordinates are above the reference coordinates in the input image, the image processing unit 108 superimposes the superimposed image from the upper end of the partner image, and the face area coordinates are below the reference coordinates in the input image. In some cases, the superimposed image is superimposed from the lower end of the partner image. At this time, the image processing unit 108 superimposes the superimposed image on the partner image so that the larger the absolute value of the vertical difference between the face area coordinates and the reference coordinates, the larger the area of the superimposed image.

On the other hand, referring to the face detection flag, the case where the face area does not exist is as follows.

FIG. 8 is a diagram showing an example of a processed image when the face area of the first embodiment is not detected. As shown in FIG. 8A, the past input image 501 includes a speaker image 502 and includes a face area 503. The input image 501 is, for example, an input image when the face area 503 is detected last.

As shown in FIG. 8B, the face area 513 is not included in the input image 511 when the speaker moves upward (moves from the speaker image 502 to the speaker image 512). In this case, the parameter calculation unit 123 calculates the machining parameter as follows.

1. Horizontal superimposition start position: Horizontal superimposition start position determined when the face area was detected last. Horizontal superimposition size W / N (W is the horizontal size of the screen, N is a fixed value of 1 or more)
3. Vertical superimposition start position: Vertical superimposition start position determined when the face area was detected last. Vertical overlap size: H / M (H is the vertical size of the screen, M is a fixed value of 1 or more)

More specifically, when the face area does not exist, the determination unit 121 reads the center coordinates (x1, y1) of the face area 503 shown in FIG. 8A from the buffer 122, and uses the read center coordinates as the parameter calculation unit. It outputs to 123. The parameter calculation unit 123 calculates a difference (dx and dy) between the center coordinates (x1, y1) and the reference coordinates (x0, y0), and determines a superposition start position based on the sign of the calculated difference. At this time, the superimposition size is a fixed value as described above regardless of the absolute value of the difference.

For example, in the example of FIG. 8, the difference between the past center coordinates and the reference coordinates is 0 in the horizontal direction and positive in the vertical direction. Therefore, as illustrated in FIG. 8C, the image superimposing unit 124 does not perform superimposition in the horizontal direction, but superimposes the superimposed image 523 having a fixed area from the upper end on the partner image 521 in the vertical direction. A processed image 522 is generated.

As described above, when the face detection flag indicates that the face area is not included in the input image, the image processing unit 108 starts from the left end when the face area coordinates stored in the buffer 122 are on the right side of the reference coordinates. Alternatively, when the face area coordinates stored in the buffer 122 are on the left side of the reference coordinates, a superimposed image having a predetermined area is superimposed on the counterpart image from the right end. Similarly, when the face area coordinates stored in the buffer 122 are above the reference coordinates, from the upper end, or when the face area coordinates stored in the buffer 122 are below the reference coordinates, the lower end From the above, a superimposed image having a predetermined area is superimposed on the counterpart image.

In this way, when there is no face area, it is possible to superimpose superimposed images with fixed areas in the horizontal and vertical directions.

As described above, in the present embodiment, the face detection unit 103 detects the center coordinates of the face region, and the image processing unit 108 determines the other image according to the horizontal and vertical distances and directions between the center coordinates and the reference coordinates. On the other hand, another image is superimposed.

Specifically, in the image communication apparatus 100 according to the present embodiment, when the speaker 201 is on the left side toward the monitor 110, that is, the center coordinates of the face area 303 are on the right side of the reference coordinates in the input image 301. In this case, the processed image 402 generated by superimposing the superimposed image 403 from the left end of the counterpart image 401 is displayed on the monitor 110. In this way, when viewing the opponent from the left side, it is possible to recognize that the person (speaker) is on the left side from the center by making the area on the left side of the opponent invisible. The same applies to the right side, upper side and lower side.

FIG. 9 is a schematic diagram showing how the partner image changes when the speaker according to Embodiment 1 moves.

As shown in FIG. 9A, when the speaker 201 is on the left side facing the monitor 110, the superimposed image 403 is superimposed on the partner image 401 from the left side of the partner image 401. That is, the processed image 402 generated by superimposing the superimposed image 403 on the left side of the partner image 401 is displayed on the monitor 110.

In the processed image 402 displayed on the monitor 110, the speaker 201 sees the position where the superimposed image 403 is superimposed and the area of the superimposed image 403, so that the speaker 201 can move in which direction. Judgment can be made. For example, in the example shown in FIG. 9A, the speaker 201 may move to the right side.

As shown in FIG. 9B, the area of the superimposed image 403 is reduced in the processed image 402 displayed on the monitor 110 by the speaker 201 moving to the right side. As a result, the speaker 201 can confirm that the speaker 201 is moving in an appropriate direction, and it can be understood that the speaker 201 is not yet in an appropriate position (that is, a reference position).

As described above, the image communication apparatus 100 according to the present embodiment corresponds to the position of the speaker 201, specifically, the position of the face area of the speaker 201 in the captured image captured by the camera 101. Then, the partner image 401 is processed. Specifically, the image communication apparatus 100 generates the processed image 402 by superimposing the superimposed image 403 having a larger area on the partner image 401 as the position of the speaker 201 is farther from the reference position.

Thus, for example, it is possible to give the speaker an impression that the speaker and the other party are communicating through the window frame. That is, when the speaker is in front of the window frame, the other party can be seen in front, and when the speaker is on the left side toward the window frame, the left side of the other party cannot be seen by the window frame (corresponding to the superimposed image). The same impression can be given to the speaker. Therefore, by applying the image communication apparatus 100 according to the present embodiment to the TV conference system, communication with higher presence can be performed.

As a result, the speaker does not display the self-portrait on the monitor, but looks at the position and size of the image superimposed on the partner image on the monitor to determine how much the speaker is displaced in which direction on the screen. It is possible to determine.

Further, in the present embodiment, when the face detection unit 103 cannot detect the face area, the image processing unit 108 refers to the face detection flag, and the last time the face area is detected with respect to the partner image. Another image having a certain area is superimposed and displayed based on the positional relationship between the center coordinates and the reference coordinates.

As a result, the speaker does not display his / her own image on the monitor, and the face area of the speaker is not imaged by seeing that a certain area of the partner image displayed on the monitor is superimposed on another image. It is possible to determine whether or not the image area is shifted.

Note that the buffer 122 does not store the center coordinates when the face area was last detected, but may store the image processing parameters calculated by the parameter calculation unit 123 when the face area was last detected. Good. At this time, if a face region is not detected, the image superimposing unit 124 does not superimpose a fixed area superposed image, but has an area indicated by the horizontal superimposition size and the vertical superimposition size stored in the buffer 122. The superimposed image may be superimposed on the partner image.

Alternatively, in the present embodiment, since the position at which the superimposed image is superimposed is determined based on the difference between the horizontal direction and the vertical direction, the buffer 122 includes the horizontal direction superposition start position among the image processing parameters. Only the vertical superimposition start position may be stored.

(Embodiment 2)
The image communication apparatus according to the second embodiment is an apparatus that performs projective transformation on the partner image received from another image communication apparatus based on the position of the speaker's face area in the input image.

FIG. 10 is a block diagram illustrating a configuration of the image communication apparatus 600 according to the second embodiment. The image communication apparatus 600 shown in the figure is different from the image communication apparatus 100 of FIG. 1 in that an image processing unit 608 is provided instead of the image processing unit 108. 10, components having the same reference numerals as those in FIG. 1 perform the same processes as those in the first embodiment, and thus description thereof will be omitted here and different points will be mainly described.

The image processing unit 608 is an example of an image processing unit that generates a processed image by processing the counterpart image. Specifically, the image processing unit 608 generates a processed image by projective conversion of the partner image so that the gradient of the projective conversion increases as the absolute difference between the face area coordinates and the reference coordinates increases. As illustrated in FIG. 10, the image processing unit 608 includes a determination unit 121, a buffer 122, a parameter calculation unit 623, and a projective conversion unit 624. Since the determination unit 121 and the buffer 122 perform the same processing as in the first embodiment, the description thereof is omitted.

The parameter calculation unit 623 calculates a difference between the center coordinates and the reference coordinates input from the determination unit 121, and calculates an image processing parameter using the calculated difference. Specifically, the parameter calculation unit 623 calculates an image processing parameter for projective transformation based on the calculated difference between positive and negative and the absolute value of the difference. The parameter calculation unit 623 outputs the calculated image processing parameters to the projective conversion unit 624. The process for calculating the image processing parameters for projective transformation will be described later.

The projective conversion unit 624 generates a processed image by performing projective conversion on the counterpart image generated by the image decoding unit 107 using the image processing parameters for projective conversion.

Next, the operation at the time of image reception of the image communication apparatus 600 having the above configuration will be described using the flowchart shown in FIG. FIG. 11 is a flowchart illustrating a reception process of the image communication apparatus 600 according to the second embodiment. Since the operation at the time of image transmission is the same as that in Embodiment 1 (FIG. 2), description thereof is omitted here.

The operation of the flowchart shown in FIG. 11 is stored as a control program in a storage device (not shown) such as a ROM or a flash memory, and is controlled by a CPU (not shown). In FIG. 11, the processing steps assigned with the same reference numerals as those in FIG. 3 are the same operations as those in FIG.

As in the first embodiment, when a face area exists (Yes in S202), the calculated difference, that is, at least one of the horizontal distance (dx) and the vertical distance (dy) is determined in advance. If it is larger than the threshold (Yes in S204), the parameter calculation unit 623 calculates an image processing parameter for projective transformation based on the positive / negative and absolute value of the calculated difference (S307).

If the face area does not exist (No in S202), the parameter calculation unit 623 calculates an image processing parameter for projective transformation based on the sign of the calculated difference (S307). A specific example of the image parameter calculation process will be described later.

Next, the projective transformation unit 624 performs projective transformation of the partner image using the calculated image processing parameter (S308). Then, the projective transformation unit 624 outputs the processed image generated by the projective transformation to the image output unit 109.

Thereafter, as in the first embodiment, the image output unit 109 outputs the processed image (or the partner image) to the monitor 110 and displays it on the monitor 110 (S209).

Subsequently, details of the image processing parameter calculation processing and the projective transformation processing performed by the image processing unit 608 will be described.

The image processing unit 608 calculates an image processing parameter for projective transformation using the face detection flag input from the face detection unit 103 and the center coordinates of the face area.

First, when a face area exists using the face detection flag, the parameter calculation unit 623 performs center coordinates (x1) of the face area with reference coordinates (x0, y0) according to (Expression 1) and (Expression 2). , Y1), that is, the horizontal distance (dx) and the vertical distance (dy) are calculated.

Next, the parameter calculation unit 623 performs threshold determination for each of the horizontal distance (dx) and the vertical distance (dy), and calculates an image processing parameter for performing projective transformation when the threshold is exceeded.

FIG. 12 is a diagram illustrating an example of a partner image 701 before execution of the projective transformation according to the second embodiment and a processed image 702 after execution. FIG. 12B shows a processed image 702 generated by projective transformation of the counterpart image 701 shown in FIG. In the following, an example of projective transformation of the counterpart image 701 at an angle calculated in the horizontal direction is shown.

FIG. 13 is a diagram showing an example of a graph for calculating image processing parameters for projective transformation. Specifically, an example of the relationship between the horizontal distance (dx) and the horizontal direction projection conversion width W (dx) is shown.

The image processing parameter for projective conversion is a parameter indicating in which direction of the counterpart image 701 and how much the projective conversion is performed. Here, since the projective transformation is performed in the horizontal direction, the image processing parameters are the following two points.

Image processing parameters for projective transformation Horizontal projection conversion direction: Left side or right side of partner image 701 Horizontal projection conversion width ... W (dx)

Here, the horizontal direction projective transformation direction as one of the image processing parameters being the left side of the partner image 701 is a projective transformation so that the left side of the partner image is far away. That is, projective transformation of the partner image 701 is performed so that the length of the left end of the partner image 701 is smaller than the length of the right end. On the other hand, the horizontal direction projective transformation direction being the right side of the partner image 701 is a projective transformation so that the right side of the partner image is far away (FIG. 12B). That is, projective transformation of the partner image 701 is performed so that the length of the right end of the partner image 701 is smaller than the length of the left end. Thus, the slope of the projective transformation is represented by the ratio of the length at the left end to the length at the right end.

As shown in FIG. 12, the projective transformation unit 624 has four points (upper left TL, upper right TR, lower left BL, lower right BR) of the counterpart image 701, and four points (upper left TL ′, upper right TR ′) of the processed image 702. , Lower left BL ′, lower right BR ′). FIG. 12B shows an example in which the horizontal projection conversion direction is the right side. As shown in the figure, the projective transformation can be performed so that the right side is farther as the horizontal direction projection transformation width W (dx) is larger.

The parameter calculation unit 623 calculates the image processing parameter in the horizontal direction according to the following when there is a face area.

1. Determination of horizontal projection conversion direction (left side or right side of counterpart image 701) The parameter calculation unit 623 determines the horizontal projection conversion direction according to the sign of dx. Specifically, the parameter calculation unit 623 determines the right side when the sign of dx is positive and the left side when the sign is negative.

As shown in FIG. 5, the fact that dx is positive means that the face region 303 exists on the right side in the input image 301, that is, as shown in FIG. ) Presents the speaker 201. In this case, since the partner is seen from the left side, the right side of the partner image 701 can be shown to the speaker 201 so that the right side is farther away by increasing the inclination of the projective transformation.

2. Calculation of Horizontal Projection Conversion Width W (dx) As shown in FIG. 13, when | dx | is larger than a threshold value (th_x), parameter calculation unit 623 calculates W (dx | ) Is calculated. The parameter calculation unit 623 sets W (dx) = 0 when | dx | is less than the threshold value (th_x).

Note that the calculation method is not limited to this example, and any method can be used as long as the value of the projective transformation width increases as the horizontal distance or the vertical distance increases. That is, it is only necessary that the horizontal distance | dx | and the horizontal projection transformation width W (dx) have a positive correlation. The threshold th_x may be 0.

As described above, when the face region coordinates are on the right side of the reference coordinates in the input image, the image processing unit 608 makes the right end of the counterpart image shorter than the left end, or the face region coordinates in the input image When it is on the left side of the coordinates, the processed image is generated by projective transformation of the partner image so that the left end of the partner image is shorter than the right end. At this time, the image processing unit 608 increases the inclination of the projective transformation as the absolute difference between the face region coordinates and the reference coordinates in the horizontal direction increases, in other words, the difference between the right end length and the left end length. Projective transformation of the partner image is performed so that becomes larger.

1. 1. Horizontal projection conversion direction: The side calculated when the face area was detected last. Horizontal projection conversion width ... H / K (H: vertical size of image, K: fixed value of 1 or more)

That is, as in the first embodiment, the determination unit 121 reads the previous face area coordinates from the buffer 122 and outputs the read face area coordinates to the parameter calculation unit 623. The parameter calculation unit 623 determines the horizontal projection conversion direction using the read face area coordinates. The horizontal projection transformation width is a fixed value as described above.

By doing in this way, even if there is no face area, it is possible to perform projective transformation at a constant angle in the horizontal direction.

As described above, when the face detection flag indicates that the face area is not included in the input image, the image processing unit 608 determines that the image of the partner image is in the case where the face area coordinates stored in the buffer 122 are on the right side of the reference coordinates. The other image with a predetermined inclination so that the right end is shorter than the left end, or when the face area coordinates stored in the buffer 122 are on the left side of the reference coordinates, the left end of the other image is shorter than the right end. Is projectively transformed.

It should be noted that image processing parameters can be calculated in the same way in the vertical direction, and instead of performing horizontal projective conversion, vertical projective conversion may be performed. Alternatively, both horizontal and vertical projective transformations can be used.

When using the vertical direction, the image processing parameters are the following two points.

2. Image processing parameters 3. Vertical projection conversion direction: upper end or lower end of the partner image 701 Vertical projection transformation width ... H (dy)
As described above, when the face area coordinates are above the reference coordinates in the input image, the image processing unit 608 makes the lower end of the counterpart image shorter than the upper end, or the face area coordinates in the input image When it is below the coordinates, the processed image is generated by projective transformation of the partner image so that the upper end of the partner image is shorter than the lower end. At this time, the image processing unit 608 increases the inclination of the projective transformation as the absolute value of the vertical difference between the face area coordinates and the reference coordinates increases, in other words, the difference between the upper end length and the lower end length. Projective transformation of the partner image is performed so that becomes larger.

In addition, when the face detection flag indicates that the face area is not included in the input image, the image processing unit 608 determines that the lower end of the partner image is the upper end when the face area coordinates stored in the buffer 122 are above the reference coordinates. The partner image is projected at a predetermined inclination so that the upper end of the partner image is shorter than the lower end when the face area coordinates stored in the buffer 122 are below the reference coordinates. Convert.

As described above, in the present embodiment, the face detection unit 103 detects the center coordinates of the face region, and the image processing unit 608 applies the distance and direction in the horizontal and vertical directions between the center coordinates and the reference coordinates to the partner image. Projective transformation is performed.

Specifically, in the image communication apparatus 600 of the present embodiment, when the speaker 201 is on the left side of the monitor 110, that is, when the center coordinates of the face area 303 are on the right side of the reference coordinates in the input image 301, Projective transformation is performed on the partner image 701 so that the left end of the partner image 701 is shorter than the right end. In this way, when the partner is viewed from the left side, it is possible to recognize that the right side of the partner is far from the center so that the user (speaker) is on the left side from the center. The same applies to the right side, upper side and lower side.

FIG. 14 is a schematic diagram showing how the partner image changes when the speaker according to the second embodiment moves.

As shown in FIG. 14A, when the speaker 201 is on the left side facing the monitor 110, the right end of the partner image 701 is shorter than the left end, that is, the right side of the partner image 701 is seen far away. The partner image 701 has undergone projective transformation. That is, the monitor 110 displays a processed image 702 generated by projective transformation so that the right side of the partner image 701 can be seen in the distance.

The speaker 201 determines how much to move in which direction by looking at the degree (ie, tilt) and direction of the projective transformation of the partner image 701 in the processed image 702 displayed on the monitor 110. can do. For example, in the example shown in FIG. 14A, the speaker 201 may move to the right side.

As shown in FIG. 14B, when the speaker 201 moves to the right side, in the processed image 702 displayed on the monitor 110, the degree (inclination) of the projective transformation of the counterpart image 701 is reduced. As a result, the speaker 201 can confirm that the speaker 201 is moving in an appropriate direction, and it can be understood that the speaker 201 is not yet in an appropriate position (that is, a reference position).

As described above, the image communication apparatus 600 according to the present embodiment corresponds to the position of the speaker 201, specifically, the position of the face area of the speaker 201 in the captured image captured by the camera 101. Then, the partner image 701 is processed. Specifically, the image communication apparatus 600 generates a processed image 702 by projectively transforming the partner image 701 with a larger inclination as the position of the speaker 201 is farther from the reference position.

As a result, the speaker does not display the self-portrait on the monitor, but looks at the tilt direction and the tilt magnitude of the projected image on the monitor, and how much the speaker is displaced in which direction with respect to the center of the screen. Can be determined.

Furthermore, in the present embodiment, when the face detection unit 103 cannot detect a face area, the image processing unit 608 refers to the face detection flag, and sets a fixed angle on the side set at the time of the last face detection with respect to the partner image. Perform a projective transformation.

As a result, the speaker does not display the self-portrait on the monitor, but sees that the other party's image displayed on the monitor is projective transformed at a constant angle, and the speaker's face area is not imaged, It is possible to determine which of the imaging areas is shifted. Specifically, according to the image communication apparatus 600 of the present embodiment, when the speaker is shifted from the proper position to the left side, the right side of the partner image appears far and the speaker is shifted from the proper position to the right side. If so, the partner image is processed so that the left side of the partner image can be seen far away. For this reason, the speaker can determine how much he / she is deviated from the appropriate position only by determining which part of the partner image is seen far away.

Note that the buffer 122 does not store the center coordinates when the face area was last detected, but may store the image processing parameters calculated by the parameter calculation unit 123 when the face area was last detected. Good. At this time, when the face area is not detected, the image superimposing unit 124 may perform the projective conversion on the partner image with the projective conversion width stored in the buffer 122 instead of performing the projective conversion at a constant angle.

Alternatively, in the present embodiment, since the direction for projective transformation is determined based on the sign of the difference between the horizontal direction and the vertical direction, the buffer 122 includes the horizontal direction projective transformation direction and the vertical direction among the image processing parameters. Only the projective transformation direction may be stored.

(Embodiment 3)
The image communication apparatus according to the third embodiment is an apparatus that blurs a partner image received from another image communication apparatus in accordance with the difference between the detected radius of the speaker's face area and the reference radius.

FIG. 15 is a block diagram illustrating a configuration of the image communication apparatus 800 according to the third embodiment. The image communication apparatus 800 shown in the figure is different from the image communication apparatus 100 in FIG. 1 in that an image processing unit 808 is provided instead of the image processing unit 108. In FIG. 15, the components denoted by the same reference numerals as those in FIG. 1 perform the same processing as in the first embodiment, and thus description thereof will be omitted here and different points will be mainly described.

The image processing unit 808 is an example of an image processing unit that generates a processed image by processing a counterpart image. Specifically, the image processing unit 808 generates a processed image by blurring the partner image so that the blur amount increases as the absolute value of the difference between the size of the face area and a predetermined reference size increases. To do. As illustrated in FIG. 15, the image processing unit 808 includes a determination unit 121, a buffer 122, a parameter calculation unit 823, and a blur processing unit 824. Since the determination unit 121 and the buffer 122 perform the same processing as in the first embodiment, the description thereof is omitted.

The parameter calculation unit 823 calculates a difference between the radius of the face area input from the determination unit 121 and a predetermined reference radius, and calculates an image processing parameter using the calculated difference. Specifically, a blur parameter used for blur processing, which is one of the processes for blurring an image, is calculated as the image processing parameter. The blur parameter calculation process will be described later.

The blur processing unit 824 generates a processed image by performing blur processing on the counterpart image generated by the image decoding unit 107 using the blur parameter.

As described above, the image processing unit 808 calculates the absolute difference between the radius of the face area, which is an example of the size of the face area, and the reference radius, which is an example of the reference size, and the calculated difference absolute value is determined in advance. When it is larger than the threshold value, the processed image is generated by processing the counterpart image so that the counterpart image and the processed image differ greatly as the difference absolute value increases.

Next, the operation at the time of image reception of the image communication apparatus 800 having the above configuration will be described using the flowchart shown in FIG. FIG. 16 is a flowchart illustrating a reception process of the image communication apparatus 800 according to the third embodiment. Since the operation at the time of image transmission is the same as that in Embodiment 1 (FIG. 2), description thereof is omitted here.

The operation of the flowchart shown in FIG. 16 is stored as a control program in a storage device (not shown) such as a ROM or a flash memory, and is controlled by a CPU (not shown). In FIG. 16, the processing steps given the same reference numerals as those in FIG. 3 are the same operations as those in FIG.

As in the first embodiment, when the determination unit 121 determines that a face area exists in the input image (Yes in S202), the determination unit 121 determines the radius R of the face area input from the face detection unit 103. It outputs to the parameter calculation part 823. At this time, the radius R is stored in the buffer 122.

When the face area does not exist (No in S202), the determination unit 121 reads the radius R of the face area from the buffer 122 (S403), and outputs the read radius R of the face area to the parameter calculation unit 823. As described above, the buffer 122 stores the radius R of the face area when the face area was last detected, and performs the blur process based on the last detected face area by the following process. Execute.

The parameter calculation unit 823 calculates the difference between the radius R input from the determination unit 121 and the reference radius R0 (S404).

Next, when the calculated absolute value of the difference is larger than a predetermined threshold (Yes in S405), the parameter calculation unit 823 calculates an image processing parameter based on the calculated absolute value of the difference (S406). Using the calculated image processing parameter, the blur processing unit 824 blurs the partner image (S407). For example, blur processing is executed on the partner image using the blur parameter. Then, the processed image generated by the blur processing is output to the image output unit 109.

If the absolute value of the calculated difference is equal to or smaller than the threshold (No in S405), the blur processing unit 824 outputs the partner image as it is to the image output unit 109 without performing blur processing on the input partner image.

Finally, the image output unit 109 outputs the processed image (or the partner image) input from the blur processing unit 824 to the monitor 110 and displays it on the monitor 110 (S209).

Next, details of the image processing parameter calculation processing and blur processing performed by the image processing unit 808 will be described.

The image processing unit 808 calculates an image processing parameter for blur processing using the face detection flag input from the face detection unit 103 and the face radius information.

First, referring to the face detection flag, if a face area exists, the parameter calculation unit 823 determines a blur parameter α for blur processing using the radius R of the face area as follows.

First, the parameter calculation unit 823 calculates a difference absolute value dr between a reference radius R0 and a radius R determined in advance using Expression 3.

(Formula 3) dr = | R−R0 |

Next, the parameter calculation unit 823 determines the blur parameter α using the calculated difference absolute value dr. FIG. 17 is a diagram illustrating an example of a graph for calculating the blur parameter α. In FIG. 17, dr is an absolute difference between the radius R of the face area and the reference radius R0. α (dr) is a blur parameter. th_r is a predetermined threshold value. α_max is a maximum blur parameter. As shown in FIG. 17, when the absolute difference value dr of the radius exceeds the threshold th_r, the blur parameter α has a feature that increases in proportion to the absolute difference value dr.

Note that the blur parameter α only needs to be larger as the radius R is significantly different from the reference radius R0. That is, the blur parameter α only needs to have a positive correlation with the difference absolute value dr. The threshold th_r may be 0.

Subsequently, the blur processing unit 824 generates a processed image by processing the counterpart image using Expression 4. Expression 4 is an expression used when blur processing is executed in the horizontal direction (x-axis direction).

(Formula 4) out (x, y) = {in (x-1, y) × α / 2 + in (x, y) × (α_max−α) + in (x + 1, y) × α / 2} / α_max

In Expression 4, out (x, y) is a pixel value of the image in the x, y coordinates after blurring. in (x, y) is a pixel value in the x and y coordinates of the counterpart image. The blur parameter α and the maximum blur parameter α_max are values calculated by the parameter calculation unit 823 using the graph shown in FIG.

Thus, as the blur parameter α increases, the value of the integral sum of the surrounding pixels increases, and the processed image can be an image that is more blurred than the counterpart image. Note that the blur processing unit 824 can perform blur processing not only in the horizontal direction but also in the vertical direction or in both the horizontal and vertical directions.

FIG. 18 is a diagram illustrating an example of the

input images

901 and 911, the counterpart image 921, and the processed image 922 according to the third embodiment. FIG. 18A is a diagram illustrating a case where the face area 903 of the speaker image 902 in the input image 901 is larger than the reference radius R0. FIG. 18B is a diagram showing a case where the face area 913 of the speaker image 912 in the input image 911 is smaller than the reference radius R0. FIG. 18C is a diagram illustrating an example of the partner image 921 before the blur processing according to the third embodiment is executed. FIG. 18D is a diagram illustrating an example of the processed image 922 after the blur processing is executed.

For example, in FIG. 4, when the speaker 201 is too close to the camera 101, the radius R of the face area 903 detected by the face detection unit 103 is larger than the reference radius R0, as shown in FIG. . Conversely, when the speaker 201 is too far away from the camera 101, the radius R of the face region 913 detected by the face detection unit 103 is smaller than the reference radius R0, as shown in FIG.

In any case, when the difference dr between the radius R and the reference radius R0 exceeds the threshold th_r, the parameter calculation unit 823 determines the blur parameter α according to the graph shown in FIG. Then, the blur processing unit 824 generates a processed image 922 by performing blur processing on the counterpart image 921 using the determined blur parameter α.

As described above, in the present embodiment, the image processing unit 808 performs control so that the value of the blur parameter α used for blur processing increases as the difference absolute value between the radius of the face area and the reference radius increases. Then, increase the degree of blur on the partner image.

As a result, even if the self-portrait is not displayed on the monitor, the speaker looks at the degree of blurring of the partner image displayed on the monitor, the distance between the speaker and the camera is not appropriate, and the size of the face Can be distinguished from the reference radius. Specifically, according to the image communication apparatus 800 of the present embodiment, the face of the partner image is blurred when the speaker is too close to the camera or too far away. For this reason, the speaker can determine whether or not he / she is at an appropriate position with respect to the camera only by looking at the partner image.

Even if the face detection unit 103 cannot detect the face area, the other party image can be blurred using the radius of the face area detected in the past. You can confirm that you are not there.

If no face area is detected, the partner image may be blurred using a fixed blur parameter α.

(Embodiment 4)
The image communication device according to the fourth embodiment enlarges or reduces the size of the partner image received from another image communication device according to the difference between the detected radius of the face area and the reference radius, thereby reducing the size of the partner image. It is a device to change.

FIG. 19 is a block diagram showing the configuration of the image communication apparatus 1000 according to the fourth embodiment. The image communication apparatus 1000 shown in the figure is different from the image communication apparatus 800 in FIG. 15 in that an image processing unit 1008 is provided instead of the image processing unit 808. 19, components having the same reference numerals as those in FIG. 15 perform the same processes as those in the third embodiment, and thus description thereof will be omitted here and different points will be mainly described.

The image processing unit 1008 is an example of an image processing unit that generates a processed image by processing the counterpart image. Specifically, the image processing unit 1008 enlarges the partner image when the size of the face region is larger than the reference size, and reduces the partner image when the size of the face region is smaller than the reference size. Generate an image. At this time, the larger the difference absolute value between the size of the face area and the reference size, the larger the enlargement ratio or reduction ratio.

As shown in FIG. 19, the image processing unit 1008 includes a determination unit 121, a buffer 122, a parameter calculation unit 1023, and a size changing unit 1024. The determination unit 121 and the buffer 122 perform the same processing as in the third embodiment, and thus description thereof is omitted.

The parameter calculation unit 1023 calculates a difference between the radius of the face area input from the determination unit 121 and a predetermined reference radius, and calculates an image processing parameter using the calculated difference. Specifically, an enlargement / reduction parameter Z for changing the size of the counterpart image is determined as the image processing parameter. Specifically, the enlargement / reduction parameter Z is an enlargement rate or a reduction rate. When the enlargement / reduction parameter Z is a value smaller than 1, the counterpart image is reduced. When the enlargement / reduction parameter Z is a value larger than 1, the counterpart image is enlarged.

The size changing unit 1024 generates a processed image by changing the size of the counterpart image generated by the image decoding unit 107 using the enlargement / reduction parameter. The enlargement / reduction parameter is also a horizontal size ratio (or a vertical size ratio) between the generated processed image and the counterpart image.

Next, the operation at the time of image reception of the image communication apparatus 1000 having the above configuration will be described using the flowchart shown in FIG. FIG. 20 is a flowchart illustrating a reception process of the image communication apparatus 1000 according to the fourth embodiment. Since the operation at the time of image transmission is the same as that in Embodiment 1 (FIG. 2), description thereof is omitted here.

The operation of the flowchart shown in FIG. 20 is stored as a control program in a storage device (not shown) such as a ROM or a flash memory, and is controlled by a CPU (not shown). In FIG. 20, the processing steps to which the same reference numerals as those in FIG. 16 are given are the same operations as those in FIG.

As in the third embodiment, the parameter calculation unit 1023 calculates the difference between the radius R input from the determination unit 121 and the reference radius R0 (S404).

The parameter calculation unit 1023 calculates an image processing parameter based on the calculated difference when the calculated difference is larger than the predetermined first threshold (“larger than the first threshold” in S505) (S506). Then, using the calculated image processing parameter, the size changing unit 1024 generates a processed image by enlarging the partner image (S507). Then, the size changing unit 1024 outputs the generated processed image to the image output unit 109.

The parameter calculation unit 1023 calculates an image processing parameter based on the calculated difference when the calculated difference is smaller than the predetermined second threshold (“smaller than the second threshold” in S505) (S508). Then, using the calculated image processing parameter, the size changing unit 1024 generates a processed image by reducing the partner image (S509). Then, the size changing unit 1024 outputs the generated processed image to the image output unit 109.

When the calculated difference is greater than or equal to the second threshold and less than or equal to the first threshold (“second threshold and greater than or equal to the first threshold” in S505), the size changing unit 1024 does not change the size of the input partner image. The image is output as it is to the image output unit 109.

Finally, the image output unit 109 outputs the processed image (or the partner image) input from the size changing unit 1024 to the monitor 110 and displays it on the monitor 110 (S209).

Next, details of the image processing parameter calculation process and the size change process performed by the image processing unit 1008 will be described.

The image processing unit 1008 calculates the image processing parameters for the size change processing using the face detection flag input from the face detection unit 103 and the face radius information.

First, referring to the face detection flag, if a face area exists, the parameter calculation unit 1023 determines the enlargement / reduction parameter Z using the face radius information R as follows.

First, the parameter calculation unit 1023 calculates a difference value dR between a reference radius R0 and a radius determined in advance using Equation 5.

(Formula 5) dR = R−R0

Next, the parameter calculation unit 1023 determines the enlargement / reduction parameter Z using the calculated difference value dR. FIG. 21 is a diagram illustrating an example of a graph for calculating the enlargement / reduction parameter Z. In FIG. 21, dR is a difference value between the radius R and the reference radius R0, Z (dR) is an enlargement / reduction parameter, th_R is a preset threshold value, and Z_max and Z_min are the maximum value and the minimum value of the enlargement / reduction parameter, respectively. That is, th_R corresponds to the first threshold shown in FIG. 20, and −th_R corresponds to the second threshold.

As shown in FIG. 21, when the absolute value of the difference value dR is less than or equal to the threshold th_R, the enlargement / reduction parameter is 1. When the face radius R is larger than the reference radius R0 and the difference value dR exceeds the threshold th_R, Z is a value of 1 or more. On the other hand, when the face radius R is smaller than the reference radius R0 and the difference value dR is less than the threshold −th_R, Z is a value of 1 or less.

Note that the absolute values of the first threshold value (th_R) for determining whether to enlarge or not and the second threshold value (-th_R) for determining whether to reduce or not should not be equal. Further, both the first threshold value and the second threshold value may be zero.

Using the enlargement / reduction parameter Z determined in this way, the size changing unit 1024 changes the size of the counterpart image using Equation 6 and Equation 7.

(Expression 6) W_out = W_in × Z
(Expression 7) H_out = H_in × Z

In Equations 6 and 7, W_in is the horizontal size of the counterpart image, and H_in is the vertical size of the counterpart image. W_out is the horizontal size of the processed image generated by enlargement or reduction, and H_out is the vertical size of the processed image generated by enlargement or reduction. Z is an enlargement / reduction parameter determined by the parameter calculation unit 1023.

FIG. 22 is a diagram illustrating an example of the

input images

1101 and 1111, the counterpart image 1121, and the processed

images

1122 and 1123 according to the fourth embodiment. FIG. 22A is a diagram illustrating a case where the face area 1103 of the speaker image 1102 in the input image 1101 is larger than the reference radius R0. FIG. 22B is a diagram illustrating a case where the face area 1113 of the speaker image 1112 in the input image 1111 is smaller than the reference radius R0. FIG. 22C is a diagram illustrating an example of the partner image 1121 before executing the size changing process of the fourth embodiment. FIG. 22D is a diagram illustrating an example of the processed image 1122 generated by enlarging the partner image 1121. FIG. 22E is a diagram showing an example of a processed image 1123 generated by reducing the partner image 1121.

As shown in FIG. 22 (a), when the difference value dR is positive and larger than the threshold th_R, the other image 1121 is enlarged and arranged at the center of the screen, whereby the processing shown in FIG. 22 (d) is performed. An image 1122 is generated. At this time, when the size of the processed image 1122 becomes larger than the screen of the monitor 110 due to enlargement, the portion outside the screen is deleted.

As shown in FIG. 22B, when the difference value dR is negative and smaller than the threshold value th_R, the other image 1121 is reduced and placed in the center of the screen, whereby the processing shown in FIG. An image 1123 is generated. At this time, when the size of the processed image 1123 becomes smaller than the screen of the monitor 110 due to the reduction, a predetermined image (single color image or the like) is superimposed on the remaining area.

As described above, in the present embodiment, the image processing unit 1008 uses the radius of the face region to enlarge and display the partner image when it is larger than the reference radius, and conversely, when it is smaller than the reference radius. The other party's image is reduced and displayed.

As a result, even if the speaker does not display his / her own image on the monitor, the speaker's distance between the speaker and the camera is appropriate by looking at the size of the partner image's face on the monitor which changes as the partner image is enlarged or reduced. It is possible to determine that the size of the face is larger or smaller than the reference radius range. Specifically, according to the image communication apparatus 1000 of the present embodiment, when the speaker is too close to the camera, the face of the partner image becomes large, and when the speaker is too far away from the camera, the face of the partner image is small. Become. For this reason, the speaker can determine whether his / her position is too close or too far away from the camera simply by looking at the partner image.

Similarly to the third embodiment, when the face detection unit 103 cannot detect the face area, the size of the partner image can be changed using the radius of the face area detected in the past. Can confirm that he is not in the right position.

If no face area is detected, the partner image may be enlarged or reduced using a fixed enlargement rate or reduction rate. At this time, whether to enlarge or reduce can be determined based on whether the radius of the past face area held in the buffer 122 is larger or smaller than the reference radius.

(Embodiment 5)
The image communication apparatus according to the fifth embodiment is an apparatus that sets the reference coordinates and the reference radius used when calculating the image processing parameters using the face area coordinates and the face area radius of the counterpart image.

FIG. 23 is a block diagram illustrating a configuration of the image communication apparatus 1200 according to the fifth embodiment. The image communication apparatus 1200 shown in the figure is different from the image communication apparatus 100 in FIG. 1 in that an image processing unit 1208 is provided instead of the image processing unit 108. In FIG. 23, the constituent elements denoted by the same reference numerals as those in FIG. 1 perform the same processing as in the first embodiment, and therefore description thereof will be omitted here and different points will be mainly described.

The image processing unit 1208 is an example of an image processing unit that generates a processed image by processing the counterpart image. Specifically, the image processing unit 1208 detects a face area from the partner image, and sets face area coordinates indicating the position of the detected face area as reference coordinates. As shown in FIG. 23, the image processing unit 1208 includes a determination unit 121, a buffer 122, a parameter calculation unit 123, an image superimposition unit 124, and a reference coordinate setting unit 1225. Since the determination unit 121, the buffer 122, the parameter calculation unit 123, and the image superimposing unit 124 perform the same processing as in the first embodiment, the description thereof is omitted.

The reference coordinate setting unit 1225 detects a face area from the partner image input from the image decoding unit 107. Similar to the face detection unit 103, the reference coordinate setting unit 1225 detects a face region by a method such as template matching or ellipse detection. Then, the reference coordinate setting unit 1225 outputs the detected center coordinate and radius of the face area to the parameter calculation unit 123 as the reference coordinate and the reference radius, respectively. Further, the partner image is output to the image superimposing unit 124. When the size of the counterpart image and the input image are different, the reference coordinates and the reference radius are corrected according to the size of the input image.

Next, the operation at the time of image reception of the image communication apparatus 1200 having the above configuration will be described using the flowchart shown in FIG. FIG. 24 is a flowchart illustrating a reception process of the image communication apparatus 1200 according to the fifth embodiment. Since the operation at the time of image transmission is the same as that in Embodiment 1 (FIG. 2), description thereof is omitted here.

The operation of the flowchart shown in FIG. 24 is stored as a control program in a storage device (not shown) such as a ROM or a flash memory, and is controlled by a CPU (not shown). In FIG. 24, processing steps assigned with the same reference numerals as those in FIG. 3 are the same operations as those in FIG.

The image decoding unit 107 generates a partner image by decoding image data received from another image communication apparatus via the network 111, and outputs the partner image to the reference coordinate setting unit 1225 (S201).

The reference coordinate setting unit 1225 sets a reference coordinate and a reference radius by detecting a face area from the partner image (S602). At this time, only one of the reference coordinates and the reference radius may be set depending on the type of processing.

Thereafter, similarly to the first embodiment, a processed image is generated (S202 to S209).

Next, details of the process for setting the reference coordinates and the reference radius will be described.

The reference coordinate setting unit 1225 detects the face area from the partner image, calculates the center coordinates (X0, Y0) and the radius R0 of the face area, and calculates the calculated center coordinates (X0, Y0) and the radius R0, respectively. , The reference coordinates (x0, y0) and the reference radius R0 are set. When the face area cannot be detected, the reference coordinates are not updated. In addition, it is possible to update only the horizontal coordinates and only the vertical coordinates as the reference coordinates.

25A and 25B are diagrams showing the relationship between the reference coordinates and the reference radius in the counterpart image 1301 and the input image 1311 of the fifth embodiment. FIG. 25A is a diagram illustrating an example of the partner image 1301. As shown in the figure, the partner image 1301 includes a partner person image 1302 and a face area 1303. FIG. 25B is a diagram illustrating an example of the input image 1311. As shown in the figure, the input image 1311 includes a speaker image 1312 and a face area 1313.

As shown in FIGS. 25A and 25B, the reference coordinate setting unit 1225 uses the center coordinates (X0, Y0) and the radius R0 obtained by detecting the face from the counterpart image 1301 as the reference coordinates (x0, y0) and the reference radius R0 are set.

As described above, in the present embodiment, the reference coordinate setting unit 1225 sets the partner face area in the partner image to the reference coordinate and the reference radius, and the speaker face area detected by the face detection unit 103 is set. The superimposed image is displayed on the partner image according to the deviation direction from the face area coordinates and the difference value.

As a result, even if the speaker does not display the self-portrait on the monitor, the speaker sees the partner image superimposed on the monitor to determine how much the speaker's face area is shifted with respect to the partner's face. It can be determined.

(Embodiment 6)
The image communication apparatus according to the sixth embodiment is an apparatus that detects a face area from an input image obtained by capturing a wider area than an image transmitted to another image communication apparatus. That is, the image communication apparatus according to the sixth embodiment cuts out a part of the input image and transmits the cut image to another image communication apparatus.

FIG. 26 is a block diagram illustrating a configuration of the image communication apparatus 1400 according to the sixth embodiment. The image communication apparatus 1400 shown in the figure is different from the image communication apparatus 100 shown in FIG. 1 in that an image cutout unit 1412 is newly provided. In FIG. 26, the constituent elements denoted by the same reference numerals as those in FIG. 1 perform the same processing as in the first embodiment, and therefore description thereof will be omitted here and different points will be mainly described.

The image cutout unit 1412 cuts out a part of the input image input from the image input unit 102 via the face detection unit 103 and outputs the cut out image to the image encoding unit 104. For example, the image cutout unit 1412 cuts out a region having a constant area centered on the reference coordinates or a region having a constant area centered on the center coordinates of the input image.

Next, the operation at the time of image transmission of the image communication apparatus 1400 having the above configuration will be described with reference to the flowchart shown in FIG. FIG. 27 is a flowchart showing a transmission process of the image communication apparatus 1400 according to the sixth embodiment. Since the operation at the time of image reception is the same as that in Embodiment 1 (FIG. 3), description thereof is omitted here.

The operation of the flowchart shown in FIG. 27 is stored as a control program in a storage device (not shown) such as a ROM or a flash memory, and is controlled by a CPU (not shown). In FIG. 27, processing steps assigned with the same reference numerals as those in FIG. 2 are the same operations as those in FIG.

The image input unit 102 acquires an uncompressed captured image from the camera 101 in units of frames, and outputs the acquired image to the face detection unit 103 (S101). Note that the camera 101 preferably captures as wide a region as possible.

The face detection unit 103 performs face region detection from the uncompressed captured image acquired from the image input unit 102 by a method such as template matching or ellipse detection (S102). Then, the face detection unit 103 calculates information such as the face detection flag, the center coordinates of the face region, and the radius of the face region, and outputs the calculated information to the image processing unit 108 and also cuts the input image. Output to the output unit 1412.

The image cutout unit 1412 cuts out a preset transmission image area from the input image acquired from the image input unit 102 via the face detection unit 103 and outputs the cut out transmission image to the image encoding unit 104 ( S703).

The image encoding unit 104 converts the transmission image input from the image cutout unit 1412 to H.264. The image data is encoded using the H.264 compression encoding method, and the image data is output to the image transmission unit 105. The image transmitting unit 105 converts the image data input from the image encoding unit 104 into RTP packets according to a packet transmission method such as RTP, and outputs the packet to the network 111 (S103).

FIG. 28 is a diagram illustrating an example of an input image 1501 and a transmission image 1502 according to the sixth embodiment. As shown in the figure, the transmission image 1502 is a part of the input image 1501, is cut out from the input image by the image cutout unit 1412, and is output to the image encoding unit 104.

Since the input image 1501 includes a speaker image 1503 and a face area 1504, the face detection unit 103 can detect the center coordinates (x1, y1) and the radius R of the face area from the input image 1501. . Therefore, since the face area can be detected from a wider range of images, the partner image can be processed based on the actual speaker position.

As described above, in the present embodiment, the face detection unit 103 detects a face region from a wide range including the outside of the transmission image region, and the image processing unit 108 detects the horizontal and horizontal coordinates of the center coordinate and the reference coordinate of the face region. Another image is superimposed on the partner image according to the distance and direction in the vertical direction.

As a result, even if the speaker does not display the self-portrait on the monitor, the speaker can see the position and size of the image superimposed on the partner image on the monitor and how much the speaker is shifted in which direction on the screen. Can be determined. At this time, even if the speaker's face area exists outside the transmission image area, it is possible to perform superimposed image display according to the position of the face area accurately.

(Embodiment 7)
The image communication apparatus according to the seventh embodiment is an apparatus that detects a face region from a captured image acquired from a camera that captures a wider range using two cameras having different imaging ranges.

FIG. 29 is a block diagram illustrating a configuration of the image communication apparatus 1600 according to the seventh embodiment. The image communication apparatus 1600 shown in the figure acquires not only the first captured image captured by the camera 101 but also the second captured image captured by the camera 1601. As shown in FIG. 29, the image communication apparatus 1600 is different from the image communication apparatus 100 of FIG. 1 in that an image input unit 1602 and a face detection unit 1603 are newly provided. 29, the same reference numerals as those in FIG. 1 perform the same processing as that in the first embodiment, and thus description thereof will be omitted here, and different points will be mainly described.

The image input unit 1602 is an interface connected to a camera 1601 that captures an image, and acquires a second captured image captured by the camera 1601. The image input unit 1602 outputs the second captured image to the face detection unit 1603 as a second input image.

Note that the image input unit 102 outputs the first captured image captured by the camera 101 to the face detection unit 1603 as a first input image.

Here, FIG. 30 is a diagram showing a positional relationship between the two

cameras

101 and 1601 and the speaker 1701. As shown in FIG. 30, the camera 101 and the camera 1601 are installed in the upper center of the monitor 110. An area that can be imaged by the camera 101 is an imaging area 202, and an area that can be imaged by the camera 1601 is an imaging area 1702.

30, it is assumed that the camera 1601 is installed in the vicinity of the camera 101 and further has a zoom rate smaller than that of the camera 101 or is equipped with a wide-angle lens. Thus, the imaging area 1702 of the camera 1601 substantially includes the imaging area 202 of the camera 101 and is wider than the imaging area 202.

29, the face detection unit 1603 detects a face area from the second input image input from the image input unit 1602. That is, the face detection unit 1603 detects a face area from the second captured image captured by the camera 1601 that captures a wider range. For example, the face detection unit 1603 performs face area detection processing by a method such as template matching or ellipse detection, and information such as a face detection flag indicating the presence or absence of a face area, center coordinates, and the radius of the face area is an image processing unit. It outputs to 108.

Further, the face detection unit 1603 outputs the first input image input from the image input unit 102 to the image encoding unit 104. That is, the face detection unit 1603 does not perform face detection processing from the first captured image captured by the camera 101. Then, the image encoding unit 104 encodes the first input image input from the image input unit 102 via the face detection unit 1603.

Thereby, the image communication apparatus 1600 according to the seventh embodiment can detect a face area from a wider range, and thus can generate a processed image reflecting the position of the speaker 1701 more accurately.

Note that the second captured image captured by the camera 1601 is not transmitted to another image communication apparatus or the like and is used only for detecting the face area, so that the image quality may not be good. That is, an inexpensive camera with low accuracy can be used.

Next, the operation at the time of image transmission of the image communication apparatus 1600 having the above configuration will be described using the flowchart shown in FIG. FIG. 31 is a flowchart showing transmission processing of the image communication apparatus 1600 according to the seventh embodiment. Since the operation at the time of image reception is the same as that in Embodiment 1 (FIG. 3), description thereof is omitted here.

The operation of the flowchart shown in FIG. 31 is stored as a control program in a storage device (not shown) such as a ROM or a flash memory, and is controlled by a CPU (not shown).

The image input unit 102 acquires an uncompressed image in units of frames from the camera 101 and outputs it to the face detection unit 1603. Further, the image input unit 1602 acquires an uncompressed image from the camera 1601 in units of frames, and outputs it to the face detection unit 1603 (S801).

The face detection unit 1603 performs face area detection from the uncompressed second captured image input from the image input unit 1602 by a technique such as template matching or ellipse detection (S802). Then, the face detection unit 1603 calculates information such as a face detection flag, the center coordinates of the face area, and the radius of the face area. The face detection unit 1603 outputs the face detection flag, the face area coordinates converted into the coordinate system of the first input image 1801, and information such as the face radius to the image processing unit 108 and is input from the image input unit 102. The first input image 1801 is output to the image encoding unit 104.

The image encoding unit 104 encodes the first input image 1801 having a small size, and outputs the image data to the image transmission unit 105. The image transmitting unit 105 converts the image data input from the image encoding unit 104 into RTP packets according to a packet transmission method such as RTP, and outputs the packet to the network 111 (S803).

FIG. 32 is a diagram illustrating an example of the first input image 1801 and the second input image 1802 according to the seventh embodiment.

The first input image 1801 is an image for transmission that is captured by the camera 101, acquired by the image input unit 102, and transmitted to another image communication apparatus. The second input image 1802 is a face detection image that is captured by the camera 1601, acquired by the image input unit 1602, and for detecting a face area by the face detection unit 1603. As shown in FIG. 32, since the second input image 1802 input from the image input unit 1602 has a wider angle than the first input image 1801, the face detection unit 1603 determines the center of the face area according to Equations 8 to 10. The coordinates and the radius information of the face area are converted into the coordinate system of the first input image 1801.

(Formula 8) x1 ′ = M × x1
(Formula 9) y1 ′ = M × y1
(Formula 10) R ′ = M × R

In Expressions 8 to 10, x1 ′ and y1 ′ are the coordinates of the face area after conversion into the coordinate system of the first input image 1801, and R ′ is the radius of the face area after conversion. By multiplying the magnification M by the face region coordinates (x1, y1) in the input coordinate system of the second input image 1802 and the radius R, it is calculated as being approximate to the coordinate system of the first input image 1801. The magnification M is calculated by the following formula 11 or formula 12.

(Formula 11) M = width_1 / width_2
(Formula 12) M = height_1 / height_2

32, width_1 and height_1 are the horizontal size and vertical size of the first input image 1801, respectively, and width_2 and height_2 are the horizontal size and vertical size of the second input image 1802, respectively.

Note that the transformation of the coordinate system can be used elsewhere, and is not limited to the method disclosed in this embodiment.

As shown in FIG. 30, even if the speaker 1701 is out of the imaging area 202 of the camera 101, when the speaker 1701 is within the imaging area 1702 of the camera 1601, as shown in FIG. A speaker image 1803 exists at 1802. Since the face detection unit 1603 detects the face area 1804 from the second input image 1802, it correctly detects the face area 1804 even when the speaker image 1803 does not exist in the first input image 1801 for transmission. Can do.

As described above, in this embodiment, the face detection unit 1603 performs the face detection process from the second input image acquired from the camera 1601 that captures an image having a wider angle than the camera 101. That is, the center coordinates of the face area are determined from a wide area including outside the area of the transmission image (first input image), and the image processing unit 108 determines the horizontal and vertical distances between the center coordinates of the face area and the reference coordinates. In accordance with the direction, another image is superimposed and displayed on the partner image.

As a result, the speaker does not display the self-portrait on the monitor, but looks at the position and size of the image superimposed on the partner image on the monitor to determine how much the speaker is displaced in which direction on the screen. Can be determined. At this time, even if the speaker's face area exists outside the transmission image area, it is possible to perform superimposed image display according to the position of the face area accurately.

As mentioned above, although the image communication apparatus and the image communication method of the present invention have been described based on the embodiments, the present invention is not limited to these embodiments. Unless it deviates from the meaning of this invention, the form which carried out the various deformation | transformation which those skilled in the art will think to the said embodiment, and the form constructed | assembled combining the component in a different embodiment is also contained in the scope of the present invention. .

For example, the image communication apparatus of the present invention may enlarge the partner image based on the difference between the center coordinates of the face area and the reference coordinates. Specifically, it is as follows.

FIG. 33 is a diagram illustrating an example of the partner image 1901 and the processed image 1911. FIG. 33B is a view in which a region 1903 excluding the predetermined region 1902 is cut out from the counterpart image 1901 in FIG. 33A and the cut-out region 1903 is enlarged.

For example, the parameter calculation unit 123 calculates the following four as image processing parameters.

Image processing parameters for clipping and enlargement Horizontal cut-out position: left end or right end of counterpart image 1901 Horizontal cut-out size W (dx)
3. Vertical cut-out position: upper end or lower end of the partner image 1901 Vertical cut-out size ... H (dy)

Each image processing parameter is calculated by the parameter calculation unit 123 in the same manner as in the first embodiment, for example. In other words, parameter calculation is performed by replacing the horizontal cutout position with the horizontal overlap start position, the horizontal cutout size with the horizontal overlap size, the vertical cutout position with the vertical overlap start position, and the vertical cutout size with the vertical overlap size. The unit 123 can calculate each image processing parameter.

In this way, when the speaker is on the left side of the reference position, that is, when the face area is on the right side of the reference coordinates in the captured image, an area having a width W (dx) from the left end of the counterpart image 1901 The area excluding is expanded. Conversely, if the speaker is on the right side of the reference position, that is, if the face area is on the left side of the reference coordinates in the captured image, the area obtained by removing the area of the width W (dx) from the right side of the partner image 1901 Is enlarged.

Similarly, when the speaker is above the reference position, that is, when the face area exists above the reference coordinates in the captured image, the area obtained by removing the area of the width H (dy) from the upper end of the partner image 1901 Is enlarged. Similarly, when the speaker is below the reference position, that is, when the face area is below the reference coordinates in the captured image, the area of the width H (dy) is excluded from the lower end of the partner image 1901. The area is enlarged.

Note that the method of determining the horizontal cutout position and the vertical cutout position may be the reverse of the first embodiment. That is, it may be the right end when dx is positive and the left end when dx is negative. Similarly, the lower end may be used when dy is positive, and the upper end may be used when negative.

In each embodiment, the center coordinates indicating the center of the face area are used as the face area coordinates. However, any value may be used as long as the value indicates the position of the face area. In each embodiment, the radius of the face area is used as the size of the face area. However, any shape may be used as long as it indicates the size of the face area. For example, the area of the face region or the number of pixels may be used.

In the embodiment of the present invention, the superimposition process, the projective transformation process, the blur process, and the size change process have been described as the techniques for processing the counterpart image. However, the counterpart image may be processed by other techniques. .

At this time, the image processing unit included in the image communication apparatus of the present invention calculates a difference between at least one of the position and size of the face area detected by the face detection unit and a predetermined reference, and the larger the calculated difference, the more the processing is performed. The partner image is processed so that the previous partner image and the processed image after processing are greatly different. Further, at this time, if the position of the face area is away from the reference position in a predetermined direction, the partner image is processed so as to indicate the direction to be returned so that the speaker can return to the reference position. Is preferred.

Also, the coordinates of the face may be detected using an IC tag that can specify the speaker position.

As described in the seventh embodiment, the image communication apparatus of the present invention may acquire captured images from two physically different cameras. That is, you may acquire the 1st captured image imaged with the 1st camera, and the 2nd captured image imaged with the 2nd camera.

Also, the image communication apparatus of the present invention may not include the image encoding unit 104 and the image decoding unit 107. FIG. 34 is a block diagram showing an example of a different form of the image communication apparatus of the present invention.

As shown in FIG. 34, image communication apparatus 2000 is not provided with image encoding section 104 and image decoding section 107, as compared with image communication apparatus 100 shown in FIG. Is different. That is, the image transmission unit 105 transmits the captured image acquired by the image input unit 102 without encoding. Similarly, the image receiving unit 106 receives a non-encoded counterpart image from another image communication apparatus via the network 111.

As described above, for example, when there is a margin in the bandwidth of the network 111, it is not necessary to perform encoding and decoding when performing image communication. Therefore, the image communication apparatus performs image coding that performs image compression coding. And an image decoding unit that performs decompression decoding of an image may not be provided.

As described above, the present invention can be realized not only as an image communication apparatus and an image communication method, but also as a program for causing a computer to execute the image communication method according to the present embodiment. Further, it may be realized as a computer-readable recording medium such as a CD-ROM for recording the program. Furthermore, it may be realized as information, data, or a signal indicating the program. These programs, information, data, and signals may be distributed via a communication network such as the Internet.

Further, in the present invention, some or all of the constituent elements constituting the image communication apparatus may be configured from one system LSI. The system LSI is an ultra-multifunctional LSI manufactured by integrating a plurality of components on a single chip. Specifically, the system LSI is a computer system including a microprocessor, a ROM, a RAM, and the like. .

The image transmission apparatus of the present invention has an effect that it is possible to guide the speaker to the reference position while viewing the processed partner image without displaying the self-portrait. It can be used for a TV conference device.

100, 600, 800, 1000, 1200, 1400, 1600, 2000

Image communication apparatus

101, 1601

Camera

102, 1602

Image input unit

103, 1603 Face detection unit 104 Image encoding unit 105 Image transmission unit 106 Image reception unit 107 Image

decoding Conversion unit

108, 608, 808, 1008, 1208 image processing unit 109 image output unit 110 monitor 111 network 121 determination unit 122

buffer

123, 623, 823, 1023 parameter calculation unit 124

image superimposition unit

201, 1701

speaker

202, 1702

Areas

301, 501, 511, 901, 911, 1101, 1111, 1311, 1501

Input images

302, 502, 512, 902, 912, 1102, 1112, 1312, 1503, 1803 Speaker images 303, 50 513, 903, 913, 1103, 1113, 1303, 1313, 1504, 1804

Facial area

401, 521, 701, 921, 1121, 1301, 1901

Corresponding image

402, 522, 702, 922, 1122, 1123, 1911 Processed

image

403, 523 Superimposed image 624 Projection conversion unit 824 Blur processing unit 1024 Size change unit 1225 Reference coordinate setting unit 1302 Counter image 1412 Image cutout unit 1502 Transmission image 1801 First input image 1802

Second input image

1902, 1903

Claims

An image communication device for communicating image data with another image communication device via a network,
An image input unit for acquiring a captured image captured by the camera;
An image transmission unit that transmits first image data including a captured image acquired by the image input unit;
A face detection unit for detecting a face region from the captured image acquired by the image input unit;
An image receiving unit for receiving second image data including the counterpart image transmitted from the other image communication device;
An image processing unit that generates a processed image by processing the counterpart image included in the second image data received by the image receiving unit;
An image output unit that outputs a processed image generated by the image processing unit to a display device;
The image processing unit
The difference between at least one of the position and size of the face area detected by the face detection unit and a predetermined reference is calculated, and the larger the calculated difference is, the greater the difference between the counterpart image and the processed image is. An image communication apparatus for processing the counterpart image.
The face detection unit detects face area coordinates indicating the position of the face area in the captured image;
The image processing unit calculates a difference absolute value between the face region coordinates and a predetermined reference coordinate, and when the calculated difference absolute value is larger than a predetermined threshold, the larger the difference absolute value, The image communication apparatus according to claim 1, wherein the processed image is generated by processing the counterpart image so that the counterpart image and the processed image are greatly different.
The image processing unit superimposes the superimposed image on the counterpart image such that the larger the absolute value of the difference between the face area coordinates and the reference coordinates, the larger the predetermined superimposed image area. The image communication apparatus according to claim 2, wherein an image is generated.
The image processing unit, when the face region coordinates are on the right side of the reference coordinates, from the left end of the counterpart image, or when the face region coordinates are on the left side of the reference coordinates, from the right end of the counterpart image, The image communication apparatus according to claim 3, wherein the superimposed image is superimposed on the partner image so that the area of the superimposed image increases as the absolute difference value in the horizontal direction between the face region coordinates and the reference coordinates increases.
The image processing unit, when the face area coordinates are above the reference coordinates, from the upper end of the counterpart image, or when the face area coordinates are below the reference coordinates, from the lower end of the counterpart image The image communication apparatus according to claim 3, wherein the superimposed image is superimposed on the partner image such that the larger the absolute difference in the vertical direction between the face region coordinates and the reference coordinates, the larger the area of the superimposed image.
The face detection unit further determines whether or not a face area is in the captured image, generates a flag indicating the presence or absence of the face area,
The image communication device according to claim 3, wherein the image processing unit superimposes the superimposed image of a predetermined area on a predetermined region of the counterpart image when the flag indicates that there is no face region.
The image communication device further includes:
A buffer for storing the face area coordinates detected by the face detection unit;
When the flag indicates that a face area is not in the captured image, the image processing unit starts from the left end of the counterpart image when the face area coordinates stored in the buffer are on the right side of the reference coordinates. Or from the right end of the counterpart image when it is on the left side of the reference coordinates, and from the top end of the counterpart image when the face area coordinates stored in the buffer are above the reference coordinates, or The image communication apparatus according to claim 6, wherein the superimposed image having a predetermined area is superimposed on the partner image from a lower end of the partner image when the coordinate is below the reference coordinates.
The image processing unit generates the processed image by projective transformation of the counterpart image such that the gradient of the projective transformation increases as the difference absolute value between the face area coordinates and the reference coordinates increases. The image communication apparatus described.
The image communication apparatus according to claim 2, wherein the image processing unit enlarges the counterpart image with a larger enlargement ratio as the difference absolute value between the face area coordinates and the reference coordinates is larger.
The image communication device further includes:
A reference coordinate setting unit configured to detect a face region from a partner image included in second image data received by the image receiving unit and set a face region coordinate indicating a position of the detected face region as the reference coordinate. 2. The image communication apparatus according to 2.
The face detection unit detects a size of the face region;
The image processing unit calculates a difference absolute value between the size of the face area and a predetermined reference size, and when the calculated difference absolute value is larger than a predetermined threshold, the larger the difference absolute value, The image communication apparatus according to claim 1, wherein the processed image is generated by processing the counterpart image so that the counterpart image and the processed image are greatly different.
The said image process part produces | generates the said processed image by blurring the said other party image so that the amount of blurring becomes large, so that the difference absolute value of the size of the said face area | region and the said reference | standard size is large. Image communication device.
The image processing unit enlarges the counterpart image when the size of the face area is larger than the reference size, and reduces the counterpart image when the size of the face area is smaller than the reference size. Generating the processed image;
The image communication apparatus according to claim 11, wherein the enlargement ratio or the reduction ratio increases as the difference absolute value between the size of the face area and the reference size increases.
An image communication method for communicating image data with another image communication apparatus via a network,
An image input step for acquiring a captured image captured by the camera;
An image transmission step of transmitting first image data including the captured image acquired in the image input step;
A face detection step of detecting a face region from the captured image acquired in the image input step;
An image receiving step of receiving second image data including the counterpart image transmitted from the other image communication device;
An image processing step for generating a processed image by processing the counterpart image included in the second image data received in the image receiving step;
An image output step of outputting the processed image generated in the image processing step to a display device,
In the image processing step,
A difference between at least one of the position and size of the face area detected in the face detection step and a predetermined reference is calculated, and the larger the calculated difference is, the larger the partner image and the processed image differ. An image communication method for processing the counterpart image.
An integrated circuit that communicates image data with another image communication device via a network,
An image input unit for acquiring a captured image captured by the camera;
An image transmission unit that transmits first image data including a captured image acquired by the image input unit;
A face detection unit for detecting a face region from the captured image acquired by the image input unit;
An image receiving unit for receiving second image data including the counterpart image transmitted from the other image communication device;
An image processing unit that generates a processed image by processing the counterpart image included in the second image data received by the image receiving unit;
An image output unit that outputs a processed image generated by the image processing unit to a display device;
The image processing unit
The difference between at least one of the position and size of the face area detected by the face detection unit and a predetermined reference is calculated, and the larger the calculated difference is, the larger the partner image and the processed image differ. An integrated circuit for processing the counterpart image.