JP2017022600A - Image communication device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a composite image that does not cause discomfort during communication between users.SOLUTION: An image communication device 1 comprises: a plurality of imaging cameras 9 arranged along a screen 3; a position detection unit 23 for detecting positional information on the position of a person on the phone relative to the screen 3; a direction detection unit 24 for detecting direction information on the direction of the person on the phone relative to the screen 3; an image synthesizing unit 27 for selecting an image of the person on the phone from images taken by the plurality of imaging cameras 9 on the basis of the positional information and the direction information on the person on the phone and for generating a composite image by combining the selected image of the person on the phone and a plurality of images; an image transmission unit 29 for transmitting the composite image to an opposite person side; and an image reception unit 31 for receiving a composite image from the opposite person side and displaying it on the screen 3.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an image communication apparatus that performs bidirectional communication using an image.

  In recent years, it has become common to use a videophone system that performs bi-directional communication using images when performing a conference or a personal call. When a videophone system is used by a plurality of users, it is required to acquire an image with a wide field of view. Therefore, a configuration in which a plurality of cameras are arranged around the display screen is used. At that time, images acquired by a plurality of cameras are transmitted to the other party's videophone system.

  On the other hand, as a technique for synthesizing a plurality of images to obtain one composite image, those described in Patent Documents 1 and 2 below are known. According to such a technique, a plurality of images acquired by a plurality of cameras can be connected and combined with high accuracy.

JP 2011-124837 A JP 2014-86097 A

  However, simply applying the above-described conventional image composition technique to a videophone system makes it difficult to obtain a composite image that does not give a sense of incongruity between callers when a plurality of users move around freely. For example, the image of the caller in the composite image is taken from various directions, and it is likely that communication is difficult.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to provide an image communication apparatus capable of obtaining a composite image without a sense of incongruity during communication between users.

  In order to solve the above-described problem, an image communication apparatus according to an aspect of the present invention includes a plurality of imaging cameras arranged along a screen, a position detection unit that detects position information regarding the position of a caller with respect to the screen, Based on the direction detection unit that detects the direction information about the direction of the caller with respect to the screen, the position information and the direction information, the caller's image is selected from a plurality of images acquired by a plurality of imaging cameras, An image synthesizing unit that synthesizes the selected caller image and a plurality of images to generate a composite image, and an image that transmits one of the composite image and a plurality of images acquired by a plurality of imaging cameras A transmission unit, and an image display unit that displays on the screen a composite image generated based on an image of a conversation party who is a communication partner of the caller.

  According to the image communication apparatus of the above aspect, information on the position of the caller with respect to the screen is detected, information on the direction of the caller with respect to the screen is acquired, and a plurality of pieces arranged along the screen based on the information The image of the caller is selected from the images acquired by the imaging cameras, and a composite image is generated using the images and the plurality of images acquired by the plurality of imaging cameras. Images or composite images acquired by a plurality of imaging cameras are transmitted to the conversation person side, and a composite image generated based on the conversation person side images is displayed on the screen. With such a configuration, an image of the caller who does not feel uncomfortable during communication for the conversation person can be included in the composite image and displayed based on the position and orientation of the caller. As a result, smooth communication between the interlocutor and the caller can be realized.

  Here, the image composition unit may select the image of the imaging camera closest to the line extended in the direction specified by the direction information from the position of the caller specified by the position information as the image of the caller. Good. In this way, the image of the caller can be selected as a directly facing image, and the image of the caller who does not feel uncomfortable during communication for the conversation person can be included in the composite image and displayed. As a result, smooth communication between the interlocutor and the caller can be realized accurately.

  In addition, an information acquisition unit that acquires position information regarding the position of the conversation person on the screen may be further provided, and the image composition unit may correct the composite image based on the position information of the conversation person. In this case, based on the positional relationship between the interlocutor and the caller, an image of the caller who does not feel uncomfortable during communication for the interlocutor can be included in the composite image and displayed. As a result, smooth communication between the interlocutor and the caller can be realized accurately.

  Further, the image composition unit reflects the position of the conversation person and the position of the caller in the virtual space across the screen, and the image of the imaging camera closest to the line connecting the conversation person and the caller The composite image may be corrected so that is selected as the caller's image. In this case, an image of the conversation person seen from the front can be displayed as a composite image for the conversation person. As a result, it is possible to display an image of the caller who is close to the actual conversation in the virtual space.

  Moreover, the position detection unit may include a camera that acquires a distance image arranged along the screen, and the direction detection unit may include a plurality of directional microphones arranged. In this case, information regarding the position or orientation of the caller with respect to the screen can be acquired efficiently.

  ADVANTAGE OF THE INVENTION According to this invention, the composite image without a sense of incongruity at the time of communication between users can be obtained.

1 is a perspective view illustrating a configuration of an image communication apparatus according to a preferred embodiment of the present invention. FIG. 3 is a plan view of the field of view of the imaging camera 9 of FIG. 1 viewed from above along the screen 3. FIG. 3 is a side view of the field of view of the imaging camera 9 and the distance measuring camera 11 of FIG. 1 as viewed from the side along the screen 3. It is a block diagram which shows the function structure of the image processing apparatus 7 of FIG. It is a conceptual diagram which shows the method of the image selection by the image selection part 25 of FIG. It is a figure which shows the image of the production | generation of the composite image by the image composition part 27 of FIG. 3 is a flowchart showing an operation procedure of image communication by the image communication apparatus 1 of FIG. 1. It is a side view which shows the structure of the imaging camera 9 and the ranging camera 11 concerning the modification of this invention. It is a side view which shows the structure of the camera 35 concerning the modification of this invention. It is a perspective view which shows arrangement | positioning of the ranging camera 11 and the imaging camera 9 in the modification of this invention. It is a perspective view which shows arrangement | positioning of the ranging camera 11 and the imaging camera 9 in the modification of this invention. It is a perspective view which shows arrangement | positioning of the ranging camera 11 in the modification of this invention.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of an image communication apparatus according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are denoted by the same reference numerals, and redundant description is omitted.

  An image communication apparatus 1 according to a preferred embodiment of the present invention is an apparatus for performing communication by transmitting and receiving images between remote locations. By providing this image communication device 1 as a set, a remote location is provided. A videophone system that communicates with reference to images among a plurality of users is configured.

  FIG. 1 shows a schematic configuration of an image communication apparatus 1 according to the present embodiment. As shown in FIG. 1, the image communication apparatus 1 includes a screen 3 arranged so as to face a plurality of users U, and an image display that displays an image by projecting image light onto the front surface of the screen 3. An apparatus (projector, image display unit) 5 and an image processing apparatus 7 that executes image processing are provided. The image communication apparatus 1 is connected to an apparatus having the same configuration (hereinafter referred to as “opposing apparatus”) via a communication network, and transmits a composite image generated by the image communication apparatus 1 to the opposing apparatus. At the same time, the composite image generated by the opposite device is received, and the composite image is displayed on the screen 3.

  The screen 3 includes a plurality of imaging cameras 9 that capture images on the front surface (image display surface) side of the screen 3, a plurality of ranging cameras 11 that acquire distance images on the front surface side of the screen 3, A pulse light source 13 that emits near-infrared pulsed light and a plurality of directional microphones 15 that detect the direction of a speaker who is speaking out of a plurality of users U are attached. The ranging camera 11 and the pulse light source 13 function as a position detection unit that detects position information regarding the position of the caller with respect to the screen 3, and the directional microphone 15 indicates direction information regarding the direction of the caller with respect to the screen 3. It functions as a direction detection unit for detection.

  The imaging camera 9 is an image sensor that is two-dimensionally arranged along the front surface of the screen 3 at equal intervals so that an image on the front surface side of the screen 3 can be captured through a minute hole formed in the screen 3. It is fixed to the rear side of the screen 3. For example, five imaging cameras 9 are arranged at equal intervals along the long side direction of the screen 3, and the five arrays are arranged in two rows in the short side direction.

  Two distance measuring cameras 11 are fixed side by side at the upper center of the front surface of the screen 3, and a pulse light source 13 is disposed between the distance measuring cameras 11. The distance measuring camera 11 obtains a distance image including distance information for each pixel by measuring in real time for each pixel the time when the light pulse emitted by the pulse light source 13 returns to the target. ) Type known image sensor. The operations of the distance measuring camera 11 and the pulse light source 13 are controlled by the image processing device 7, whereby a distance image is acquired in real time.

  A plurality of directional microphones 15 are fixed side by side at the lower end of the front surface of the screen 3 and detect an audio signal emitted from a sound source including the user U on the front surface side of the screen 3. The sound signal detected by these directional microphones 15 is processed by the image processing device 7 so that the arrival direction of the sound from the sound source including the user U with respect to the surface of the screen 3 is detected in real time. As this detection method, a method of calculating the direction of arrival of sound using a known microphone array (for example, correlation function method, delay sum array method) is used.

FIG. 2 is a plan view of the imaging camera 9 viewed from above along the screen 3, and FIG. 3 is a side view of the imaging camera 9 and ranging camera 11 viewed from the side along the screen 3. FIG. As shown in these drawings, the imaging camera 9, the distance from the screen 3 is the focal length and the angle so as to enable imaging an image of the space S ₀ of the predetermined range is set. Furthermore, imaging camera 9, in the space S _0, their optical axis and angle as the viewing V ₁ of the imaging camera 9 adjacent vertically and horizontally overlap is set. Thereby, when the image acquired by the imaging camera 9 is combined to generate a combined image, the discontinuity of the object captured in the image can be reduced. In addition, in order to reduce the generation of blind spots in the space S ₀ , the ranging camera 11 is set to have a field angle that has a wider field of view V ₂ than the imaging camera 9, and The optical axis is directed downward.

  Next, the configuration of the image processing device 7 will be described with reference to FIG. FIG. 4 is a block diagram showing a functional configuration of the image processing apparatus 7. The image processing apparatus 7 is physically composed of a data processing apparatus such as a personal computer or an image processing device, an arithmetic processing circuit such as a CPU, a memory such as a RAM or a ROM, a data storage device such as a hard disk apparatus, and a communication. Built-in device. As shown in FIG. 4, the image processing apparatus 7 includes a position information acquisition unit 21, a position detection unit 23, a direction detection unit 24, an image selection unit 25, an image composition unit 27, and an image transmission unit as shown in FIG. 29 and an image receiving unit (image display unit) 31.

The position detection unit 23 of the image processing device 7 is a caller with respect to the screen 3 using the distance images detected by the plurality of ranging cameras 11 and the plurality of images acquired by the plurality of imaging cameras 9. Position information regarding the positions of a plurality of users U is detected. Specifically, the position detection unit 23 performs a known face recognition process on each image acquired by the plurality of imaging cameras 9 to detect the face positions of the plurality of users U on each image. . When recognizing the face of the user U from the image, a known face recognition method using machine learning is used. Thereafter, the position detecting unit 23 starts the center position in the three-dimensional space based on the face position of the user U recognized on the image and the center position of the lens of the imaging camera 9 corresponding to the image. The position of the straight line passing through the user U's face position is specified. Further, the position detection unit 23 combines a plurality of distance images to generate a combined distance image (depth map) indicating the three-dimensional shape of the object located in the space S ₀ , and determines the combined distance image and each determined distance image. By using the information specifying the straight line corresponding to the face position of the user U, three-dimensional coordinates based on the screens 3 of the faces of the plurality of users U are calculated as position information. That is, the position detection unit 23 extracts an object that is determined to intersect the straight line from objects recognized to approximate a human shape in the composite distance image, and the three-dimensional coordinates of the object are determined by the user U's. Calculated as three-dimensional coordinates of the face. In addition, the position detection unit 23 outputs the detected three-dimensional coordinates of the faces of the plurality of users U to the direction detection unit 24 and the image selection unit 25. Here, the position detection unit 23 may detect the three-dimensional coordinates of the face of the user U by separately processing a plurality of distance images without synthesizing the distance images.

  The direction detection unit 24 continuously detects the arrival directions of sound from the sound source with respect to the surface of the screen 3 based on the plurality of audio signals obtained by the directional microphone 15 for the plurality of sound sources. Then, the direction detection unit 24 has the face directions of the plurality of users U detected by the position detection unit 23 based on the detected sound arrival directions equal to the detected sound arrival directions from the plurality of sound sources. The three-dimensional coordinates of the faces of the plurality of users U and the direction information related to the orientations of the faces of the users U are output to the image selection unit 25. Furthermore, the direction detection unit 24 transmits the acquired position information (three-dimensional coordinates) of the user U toward the opposing device.

  The position information acquisition unit 21 receives (acquires) position information related to the position of the conversation person who is the communication partner of the caller with respect to the screen from the opposite device. This position information is obtained as the position information of the caller's face in the opposite apparatus in the same manner as the image communication apparatus 1. Specifically, the position information acquisition unit 21 acquires three-dimensional coordinates based on the screen on the opposite device side of the conversation person's face.

The image selection unit 25 selects a synthesis source image from among a plurality of images acquired by the imaging camera 9 for each of a plurality of users U detected by the position detection unit 23. For example, for the user U whose face orientation is not detected, the image of the imaging camera 9 closest to the three-dimensional coordinate is selected based on the three-dimensional coordinate calculated for the user U. On the other hand, for the user U whose face orientation is detected, an image is selected as follows. That is, based on the three-dimensional coordinates of the face of the user U output from the direction detection unit 24 and the direction information of the user U, an image to be synthesized is selected from a plurality of images. FIG. 5 is a conceptual diagram showing an image selection method by the image selection unit 25. As shown in the figure, the image selection unit 25 sets the coordinates of the screen 3 and the coordinates of the imaging camera 9 in the virtual space, and further sets the coordinates of the screen 3 in the virtual space sandwiching the screen 3. The coordinates P ₂ corresponding to the three-dimensional coordinates of the _two users U ₁ and U ₂ detected by the position detection unit 23 are reflected on the reference. The virtual space here means that the talker is virtually arranged based on the position information with the screen 3 sandwiched in the real space including the users U ₁ and U ₂ who are callers, the screen 3 and the imaging camera 9. It is the space that was made. Then, the image selection unit 25 determines the position of the line L extended from the coordinate P ₂ of the user U ₁ in the virtual space in the direction specified by the direction information of the user U ₁ (eg, angle information α with respect to the screen 3). The imaging camera 9 corresponding to the position closest to the line L is recognized and selected. According to the example of FIG. 5, the third imaging camera 9 from the left is selected. Here, the imaging camera 9 which is selected for the user U _1, limited to imaging camera 3-dimensional coordinates of the face of the user U ₁ is within the angle of view. That is, the number of imaging camera 9 to be selected for the user U ₁ is dependent on the distance from the screen 3 of the user U _1.

Further, the image selection unit 25 may correct the selection result of the imaging camera 9 obtained as described above based on the position information regarding the position of the conversation person acquired by the position information acquisition unit 21. Then, the image selection unit 25 may select a synthesis source image from a plurality of images based on the corrected selection result. In this case, the image selection unit 25 further reflects the coordinates P ₁ corresponding to the three-dimensional coordinates interlocutor obtained in the virtual space. Then, the image selection unit 25 recognizes the position of the line L ₁ connecting the coordinate P ₂ of the coordinates P ₁ and user U ₁ of the interlocutor in the virtual space, which corresponds to the position closest to the line L ₁ to select the imaging camera 9 to the user U _1, corrects the selection result of the imaging camera 9. According to the example of FIG. 5, the fourth imaging camera 9 from the left is selected. Thereafter, the image selection unit 25 outputs the information of the imaging camera 9 selected for each of the plurality of users U ₁ and U ₂ and the images acquired by all the imaging cameras 9 to the image synthesis unit 27 in real time. .

  Returning to FIG. 4, the image composition unit 27 synthesizes the images acquired by the imaging camera 9 using the information of the imaging camera 9 selected for each of the plurality of users U, and generates a composite image. At this time, the image composition unit 27 combines the images corresponding to the background by combining the images of the imaging cameras 9 arranged in a two-dimensional manner on the screen 3 so as to correspond to the arrangement. Generate. At that time, the image composition unit 27 refers to the composite distance image so that the corresponding points coincide with each other at the boundary between the two images of the adjacent imaging cameras 9, that is, so as to correct the parallax. Adjust the position of the combination. Further, the image composition unit 27 superimposes images of a plurality of users U on the background composite image created in this way. Specifically, for each user U, an image of a part including the face of the user U is cut out from the selected image of the imaging camera 9 and cut out at a position corresponding to the three-dimensional coordinates of the face of the user U on the composite image. Superimpose images. When the image of the user U is cut out from the image, the search result of the three-dimensional coordinates of the face of the user U targeted for the composite distance image by the position detection unit 23 is referred to, and the three-dimensional coordinates are extracted on the image to be cut out. The position of the image of the face of the user U is specified by converting into the coordinates.

  The unit of image selection can be set as appropriate, and the image may be selected in units of the entire user U, or the image may be selected in units of the user U's part (face, neck, body, etc.). May be cut out. However, in the case where images are cut out and overlapped in units of user U, in order to prevent discontinuity of the images in the composite image, they correspond to each other at the boundary between the multiple images to be overlapped while referring to the composite distance image. It is preferable to adjust the overlapping position of the images so that the points coincide, that is, to correct the parallax.

FIG. 6 shows a generation image of the composite image by the image composition unit 27. FIGS. 6A to 6C show images acquired by the second imaging camera 9 from the left, the middle, and the second from the right, corresponding to FIG. 5, respectively. ) Shows a composite image generated by the image composition unit 27 using these images. Thus, with respect to the user U _1, as shown in FIG. 5, the image of the imaging camera 9 in the middle closest to the line extending from the user in the virtual space in the direction of orientation of the face, ie, and FIG. 6 (b) image is selected, the image G ₁ cut out from the image is synthesized in the composite image. On the other hand, for the user U ₂ whose face orientation is not detected, the image of the second imaging camera 9 from the right closest to the position of the user U ₂ , that is, the image of FIG. 6C is selected. the image G ₂ cut out from the image is synthesized in the composite image.

  Returning to FIG. 4 again, the image transmission unit 29 is obtained by repeating the processing of the position information acquisition unit 21, the position detection unit 23, the direction detection unit 24, the image selection unit 25, and the image composition unit 27. A plurality of composite images of the series are transmitted as continuous moving images to the opposing device on the conversation party side. The image receiving unit 31 continuously receives a time-series composite image generated based on the image of the conversation person from the opposite device, and sends the time-series composite image to the image display device 5 to transmit the screen 3. Displayed as a moving image.

  An operation procedure of image communication by the image communication apparatus 1 described above will be described with reference to FIG. FIG. 7 is a flowchart showing an operation procedure of image communication by the image communication apparatus 1.

  First, when the image processing apparatus 7 receives an instruction to start communication from the user U, the image communication process is started (step S01). Then, the position of the face of the user U who is a plurality of callers on the front side of the screen 3 is detected by the position detector 23 of the image processing device 7 (step S02). Next, the direction information regarding the orientation of the face of the user U is estimated by the direction detection unit 24 of the image processing device 7 (step S03). At the same time, the position information acquisition unit 21 of the image processing device 7 acquires information on the three-dimensional coordinates with respect to the screen of the conversation person from the opposing device (step S04).

  Thereafter, the image selection unit 25 of the image processing device 7 uses the positional information related to the positions of the faces of the plurality of users U and the direction information related to the orientations of the faces of the plurality of users U to perform imaging for each user U. A synthesis source image is selected from the images acquired by the camera 9 (step S05). Then, the image composition unit 27 of the image processing device 7 combines the images acquired by the plurality of imaging cameras 9 based on the selection result by the image selection unit 25 to generate a composite image (step S06). .

  Further, the generated composite image is transmitted to the opposite device by the image transmission unit 29 of the image processing device 7 each time (step S07). At the same time, the composite image is received from the opposing device by the image receiving unit 31 of the image processing device 7, and the received composite image is displayed on the screen 3 (step S08). The processes in steps S02 to S08 are repeated until an instruction to end communication from the user U is received (step S09), and real-time image communication between remote locations is executed.

  According to the image communication apparatus 1 described above, the position information related to the position of the caller relative to the screen 3 is detected, and the direction information related to the direction of the caller relative to the screen 3 is acquired. Based on the information, the screen 3 The image of the caller is selected from the images acquired by the plurality of imaging cameras 9 arranged along the line, and is synthesized using the image and the plurality of images acquired by the plurality of imaging cameras 9. An image is generated. Further, the composite image is transmitted to the dialog side, and the composite image received from the dialog side is displayed on the screen 3. With such a configuration, an image of the caller who does not feel uncomfortable during communication for the conversation person can be included in the composite image and displayed based on the position and orientation of the caller. In addition, it is possible to display an image of a conversation person who does not feel strange at the time of communication for the caller. As a result, smooth communication between the interlocutor and the caller can be realized.

  Specifically, based on the position of the caller's face relative to the screen and the direction of the caller's face, an image of the caller who does not feel uncomfortable during communication can be included in the composite image and displayed. . As a result, smooth communication between the interlocutor and the caller can be realized more accurately. In particular, the position of the caller is reflected in the virtual space, and the image of the imaging camera 9 closest to the line extending from the caller's position in the virtual space toward the caller is selected as the caller's image. Therefore, it is easy to reflect on the composite image an image obtained by viewing the caller from the front with respect to the talker during the dialogue between the talker and the talker. As a result, it is possible to display an image of the caller who is close to the actual conversation in the virtual space.

  Further, the position of the conversation person is further reflected in the virtual space across the screen, and the image of the imaging camera 9 closest to the line connecting the position of the conversation person and the position of the talker in the virtual space is called. So that it can be selected as the person's image. Thereby, it is possible to display an image obtained by viewing the caller from the front as a composite image with respect to the conversation person. As a result, it is possible to display an image of the caller who is close to the actual conversation in the virtual space.

  Further, since the position detecting unit 11 includes the distance measuring camera 11 and the direction detecting unit includes the directional microphone 15, information on the position or direction of the caller with respect to the screen 3 can be efficiently acquired. Can do.

  The present invention is not limited to the embodiment described above.

  In the above embodiment, the ranging camera 11 as the position detection unit and the imaging camera 9 for acquiring the synthesized image of the synthesized image are separately arranged, but these are the same on the screen 3. You may arrange at the place. FIG. 8 is a side view showing configurations of the imaging camera 9 and the distance measuring camera 11 according to the modification of the present invention. In the modification shown in the figure, a lens 32 and a dichroic mirror 33 that receive light that has passed through the hole 3a are arranged on the rear surface side of the hole 3a of the screen 3. The dichroic mirror 33 transmits near-infrared light for generating a distance image and reflects visible light detected by the imaging camera 9. Further, an imaging camera 9 including an imaging lens 9b and an imaging element 9a is disposed at a position adjacent to the reflection direction of visible light that has passed through the hole 3a with respect to the dichroic mirror 33. A distance measuring camera 11 including an imaging lens 11b and an imaging element 11a is disposed at a position adjacent to the transmission direction of near infrared light that has passed through the hole 3a with respect to the dichroic mirror 33. According to such a modification, the field of view can be reliably matched between the imaging camera 9 and the distance measuring camera 11, so that the parallax correction processing when generating the composite image can be simplified.

  In addition, as shown in FIG. 9, an imaging camera 9 and a distance measuring camera 11 may be replaced with an integrated camera. In the modification shown in the figure, a camera 35 including a lens 35 a that receives light that has passed through the hole 3 a and an imaging element 35 b is disposed on the rear surface side of the hole 3 a of the screen 3. This camera is an integrated image sensor having a function of generating a distance image by the TOF method and a function of generating a normal image by detecting visible light.

  In the above embodiment, the number and arrangement of the distance measurement camera 11 and the imaging camera 9 may be variously changed. 10 to 11 show the arrangement of the distance measuring camera 11 and the imaging camera 9 in a modification of the present invention. In these drawings, the directional microphone 15 is not shown.

  In the modification shown in FIG. 10, the number of ranging cameras 11 is set to four, arranged separately at the upper end and the lower end of the screen 3, and the imaging cameras 9 are arranged in one row in the short side direction of the screen 3. A configuration with additional minutes (a total of 5) is adopted. In the modification shown in FIG. 11, a configuration is adopted in which the imaging camera 9 and the distance measuring camera 11 are alternately arranged on the screen 3 along the short side direction and the long side direction.

  In the above embodiment, the distance measuring camera 11 may not be included. FIG. 12 shows the arrangement of the imaging camera 9 in a modification of the present invention. In FIG. 12, the directional microphone 15 is not shown.

  In the modification shown in FIG. 12, only the imaging camera 9 is provided on the screen 3. Then, a composite image is generated by combining the images acquired by these imaging cameras 9. These imaging cameras 9 also function as a position detection unit that detects the positions of the faces of a plurality of users U. That is, in this modification, the position detection unit 23 of the image processing device 7 generates a distance image by performing matching processing on the images of the two adjacent imaging cameras 9 as images of a stereo camera, and a plurality of imaging images A composite distance image is generated by combining a plurality of distance images obtained by the pair of cameras 9.

  In the above embodiment, the image selection unit 25 of the image processing apparatus 7 selects the image of the imaging camera 9 closest to the position of the user U when selecting the image of the user U. It is not limited to such a process. That is, for the user U, the image selection unit 25 of the above embodiment selects the image of the imaging camera 9 closest to the three-dimensional coordinate based on the three-dimensional coordinate calculated for the user U. On the other hand, as a modification, various image selection methods may be employed. For example, the direction of the face of the user U is detected using the images obtained by the plurality of imaging cameras 9 or the distance images obtained by the distance measuring camera 11, and imaging at a position close to the face direction is detected. An image of the camera 9 may be selected. For example, the face direction is detected by specifying the position of a part of the face such as the mouth and eyes of the user U and calculating the relationship between the positions. As a further modification, the line-of-sight direction of the user U is detected using the image obtained by the imaging camera 9 or the distance image obtained by the distance measuring camera 11, and the position of the user U's face is detected. You may select the image of the camera 9 for imaging in the position close | similar from the gaze direction vector extended in the detected gaze direction. An existing technique (for example, the configuration described in Japanese Patent No. 4604190, the configuration described in Japanese Patent No. 4517049) can be used as a mechanism for detecting the user's U gaze direction. According to such a modification, it is possible to display an image that facilitates communication with all users U for the interlocutor.

  In addition, the direction detection unit 24 of the above embodiment estimates the orientation of the user U's face based on the audio signal detected by the directional microphone 15, but the orientation of the user U's face is detected by the following method. May be. For example, the direction detection unit 24 may detect the orientation of the face of the user U by image processing using the images obtained by the plurality of imaging cameras 9 or the distance images obtained by the distance measuring camera 11. Good. At this time, the orientation of the face is detected by specifying the position of the face part such as the mouth and eyes of the caller on the image and calculating the relationship between these positions. In addition, the direction detection unit 24 detects the gaze direction of the user U using the image obtained by the imaging camera 9 or the distance image obtained by the distance measurement camera 11, and the direction of the face from the gaze direction. May be estimated.

  In the above embodiment, the image communication device 1 on the caller side generates a composite image including the image of the caller and transmits the composite image to the opposite device. The function of generating an image may be provided on the opposite device side. In this case, the image transmission unit 29 of the image communication apparatus 1 transmits the images of the plurality of imaging cameras 9 and the information detected by the position detection unit 23 and the direction detection unit 24 to the opposite device.

  In addition, the screen size and the resolution of the image displayed on the screen may be different between the image communication device 1 on the caller side and the opposite device on the dialog party side as the communication partner. Further, the arrangement and number of the plurality of imaging cameras 9 may be different between the image communication apparatus 1 on the caller side and the opposite apparatus as the communication counterpart, and the arrangement and number of the ranging cameras 11 may be different. May be different. In such a case, the image compositing unit 27 of the image communication device 1 generates the composite image when the user U in the virtual space separated by the screen 3 of the image communication device 1 and the screen of the opposite device. It is desirable to execute coordinate conversion processing, pixel interpolation processing, image enlargement processing, or image reduction processing so that the image of the user U is displayed at a position on the display of the opposing device corresponding to the position.

DESCRIPTION OF SYMBOLS 1 ... Image communication apparatus, 3 ... Screen (screen), 5 ... Image display apparatus (image display part), 7 ... Image processing apparatus, 9 ... Camera for imaging (position detection part), 11 ... Camera for distance measurement (position detection) Part), 13 ... pulse light source, 15 ... directional microphone (direction detection part), 21 ... position information acquisition part, 23 ... position detection part, 24 ... direction detection part, 25 ... image selection part (image composition part), 27 ... image combining unit, 29 ... image transmitting unit, 31 ... image receiving unit, 35 ... camera, U, _{U 1,} U 2 _... user (caller).

Claims (6)

A plurality of imaging cameras arranged along the screen;
A position detector for detecting position information related to the position of the caller with respect to the screen;
A direction detection unit for detecting direction information regarding the direction of the caller with respect to the screen;
Based on the position information and the direction information, the caller's image is selected from a plurality of images acquired by the plurality of imaging cameras, and the selected caller's image and the plurality of images are selected. An image composition unit that generates a composite image by combining
An image transmission unit that transmits one of the composite image and the plurality of images acquired by the plurality of imaging cameras;
An image display unit for displaying on the screen a composite image generated on the basis of an image of a conversation party that is a communication partner of the caller;
An image communication apparatus comprising: The image composition unit selects, as the caller's image, the image of the imaging camera that is closest to the line extended in the direction specified by the direction information from the position of the caller specified by the position information.
The image communication apparatus according to claim 1. An information acquisition unit that acquires position information related to the position of the dialog person on the screen;
The image composition unit corrects the composite image based on the position information of the interactor.
The image communication apparatus according to claim 1 or 2. The image synthesizing unit captures the image closest to a line connecting the conversation person and the talker when the position of the talker and the position of the talker are reflected on a space sandwiching the screen. Correcting the composite image so as to select the camera image as the caller image,
The image communication apparatus according to claim 3. The position detection unit includes a camera that acquires a distance image arranged along a screen.
The image communication apparatus according to claim 1. The direction detection unit includes a plurality of directional microphones arranged,
The image communication apparatus according to claim 1.

Images