JPWO2009119288A1 - Communication system and communication program - Google Patents

Abstract

The installation direction of the display 202 is detected based on the location of the person B in the living room Y, and the shooting direction of the camera 104 is changed based on the detection result, and the range of the scene of the living room X displayed on the display 202 is adjusted. . As a result, scenes at different points can be displayed on the display as a realistic image without a sense of incongruity.

Description

  The present invention relates to a communication system and a communication program that enable communication between different points via a network.

  A communication system that enables communication between different points via a network is progressing day by day due to an increase in communication line capacity represented by broadband, high performance of computers in the system, and the like.

  Some communication systems communicate high-quality color images bidirectionally in real time with audio. For example, in a video conference system in the business field, there is one that can talk with a plurality of life-size displayed persons with realism. is there. As a video conference system capable of bidirectional communication, there is a technique described in Patent Document 1. In the video conference system described in Patent Document 1, participants who gather at each conference place can interact through a display or a microphone.

  In addition, there is a communication system in which remote offices are always connected using a large screen display, etc., so that people in different offices can talk to each other so that they are in the same office. Is possible.

  In a communication system that connects remote offices, a scene of one office is photographed by a camera, and the photographed scene is displayed on the display of the other office. Usually, the scene displayed on the display is a range of scenes taken with the camera fixed, and the same image is seen from any position of the display, so that the presence as an image is poor.

Therefore, there is a technique of adjusting the captured image by controlling the operation of the remote camera in accordance with the user's line of sight, as in the technique described in Patent Document 2. By adjusting the captured image according to the user's line of sight, the user can see a realistic scene. However, the user's line of sight changes frequently, and if the shooting range of the camera is changed every time the line of sight is changed, there is a problem that the image on the display is frequently changed and is troublesome.
JP-T-2001-517395 JP-A-2005-27166

  Accordingly, it is an object and an object of the present invention to provide a communication system and a communication program for projecting scenes at different points on a display as a realistic image with no sense of incongruity.

In order to achieve the above object, a communication system according to the present invention provides:
A communication system that enables communication between a first point and a second point via a network,
A camera installed at the first point and shooting a scene of the first point;
A display installed at the second point and displaying a scene of the first point taken by the camera;
A detection unit that detects an installation direction of the display based on a location of the person at the second point;
An adjustment unit that adjusts the direction of the scene of the first point displayed on the display based on the result detected by the detection unit;
It is characterized by having.

In addition, the communication program according to the present invention includes:
A camera installed at a first point and shooting a scene of the first point;
A display installed at a second point and displaying a scene of the first point taken by the camera;
A communication program for enabling communication between the first point and the second point using a computer,
A detection step of detecting an installation direction of the display based on a location of the person at the second point;
An adjustment step of adjusting the direction of the scene of the first point displayed on the display based on the result detected by the detection step;
Is executed by a computer.

  According to the communication system and the communication program according to the present invention, scenes at different points can be displayed on the display as an image with no sense of incongruity and presence.

It is the schematic of a communication system. It is another schematic diagram of a communication system. It is the schematic which looked at the room X from upper direction. It is a flowchart figure which shows the operation | movement which detects the installation direction of the display on the basis of the whereabouts of the person B, and adjusts the range of the scene of the room X projected on a display. It is the schematic which looked at the room X and the room Y from upper direction. It is explanatory drawing which shows the state which installs the several camera of living room X and image | photographs a different direction with each camera. It is explanatory drawing which shows the image image | photographed with the camera. It is a flowchart figure which shows the operation | movement which adjusts the range of the scene of the living room X shown on a display in consideration of a person's position and face direction.

Explanation of symbols

1, 2 Communication device 101, 201 PC
102, 202 Display 103, 203 Speaker 104, 204 Camera 105, 205 Microphone 101A, 201A CPU
101B, 201B ROM
101C, 201C RAM
101D, 201D HDD

  [Outline of Communication System] FIG. 1 is a schematic view of a communication system according to the present invention.

  The room X (first point) and the room Y (second point) at different points are connected via the network 3 so that two-way communication is possible.

  A display 102 is installed in the living room X, and a display 202 is installed in the living room Y. The display 102 displays the scene of the room Y, and the display 202 displays the scene of the room X. For example, the person A enrolled in the room X can interact with the person B enrolled in the room Y using the communication system shown in FIG. Even if the person A is not in the room X, the person B in the room Y can see the scene of the room X.

  FIG. 2 is another schematic diagram of the communication system according to the present invention, and shows a typical configuration.

  The communication device 1 is installed in the living room X, and the communication device 2 is installed in the living room Y. The communication device 1 and the communication device 2 are connected via a network 3 to constitute a communication system as a whole.

  Since the communication device 1 and the communication device 2 have the same configuration, each configuration will be described based on the communication device 1.

  The communication device 1 includes a PC (computer) 101, a display 102, a speaker 103, a camera 104, and a microphone 105.

  The PC 101 transmits a signal to the communication device 2 and receives a signal from the communication device 2. The display 102, the speaker 103, and the like are connected to the PC 101, and the operation of the display 102 to which the PC 101 is connected is controlled by a predetermined program.

  The PC 101 incorporates a CPU (Central Processing Unit) 101A, a ROM (Read Only Memory) 101B, a RAM (Random Access Memory) 101C, an HDD (Hard Disc Drive) 101D, and the like.

  The CPU 101A controls the overall operation of the PC 101, and is connected to the ROM 101B, the RAM 101C, and the like. The CPU 101A reads various programs stored in the ROM 101B, develops them in the RAM 101C, and controls the operation of each unit. Further, the CPU 101A executes various processes in accordance with the program expanded in the RAM 101C, and stores the processing results in the RAM 101C. Then, the processing result stored in the RAM 101C is stored in a predetermined storage destination.

  The ROM 101B is a storage medium that stores programs, data, and the like in advance, and is configured by a magnetic or optical recording medium or a semiconductor memory.

  The RAM 101C forms a work area for temporarily storing data processed by various programs executed by the CPU 101A.

  The communication program according to the present invention is stored in the hard disk HDD 101D.

  The CPU 101A in the PC 101 controls the operation of the camera 104 and the like according to a predetermined program, thereby realizing communication with the communication device 2 in the living room Y.

  The display 102 displays the video of the room Y and the video of the person in the room Y, and the video displayed is taken by the camera 204 in the communication device 2. The display 102 is preferably a large screen with a high resolution in order to enhance a sense of reality, for example, a 50-inch display.

  The speaker 103 provides sound generated in the room Y in the room X, and may be built in the display 102 or may be separate from the display 102. The speaker 103 preferably has two or more channels.

  The camera 104 captures the room X or a person in the room X, and distributes the captured scene to the communication device 2 via the network 3.

  The microphone 105 collects sound generated in the room X, and the sound collected by the microphone 105 is distributed to the speaker 203 in the communication device 2 via the network 3. The microphone 105 is preferably a stereo microphone in order to enhance the sense of reality.

  FIG. 3 is a schematic view of the room X as viewed from above. As shown in FIG. 3, the camera 104 and the plurality of microphones 105 can rotate, and the CPU 101 </ b> A of the PC 101 rotates the camera 104 and the plurality of microphones 105 based on the location (position) of the person B in the living room Y. Let

  [Adjustment Operation of Image on Display] By the way, when the person B in the room Y sees the scene of the room X displayed on the display 202, the display 202 is regarded as one window, and the room X appears as if it is ahead of the display 202. It seems preferable that it looks like it from the viewpoint of presence. Therefore, based on the installation direction of the display 202 based on the location of the person B in the room Y, the shooting direction by the camera 104 is controlled to adjust the range of the scene of the room X displayed on the display 202. This point will be described in detail below.

  FIG. 4 is a flowchart showing an operation of detecting the installation direction of the display 202 based on the location of the person B and adjusting the range of the scene of the room X displayed on the display 202. The determination steps (steps S1 and S2) in FIG. 4 are executed by the CPU 201A of the PC 201 in cooperation with the ROM 201B and the RAM 201C based on a predetermined program.

  First, in the living room Y, it is determined whether or not there is a person within a predetermined range from the display 202 (step S1). A person far away from the display 202 is regarded as not seeing the scene of the room X through the display 202, and the range of the scene of the room X displayed on the display 202 for a person whose distance from the display 202 is within a predetermined range. Adjust. Whether or not a person is within a predetermined range from the display 202 is determined by analyzing the image taken by the camera 204 installed at the center position of the display 202 and the PC 201 based on a predetermined program. Specifically, for example, a known person or a person face detection technique is used, and a person whose size is larger than a predetermined value is targeted. Alternatively, the distance of the actual person may be calculated by using a plurality of cameras separately provided, and the person whose distance is within the threshold may be targeted.

  Whether or not a person is within a predetermined range may be another method, for example, a light emitting element and a light receiving element are installed on the display 202, and when light is emitted from the light emitting element, the reflected light of the light is received from the light receiving element. It is also possible to measure the time interval until the time of detection at, and determine whether there is a person within a predetermined range based on the time interval. Note that the present invention should not be operated when there are a plurality of persons as a result of the determination.

  When it is determined in step S1 that there is a person within a predetermined range of the distance from the display 202 (step S1; Yes), it is determined whether the motion amount of the person in the predetermined range is equal to or less than a predetermined value (step S2; Detection step). Even if it is determined that there is a person within the predetermined range, it may be considered that the person simply passes in front of the display 202 and does not see the scene of the room X displayed on the display 202. In addition, it is considered that a person who sees the scene of the room X displayed on the display 202 often does not move his body, such as sitting on a chair. Therefore, the scene of the room X displayed on the display 202 is adjusted for a person whose movement amount is equal to or less than a predetermined value, that is, a person who is not moving much.

  For the detection of the operation amount, a known technique, for example, Japanese Patent Application Laid-Open No. 2008-288797 can be used. Specifically, the determination is made based on the amount of movement of the person or the person's face in a certain time. For example, when a motion is detected at a certain interval, and a specific pixel in the image has not moved more than a certain time, the scene of the room X displayed on the display 202 is adjusted for that person.

  If the scene X of the living room X displayed on the display 202 is adjusted based on the movement amount of the person in this way, unnecessary adjustment operations for unrelated persons need not be executed. The determination operation in step S2 is executed by calculating the amount of movement of the person based on the image captured by the camera 204, but is not limited to this method.

  In step S2, when the movement amount of the person within the predetermined range is equal to or less than the predetermined value (step S2; Yes), the installation direction of the display 202 based on the person's whereabouts is detected (step S3; detection process). ). This point will be described with reference to FIG. In this embodiment, the CPU 201A functions as a detection unit by cooperating with the ROM 201B and the RAM 201C.

  FIG. 5 is a schematic view of the room X and the room Y as viewed from above. A camera 204 is installed at the central position a of the display 202. Assuming that the target person is B, in order to detect the installation direction of the display 202 based on the whereabouts of the person B, the whereabouts (position) of the person B in the photographed image is first detected. The whereabouts (position) of the person B is the position in the two-dimensional space in the horizontal direction, and it is not necessary to calculate a highly accurate angle. Therefore, if the distance from the camera 204 to the person is a fixed value, You can determine which angle you are at relative to the camera. Of course, the distance to the target person may be calculated by a known method as described above, and that value may be used. Next, the rotation angle of the camera 204 at the time of detection when the case where the camera 204 is oriented in the vertical direction with respect to the display 202 is set as a reference (for example, 0 degree) is acquired (however, there is no target person) Is preferably returned in the vertical direction). The sum of these two angles is an angle b (an angle formed by a line connecting the location of the person B and the center position a and the plane of the display 202) with the location of the person B as a reference. 202 is the installation direction.

  When the installation direction of the display 202 is detected with reference to the location of the person B in step S3, the direction of the camera 104 in the room X is controlled and detected in order to adjust the range of the scene of the room X displayed on the display 202. A scene in the room X in the direction that coincides with the installation direction is photographed (step S4; adjustment step). And the scene of the room X image | photographed with the camera 104 is displayed on the display 204 (step S5). Referring to FIG. 5, the camera 104 is rotated in a direction coinciding with the installation direction (angle b) of the display 202 with respect to the person B's whereabouts, and the scene of the room X is photographed. This is displayed on the display 202. The rotation of the camera 104 is controlled by the CPU 101A in the PC 101 based on a predetermined program (in this embodiment, the CPU 101A functions as an adjustment unit in cooperation with the ROM 101B and the RAM 101C). Thus, if the scene of the room X displayed on the display 202 is adjusted so as to coincide with the installation direction of the display 202 based on the location of the person B, the display 202 is regarded as one window, as if the display 202 It becomes visible to the person B as if there is a living room X first, which is preferable in terms of realism. In addition, since the scene displayed on the display 202 does not change frequently, it is possible to view a video with no sense of incongruity.

  The description of the flowchart in FIG. 4 is continued. After changing the shooting direction of the camera 104 as shown in steps S4 and S5, the operation of the microphone 105 in the living room X is controlled. In order to adjust the range of sound collected by the microphone 105, the direction of the microphone 105 in the room X is controlled, and the sound in the direction matching the installation direction detected in step S3 is collected (step S6; adjustment process). The sound collected by 105 is output by the speaker 203 in the living room Y (step S7). Referring to FIG. 5, sound is collected by rotating the microphone 105 in a direction that matches the installation direction (angle b) of the display 202 with respect to the location of the person B, and the collected sound is collected by the speaker 203 in the room Y. Output. The rotation of the microphone 105 is controlled by the CPU 101A in the PC 101 based on a predetermined program. Thus, if the range collected by the microphone 105 is adjusted so as to match the installation direction of the display 202 based on the location of the person B, the sound in the same direction as the direction seen by the person B is collected without omission. This is preferable from the viewpoint of the presence of sound.

  In FIG. 4, steps S6 and S7 are executed after steps S4 and S5. However, steps S6 and S7 are executed simultaneously with steps S4 and S5, or steps S6 and S7 are executed before steps S4 and S5. You may do it.

  Also, in FIG. 4, in order to adjust the range of the scene of the room X displayed on the display 202, the direction of the camera 104 in the room X is controlled, and the scene in the room X in the direction matching the detected installation direction is photographed. However, other methods are possible. For example, as shown in FIG. 6, a plurality of cameras 104 in the living room X are installed, and each camera is set to shoot in different directions. Then, the images taken by the respective cameras may be combined, and an image that matches the installation direction of the display 202 based on the whereabouts of the person B may be extracted from the combined image and displayed on the display 202. In addition, a wide-angle camera 104 is installed in the room X, and an image G1 that matches the installation direction of the display 202 based on the location of the person B is extracted from an image G (see FIG. 7) captured by the camera. You may display on the display 202. FIG.

  Further, in FIG. 4, in order to adjust the range of sound collected by the microphone 105, the direction of the microphone 105 in the room X is controlled, and the sound in the room X in the direction matching the detected installation direction is collected. Other methods are also conceivable. For example, the direction of the microphone 105 may be fixed, and the sensitivity of the microphone 105 may be changed to increase the sensitivity in a direction that matches the installation direction of the display 202 with respect to the location of the person B.

  Moreover, although it is the magnification of the image | video in the display 202, adjusting based on the distance of the display 202 and the person B in the living room Y is also considered. The distance between the display 202 and the person B in the room Y is detected based on the location (position) of the person B detected in step S3 in FIG. Specifically, the magnification of the image will be described. If the distance between the display 202 and the person B is short, the image on the display 202 is easy to see for the person B. Therefore, the shooting range of the camera 104 is widened and the image magnification is small and a wide range of scenes. Is displayed on the display 202. On the other hand, if the distance between the display 202 and the person B is long, the image on the display 202 is difficult for the person B to see. Therefore, the shooting range of the camera 104 is narrowed, and the scene with a large image magnification is displayed on the display 202. . Thus, by adjusting the magnification of the video, an image having a size suitable for the person B in the room Y can be displayed on the display 202. As a distance measuring method, the distance can be calculated by using a plurality of cameras as described above, and since a highly accurate distance is not necessary for this purpose, a known detection for detecting a person or a person's face The approximate distance may be obtained by using a technique and referring to the size and distance table.

  [Image Adjustment Operation Considering Face Direction of Person] In FIG. 4, the range of the scene of the room X displayed on the display 202 is adjusted based on the whereabouts of the person B. It is preferable to adjust the range of the scene to be projected depending on the orientation of the face in order to obtain a realistic image. Therefore, an operation for adjusting the range of the scene of the room X displayed on the display 202 in consideration of the direction of the face of the person B will be described.

  FIG. 8 is a flowchart showing an operation of adjusting the scene range of the room X displayed on the display 202 in consideration of the position of the person and the face orientation. The determination steps (steps S11, S12, and S13) in FIG. 8 are executed by the CPU 201A of the PC 201 in cooperation with the ROM 201B and the RAM 201C based on a predetermined program.

  Steps S11 and S12 in FIG. 8 are the same operations as steps S1 and S2 in FIG.

  In step S12, when the motion amount of the person within the predetermined range is equal to or less than the predetermined value (step S12; Yes), it is determined whether or not the person is facing the display 202 (step S13). Since there is a person who is not facing the display 202 even if the amount of movement is equal to or less than a predetermined value, the scene of the room X displayed on the display 202 is adjusted for a person whose person is facing the display 202. .

  Whether or not the person B in the room Y is looking at the display 202 is obtained by extracting the face of the person B from the image captured by the camera 204, and extracting the image information of the extracted face and the image pattern of the face facing the display 202. Judge by comparing.

  If it is determined in step S13 that the person B is facing the display 202 (step S13; Yes), the installation direction of the display 202 and the face direction of the person B are detected based on the whereabouts of the person B (step S14). The method for detecting the installation direction of the display 202 based on the location of the person B is as described in FIG. The direction of the face of the person B is executed by analyzing a photographed image of the camera 204 installed at the central position a. The orientation of the face of the person B is detected from the position of eyes, nose, mouth, etc. by analyzing the image taken by the camera 204 (for example, detected by a technique described in Japanese Patent Laid-Open No. 2000-347692). In step S14, the sum of the face orientation angle obtained from the face orientation detection result and the angle with respect to the display obtained as described above is an angle necessary for the operation of the camera 104 and the microphone 105 in the living room X.

  Then, in order to adjust the range of the scene X of the room X displayed on the display 202, the direction of the camera 104 in the room X is controlled, and the scene in the room X in the direction coinciding with the specific direction determined in step S14 is photographed. (Step S15; adjustment process). Then, the scene of the room X taken by the camera 104 is displayed on the display 204 (step S16). Further, in order to adjust the range of sound collected by the microphone 105, the direction of the microphone 105 in the living room X is controlled, and the sound in the direction matching the specific direction determined in step S14 is collected (step S17; adjustment process) ), The sound collected by the microphone 105 is output by the speaker 203 in the room Y (step S18). As described above, by adjusting the range to be projected in consideration of the face direction of the person B, it is possible to project a realistic image on the display 202.

  The embodiments of the present invention have been described with reference to the drawings. However, the present invention is not limited to the embodiments, and the present invention can be modified or added without departing from the gist of the present invention. included.

Claims (14)

A communication system that enables communication between a first point and a second point via a network,
A camera installed at the first point and shooting a scene of the first point;
A display installed at the second point and displaying a scene of the first point taken by the camera;
A detection unit that detects an installation direction of the display based on a location of the person at the second point;
An adjustment unit that adjusts the direction of the scene of the first point displayed on the display based on the result detected by the detection unit;
A communication system characterized by comprising: 2. The communication system according to claim 1, wherein the adjustment unit controls an operation of the camera and adjusts a direction of a scene of the first point photographed by the camera. The communication system according to claim 2, wherein a direction of the scene of the first point photographed by the camera with respect to the camera matches the installation direction. The detection unit detects a movement amount of a person at the second point,
The said adjustment part adjusts the direction of the scene of the said 1st point projected on the said display with respect to the person whose operation amount detected by the said detection part is below a predetermined value, The range 1st-3rd Claim The communication system according to any one of the above. The communication system according to any one of claims 1 to 4, wherein the detection unit detects the installation direction and detects the orientation of the face of a person at the second point. A microphone installed at the first point for converting the collected sound into an audio signal;
The communication system according to any one of claims 1 to 5, wherein the adjustment unit adjusts a direction in which sound is collected by the microphone based on a result detected by the detection unit. The detection unit detects a distance between the display and the person at the second point,
The communication system according to any one of claims 1 to 6, wherein the adjustment unit adjusts a magnification of an image on the display based on a distance detected by the detection unit. A camera installed at a first point and shooting a scene of the first point;
A display installed at a second point and displaying a scene of the first point taken by the camera;
A communication program for enabling communication between the first point and the second point using a computer,
A detection step of detecting an installation direction of the display based on a location of the person at the second point;
An adjustment step of adjusting the direction of the scene of the first point displayed on the display based on the result detected by the detection step;
A communication program characterized by causing a computer to execute. The communication program according to claim 8, wherein the adjusting step controls the operation of the camera and adjusts the direction of the scene of the first point photographed by the camera. The communication program according to claim 9, wherein a direction of the scene of the first point photographed by the camera with respect to the camera matches the installation direction. In the detection step, the movement amount of the person at the second point is detected,
The communication program according to any one of claims 8 to 10, which causes a computer to execute the adjustment step for a person whose operation amount detected by the detection step is a predetermined value or less. The communication program according to any one of claims 8 to 11, wherein the detecting step detects the installation direction and detects a face orientation of a person at the second point. The communication system includes a microphone that converts sound collected and collected at the first point into an audio signal;
13. The method according to any one of claims 8 to 12, wherein in the adjustment step, the computer is caused to adjust a direction in which sound is collected by the microphone based on a result detected by the detection step. The described communication program. In the detection step, the distance between the display and the person at the second point is detected,
The said adjustment process WHEREIN: Based on the distance detected by the said detection process, it adjusts the magnification of the image | video in the said display, The computer of Claim 8 thru | or 13 Communication program.