JP4438343B2 - Imaging display device and video conference system - Google Patents

Description

  The present invention relates to an image pickup display device and a video conference system, and for example, relates to an image pickup display device and a video conference system for picking up images of participants in a video conference and displaying picked-up images of other participants on the participants. is there.

  FIG. 16 is a diagram illustrating an example of an imaging display device used in a general video conference system.

In the example of FIG. 16, images 2 </ b> A to 2 </ b> D obtained by capturing the participants of the conference are arranged and arranged on the screen 1 of the display device.
In addition, an image pickup device 3 for picking up an image of a conference participant facing the screen 1 is installed on the upper portion of the display device. The participant's image picked up by the image pickup device 3 is distributed to other participant's terminal devices via the communication line, and displayed on the screen of the display device in the same manner as in FIG.

  With respect to a video conference system capable of such screen display, for example, there is a conventional technique shown in Patent Document 1 (FIG. 39).

  FIG. 17 is a diagram illustrating an example of an imaging display device using a plurality of display devices and imaging devices.

In the example of FIG. 17, a dedicated display device and imaging device are provided for each participant's terminal device. That is, the display devices 1A to 1C and the imaging devices 3A to 3C are provided corresponding to the three terminal devices of the communication partner. Each display device displays an image captured by a dedicated imaging device of the corresponding terminal device. Moreover, the image imaged by each imaging device is displayed on a dedicated display device of the corresponding terminal device.
JP 2003-18561 A

  The conventional imaging display device and video conference system described above have the following disadvantages.

  In the imaging display device shown in FIG. 16, the participant's line of sight in the captured image is always fixed to the front, and thus natural dialogue cannot be performed. In general, the direction of the line of sight of the conversation partner is one of important information for communication. If this information is missing, smooth conversation is hindered. For example, it is not possible to determine who the participant's remarks are directed to based on the participant's image alone, and it is necessary to confirm each by voice.

On the other hand, in the imaging display device shown in FIG. 17, since a dedicated display device and an imaging device are provided for each communication partner terminal device, it is possible to match each other's line of sight.
However, in the configuration as shown in FIG. 17, the number of display devices and imaging devices increases in proportion to the number of participants, so the configuration becomes larger and the cost of the device increases as the number of participants in the conference increases. To do. In addition, since the amount of data of the captured image that is transmitted and received is extremely large, the load on the communication line becomes heavy.

  In addition, the imaging display devices shown in FIGS. 16 and 17 both have a disadvantage in that only an image captured from one direction is displayed on a flat screen, and the viewing angle is narrow and the sense of reality is poor.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an imaging display device and a video conference that improve the sense of reality without complicating the device configuration and enable a natural dialogue between users. To provide a system.

  In order to achieve the above object, an imaging display device according to a first aspect of the present invention includes an imaging unit, a light projection unit that projects light according to an image of a communication partner to be displayed to a user, and a user's An image of the user having a light reflecting surface surrounding part or all of the surroundings, reflecting the light from the light projecting means on the light reflecting surface and entering the user, and reflecting the light on the light reflecting surface Optical means for making the light incident on the imaging means.

  According to the first aspect of the present invention, an image of the user is reflected on a light reflecting surface surrounding a part or all of the periphery of the user, is incident on the imaging device, and is captured by the imaging device. The In addition, light corresponding to the image of the communication partner is projected from the light projection means, reflected by the light reflecting surface, and incident on the user. As a result, the image of the communication partner is displayed on the light reflecting surface surrounding the user.

The optical means converges light from the user incident on at least a part of the curved surface of the cylindrical space surrounding the user in the direction of the central axis of the cylindrical space, and the imaging means includes the optical means. The focused light may be imaged.
Thereby, in the said imaging means, the light converged in one direction is imaged.

The light projection means may project light in a direction passing through the central axis and entering the user via the optical means.
Thereby, in the said light projection means, the light of a display image is projected to one direction.

Moreover, the 1st viewpoint of this invention may have an image synthetic | combination means which synthesize | combines the picked-up image of several communication partners.
This captured image is a captured image obtained by converging light from the communication partner incident on at least a part of the curved surface of the cylindrical space surrounding each communication partner in the direction of the central axis of the cylindrical space, The image synthesizing unit extracts images included in a predetermined angle range when viewed from the center of the captured image corresponding to the central axis, from the captured images of the respective communication partners. Then, the extracted images are synthesized by matching the respective image centers.
In this case, the light projection unit projects the light of the display image generated according to the synthesized image of the image synthesis unit in the direction in which the light path corresponding to the image center coincides with the central axis.
Thereby, the captured image from the predetermined direction of the communication partner is displayed at a predetermined position on the light reflecting surface.

  Further, the imaging unit may change the resolution when imaging the light converged by the optical unit according to the distance from the central axis.

  A second aspect of the present invention is a video conference system having a plurality of terminal devices that captures an image of one or more users and displays captured images of other users on the user. The terminal device includes an imaging unit, a light projection unit that projects light according to an image captured by the imaging unit of another terminal device, and a light reflecting surface that surrounds part or all of the periphery of the user Optical means for reflecting the light from the light projecting means on the light reflecting surface to be incident on the user, and causing the image reflected on the light reflecting surface to be incident on the imaging means. Have.

  According to the second aspect of the present invention, the image of the user is reflected by a light reflecting surface provided at a part or all of the periphery of the user of each terminal device, enters the imaging device, and the imaging is performed. It is imaged in the device. Further, light corresponding to an image picked up by the image pickup means of another terminal device is projected from the light projection means, reflected by the light reflection surface, and incident on the user. Thereby, an image of a user of another terminal device is displayed on the light reflecting surface surrounding the user.

Preferably, the optical means converges light from the user incident on at least a part of the curved surface of the cylindrical space surrounding the user in the direction of the central axis of the cylindrical space, and the imaging means includes: The light converged by the optical means is imaged, and the light projecting means projects light in a direction passing through the central axis and entering the user via the optical means.
Thereby, in the said imaging means, the light converged in one direction is imaged. In the light projection means, the light of the display image is projected in one direction.

The second aspect of the present invention may include an image composition device that supplies a composite image to each of the plurality of terminal devices.
The image composition device extracts images included in a predetermined angle range as seen from the center of the captured image corresponding to the central axis, from captured images in a plurality of other terminal devices excluding the supply destination of the composite image. . Then, the extracted images are synthesized by matching the respective image centers.
In this case, the light projecting means projects the light of the display image generated according to the composite image supplied from the image synthesizing means in a direction in which the light path corresponding to the image center and the central axis coincide with each other. To do.
Thereby, the image which imaged the user of the other terminal device from the predetermined direction is displayed on the predetermined position on the light reflection surface.

ADVANTAGE OF THE INVENTION According to this invention, although it is a simple structure, it becomes possible to look over the captured image from the various directions of a communicating party, and can enhance a sense of reality.
In addition, according to the present invention, a part or all of the periphery of the imaging target is captured with a simple configuration, and a captured image from the desired direction of the imaging target is extracted from the captured image, and the screen is displayed. Can be displayed at a desired position. As a result, it becomes possible to match the lines of sight of the users, and a more natural dialogue can be realized.

  Hereinafter, five embodiments of the present invention will be described with reference to the drawings.

<First Embodiment>
FIG. 1 is a block diagram illustrating an example of a configuration of a video conference system according to an embodiment of the present invention.
The video conference system illustrated in FIG. 1 includes a system server 20 and terminal devices 30_1 to 30_N (N represents an integer of 2 or more).

[System Server 20]
The system server 20 performs various processes related to the overall operation of the video conference system.

  For example, the system server 20 generates information for identifying the terminal devices 30_1 to 30_N (hereinafter, referred to as identification information), and gives the information to the terminal devices 30_1 to 30_N, respectively. Also, information indicating the positions where the users of the terminal devices 30_1 to 30_N are arranged in the virtual space (hereinafter referred to as virtual arrangement information) is generated and given to the terminal devices 30_1 to 30_N, respectively.

[Terminal device 30_n]
The terminal device 30_n (n represents an integer from 2 to N) captures the user's voice and image, transmits the voice / image data to the other terminal device, and transmits the voice / image from the other terminal device. Data is received, played and displayed.

The system server 20 and the terminal devices 30_1 to 30_N described above are connected via the network 10 and exchange information (voice / image data, control information, etc.) necessary for system processing via the network 10.
For example, the network 10 may be configured by a communication line dedicated for a video conference system, or a public network such as the Internet may be used as the network 10.

  Next, a detailed configuration of the terminal device 30_n will be described.

FIG. 2 is a block diagram showing an example of the configuration of the terminal device 30_n according to the embodiment of the present invention.
The terminal device 30_n illustrated in FIG. 2 includes a control unit 301, an image composition unit 302, and an imaging display device 303.
The image composition unit 302 is an embodiment of the image composition means of the present invention.
The imaging display device 303 is an embodiment of the imaging display device of the present invention.

{Control unit 301}
The control unit 301 performs various controls related to the overall operation of the terminal device 30_n.

For example, the control unit 301 communicates with the system server 20 and other terminal devices via the network 10 to perform processing for exchanging information such as voice / image data, virtual arrangement information, and identification information.
In addition, the control unit 301 gives the image composition unit 302 various information necessary for image composition such as captured image data received from other terminal devices and virtual layout information received from the system server 20. Then, the image composition processing is executed.
Furthermore, the control part 301 performs the process which distributes the data of the image imaged in the imaging display apparatus 303 to another terminal device with the identification information mentioned above.
In addition to this, the control unit 301 inputs audio data received from another terminal device to an audio reproduction device (not shown) and reproduces the audio, and user's data collected by the audio input device (not shown). A process of distributing the audio data to another terminal device is performed.

{Image composition unit 302}
The image composition unit 302 inputs data of captured images in other terminal devices, extracts images in a predetermined range from the plurality of input captured images, and performs a process of combining them. This image composition processing will be described in detail later with reference to FIG. 6 and FIG.

{Imaging display device 303}
The imaging display device 303 captures an image of the user of the terminal device, outputs the captured image data S304 to the control unit 301, and reproduces an image corresponding to the composite image data S302 in the image composition unit 302. This is displayed to the user.

FIG. 3 is a block diagram illustrating an example of the configuration of the imaging display device 303.
An imaging display device 303 illustrated in FIG. 3 includes an imaging unit 304, a light projection unit 305, a half mirror 306, and an optical system 307.
The imaging unit 304 is an embodiment of the imaging unit of the present invention.
The light projection unit 305 is an embodiment of the light projection means of the present invention.
The unit including the half mirror 306 and the optical system 307 is an embodiment of the optical means of the present invention.

{Imaging unit 304}
The imaging unit 304 captures an image of a user that is emitted from the optical system 307 and is incident through the half mirror 306.

{Light projection unit 305}
The light projection unit 305 generates light corresponding to the composite image data S302 of the image composition unit 302 and projects the light toward the half mirror 306.

{Half mirror 306}
The half mirror 306 reflects a part of the light of the composite image projected from the light projection unit 305 and enters the optical system 307 and transmits a part of the user image emitted from the optical system 307. , Enters the imaging unit 304.

  Note that the projection direction of the display image in the light projection unit 305, the installation position of the half mirror 306, and the inclination of the mirror surface are such that the light of the composite image projected in the light projection unit 305 passes through a central axis described later of the optical system 307. Adjustment is made so as to enter the user via the optical system 307.

{Optical system 307}
The optical system 307 has a light reflecting surface that surrounds the user, and reflects light from the light projection unit 305 by the reflecting surface so as to enter the user. Further, the user image reflected by the light reflecting surface is made incident on the imaging unit 304.

For example, the optical system 307 reflects light from a user incident on at least a part of a curved surface of a cylindrical space surrounding the user and converges the light in the direction of the central axis of the cylindrical space.
In this case, for example, the optical system 307 reflects the image of the user who goes to the outside of the curved surface of the cylindrical space described above toward the inside of the curved surface of the cylindrical space, and further reflects the reflected image to the center of the cylindrical space. Reflects in the axial direction.
The light reflection described above is realized by combining a mirror having a mirror surface formed on an inclined surface such as a cone as described below.

FIG. 4 is a diagram illustrating an example of the configuration of the optical system 307 in the imaging display device according to the first embodiment of the present invention.
The optical system 307 illustrated in FIG. 4 includes mirrors M1 to M3.
The mirror M1 is an embodiment of the first mirror of the present invention.
Mirror M2 is an embodiment of the second mirror of the present invention.
The mirror M3 is an embodiment of the third mirror of the present invention.

In FIG. 4, reference numeral “308” is a unit that houses the imaging unit 304, the light projection unit 305, and the half mirror 306 already described. In the following description, this is referred to as an image input / output unit 308.
Symbols “F1” and “F2” indicate supports that support the mirrors M1 to M3 and the image input / output unit 308.
The code “O1” indicates a user of the terminal device 30_n.

FIG. 4A is a view of the optical system 307 looking down from the direction in which the user image exits toward the imaging unit 304. However, the right side of the dotted line C1-C2 is a diagram in which the mirror M2 and the image input / output unit 308 illustrated on the left side are removed, and the mirror M1 and the mirror M3 hidden under the mirror are exposed.
FIG. 4B is a diagram illustrating a cross section of the optical system 307 viewed from a direction perpendicular to the emission direction of the user image.

{Mirror M1}
The mirror M1 is a mirror surface formed in a predetermined region on the inclined surface of the cone (first cone) whose center axis coincides with the axis indicated by the dotted line C1-C2 so as to face the inside of the first cone. Have

In the example of FIG. 4, the apex of the first cone is directed in the direction opposite to the imaging unit 304 side from which the user image is emitted.
The mirror surface of the mirror M1 is formed so as to surround the entire circumference of the user O1 in a band-like region of the inner slope of the cone, the distance from the apex of the first cone being included in a predetermined range.
The band-like mirror surface of the mirror M1 reflects the light emitted in the horizontal direction from the head of the user O1 in the example of FIG. 4B and enters the mirror M2.

  For example, it is assumed that the incident angle and the reflection angle of light in the mirror M1 are equal, and the angle formed by the slope of the first cone and the central axis is 45 °. In this case, the light emitted in the horizontal direction from the head of the user O1, that is, the light emitted in the direction perpendicular to the central axis of the first cone is reflected by the mirror M1, so that FIG. As shown in B), the light is parallel to the central axis.

{Mirror M2}
The mirror M2 directs light from the mirror M1 to the inside of the second cone on a predetermined area on the slope of the cone (second cone) whose center axis coincides with the axis indicated by the dotted line C1-C2. And a mirror surface formed so as to be reflected.

In the example of FIG. 4, the vertex of the second cone faces the direction of the imaging unit 304 from which the user image is emitted. That is, the second cone is arranged such that the position of the vertex with respect to the bottom surface is opposite to that of the first cone.
The mirror surface of the mirror M2 is formed in a belt-like region on the inner slope of the cone, the distance from the apex of the second cone being included in a predetermined range.
The belt-like mirror surface of the mirror M2 is disposed closer to the imaging unit 304 than the belt-like mirror surface of the mirror M1.

  For example, similarly to the mirror M1, it is assumed that the incident angle and the reflection angle of light in the mirror M2 are equal, and the angle formed between the inclined surface of the second cone and the central axis is 45 °. In this case, the light parallel to the central axis emitted from the mirror M1 is reflected by the mirror M2, and as shown in FIG. 4 (B), is perpendicular to the central axis and inside the second cone. Light that travels toward.

{Mirror M3}
The mirror M3 is a mirror surface formed on the inclined surface of a cone (third cone) whose center axis coincides with the axis indicated by the dotted line C1-C2 so as to reflect the light from the mirror M2 in the direction of the center axis. Have

In the example of FIG. 4, the vertex of the third cone faces the direction of the imaging unit 304 from which the user image is emitted. That is, the position of the vertex with respect to the bottom surface of the third cone is arranged in the same direction as the second cone. In addition, the third cone has a smaller outer shape than the second cone, and is disposed inside the third cone.
The mirror surface of the mirror M3 is formed on the outer slope of the third cone so as to face the belt-like mirror surface of the mirror M2.

  For example, similarly to the mirrors M1 and M2, it is assumed that the incident angle and the reflection angle of light in the mirror M3 are equal, and the angle formed between the inclined surface of the third cone and the central axis is 45 °. In this case, the light perpendicular to the central axis emitted from the mirror M2 is reflected by the mirror M3, and becomes parallel to the central axis as shown in FIG. 4B. The reflected light emitted from the mirror M3 becomes light converged to the central axis side more than the reflected light of the mirror M1.

  Next, the operation of the video conference system having the above-described configuration will be described.

First, a terminal device used for a video conference is determined, and the information is input to the system server 20. In addition, the arrangement of each terminal device in the virtual space is determined, and the information is input to the system server 20.
Based on the input information, the system server 20 generates identification information for identifying each terminal device used in the video conference, and transmits the generated identification information to the corresponding terminal device. Further, the system server 20 generates virtual arrangement information indicating a position where the user of the terminal device is arranged in the virtual space, and transmits this to all the terminal devices used for the video conference.

  After such an initial setting is performed, an imaging process and a display process described below are performed in parallel.

[Imaging processing]
As shown in FIG. 4, the image of the user located near the central axis of the optical system 307 is reflected in a direction parallel to the central axis by the mirror M1 disposed around the entire periphery. The image reflected by the mirror M1 is reflected by the mirror M2 in a direction perpendicular to the central axis, and further reflected by the mirror M3 in a direction parallel to the central axis. Accordingly, a circular image that reflects the entire periphery of the user is emitted from the mirror M3.

FIG. 5 is a diagram illustrating an example of an image of the user's head emitted from the mirror M3. As shown in this example, in the optical system 307 of FIG. 4, a circular image with the user's forehead facing the outer periphery and the jaw facing the center is obtained.
For example, when the user turns his / her face around the neck, the user's face rotates around the center of the circle in the all-round image shown in FIG. When the user approaches the mirror M1, the front image of the face is reduced and the occipital image is enlarged in the all-round image shown in FIG.

Such an entire circumference image of the user emitted from the optical system 307 passes through the half mirror 306 and enters the imaging unit 304.
The imaging unit 304 captures this image, converts it into captured image data S304, and inputs it to the control unit 301.
The control unit 301 adds the above-described identification information to the captured image data S304 input from the imaging unit 304, and transmits this to other terminal devices.

[Display processing]
The control unit 301 inputs captured image data and identification information received from another terminal device via the network 10 and virtual arrangement information received from the system server 20 in the initial setting to the image composition unit 302, respectively.

  In generating a composite image, the image composition unit 302 extracts a range of captured images received from other terminal devices, and determines in which position of the composite result the extracted image is to be arranged. For this determination, the above-described virtual arrangement information, identification information of the terminal device of its own, and identification information of the imaging source terminal device input along with each captured image data are used.

  FIG. 6 is a diagram illustrating an example of the arrangement of users in the virtual space.

  First, the image composition unit 302 checks in which position in the virtual space the user of the terminal is placed based on identification information of the terminal device (hereinafter referred to as the terminal itself). In the example of FIG. 6, the user of the terminal is placed at the position P1.

  In addition, the image composition unit 302 determines the position in the virtual space of the user of another imaged terminal device (hereinafter referred to as another terminal) based on the identification information input accompanying the captured image data. Investigate. In the example of FIG. 6, users of other terminals are placed at positions P2 to P4.

  When the positions of the user of the own terminal and the other terminal in the virtual space are specified, the image composition unit 302 uses the other terminal to be displayed to the user of the own terminal based on the position in the virtual space. The angle range of the person image is determined respectively.

  For example, the image of the user at the position P3 that falls within the field of view of the user at the position P1 has an angle range of about 180 ° surrounding the user at the position P3, with a straight line that virtually connects the position P1 and the position P3 as the center. included.

  Also, the image composition unit 302 determines areas on which a user of another terminal should be displayed on the virtual screen of the terminal itself (actually the mirror M1) based on the position in the virtual space.

  For example, on the virtual screen SC1 viewed by the user at the position P1, the users at the positions P2 to P4 are displayed on the cylindrical curved surfaces in the areas A2 to A4, respectively.

  When the angle range of the user image of the other terminal to be displayed and the region on the virtual screen on which the image is to be displayed are determined for each captured image in this way, the image composition unit 302 then determines The images in the angle range are extracted from the captured images, and the extracted images are combined so that these images are projected in the determined area on the screen, thereby creating an all-round image.

A captured image in a predetermined angular range surrounding the imaging target is a predetermined angular range viewed from the center of the image (this image center corresponds to the central axis of the optical system 307) in the entire circumference image as shown in FIG. Will be a fan-shaped image.
Further, if the region on the virtual screen for displaying an image to the user is a cylindrical curved surface, such a cylindrical curved surface becomes a fan-shaped image in the entire circumference image.
Accordingly, in this case, the image composition unit 302 extracts a fan-shaped image corresponding to the determined angle range from each captured image (circumferential image), and the extracted image corresponds to the determined region on the virtual screen. To the fan-shaped area on the entire circumference image.

When mapping is performed, the image composition unit 302 matches the centers (fan-shaped vertices) of the captured image corresponding to the central axis of the optical system 307.
Further, when the size of the extracted image and the size of the mapping area are different, the image composition unit 302 enlarges / reduces the extracted image so that the size of the extracted image and the mapping area matches.
Furthermore, the image composition unit 302 may perform mapping after performing appropriate correction on the extracted image so that the image displayed on the screen is closer to the real object.

FIG. 7 is a diagram illustrating an example of the operation of the image composition unit 302.
In the example of FIG. 7, the image composition unit 302 extracts fan-shaped images I2C, I3D, I4E, I5F, I6A, and I7B from the all-round images I2, I3,. The composite image SI1 is generated by mapping this to an appropriate region.

  The image data S302 synthesized by the image synthesis unit 302 in this way is input to the light projection unit 305.

  The light projection unit 305 generates an all-round image corresponding to the composite image data S302 of the image composition unit 302, projects the light, reflects the light at the half mirror 306, and causes the light to enter the optical system 307.

In this case, by appropriately adjusting the projection direction of the light projection unit 305, the installation position of the half mirror 306, the inclination of the mirror surface, and the like, the light path corresponding to the center of the projected all-round image is optical. This coincides with the central axis of the system 307.
As a result, an image emitted from the optical system 307 of a certain terminal device enters the optical system 307 of another terminal device under the same conditions as those at the time of emission.

  The all-round image incident on the optical system 307 follows an optical path opposite to that in the case of display processing, and enters the user of the terminal device. Thereby, the user of another terminal device is displayed with respect to the user of a terminal device.

As described above, according to the imaging display device 303 according to the present embodiment, the image of the entire circumference of the user is incident on the mirror M1 surrounding the user, and this image is reflected by the mirrors M2 and M3. The image is picked up by the image pickup unit 304. Further, an image of the entire periphery of the communication partner is projected from the light projection unit 305, reflected by the half mirror 306 and the mirrors M3 and M2, and projected on the mirror M1 surrounding the user.
Therefore, captured images from various directions of the communication partner can be displayed on the mirror M1 surrounding the user, and an unprecedented presence can be created.

  Further, in the optical system 307, the light from the user incident on the curved surface of the cylindrical space surrounding the user converges in the direction of the central axis of the cylindrical space, and the converged light is imaged in the imaging unit 304. The For this reason, as shown in the conventional example of FIG. 17, it is possible to capture an image of the entire circumference of the user without providing a plurality of imaging devices, and the configuration of the device can be simplified. In addition, the load on the communication line can be reduced compared to the case where a plurality of captured images are transmitted.

  Moreover, since the image captured by the imaging unit 304 is an image obtained by capturing the periphery of the imaged object, there are few wasted images other than the imaged object. Therefore, necessary image information can be efficiently transmitted.

Further, in the image composition unit 302, images included in a predetermined angle range as viewed from the center of the image are extracted from all-round images of the user in other terminal devices, and the extracted images are displayed at the respective image centers. Are combined in a state where they are matched. Light corresponding to the composite image is projected from the light projection unit 305, reflected by the half mirror 306, and enters the optical system 307. In the composite image incident on the optical system 307, the light path at the center of the image coincides with the direction of the central axis of the optical system 307.
Therefore, an image captured from a desired direction of the imaging target can be displayed at a desired position on the screen. Accordingly, by appropriately setting the angle range for extracting the captured image and the position for mapping to the composite image, it becomes possible to match the lines of sight of the users, and a more natural dialogue can be realized. .

Moreover, according to the video conference system according to the present embodiment, the users of the terminal devices 30_1 to 30_N including the imaging display device 303 are arranged in the virtual space, and the virtual arrangement information and the identification information of each terminal device are used. Based on this, the above-described combination of the captured images is performed in each terminal device.
As a result, users of multiple terminal devices can interact with each other in the same virtual environment as if they were in a common virtual conference room. Can be increased. In addition, such an improvement in the sense of reality brings about an effect of increasing the frequency of use of the video conference and an effect of improving the quality of discussion.

  In the video conference system according to the present embodiment, the number of users in one terminal device can be arbitrarily increased as necessary.

FIG. 8 is a diagram illustrating an example in which the number of users of one terminal device is one.
In this example, the size of the optical system 307 surrounds the user's head, and an image of the head of the user of the other terminal is displayed on the screen of the optical system 307. According to such a conference system, it is possible to obtain a sense of reality that a plurality of users (O1 to O5) surround the round table in one virtual conference room.

On the other hand, FIG. 9 is a diagram illustrating an example in which the number of users of one terminal device is plural.
In this example, the optical system 307 is about the same size as one conference room, and a seat is provided therein, for example, like a normal conference room. A similar conference room of another terminal is displayed on the screen of the optical system 307, and users (O1A to O5A) of a plurality of conference rooms face each other in a virtual space. According to such a conference system, it is possible to obtain a sense of realism as if it was being discussed in a large conference hall.

It is also possible to construct a video conference system in which a terminal device that can be used by one person and a terminal device that can be used by a plurality of people are mixed. It becomes possible.
In this system, since the size of the subject is different for each terminal device, there is a possibility that the size of the human image displayed on the screen of the optical system 307 may be uneven. In this case, for example, by appropriately setting the image extraction range in the image composition unit 302 and the enlargement / reduction ratio of the extracted image, the sizes of the human images can be made uniform.

<Second Embodiment>
Next, a second embodiment of the present invention will be described.
In the second embodiment, the conical mirror used in the optical system of the first embodiment is replaced with a pyramidal mirror.

FIG. 10 is a diagram illustrating an example of an optical system in an imaging display device according to the second embodiment of the present invention. The same reference numerals in FIGS. 4 and 10 denote the same components.
An optical system 307A illustrated in FIG. 10 includes mirrors M1A to M3A.
Mirror M1A is an embodiment of the first mirror of the present invention.
Mirror M2A is an embodiment of the second mirror of the present invention.
Mirror M3A is an embodiment of the third mirror of the present invention.

FIG. 10A illustrates the optical system 307A as viewed from the imaging unit 304, as in FIG. 4A. Further, the right side of the dotted line C1-C2 is a diagram in which the mirror M2A and the image input / output unit 308 illustrated on the left side are removed, and the mirror M1A and the mirror M3A hidden under the mirror M2A are exposed.
FIG. 10B is a diagram showing a cross section of the optical system 307A viewed from a direction perpendicular to the exit direction of the user's image, as in FIG. 4B.

{Mirror M1A}
The mirror M1A is a mirror surface formed in a predetermined region on the slope of the pyramid (first pyramid) whose axis shown by the dotted line C1-C2 coincides with the central axis, toward the inside of the first pyramid. Have

In the example of FIG. 10, the first pyramid has a regular heptagonal bottom surface, and the apex thereof faces in the direction opposite to the imaging unit 304 side from which the user image is emitted. .
The mirror surface of the mirror M1A is formed so as to surround the entire circumference of the user O1 in a band-like region on the inner slope of the pyramid, the distance from the bottom of the first pyramid being included in a predetermined range.
In the example of FIG. 10B, the belt-like mirror surface of the mirror M1A reflects light emitted from the head of the user O1 in the horizontal direction and enters the mirror M2A.

{Mirror M2A}
The mirror M2A directs the light from the mirror M1A to the inside of the second pyramid on a predetermined area on the slope of the pyramid (second pyramid) whose center axis coincides with the axis indicated by the dotted line C1-C2. And a mirror surface formed so as to be reflected.

In the example of FIG. 10, the second pyramid has a regular heptagonal bottom surface similar to the first pyramid, and the positional relationship of each vertex with respect to the center is the same between the two bottom surfaces. That is, when the bottom surface of the second pyramid is orthogonally projected onto the bottom surface of the first pyramid, the seven lines connecting the vertices and the center of the image are the vertices on the bottom surface of the first pyramid. It overlaps each of the seven lines connecting the center.
Further, the vertex of the second pyramid faces the direction of the imaging unit 304 from which the user image is emitted. That is, the second pyramid is arranged such that the position of the vertex with respect to the bottom surface is opposite to that of the first pyramid.
The mirror surface of the mirror M2A is formed in a band-like region on the inner slope of the pyramid that includes the bottom surface from the apex of the second pyramid in a predetermined range.
The belt-like mirror surface of the mirror M2A is disposed closer to the imaging unit 304 than the belt-like mirror surface of the mirror M1A.

{Mirror M3A}
The mirror M3A reflects the light from the mirror M2A in the direction of the central axis in a predetermined region on the inclined surface of the pyramid (third pyramid) whose central axis coincides with the axis indicated by the dotted line C1-C2. It has a formed mirror surface.

In the example of FIG. 10, the third pyramid has a regular heptagon base similar to that of the first pyramid, and the positional relations of the vertices with respect to the center of the two bottoms are equal to each other. That is, when the bottom surface of the third pyramid is orthogonally projected onto the bottom surface of the first pyramid, the seven lines connecting the vertices and the center of the image are the vertices on the bottom surface of the first pyramid. It overlaps each of the seven lines connecting the center.
Further, the vertex of each third cone is directed toward the imaging unit 304 from which the user image is emitted. That is, the third cones are arranged in the same direction as the second cones at the positions of the vertices with respect to the bottom surface. Moreover, each 3rd cone is smaller than each 2nd cone, and is arrange | positioned inside it.
The mirror surface of the mirror M3A is formed on the outer slope of such a third cone so as to face the band-shaped mirror surface of the mirror M2A.

In the optical system 307A described above, similarly to the optical system 307 described above, the light emitted from the user located near the central axis is reflected by the mirrors M1A, M2A, and M3A and converges in the central axis direction. And enters the imaging unit 304. Moreover, the light of the perimeter image projected from the light projection part 305 is reflected in mirror M3A, M2A, M1A, and injects into the user located near central axis from the perimeter.
Therefore, also in the imaging display device according to the present embodiment, the imaging display device according to the first embodiment can display captured images from various directions of the communication partner on the mirror M1A surrounding the user. It is possible to achieve the same effect as.

  In addition, since the mirror surfaces of the mirrors M1A, M2A, and M3A are all flat, there is an advantage that the device can be manufactured and installed more easily than the curved mirror surface in the first embodiment.

<Third Embodiment>
Next, a third embodiment of the present invention will be described.
In the third embodiment, the diameter of the light emitted from the optical system can be reduced.

FIG. 11 is a diagram illustrating an example of an optical system in an imaging display device according to the third embodiment of the present invention. The same reference numerals in FIGS. 4 and 11 denote the same components.
An optical system 307B illustrated in FIG. 11 includes the mirror M1 and the mirrors M2B and M3B described above.
Mirror M2B is an embodiment of the second mirror of the present invention.
Mirror M3B is an embodiment of the third mirror of the present invention.

{Mirror M2B}
The mirror M2B is a mirror having a belt-like mirror surface similar to the above-described mirror M2, and the difference between the mirror M2 and the mirror M2B is the angle formed by the central axis of the second cone on which the mirror surface is formed and the inclined surface. It is in. That is, in the case of the mirror M2 illustrated in FIG. 4, the above-described angle of the second cone is equal to that of the first cone, whereas in the mirror M2B illustrated in FIG. In the example of FIG. 11, the above-described angle of the second cone is smaller than that of the first cone.

{Mirror M3B}
The mirror M3B is a mirror having a conical mirror surface similar to the above-described mirror M3, and the difference between the mirror M3 and the mirror M3B is formed by the central axis of the third cone on which the mirror surface is formed and the inclined surface. Is at an angle. That is, in the case of the mirror M3 illustrated in FIG. 4, the above-described angle of the third cone is equal to that of the first cone, whereas in the mirror M3B illustrated in FIG. In the example of FIG. 11, the above-mentioned angle of the third cone is smaller than that of the first cone, and is equal to the second cone.

Thus, even when the cone in which the mirrors M2B and M3B are formed and the cone in which the mirror M1 is formed are not similar, the light emitted from the user can be converged in the direction of the central axis. Is possible. Further, by appropriately selecting the angle and arrangement between the central axis of the cone and the inclined surface, the diameter of the light can be reduced as shown in FIG.
By reducing the diameter of the light, the light receiving surface of the imaging unit 304 can be downsized, so that the device cost can be reduced.
In the image input / output unit 308B in FIG. 11, the imaging unit with the light receiving surface reduced in size is housed.

FIG. 12 is a diagram illustrating another configuration example of the optical system in the imaging display device according to the third embodiment of the present invention. The same reference numerals in FIGS. 4 and 12 denote the same components.
An optical system 307C illustrated in FIG. 12 includes the mirrors M1 and M2 and the mirror M3C already described.
Mirror M3C is an embodiment of the fourth mirror of the present invention.

{Mirror M3C}
The mirror M3C has a mirror surface formed so as to reflect light from the mirror M2 in the direction of the central axis on a curved surface obtained by bending the slope of the third cone described above toward the central axis.

  As described above, the diameter of the light incident on the imaging unit can be reduced by providing the mirror M3C whose mirror surface is curved in the central axis direction.

<Fourth Embodiment>
Next, a fourth embodiment of the present invention will be described.
In the fourth embodiment, the light reflecting surface is provided only on a part of the entire circumference of the user.

FIG. 13 is a diagram illustrating an example of a configuration of an optical system in an imaging display device according to the fourth embodiment of the present invention.
An optical system 307D illustrated in FIG. 13 includes mirrors M1D to M3D.
Mirror M1D is an embodiment of the first mirror of the present invention.
Mirror M2D is an embodiment of the second mirror of the present invention.
Mirror M3D is an embodiment of the third mirror of the present invention.

  FIG. 13 is a view of the optical system 307D looking down from the imaging unit 304, as in FIG. 4A. The right side of the dotted line C1-C2 is the mirror M2D and the image input / output unit 308 illustrated on the left side. FIG. 6 is a diagram in which the mirror M1D and the mirror M3D that are hidden underneath are removed and are exposed.

The mirrors M1D to M3D in FIG. 13 have the same shape as the mirrors M1 to M3 in FIG. The difference between the two is that the mirrors M1 to M3 have a mirror surface on the entire circumference of the cone, whereas the mirrors M1D to M3D have a mirror surface only on a part of the entire circumference of the cone. is there. In the example of FIG. 13, the mirror surface is provided only in an angle range within 180 ° in front of the user.
For example, in the case of a round table conference, when only the front image of the user is displayed and the back image is unnecessary, as shown in FIG. 13, a light reflecting surface may be provided only in the necessary range of the entire circumference. good. Thereby, the apparatus configuration can be simplified and the cost can be reduced.

FIG. 14 is a diagram illustrating another configuration example of the optical system in the imaging display device according to the fourth embodiment of the present invention.
An optical system 307E illustrated in FIG. 14 includes mirrors M1E to M3E.
Mirror M1E is an embodiment of the first mirror of the present invention.
Mirror M2E is an embodiment of the second mirror of the present invention.
Mirror M3E is an embodiment of the third mirror of the present invention.

  FIG. 14 is a view of the optical system 307E looking down from the imaging unit 304 as in FIG. 4A. The right side of the dotted line C1-C2 is the mirror M2E and the image input / output unit 308 illustrated on the left side. FIG. 6 is a diagram in which the mirror M1E and the mirror M3E that are hidden underneath are removed and are exposed.

  The mirrors M1D to M3D in FIG. 14 have the same shape as the mirrors M1 to M3 in FIG. 4, and the difference between them is that the mirrors M1 to M3 have mirror surfaces on the entire circumference of the cone. On the other hand, the mirrors M1D to M3D have a non-reflective region in one of two regions facing each other across the central axis.

When a light reflecting surface is provided on the entire circumference of the user, the display image of the light reflecting surface facing the center axis along with the light of the user to be imaged is also incident on the imaging unit 304, so the image becomes unclear. Or may cause an oscillation phenomenon.
On the other hand, as shown in FIG. 14, if a non-reflective area is provided in a part of the entire circumference, the light of the display image does not enter from the opposite surface across the central axis, thereby preventing the above phenomenon from occurring. It is possible to improve the quality of the display image.

  The non-display area may be an area where a mirror surface is cut out, or an area where an object in which light reflection is suppressed is arranged.

<Fifth Embodiment>
Next, a fifth embodiment of the present invention will be described.
In the fifth embodiment, the resolution when capturing an image of light converged in the optical system is changed according to the distance from the central axis of the optical system.

FIG. 15 is a diagram illustrating an example of the operation of the imaging unit in the imaging display device according to the fifth embodiment of the present invention.
The imaging unit 304A according to the present embodiment is the same as the imaging unit 304 described above in that the light emitted from the optical system 307 is imaged, but at the time of this imaging, the distance from the central axis of the optical system 307 is set. It differs from the imaging unit 304 in that the resolution is changed accordingly.

  In the example of FIG. 15A, the imaging unit 304A is fixed at a predetermined position on the central axis at a single point, and the operation of a driving mechanism (not shown) having an attitude control motor and the like around this fixed point. To rotate. By this rotation, the imaging area by the imaging unit 304A moves in the region of light emitted from the optical system 307 as shown in FIG. The imaging unit 304A divides the region of the emitted light into a plurality of imaging areas and captures images while sequentially scanning each imaging area. At this time, for example, the resolution is changed according to the distance between the imaging area and the central axis by changing the zoom rate.

As shown in the example of FIG. 5, since the circular image emitted from the optical system 307 is reduced as it is closer to the center, the whole image is displayed on the screen of the optical system 307 when captured at the same resolution. There is a possibility that the image to be displayed becomes an unnatural image with lower resolution as viewed from the user. For example, in a video conference system with a large number of people as shown in FIG. 9, since individual images are small, such low resolution is a major factor that impairs the sense of reality.
Therefore, for example, by using a method as shown in FIG. 15 and picking up an image with higher resolution as it approaches the central axis of the optical system 307, it becomes possible to make the resolution of the entire display image uniform. As a result, the quality of the display image is improved, and a more realistic video can be displayed.

  The present invention is not limited to the above-described embodiment, and various modifications can be made.

In the embodiment described above, the image composition unit is incorporated in each terminal device, but the present invention is not limited to this example. For example, these image composition units may be accommodated in one apparatus on the network as an image composition server.
In this case, all images captured by each terminal device are transmitted to this image composition server. The image composition server performs the above-described composition processing on the received captured image, generates a composite image for each terminal device, and transmits it to the corresponding terminal device.
As described above, when the image composition unit is accommodated in one server, each terminal device needs to transmit a captured image only to this server. Therefore, when transmitting the same captured image to a plurality of other terminals. In comparison, the network load can be reduced.
Further, the image composition server and the system server may be integrated into a common server. This simplifies the system configuration and facilitates its management.

  In the optical system of the above-described embodiment, a mirror formed on the inclined surface of a cone or a pyramid is used. However, the present invention is not limited to this example. For example, the optical system is formed on an inclined surface of a cone having a nonlinear curved shape. A mirror may be used.

  Further, the mirror of the optical system illustrated in FIG. 10 is formed on the slope of a pyramid having a regular heptagonal bottom, but the present invention is not limited to this example, and a pyramid having another polygonal bottom. A mirror formed on the slope of the body may be used.

  The positional relationship between the imaging unit and the light projection unit illustrated in FIG. 3 may be reversed. That is, the position of the light projection unit may be changed so that the direction in which the light is emitted coincides with the central axis of the optical system, and the direction in which the light enters the imaging unit is orthogonal to the central axis. .

It is a block diagram which shows an example of a structure of the video conference system which concerns on embodiment of this invention. It is a block diagram which shows an example of a structure of the terminal device which concerns on embodiment of this invention. It is a block diagram which shows an example of a structure of the imaging display apparatus which concerns on embodiment of this invention. It is the figure which illustrated an example of the structure of the optical system in the imaging display apparatus which concerns on the 1st Embodiment of this invention. It is the figure which illustrated an example of the image of the user's head radiate | emitted from an optical system. It is the figure which illustrated an example of arrangement | positioning of the user on virtual space. It is the figure which illustrated an example of operation | movement of an image synthetic | combination part. It is the figure which illustrated the example in case the number of users of one terminal device is one person. It is the figure which illustrated the example in case the number of users of one terminal device is multiple persons. It is the figure which illustrated an example of the optical system in the imaging display apparatus which concerns on the 2nd Embodiment of this invention. It is the figure which illustrated an example of the optical system in the imaging display apparatus which concerns on the 3rd Embodiment of this invention. It is the figure which illustrated the other structural example of the optical system in the imaging display apparatus which concerns on the 3rd Embodiment of this invention. It is the figure which illustrated an example of the structure of the optical system in the imaging display apparatus which concerns on the 4th Embodiment of this invention. It is the figure which illustrated the other structural example of the optical system in the imaging display apparatus which concerns on the 4th Embodiment of this invention. It is the figure which illustrated an example of operation | movement of the imaging part in the imaging display apparatus which concerns on the 5th Embodiment of this invention. It is the figure which illustrated an example of the imaging display apparatus used for a general video conference system. It is the figure which illustrated an example of the imaging display apparatus which uses multiple display apparatuses and imaging devices.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Network, 20 ... System server, 30_1-30_N ... Terminal device, 301 ... Control apparatus, 302 ... Image composition part, 303 ... Imaging display apparatus, 304, 304A ... Imaging part, 305 ... Light projection part, 306 ... Half mirror , 307, 307A to 307E ... Optical system, M1 to M3, M1A to M3A, M2B, M3B, M3C, M1D to M3D, M1E to M3E ... Mirror

Claims (18)

Imaging means;
Light projection means for projecting light according to the image of the communication partner to be displayed to the user;
The use having a light reflecting surface surrounding a part or all of the periphery of the user, reflecting the light from the light projecting means on the light reflecting surface and entering the user, and reflecting the light on the light reflecting surface Optical means for causing an image of the person to enter the imaging means;
An imaging display device having The optical means converges the light from the user incident on at least a part of the curved surface of the cylindrical space surrounding the user in the direction of the central axis of the cylindrical space,
The imaging means images the light converged by the optical means;
The imaging display device according to claim 1. The light projecting means projects light in a direction passing through the central axis and incident on the user via the optical means;
The imaging display device according to claim 2. The optical means is
First light reflecting means for reflecting the image of the user heading toward the outer side of the curved surface of the cylindrical space toward the inner side of the curved surface of the cylindrical space;
And second light reflecting means for reflecting the image of the user reflected by the first light reflecting means in the direction of the central axis.
The imaging display device according to claim 3. The first light reflecting means is
A first mirror having a mirror surface formed at least in a region on the slope of the first cone having a common central axis with the cylindrical space toward the inside of the first cone;
A second cone having a central axis in common with the cylindrical space, at least a portion of the second cone having a vertex position with respect to the bottom surface arranged on the opposite side of the first cone; And a second mirror formed with a mirror surface that reflects light from the first mirror toward the inside of the second cone.
The imaging display device according to claim 4. The second light reflecting means applies light from the second mirror in the direction of the central axis to at least a part of the slope of the third cone having a central axis common to the cylindrical space. Including a third mirror formed with a reflecting mirror surface;
The imaging display device according to claim 5. The second light reflecting means includes the second mirror on at least a part of a curved surface obtained by curving the slope of the third cone having a central axis common to the cylindrical space toward the central axis. Including a fourth mirror formed with a mirror surface that reflects the light from the center axis direction,
The imaging display device according to claim 5. The first cone in which the mirror surface of the first mirror is formed, the second cone in which the mirror surface of the second mirror is formed, and the mirror surface of the third mirror is formed The third cones have similar shapes to each other,
The imaging display device according to claim 6. The first cone in which the mirror surface of the first mirror is formed, the second cone in which the mirror surface of the second mirror is formed, and the mirror surface of the third mirror is formed The third cone is a cone.
The imaging display device according to claim 6. The first cone in which the mirror surface of the first mirror is formed, the second cone in which the mirror surface of the second mirror is formed, and the mirror surface of the third mirror is formed The third cone is a pyramid.
The imaging display device according to claim 6. The first mirror and the second mirror have a non-reflective region in at least one of two regions facing each other across the central axis.
The imaging display device according to claim 5. The captured images of a plurality of communication partners are captured by converging light from the communication partner incident on at least a part of the curved surface of the cylindrical space surrounding each communication partner in the direction of the central axis of the cylindrical space. Image synthesizing means for extracting, from the captured image, images included in a predetermined angle range when viewed from the center of the captured image corresponding to the central axis, and synthesizing the extracted images by matching the respective image centers. Have
The light projecting means projects the light of the display image generated according to the synthesized image of the image synthesizing means in a direction in which the light path corresponding to the center of the image coincides with the central axis.
The imaging display device according to claim 3. The imaging means changes the resolution when imaging the light converged by the optical means according to the distance from the central axis,
The imaging display device according to claim 2. The imaging means divides the light converged by the optical means into a plurality of imaging areas and changes the resolution at the time of imaging according to the distance between the imaging area and the central axis.
The imaging display device according to claim 13. A video conference system having a plurality of terminal devices that captures images of one or more users and displays captured images of other users on the user,
The terminal device is
Imaging means;
Light projection means for projecting light according to the image picked up by the image pickup means of another terminal device;
The use having a light reflecting surface surrounding a part or all of the periphery of the user, reflecting the light from the light projecting means on the light reflecting surface and entering the user, and reflecting the light on the light reflecting surface Optical means for causing an image of the person to enter the imaging means,
Video conference system. The optical means converges the light from the user incident on at least a part of the curved surface of the cylindrical space surrounding the user in the direction of the central axis of the cylindrical space,
The imaging means images the light converged by the optical means,
The light projecting means projects light in a direction passing through the central axis and entering the user via the optical means.
The video conference system according to claim 15. An image composition device that supplies a composite image to each of the plurality of terminal devices, and is viewed from the center of the captured image corresponding to the central axis from the captured images of a plurality of other terminal devices excluding the supply destination of the composite image. Each of the images included in a predetermined angle range, and has an image composition device for synthesizing the extracted images by matching the respective image centers,
The light projection unit projects the light of the display image generated according to the combined image supplied from the image combining unit in a direction in which the light path corresponding to the center of the image and the central axis coincide with each other.
The video conference system according to claim 16. The image synthesizing unit includes information indicating positions where users of the plurality of terminal devices are arranged in a virtual space, information for identifying a terminal device to which a synthesized image is supplied, and a terminal that is an imaging source of the captured image Based on the information for identifying the device, the angular range for extracting the image from the captured image and the arrangement of the extracted image in the image to be combined are determined.
The video conference system according to claim 17.

Images