JP4595313B2 - Imaging display apparatus and method, and image transmission / reception system - Google Patents

Description

  The present invention is applied to, for example, a video conference system or a video phone system, and an image transmission / reception system for bidirectionally transmitting / receiving an image via a network, and an image to be transmitted / received are captured, and a spatial connection effect similar to that of a window glass The present invention relates to an imaging display device and method for reconfiguring this to obtain.

  As represented by a videophone system, a video conference system, and the like, a system has been proposed in which a plurality of users can remotely interact with each other while viewing a display image of the other party from a location apart from each other. In such a system, a display image of the other party is displayed on the display, and a user who visually recognizes the display is imaged as a subject to be photographed, and the obtained image signal is transmitted through a network such as a public line or a dedicated line. By transmitting to the terminal device, it is possible to give both users a sense of reality.

  In a conventional video conference system, a user who visually recognizes the display image of the other party displayed near the center of the display is captured by the camera at the top of the display, so that the image of the user facing down is displayed on the other party's display. Will be displayed. For this reason, there is a problem in that the users who actually view the display are interacted with each other in a state where their lines of sight are inconsistent, giving a sense of discomfort to each other.

  Ideally, if a camera is installed in the vicinity of the center of the display where the display image of the other party is projected, it is possible to realize a conversation in a state where the line of sight of both users is matched. However, it is physically difficult to install a camera near the center of such a display.

  In order to solve such a problem related to the line-of-sight mismatch, for example, a videophone device (for example, refer to Patent Document 1) that uses a half mirror to match the camera direction and the display screen, the light transmission state and the light scattering state can be controlled. An image display / control device that performs display and imaging in time series using a screen and a projector (see, for example, Patent Document 2), with an imaging function that can realize both display and imaging simultaneously by using a hologram screen and a projector. A display device (see, for example, Patent Document 3) has been proposed.

  In addition, a bidirectional communication system, a terminal device, and a control method have been proposed that match the line of sight with the display screen by controlling the optical axis of the camera on the other side according to the line of sight and the position of the face (for example, Patent Documents). 4).

  Also, three-dimensional information of the subject is extracted based on input images taken by a plurality of cameras arranged on both sides of the display, and an output image of the subject is determined according to the extracted three-dimensional information and information on the viewpoint position of the receiver. Has been proposed (see Patent Document 5, for example). In this image processing apparatus, by synthesizing a virtual viewpoint camera image centered on the screen using an epipolar plane image generated from a plurality of camera images arranged on a straight line, the user's line of sight is made coincident and a sense of presence is realized. High communication can be realized.

In addition, an image processing method and apparatus for displaying an image according to the viewpoint position of the observer by switching and displaying an image according to the viewpoint position of the observer from the input image group (see, for example, Patent Document 6). Has also been proposed. In this image processing method or the like, an epipolar plane image can be similarly used to facilitate the search for corresponding points.
In addition, in order to match each other's line of sight in a video conference, an image communication apparatus that generates three-dimensional position information based on images taken by two cameras installed on the left and right of the screen (see, for example, Patent Document 7). Has also been proposed.

  By the way, the above-described video conference system is a system in which an image in the camera field of view captured by a camera arranged at a shooting site is transmitted and displayed on a display device installed at a different point. For this reason, even when the viewing direction of the user with respect to the display device changes, the content of the displayed image is the same, so that there is a problem of giving an impression as if viewing a painting.

  For this reason, in recent years, a system has been proposed in which a window-like image display device is installed in two spaces separated via a network, and the space is felt as if they are connected together through a window. (For example, refer to Patent Document 8).

  In general, when viewing an object on the opposite side through a window glass, for example, as shown in FIG. 16, light incident at an angle (θ °, φ °) on a position (x, y) on the window glass surface is Are emitted at angles (180 ° -θ, 180 ° -φ °) from the same position. That is, in order to obtain a spatial connection effect similar to that of a window glass, a luminance level is identified for each incident angle (θ °, φ °), and the identified luminance level is determined as an incident angle (θ °, (φ °) and transmit to the display side. Then, light that is 180 ° out of phase with the incident angle (θ °, φ °) is emitted from the position corresponding to the above (x, y) on the display side panel at the identified luminance level.

  In the window-shaped imaging display device proposed in Patent Document 8, two imaging display panels are arranged back to back as shown in FIG. 17, for example. In FIG. 17, the upper panel shows a portion where an image sensor is formed, and the lower panel shows a portion where a display element is formed. Incidentally, both the image sensor and the display element are formed on the surface of a hemispherical protruding surface having the same size.

  Here, when incident light enters from the upper right to the lower left as indicated by the arrows in the figure, it enters the corresponding image sensor in the right direction on the hemispherical protruding surface of the image display panel. The incident position of incident light can be specified by assigning a number (address) to the hemispherical protruding surface, and the incident angle can be detected by the position of the image sensor on the protruding surface.

  In the figure, the protrusion surfaces Ca1 to Cay of the (b-1), (b), and (b + 1) columns of the a-th row of the hemispherical protrusion surfaces constituting the imaging element group arranged in a matrix of k rows and j columns. Is formed on the display panel T1 (upper side). Further, the protrusion surfaces La1 to Ray of the (j−b + 2), (j−b + 1), and (j−b) columns of the a-th row of the hemispherical protrusion surfaces constituting the display element group arranged in the k-row and j-column matrix. Is formed on the display panel T2 (lower side). The display panels T1 and T2 may be provided at different locations W1 and W2.

  FIG. 18 shows in detail the display element group formed on the hemispherical protruding surface. As shown in FIG. 18 (a), the hemispherical projecting surface is configured as an assembly of an image sensor and a display element on which both the image sensor and the display element are mounted. In addition, these image pickup element / display element assemblies are formed so that a display element is formed with an LCD or the like on a hemispherical projecting surface, and is covered with a light-transmitting layer, and further, it is not overlapped with a virtual display element. An image sensor is formed by a CCD or the like, and an adjacent image sensor and display element are handled as a pair of elements.

  Since the light that has passed through the display element travels straight as shown in FIG. 18B, the emission direction can be specified by identifying the address at which the display element is emitted. become. FIG. 18C shows a driving circuit for such an LCD display element, in which position data of the display element to be driven is set in the register RG1, and scanning data is set in the register RG2.

As an example of arranging the above-described element assemblies on the imaging display panel, for example, as shown in FIG. 19, imaging element assemblies C _{11 to} C _kj and display element assemblies L _{11 to} L _kj are separately formed. The two-element type, and the one-element type composed of the imaging display elements D _{11 to} D _{kj in} which the imaging element and the display element are formed as one assembly, but when any type is adopted, By installing each image display panel of the same type at two different points via a network, the incident angle is determined by identifying the address of the image sensor on which the light is incident, and the luminance level for the incident light Can be identified and sent to and received from each other. In addition, by matching the brightness level of light emitted from the display element, the brightness level received from the other party, and the incident angle, an ideal spatial connection effect can be obtained, and these can be used for videophones and videoconferencing systems. It can also be applied.

Japanese Patent Laid-Open No. 61-65683 JP-A-4-11485 JP-A-9-168141 JP 2000-83228 A JP 2001-52177 A JP 7-296139 A JP-A-7-99644 JP 2002-300602 A

  However, in the window-shaped imaging display device described in Patent Document 8, since it is necessary to arrange the same number of imaging elements as display elements on the imaging display panel, the signal wiring becomes complicated and the manufacturing difficulty level is improved. There was a problem. Further, as a result of light leakage from the display element to the image pickup element, there is a problem that the above-described spatial connection effect cannot be obtained with high efficiency.

  Therefore, the present invention has been devised in view of the above-described problems, and in particular, an image sensor is omitted in an image transmission / reception system, an imaging display apparatus, and a method for obtaining a spatial connection effect similar to that of a window glass. Therefore, it is an object of the present invention to provide an imaging display device and method, and an image transmission / reception system that can be expected to facilitate manufacture by simplifying the device configuration.

  In order to solve the above-described problems, the present invention captures a subject to be photographed from different viewpoints by at least three cameras arranged around the display unit, and at least a method of each image captured by each of the cameras. Correction is performed by matching the line direction to one direction, and the corrected images are associated with each other while being associated with the object to be imaged, and relative to each camera at the optical center in the virtual camera that is virtually installed Relative position information indicating a specific positional relationship, and according to the relative position information generated from the pixel positions associated with each other and their luminance components, the pixel positions constituting the virtual viewpoint image to be generated by the virtual camera and the A luminance component is obtained, and a virtual viewpoint image composed of the obtained pixel position and the luminance component is transmitted via the network. Each of the pixels constituting the virtual viewpoint image by receiving the virtual viewpoint image generated in the other electronic device via the network, specifying the emission direction in the display unit according to each pixel position constituting the received virtual viewpoint image By emitting light in accordance with the luminance component at the position from the display unit in the designated emission direction, a spatial connection effect is obtained without forming an image sensor on the display unit.

That is, the imaging display device to which the present invention is applied has a two-dimensional array of protrusions in which a plurality of display elements are formed on a hemispherical protrusion surface, and selects an address assigned to the display element formed in each protrusion. The imaging object is different from each other by display means for displaying an image by emitting light according to a luminance component at a pixel position in a designated emission direction and at least four cameras arranged around the display means. Imaging means for imaging from a viewpoint, information generation means for generating relative position information indicating the relative positional relationship of the optical center of the virtual camera virtually installed with respect to each of the cameras, and images captured by at least each of the cameras Between the image correcting means for aligning the normal direction of each image in one direction and each image corrected by the image correcting means, the pixels are associated with the photographing object. A virtual viewpoint image to be generated by the virtual camera according to the generated relative position information from a matching unit that associates each position and a pixel position and a luminance component associated with each other by the matching unit. An image generation means for obtaining a pixel position and its luminance component to be configured, and a virtual viewpoint image constituted by the pixel position and its luminance component obtained by the image generation means are transmitted to another electronic device, or the other electronic A transmission / reception means for receiving a virtual viewpoint image generated by the device, and a display element for each protrusion two-dimensionally arranged in the display means according to each pixel position constituting the virtual viewpoint image received by the transmission / reception means. The display means is controlled so that emitted light corresponding to the luminance component at the pixel position is emitted by each display element. Display processing means, and the image correction means selects two sets of three cameras from the four cameras based on the relative position information generated by the information generation means, and selects the selected set. Every time, a projection transformation matrix for projecting the normal direction of each image captured by the three cameras onto a virtually set virtual plane is obtained, and the normal of each image is obtained based on the obtained projection transformation matrix. the combined direction in the normal direction of the virtual plane, when combined normal direction is different for each set, the commonly captured by one camera is not selected by the projective transformation matrix of each other pair so that a normal line in the same direction as that of the projected image on the basis of, corrects each normal line direction of the projected image based on the captured by the other three cameras the projective transformation matrix.

In addition, an imaging display device to which the present invention is applied has an imaging means for imaging a subject to be photographed from different viewpoints by at least four cameras, and a protruding portion in which a plurality of display elements are formed on a hemispherical protruding surface. Display means for displaying an image by selecting an address assigned to a display element formed in each protruding portion and emitting light according to a luminance component at a pixel position in a designated emission direction; Information generating means for generating relative position information indicating the relative positional relationship of the optical center with respect to each of the cameras in a virtual camera that is installed, and at least one normal direction of each image captured by each of the cameras Matching is performed for each pixel position between the image correction unit matched to the image and each image corrected by the image correction unit, in association with the shooting target. The pixel position and the luminance of the virtual viewpoint image to be generated by the virtual camera according to the generated relative position information from the pixel position and the luminance component associated with each other by the matching means and the matching means According to the image generation means for obtaining the component and each pixel position constituting the virtual viewpoint image obtained by the image generation means, the address of the display element is selected for each of the two-dimensionally arranged protrusions in the display means, Display processing means for controlling the display means so that emitted light corresponding to the luminance component of the pixel position is emitted by each display element, and the image correction means includes relative position information generated by the information generation means. based on, among four cameras, three cameras select two sets, for each set selected, the normal direction of the image captured by the three cameras The virtually determined projective transformation matrix for projective to set virtual plane, the normal direction of each image based on the projection transformation matrix determined in accordance with the normal direction of the virtual plane, for each set When the combined normal directions are different, the normal direction of the image captured by one camera not selected in common with each other and projected based on the projective transformation matrix is the same as the normal direction. The normal directions of the images captured by the other three cameras and projected based on the projective transformation matrix are corrected.

That is, in the imaging display method to which the present invention is applied, the projections having a plurality of display elements formed on the hemispherical projection surface are two-dimensionally arranged, and addresses assigned to the display elements formed in the projections are selected. Then, the imaging object is imaged from different viewpoints by at least four cameras arranged around the display unit that displays an image by emitting light according to the luminance component at the pixel position in the designated emission direction. An imaging step, a generation step for generating relative position information indicating a relative positional relationship of the optical center with respect to each of the cameras in a virtually installed virtual camera, and a normal direction of each image captured by at least each of the cameras The image correction step for correcting the image by aligning the image in one direction and the image to be imaged between the images corrected by the image correction step. The virtual camera should be generated according to the relative position information generated by the information generation step from the matching step for performing the association for each pixel position, the pixel position associated with each other by the matching step and the luminance component thereof. An image generation step for obtaining a pixel position and a luminance component thereof constituting a virtual viewpoint image, and a transmission step for transmitting a virtual viewpoint image constituted by the pixel position and the luminance component obtained by the image generation step via a network; A reception step of receiving a virtual viewpoint image generated in another electronic device via the network, and a two-dimensional array in the display unit according to each pixel position constituting the virtual viewpoint image received by the reception step. Select the address of the display element for each protruding part, and Display processing step for controlling the display unit so that emitted light corresponding to the luminance component of the display component is emitted by each display element. In the image correction step, the relative position information generated by the information generation step is added to the relative position information. Based on this, two sets of three cameras are selected from the four cameras, and a virtual plane in which the normal direction of each image captured by the three cameras is virtually set for each selected set calculated projective transformation matrix for projective to, when it the normal direction of the image based on the projection transformation matrix determined tailored to the normal direction of the virtual plane, the normal direction to suit each set is different from Imaged by the other three cameras so as to be in the same direction as the normal direction of the image captured by one camera not selected in common with each other and projected based on the projective transformation matrix. Based on the projective transformation matrix Each normal direction of the projected image is corrected.

An image transmission / reception system to which the present invention is applied is an image transmission / reception system that bidirectionally transmits / receives an image captured by each imaging display device, and the imaging display device includes a plurality of hemispherical protrusion surfaces. The protrusions on which the display elements are formed are two-dimensionally arranged, and an address assigned to the display element formed on each protrusion is selected, and light corresponding to the luminance component at the pixel position is emitted in the designated emission direction. Display means for displaying an image, image pickup means for picking up a photographing object from different viewpoints by at least four cameras arranged around the display means, and optics in a virtual camera virtually installed Information generating means for generating relative position information indicating a relative positional relationship with respect to each camera at the center, and at least each image captured by each camera An image correcting unit that matches the normal direction of the image to one direction, a matching unit that associates each image position corrected by the image correcting unit for each pixel position while being associated with the photographing object, and the matching unit Image generation means for obtaining a pixel position and its luminance component constituting a virtual viewpoint image to be generated by the virtual camera according to the generated relative position information from the pixel position and its luminance component associated with each other by The virtual viewpoint image composed of the pixel position obtained by the image generation means and the luminance component thereof is transmitted to the partner imaging display device via the network, or the virtual viewpoint image generated by the partner imaging display device is generated. Transmission / reception means for receiving viewpoint images via the network, and virtual viewpoint images received by the transmission / reception means According to each pixel position to be configured, an address of the display element is selected for each of the two-dimensionally arranged protrusions in the display unit, and emitted light corresponding to the luminance component of the pixel position is emitted by each display element. Display processing means for controlling the display means, and the image correction means includes two sets of three cameras among the four cameras based on the relative position information generated by the information generation means. For each selected group, a projection transformation matrix for projecting the normal direction of each image captured by the three cameras onto a virtually set virtual plane is obtained, and based on the obtained projection transformation matrix When the normal direction of each image is matched with the normal direction of the virtual plane and the normal direction is different for each group, the camera is not selected in common with each other. Imaged and based on the projective transformation matrix Each normal direction of the image captured by the other three cameras and projected based on the projection transformation matrix is corrected so as to be in the same direction as the normal direction of the projected image.

  The imaging display device and method, and the image transmission / reception system to which the present invention is applied have a spatial connection effect similar to that of a window glass, while simplifying the configuration of the device by omitting an imaging element, and expecting easy manufacturing. Can be obtained efficiently.

  Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

  An image transmission / reception system 1 to which the present invention is applied, for example, as shown in FIG. 1, bi-directionally transmits and receives images captured by the imaging display devices 2 a and 2 b installed at different points A and B via a network 50. By doing so, it is a system for performing a remote dialogue while visually recognizing the display image of the other party from a location distant from each other.

  The imaging display device 2a installed at the point A includes an imaging camera group 10a including eight cameras 11a to 18a that capture the user a as a subject to be photographed from different viewpoints, and a user b captured at the point B side. An image signal processing unit 6a for generating a virtual viewpoint image Ima based on the images Pa1 to Pa8 captured by the imaging camera group 10a, and an imaging signal processing unit. The virtual viewpoint image Ima generated in 6a is transmitted to the partner imaging display device 2b, or the virtual viewpoint image Imb generated by the imaging display device 2b is received by the transmission / reception unit 7a and the transmission / reception unit 7a. A display signal processing unit 8a for designating an output direction in the display 5a according to each pixel position constituting the virtual viewpoint image Imb. To have.

  The imaging display device 2b installed at the point B includes an imaging camera group 10b including eight cameras 11b to 18b that capture the user b as an imaging target from different viewpoints, and a user a captured at the point A side. An image signal processing unit 6b that generates a virtual viewpoint image Imb based on the images Pb1 to Pb8 captured by the imaging camera group 10b, The virtual viewpoint image Imb generated in 6b is transmitted to the partner imaging display device 2a, or received by the transmission / reception unit 7b, which receives the virtual viewpoint image Ima generated by the imaging display device 2a. A display signal processing unit 8b for designating an output direction in the display 5b according to each pixel position constituting the virtual viewpoint image Ima. To have.

  The cameras 11a to 18a constituting the imaging camera group 10a generate images Pa1 to Pa8 based on imaging signals obtained by converting incident subject images into electrical signals, respectively. The cameras 11b to 18b constituting the imaging camera group 10b generate images Pb1 to Pb8 based on imaging signals obtained by converting incident subject images into electrical signals, respectively.

  The imaging camera groups 10a and 10b are arranged around the displays 5a and 5b as shown in FIG. The cameras 11a and 11b are respectively installed on the upper left side of the displays 5a and 5b when viewed from the users a and b, the cameras 12a and 12b are on the upper side, the cameras 13a and 13b are on the upper right side, and the cameras 14a and 14b are on the left side. The cameras 15a and 15b are installed on the right side, the cameras 16a and 16b on the lower left side, the cameras 17a and 17b on the lower side, and the cameras 18 and 18b on the lower right side. These imaging camera groups 10a and 10b are installed with the shooting direction and the shooting angle of view fixed, but these may be freely changed based on information input from the users a and b. Good. Incidentally, the number of cameras 11 to 18 constituting the imaging camera group 10 may be any number as long as it is at least three.

  Note that the virtual viewpoint images Ima and Imb are virtually installed at respective positions on the display screen indicated by the oblique lines of the displays 5a and 5b in the positional relationship with the imaging camera groups 10a and 10b shown in FIG. It corresponds to an image captured by a camera. That is, the installation position of the virtual camera can be arbitrarily set as long as it is on this display screen. A method for generating the virtual viewpoint images Ima and Imb will be described in detail later.

  The displays 5a and 5b display images based on the virtual viewpoint images Ima and Imb supplied from the counterpart point via the network 50, for example, via a liquid crystal display surface. The liquid crystal display surfaces of the displays 5a and 5b are composed of a large number of liquid crystal display elements and the like, and the liquid crystal display elements are optically modulated in accordance with output signals based on the virtual viewpoint images Ima and Imb, thereby creating an image to be displayed to the user.

  The imaging signal processing units 6a and 6b are integrated circuits for processing the images obtained by the imaging camera groups 10a and 10b. The detailed configuration of each of the imaging display devices 2a and 2b will be described later.

  The network 50 includes an Internet network connected to the imaging display devices 2a and 2b via a telephone line, ISDN (Integrated Services Digital Network) / B (broadband) -ISDN connected to a TA / modem, and the like. It is a public communication network that enables bidirectional transmission and reception of information. Incidentally, when the communication system 1 is operated in a certain narrow area, the network 7 may be configured by a LAN (Local Area Network). Further, when transmitting moving images, the network 7 is continuously transmitted from one channel having moving images including, for example, MPEG (Moving Picture Experts Group) data, based on the Internet protocol (IP). . In addition, when transmitting a still image, the image is transmitted at regular intervals from a channel different from the channel for transmitting a moving image.

  Note that a terminal device 51, a data storage device 52, and a data distribution device 53 may be connected to the network 50.

  The terminal device 51 is constituted by a personal computer, for example, and applies desired processing to the images transmitted from the imaging display devices 2a and 2b. Further, the terminal device 51 remotely controls the imaging display devices 2 a and 2 b via the network 50.

  The data storage device 52 is a storage that stores images transmitted via the network 50. Further, the data storage device 52 may read the stored image afterwards in response to a request from the terminal device 51 or the imaging display device 2.

  The data distribution device 53 manages Internet information, for example, and distributes predetermined content information stored therein to the imaging display device 2 and the terminal device 51.

  The transmission / reception units 7a and 7b perform signal processing necessary for transmitting the virtual viewpoint images Ima and Imb supplied from the imaging signal processing devices 6a and 6b based on the communication standard and protocol in the network 50 described above, This is transmitted to the imaging display device 2 on the other side via the network 50. Further, the transmission / reception units 7a and 7b perform communication signal processing such as demodulation on the virtual viewpoint images Ima and Imb transmitted from the other side via the network 50, and output them to the display signal processing units 8a and 8b. .

  The display signal processing units 8a and 8b perform processing such as amplification and signal decompression on the virtual viewpoint images Ima and Imb from the transmission / reception units 7a and 7b. Further, the display signal processing units 8a and 8b read the luminance levels and the like constituting the virtual viewpoint images Ima and Imb, and control so as to optically modulate the liquid crystal display elements of the displays 5a and 5b via a video interface (not shown). .

  Next, the configuration of the display 5a (5b) will be described with reference to FIG.

  The display 5a employs a display panel 61 as shown in FIG. 3A as a display surface for displaying an image. The display panel 61 has a hemispherical projection 62 formed over the entire surface, and a large number of display elements 63 are formed on the surface of the projection 62. The protrusions 62 are formed in a matrix arrangement in X rows and Y columns.

  FIG. 3B shows a cross-sectional view of the protrusions 62 arranged in the a row among the protrusions 62 in the X rows and Y columns arranged in the matrix. Outgoing light is emitted from the display elements on the protrusions 62 formed in rows (b + 2), (b + 1), and b.

  FIG. 3C shows the detailed configuration of these protrusions 62. The display elements 63 are hierarchically arranged over m stages from the top to the bottom of the hemispherical protrusion 62. One display element 63 is formed at the top, and the number m of steps until reaching the bottom is determined according to the pitch of the outgoing angle of the outgoing light. In addition, the number n of display elements 63 formed in each stage is a number that gradually increases from the first stage to the m-th stage. In this protrusion 62, an address indicating in which arrangement each display element 63 is formed may be assigned.

  Since the light that has passed through the display element 63 travels straight as shown in FIG. 3D, it is possible to easily determine the emission direction by identifying which address is the light emitted from the display element 63. Can be identified.

  Incidentally, the viewing angle θd of these protrusions 62 is designed to be equivalent to the imaging angle θc of the cameras 11 to 18 as shown in FIG. Thereby, when restoring the virtual viewpoint images obtained from the cameras 11 to 18 through the light emitted from the projecting portion 62, this can be executed efficiently. Incidentally, it is designed to satisfy θd, θc <180 °.

  Next, the configuration of the imaging signal processing unit 6 will be described by taking the imaging signal processing unit 6a as an example.

  As shown in FIG. 5, the imaging signal processing unit 6a includes a correction unit 20 to which images Pa1 to Pa8 are supplied from the connected cameras 11a to 18a, a matching unit 29 connected to the correction unit 20, and A virtual viewpoint image generation unit 30 connected to the matching unit 29 and an information generation unit 33 that generates information indicating the relative positional relationship of the user a with respect to the cameras 11a to 18a are provided.

  The correction unit 20 includes geometric image correction units 41a to 48a for performing geometric image correction on the images Pa1 to Pa8 transmitted from the cameras 11a to 18a, and these geometric image correction units geometrically. And a normalization processing unit 24 for normalizing the images subjected to the image correction by the image correction units 41a to 48a.

  The geometric image correction units 41a to 48a correct the images Pa1 to Pa8 based on the control information transmitted from the camera calibration unit 26 and including the geometric positional relationship of the cameras 11a to 18a. The geometric positional relationship between the cameras 11a to 18a may be parameterized in the control information transmitted from the camera calibration unit 26 described above. Further, when imaging is performed while changing the shooting direction and / or the shooting angle of view of each of the cameras 11a to 18a, these are parameterized by the camera calibration unit 26, and these parameters are controlled when correcting the image. It may be included in the information. Accordingly, the geometric image correction units 41a to 48a can perform image correction in real time according to the shooting direction and / or the shooting angle of view of each camera 11a to 18a.

  Similarly, the camera calibration unit 26 may parameterize chromatic aberration, distortion, and optical axis shift in each lens of the cameras 11 a to 18 a and transmit them to the correction units 20.

  The normalization processing unit 24 is supplied with the images corrected by the respective geometric image correction units 41a to 48a, and performs geometric normalization processing on these images. The normalization processing unit 24 selects three cameras from the images Pa1 to Pa8 captured by the cameras, and matches the normal directions of the images captured by the selected three cameras. That is, the normalization processing unit 24 generates normalized images Pm1 to Pm8 that are normalized by matching the normal direction of each image Pa1 to Pa8 with the normal direction of the virtually set virtual plane π. To do. In such a case, the normalization processing unit 24 obtains a projective transformation matrix for projecting each of the images Pa1 to Pa8 captured by the cameras 11a to 18a onto the virtual plane π, and based on the obtained projective transformation matrix The normal direction of each image is matched with the normal direction of the virtual plane π.

  Incidentally, when a so-called fixed viewpoint camera is applied as the cameras 11a to 18a, the camera calibration unit 26 may acquire the normal directions of the images Pa1 to Pa8 by the camera calibration unit 26 in advance. Further, when imaging is performed while changing the shooting direction and / or shooting angle of view of each camera 11a to 18a, these are parameterized by the camera calibration unit 26, and these parameters are set when normalizing the image. It may be included in the control information. Accordingly, it is possible to flexibly cope with the case where imaging is performed while sequentially changing the shooting direction according to the positions of the users a and b.

  FIG. 6 is a diagram for explaining such a camera calibration method. A teaching image pattern 92 is set at a predetermined distance z from the entire surface of the cameras 11a to 18a. The teaching image pattern 92 is formed with a special pattern such as a checker pattern, for example, and the size of each checker is known. The camera 11a to 18a are used to capture images of these teaching image patterns 92 from different viewpoints, and after the distortion correction is performed, the checker pattern images are captured so that the individual checker sizes are the same. Adjust position, shooting direction, zoom, etc.

  In addition, by storing these parameters in a ROM or RAM (not shown) in the camera calibration unit 26, the correction unit 20 can refer to them at any time according to the situation, and can perform high-speed correction processing. Can be realized. The camera calibration unit 26 obtains these parameters every time the images Pa1 to Pa8 are supplied from the cameras 11a to 18a, thereby realizing a highly accurate correction process in each of the geometric image correction units 41 to 48. can do.

  The matching unit 29 is supplied with the normalized images Pm1 to Pm8 generated by the normalization processing unit 24 in the correction unit 20. The matching unit 29 associates the normalized images Pm1 to Pm8 with each other.

  This association is performed by extracting pixel positions and luminance components at the same location constituting the face of the user a between the normalized images Pm1 to Pm8. For example, when associating the pixels constituting the normalized images Pm1, Pm2, and Pm3, for example, as shown in FIG. 7, with respect to the pixel position P11 on the normalized image Pm1, the normalized image Pm2 The pixel position P11 ′ existing at the same location is specified as the corresponding point, and the pixel position P11 ″ existing at the same location on the normalized image Pm3 is specified as the corresponding point. In such a case, when associating the pixel position P11 on the normalized image Pm1 with the pixel position P11 ′ on the normalized image Pm2, the pixel information on the normalized image Pm3 is referred to. You may do it.

  That is, the matching unit 29 associates the normalized images Pm1, Pm2, and Pm3 normalized by the normalization processing unit 24 for each pixel position while associating them with the imaging target. Incidentally, the matching unit 29 may be executed only for the part where the feature is extracted for this association, or may be executed for all the pixels constituting the normalized images Pm1, Pm2, and Pm3.

  The information generation unit 33 may generate information to be generated based on the line-of-sight direction of the user a with respect to the display 5a. In such a case, the information generation unit 30 acquires the viewing direction of the user a from the images Pa1 to Pa8 supplied from the cameras 11a to 18a, and generates information on the position of the user a based on the acquired direction. The information generation unit 33 may generate information related to the position of the user a based on information input via an operation unit such as a keyboard or a mouse (not shown).

  The virtual viewpoint image generation unit 30 is input with the pixel positions and the luminance components associated with each other by the matching unit 29. Further, the virtual viewpoint image generation unit 30 obtains a pixel position and its luminance component constituting the virtual viewpoint image Ima to be newly generated from the pixel position and the luminance component associated with each other. At this time, the virtual viewpoint image generation unit 30 may obtain the pixel position constituting the virtual viewpoint image Ima and its luminance component according to the relative position information generated by the information generation unit 33. This relative position information indicates the relative positional relationship between the cameras 11 to 18 at the optical center in the virtual camera that is virtually installed, and is generated by the information generation unit 33 described above. The virtual viewpoint image generation unit 30 transmits a virtual viewpoint image Ima configured by the obtained pixel position and its luminance component to the transmission / reception unit 7a.

  Next, a specific operation in the imaging display device 2 will be described.

  In the imaging display device 2, a processing operation for obtaining a spatial connection effect described below is executed. For example, as shown in FIG. 8, the space connection effect is that a window 67 is installed on the wall surfaces of two adjacent places W1 and W2 partitioned by a wall, and the object D placed at the place W2 , E, F are assumed to be viewed through the window 67 by the users A, B, C at the location W1. In such a case, the user A can visually recognize the scene centering on the object F through the window 67, and the user B can view the scene in which all the objects D, E, and F are projected through the window 67. Furthermore, the user C can visually recognize a scene centering on the object D through the window 67.

  That is, the space connection effect means that an effect equivalent to that obtained when the object is visually recognized through these windows 67 is obtained. In general, in order to obtain such a spatial connection effect, for example, as shown in FIG. 9 (a), light incident at an angle (θ °, φ °) at an arbitrary point (x, y) on the window glass surface, What is necessary is just to control so that it may radiate | emit with an angle (180 degrees-theta, 180 degrees -phi degrees) from the same position of the panel.

  Further, for example, as shown in FIG. 9B, when the incident surface 68 is at the location W1 ′ and the emission surface 69 is at the location W2 ′, the location W1 ′ and the location W2 ′ are in remote locations. Even at a certain time, the same spatial connection effect can be obtained theoretically. That is, at an arbitrary point (x, y) on the incident surface 68, a luminance level is identified for each incident angle (θ °, φ °) with respect to incident light, and the identified luminance level is determined as an incident angle (θ °, (φ °) and transmit to the display side. Then, light that is 180 ° out of phase with the incident angle (θ °, φ °) from the point (x ′, y ′) corresponding to (x, y) on the display side panel is emitted at the identified luminance level. To do.

  Therefore, in order to create a display surface having a spatial connection effect similar to that of a window glass, the incident surface is provided with an imaging function, and incident angles (θ °, φ for incident light at all points on the incident surface). °) It is necessary to identify the brightness level every time. In the imaging display device 2 to which the present invention is applied, the imaging function on these incident surfaces is assigned to the imaging camera group 10a (10b) disposed around the displays 5a and 5b.

  The cameras 11a to 18a constituting the imaging camera group 10a (10b) shoot users as shooting targets from different viewpoints. As a result, the line of sight of the user a on the images Pa1 to Pa8 generated by the cameras 11a to 18a, the face orientation, and the like are different from each other. When such images Pa1 to Pa8 are respectively supplied to the normalization processing unit 24 in the correction unit 20, they are normalized based on the method described below.

  FIG. 10 shows a case where the images Pa1, Pa2, and Pa3 captured by the three selected cameras 11a, 12a, and 13a among the cameras 11a to 18a are normalized.

  As shown in FIG. 10, when an image is taken from different viewpoints to the M point to be photographed from different viewpoints by the optical centers C1, C2, and C3 of the cameras 11a, 12a, and 13a, they are generated. The images Pa1, Pa2, and Pa3 are parallel to the imaging surfaces of the cameras 11a, 12a, and 13a. Here, the direction of the straight line connecting each camera 11a, 12a, 13a and the M point coincides with the normal direction k1, k2, k3 of each image Pa1, Pa2, Pa3 captured by each camera, but these are mutually Pointing in a different direction. By performing geometric normalization so that the normal directions k1, k2, and k3 of these images Pa1, Pa2, and Pa3 are in the same direction, normalized images Pm1, Pm2, and Pm3 whose image planes are parallel to each other are generated. .

  This geometric normalization uses camera projection parameters P1, P2, and P3 obtained in advance by the camera correction parameter 13, and uses camera internal parameters A1, A2, and A3, rotation matrices R1, R2, and R3, and transition matrices T1 and T2. , T3 is estimated. As a result, normalized images Pm1, Pm2, and Pm3 are generated in which the normal directions k1 ', k2', and k3 'of the corrected image pickup surface are parallelized.

  Incidentally, when performing this geometric normalization, a virtual plane π surrounded by the optical centers C1, C2, and C3 is set, and each of these images Pa1, Pa2 with respect to the normal direction of the virtual plane π. , Pa3 may be normalized using the projection matrices P1, P2, P3 so that the normal direction of Pa3 is the same.

  Further, the normalization of these images may be reflected in the calibration such as the shooting direction and zoom of each camera 11a, 12a, 13a by the camera calibration unit 26. Conversely, these calibrations may be reflected in the above-described geometric normalization.

  The normalized images Pm1, Pm2, and Pm3 that have undergone projective transformation as described above in the normalization processing unit 24 are associated with each pixel position while being associated with the imaging target in the matching unit 29 as described above. In the present invention, the normalization processing unit 24 in the previous stage of the matching unit 29 is normalized in advance and the epipolar lines are parallelized. In combination with this, in the present invention, since two normalized images among the normalized images Pm1 to Pm3 are associated with each other with reference to the remaining normalized image, two cameras are used. Compared to the above case, more information is referred to at the time of matching, and the robustness of pixel search can be improved.

  FIG. 11 shows a case where the matching unit 29 associates these normalized images Pm1, Pm2, and Pm3. When matching is performed in the matching unit 29 for the normalized images Pm1 and Pm2, the corresponding point of the pixel m1 ′ on the epipolar line L1 of the normalized image Pm1 is on the epipolar line L2 of the normalized image Pm2. The pixel m2 ′ as the corresponding point can be detected by searching on L2. In such a case, the pixel m2 'may be detected by simultaneously searching for the epipolar line L2' of the normalized image Pm2 corresponding to the epipolar line L1 'of the normalized image Pm1.

  Similarly, when matching is performed in the matching unit 29 for the normalized images Pm1 and Pm3, regarding the corresponding point of the pixel m1 ′ on the epipolar line L1 of the normalized image Pm1, the epipolar line of the normalized image Pm3 The pixel m3 ′ as a corresponding point can be detected by searching on the L3.

That is, since the normalized images Pm1, Pm2, and Pm3 are subjected to geometric normalization in the normalization processing unit 24 in the previous stage, it is possible to efficiently associate each pixel. This is because each pixel on the epipolar line L1 to be scanned corresponds to each pixel on the epipolar line L2 (L3), so that the correspondence between the pixels arranged in each column is solved as a shortest path search problem by Dynamic Programming. Because it can. The matching unit 29 uses the luminance difference dg (x, y) between the images and the color information difference dI (x, y) to obtain a matching cost function MC (x, y) expressed by the following equation (1). y) is determined.
MC (x, y) = dg (x, y) + α · dI (x, y) (1)
The matching unit 29 detects corresponding points based on the obtained matching cost function MC (x, y). In such a case, input is performed from each correction processing unit 20 in order to further improve the matching accuracy. The occlusion cost may be appropriately adjusted according to each image to be processed. The occlusion cost reflects the distance of the camera group 10 with respect to the object to be imaged, parallax, or the like.

  That is, since the image transmission / reception system 1 to which the present invention is applied realizes the matching processing described above in the matching unit 29, it is possible to accurately and smoothly perform the association between the pixels. Incidentally, the pixel positions and the luminance components associated with each other by the matching unit 29 are output to the virtual viewpoint image generation unit 30, respectively.

In the virtual viewpoint image generation unit 30, based on the pixel positions associated with each other and the relative position information transmitted from the information generation unit 33, the pixel positions constituting the virtual viewpoint image Ima based on the method described below and the positions thereof Find the brightness level (shading value). FIG. 12 is a diagram for explaining a case where the pixel Cv on the virtual viewpoint image Ima is specified according to the pixel m1 ′, the pixel m2 ′, and the pixel m3 ′ associated in the matching unit 29. The pixel position of the pixel Cv shown in FIG. 12 is Pv (xv, yv), and the pixel positions of the pixels m1 ′, m2 ′, m3 ′ are P1 (x1, y1), P2 (x2, y2), When P3 (x3, y3) is set, Pv (xv, yv) can be obtained by the following equation (2).
Pv (xv, yv) = α × P1 (x1, y1) + β × P2 (x2, y2) + γ × P3 (x3, y3) (2)
Here, α, β, and γ are relative position information, α is the area of the triangle C2C3Cv / the area of the triangle C1C2C3, β is the area of the triangle C1C2Cv / the area of the triangle C1C2C3, and γ is the triangle C1C2Cv. Area / triangle C1C2C3 area. Incidentally, α + β + γ = 1.

  Here, when the pixel Cv is a pixel corresponding to the optical center of the virtual camera, the pixel position Pv (xv, yv) determined based on the equation (2) is such that the optical center of the virtual camera is the optical center of the camera 11a. It approaches P1 (x1, y1) as it approaches C1, approaches P2 (x2, y2) as it approaches the optical center C2 of the camera 12a, and approaches P3 (x3, y3) as it further approaches the optical center C3 of the camera 13a. become. That is, by freely determining the relative position information α, β, γ, the pixel position Pv (xv, yv) of the pixel Cv can be freely set within the frame of the triangle C1C2C3. It is possible to freely change the position of the user a displayed on the screen.

The luminance level of the pixel Cv is mv ′ (xv, yv), and the luminance levels of the pixels m1 ′, m2 ′, and m3 ′ are respectively Q1 (x1, y1), Q2 (x2, y2), Q3 (x3, When y3), mv ′ (xv, yv) can be obtained by the following equation (3).
mv ′ (xv, yv) = α × Q1 (x1, y1) + β × Q2 (x2, y2) + γ × Q3 (x3, y3) (3)
Here, when the pixel Cv is a pixel corresponding to the optical center of the virtual camera, the luminance level mv ′ (xv, yv) of the pixel Cv determined based on the equation (3) is obtained when the optical center of the virtual camera is the camera. 11a approaches the optical center C1 of 11a, approaches Q1 (x1, y1), approaches the optical center C2 of the camera 12a, approaches Q2 (x2, y2), and further approaches the optical center C3 of the camera 13a, Q3 (x3, y3) ). That is, since the brightness level mv ′ (xv, yv) can be freely determined according to the optical center position of the virtual camera, the brightness of the pixels constituting the user a displayed on the virtual viewpoint image Ima can be freely set. Can be changed.

  As described above, the virtual viewpoint image generation unit 30 can obtain the pixel position Pv (xv, yv) and the luminance level mv ′ (xv, yv) of the pixel Cv located in the triangle C1C2C3. In addition, the imaging display device 2 can freely set the pixel position Pv (xv, yv) of the pixel Cv within the frame of the triangle C1C2C3 by freely determining the relative position information α, β, γ. Further, the information generation unit 33 determines the relative position information α, β, γ based on the information on the position of the user, so that the virtual viewpoint image generation unit 30 performs the above-described pixel in the direction of the line of sight of the identified user a. The pixel position Pv (xv, yv) of Cv can be adjusted. Accordingly, it is possible to realize the same processing as that of adjusting the shooting direction of the virtual camera to the line of sight of the user a.

  In particular, since the cameras 11a to 18a have different shooting directions, the luminance components at the image positions w1, w2, and w3 indicating the same image are different from each other. Either one of the different luminance components is set to the minimum value, and the other is set to the maximum value, and the pixel position Pv (xv, yv) or the luminance level mv ′ according to α, β, γ based on the relative position information. By linearly increasing / decreasing (xv, yv), it is possible to accurately realize the picture processing on the virtual viewpoint image Ima according to the position of the virtual camera.

  In the imaging display device 2a, the normalization processing unit 24 generates the virtual viewpoint image Ima from the normalized images Pm1, Pm2, and Pm3 matched with the normal direction of the virtual plane π. This makes it possible to improve the efficiency of image interpolation. In addition, this makes it possible to calculate with the Z-direction coordinate of each camera center set to zero. That is, it is sufficient to focus on only the x and y direction coordinates and use the equations (2) and (3) to calculate, so that the relative amount of calculation in generating the virtual viewpoint image Ima can be reduced. It is possible to realize a rapid painting process.

  FIG. 13 shows a case where the virtual viewpoint image generator 30 generates the virtual viewpoint image Ima from the normalized images Pm1, Pm2, and Pm3 actually acquired from the three cameras 11a, 12a, and 13a. When a substantially equilateral triangle is formed by a line connecting the optical centers C1, C2, and C3 of the cameras 11a, 12a, and 13a, as shown in FIG. 13A, the optical centers C1, C2, and C3 are equidistant from each other. The virtual viewpoint image Ima obtained when the optical center CL of the virtual camera is set at a certain center of gravity position is obtained by approximately averaging the contents of the normalized images Pm1, Pm2, and Pm3 as shown in FIG. 13 (b). It becomes. That is, from the normalized image Pm1 indicating the user a facing the right direction, the normalized image Pm2 indicating the user a facing the upward direction, and the normalized image Pm3 indicating the user a facing the left direction, A virtual viewpoint image Ima indicating the user a facing directly in front is generated.

  In the imaging display device 2 to which the present invention is applied, these generated virtual viewpoint images Ima are captured by a virtual camera virtually installed at each position on the display screen of the display 5 as shown in FIG. I see. Then, by finely adjusting the relative position information α, β, and γ described above, it is possible to make a fake as if a virtual camera was installed over the entire display screen of the display 5.

  FIG. 14 shows a case where a virtual camera is virtually installed at an arbitrary position on the display screen of the display 5. For example, when the optical center of the virtual camera is set in the region F surrounded by the optical centers C1, C2, and C3 of the cameras 11, 12, and 13 in FIG. 14, the image Pa1 captured by the cameras 11, 12, and 13 is set. , Pa2 and Pa3, the virtual viewpoint image Ima is generated as described above. The detailed position of the virtual camera in the region F is determined by finely adjusting the relative position information α, β, γ described above.

  Similarly, the region G surrounded by the optical centers C2, C4, and C6 of the cameras 12, 14, and 16, the region H surrounded by the optical centers C3, C5, and C7 of the cameras 13, 15, and 17, and the cameras 16, 17, and 18 For the region I surrounded by the optical centers C6, C7, and C8, the virtual viewpoint image Ima is generated based on the images from the cameras.

  By the way, when setting the optical center of the virtual camera in the area surrounded by any four optical centers including the area J surrounded by the optical centers C2, C3, C6, C7, The image Pa captured by the four selected cameras is normalized.

  For example, as shown in FIG. 15, when the optical centers C2, C3, C6, and C7 of the cameras 12, 13, 16, and 17 are used to take images from different viewpoints with the optical axis aligned to the M point to be imaged, The generated images Pa2, Pa3, Pa6, and Pa7 are parallel to the imaging surfaces of the cameras 12, 13, 16, and 17, respectively. Here, the directions of the straight lines connecting the optical centers C2, C3, C6, C7 and the M point are the normal directions k2, k3, k6, k7 of the images Pa2, Pa3, Pa6, Pa7 captured by the cameras. Although they match, they point in different directions.

  Therefore, arbitrary three images (for example, images Pa2, Pa3, Pa7) are selected from these images Pa2, Pa3, Pa6, Pa7. Then, by performing geometric normalization so that the normal directions k2, k3, k7 of these images Pa2, Pa3, Pa7 are in the same direction ka ′, the normalized images Pm2, Pm3 whose image planes are parallel to each other are obtained. , Pm7 is created. Also, the selected three images (images Pa2, Pa6, Pa7) are selected for the remaining areas. Then, by performing geometric normalization so that the normal directions k2, k6, k7 of these images Pa2, Pa6, Pa7 are in the same direction kb ′, normalized images Pm2, Pm6 whose image planes are parallel to each other , Pm7 is created. Incidentally, the geometric normalization method here is executed by obtaining a projection matrix P in the same manner as described above, and performing projection transformation based thereon.

  Here, when the normal direction ka ′ of the normalized images Pm2, Pm3, and Pm7 is different from the normal direction kb ′ of the normalized images Pm2, Pm6, and Pm7, the normalization processing unit 24 uses the same direction as each other. It corrects so that it becomes (it becomes parallel). This correction process is also performed by obtaining the projection matrix P and performing projective transformation based thereon. As a result, the normal directions k2 ', k3', k6 ', and k7' of these four normalized images Pm2, Pm3, Pm6, and Pm7 can all be parallelized. Incidentally, the parallelization is more easily realized by adjusting the normal direction ka ′ obtained for the normalized images Pm2, Pm3, and Pm7 according to the normal direction k6 of the remaining image Pa6. Also good.

  The images normalized in this way are associated with each other in the subsequent matching unit 29 as described above. In such a case, images captured by cameras other than the three selected cameras are used. May be referred to.

  For example, when the normalized images Pm1, Pm2, and Pm3 are associated with each other, the depth information is acquired by the cameras 11, 12, and 13 corresponding thereto. Then, a parallax map is created based on the acquired depth information. In such a case, the depth information acquired by the remaining cameras 14 to 18 around each of the cameras 11, 12, and 13 can be referred to using a so-called multi-baseline method. Thereby, the association between the images can be executed by referring to the parallax maps obtained from more cameras, and the image quality of the generated virtual viewpoint image Ima can be improved.

  Further, by repeating the above process while finely adjusting the relative position information α, β, γ, the number of virtual cameras corresponding to the projections 62 of the X rows and Y columns formed over the entire surface on the display panel 61 Can be virtually installed. Then, the virtual viewpoint image Im obtained from each virtual camera through the above-described procedure is sequentially transmitted to the imaging display device 2 on the counterpart side via the network 50. For example, the virtual viewpoint image Ima generated in the imaging display device 2a is transmitted to the partner imaging display device 2b via the network 50, and is transmitted to the display signal processing unit 8b via the transmission / reception unit 7b. The display signal processing unit 8b analyzes the luminance level of these virtual viewpoint images Ima for each pixel position. At this time, in the display signal processing unit 8b, the incident angles (θ °, φ °) and the pixel positions in the virtual viewpoint image Ima are recorded in association with each other, so that these incident angles (θ °, φ) are recorded. °) Brightness level can be identified every time.

  The display signal processing unit 8b emits light corresponding to the identified luminance level from the display element 63 on the protrusion 62 to angles (180 ° −θ, 180 ° −φ °) corresponding to the incident angles (θ °, φ °). ). By making the address of the display element 63 formed on the protrusion 62 correspond to the emission angle (180 ° −θ, 180 ° −φ °), the fine emission corresponding to the incident angle (θ °, φ °) is made. Light can be emitted at an angular pitch, and the above-described spatial connection effect can be obtained.

  That is, in the image transmission / reception system 1 to which the present invention is applied, in order to generate a display surface on the display 5 having a spatial connection effect similar to that of the window glass, an imaging function to be provided on the incident surface is provided around the displays 5a and 5b. The arranged imaging camera group 10a (10b) can be used. That is, by these imaging camera groups 10a (10b), a virtual camera is set over the entire surface on the display 6, and light incident at all points on the incident surface via virtual viewpoint images obtained from each virtual camera. The luminance level can be identified for each incident angle (θ °, φ °). Then, light of the identified brightness level is emitted at an angle (180 ° −θ, 180 ° −φ °) from the protrusion 62 corresponding to the virtual camera formed on the display 6 on the other side, A spatial connection effect can be obtained. In particular, in the image transmission / reception system 1 to which the present invention is applied, the imaging device can be omitted as compared with the conventional technique, so that the apparatus configuration can be simplified and the manufacture can be expected to be simplified.

  Further, in the image transmission / reception system 1 to which the present invention is applied, the imaging camera group 10a (10b) disposed around the displays 5a and 5b can be assigned the imaging function to be given to the incident surface. As a result, light leakage from the display element 63 to the imaging element does not occur, and the above-described space connection effect can be obtained with high efficiency.

  The present invention is not limited to the embodiment described above. For example, the left eye position and the right eye position of the user are estimated based on the coordinates of the user's head position, and the relative position information α, β, γ is adjusted as relative position information, and the left eye position and the right eye position are adjusted. A virtual viewpoint image may be generated for each eye. By displaying these left-eye and right-eye virtual viewpoint images via a stereoscopic display or the like, a three-dimensional image can be provided to the user.

  Further, the present invention is not limited to the case of obtaining a spatial connection effect via the network 50 such as the Internet network. Even if the imaging display devices 2 are connected to each other without the network 50, the same effect can be obtained. Of course, it can be obtained.

It is a figure which shows the outline of the image transmission / reception system to which this invention is applied. It is a figure for demonstrating about the arrangement | sequence of the camera which comprises an imaging camera group. It is a figure for demonstrating the structure of a display. It is a figure for demonstrating about the viewing angle (theta) d of a protrusion part, and the imaging angle (theta) c of a camera. It is a figure for demonstrating per structure of an imaging signal process part. It is a figure for demonstrating per camera calibration method. It is a figure for demonstrating per-image matching in a matching part. It is a figure for demonstrating per space connection effect. It is a figure for demonstrating about the control method for obtaining the space connection effect. It is a figure for demonstrating the case where an optical axis is match | combined to M point of imaging | photography object from a mutually different viewpoint by the optical center of each camera. It is a figure for demonstrating about the case where virtual plane (pi) is set. It is a figure for demonstrating per image interpolation between virtual viewpoint images. It is a figure shown about the example of the produced | generated virtual viewpoint image. It is a figure for demonstrating about the case where a virtual camera is virtually installed in the arbitrary positions on the display screen in a display. It is a figure for demonstrating about the case where the image imaged with the four selected cameras is normalized. It is a figure for demonstrating about the case where the thing in an other side is visually recognized through a window glass. It is a figure which shows the structure which arrange | positions two imaging display panels back to back. It is a figure which shows the detail of the display element group formed in the hemispherical protrusion surface. It is a figure which shows the example which arranges an element assembly on an imaging display panel.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Image transmission / reception system, 2 Each imaging display apparatus, 5 Display, 6 Imaging signal processing part, 7 Transmission / reception part, 8 Display signal processing part, 10 Imaging camera group, 11-18 camera

Claims (4)

Projections having a plurality of display elements formed on a hemispherical projection surface are two-dimensionally arranged, and an address assigned to the display element formed on each projection is selected to emit light corresponding to the luminance component at the pixel position. Display means for displaying an image by exiting in a specified exit direction;
Imaging means for imaging a subject to be photographed from different viewpoints by at least four cameras disposed around the display means;
Information generating means for generating relative position information indicating a relative positional relationship of the optical center with respect to each of the cameras in a virtually installed virtual camera;
Image correcting means for aligning at least one normal direction of each image captured by each camera with one direction;
Matching means for associating each image position with each other between the images corrected by the image correction means, in association with the photographing object;
An image for obtaining a pixel position and its luminance component constituting a virtual viewpoint image to be generated by the virtual camera according to the generated relative position information from the pixel position and its luminance component associated with each other by the matching means. Generating means;
A transmission / reception means for transmitting a virtual viewpoint image composed of the pixel position obtained by the image generation means and its luminance component to another electronic device, or receiving a virtual viewpoint image generated by the other electronic device;
In accordance with each pixel position constituting the virtual viewpoint image received by the transmission / reception means, an address of the display element is selected for each of the two-dimensionally arranged protrusions on the display means, and the output corresponding to the luminance component at the pixel position is selected. Display processing means for controlling the display means so as to emit light by each display element,
The image correction means selects two sets of three cameras from the four cameras based on the relative position information generated by the information generation means, and three cameras are selected for each selected set. A projection transformation matrix for projecting the normal direction of each captured image onto a virtually set virtual plane is obtained, and the normal direction of each image is determined based on the obtained projection transformation matrix. When the normal direction for each set differs according to the direction , the method of the image captured by one camera not selected in common with each other and projected based on the projective transformation matrix An imaging display device that corrects each normal direction of an image captured by another three cameras and projected based on the projection transformation matrix so as to be in the same direction as the line direction . Imaging means for imaging a subject to be photographed from different viewpoints by at least four cameras;
Projections having a plurality of display elements formed on a hemispherical projection surface are two-dimensionally arranged, and an address assigned to the display element formed on each projection is selected to emit light corresponding to the luminance component at the pixel position. Display means for displaying an image by exiting in a specified exit direction;
Information generating means for generating relative position information indicating a relative positional relationship of the optical center with respect to each of the cameras in a virtually installed virtual camera;
Image correcting means for aligning at least one normal direction of each image captured by each camera with one direction;
Matching means for associating each image position with each other between the images corrected by the image correction means, in association with the photographing object;
An image for obtaining a pixel position and its luminance component constituting a virtual viewpoint image to be generated by the virtual camera according to the generated relative position information from the pixel position and its luminance component associated with each other by the matching means. Generating means;
In accordance with each pixel position constituting the virtual viewpoint image obtained by the image generation means, an address of the display element is selected for each projecting portion arranged two-dimensionally in the display means, and according to the luminance component of the pixel position. Display processing means for controlling the display means so that the emitted light is emitted by each display element,
The image correction means selects two sets of three cameras from the four cameras based on the relative position information generated by the information generation means, and three cameras are selected for each selected set. A projection transformation matrix for projecting the normal direction of each captured image onto a virtually set virtual plane is obtained, and the normal direction of each image is determined based on the obtained projection transformation matrix. When the normal direction for each set differs according to the direction , the method of the image captured by one camera not selected in common with each other and projected based on the projective transformation matrix An imaging display device that corrects each normal direction of an image captured by another three cameras and projected based on the projection transformation matrix so as to be in the same direction as the line direction . Projections having a plurality of display elements formed on a hemispherical projection surface are two-dimensionally arranged, and an address assigned to the display element formed on each projection is selected to emit light corresponding to the luminance component at the pixel position. An imaging step of imaging a subject to be photographed from different viewpoints by at least four cameras disposed around a display unit that displays an image by emitting in a designated emission direction;
An information generation step of generating relative position information indicating a relative positional relationship of the optical center with respect to each of the cameras in a virtually installed virtual camera;
An image correction step for correcting by aligning the normal direction of each image captured by each camera at least in one direction;
A matching step for associating each image position with each other between the images corrected by the image correction step, in association with the shooting target;
According to the relative position information generated by the information generation step, the pixel position and the luminance component constituting the virtual viewpoint image to be generated by the virtual camera are determined from the pixel position and the luminance component associated with each other by the matching step. A desired image generation step;
A transmission step of transmitting a virtual viewpoint image constituted by the pixel position obtained by the image generation step and its luminance component via a network;
A receiving step of receiving a virtual viewpoint image generated in another electronic device via the network;
In accordance with each pixel position constituting the virtual viewpoint image received in the reception step, an address of the display element is selected for each protrusion two-dimensionally arranged in the display unit, and emitted light corresponding to the luminance component of the pixel position A display processing step for controlling the display unit so that each display element emits
In the image correction step, two sets of three cameras are selected from the four cameras based on the relative position information generated in the information generation step, and three cameras are selected for each selected group. A projection transformation matrix for projecting the normal direction of each captured image onto a virtually set virtual plane is obtained, and the normal direction of each image is determined based on the obtained projection transformation matrix. When the normal direction for each set differs according to the direction , the method of the image captured by one camera not selected in common with each other and projected based on the projective transformation matrix An imaging display method for correcting each normal direction of an image captured by another three cameras and projected based on the projection transformation matrix so as to be in the same direction as the line direction . In an image transmission / reception system that bi-directionally transmits / receives an image captured by each imaging display device via a network,
The imaging display device
Projections having a plurality of display elements formed on a hemispherical projection surface are two-dimensionally arranged, and an address assigned to the display element formed on each projection is selected to emit light corresponding to the luminance component at the pixel position. Display means for displaying an image by exiting in a specified exit direction;
Imaging means for imaging a subject to be photographed from different viewpoints by at least four cameras disposed around the display means;
Information generating means for generating relative position information indicating a relative positional relationship of the optical center with respect to each of the cameras in a virtually installed virtual camera;
Image correcting means for aligning at least one normal direction of each image captured by each camera with one direction;
Matching means for associating each image position with each other between the images corrected by the image correction means, in association with the photographing object;
An image for obtaining a pixel position and its luminance component constituting a virtual viewpoint image to be generated by the virtual camera according to the generated relative position information from the pixel position and its luminance component associated with each other by the matching means. Generating means;
A virtual viewpoint image composed of the pixel position obtained by the image generation means and its luminance component is transmitted to the other imaging display device via the network, or a virtual viewpoint generated by the other imaging display device A transmission / reception means for receiving an image via the network;
In accordance with each pixel position constituting the virtual viewpoint image received by the transmission / reception means, an address of the display element is selected for each of the two-dimensionally arranged protrusions on the display means, and the output corresponding to the luminance component at the pixel position is selected. Display processing means for controlling the display means so as to emit light by each display element,
The image correction means selects two sets of three cameras from the four cameras based on the relative position information generated by the information generation means, and three cameras are selected for each selected set. A projection transformation matrix for projecting the normal direction of each captured image onto a virtually set virtual plane is obtained, and the normal direction of each image is determined based on the obtained projection transformation matrix. When the normal direction for each set differs according to the direction , the method of the image captured by one camera not selected in common with each other and projected based on the projective transformation matrix An image transmission / reception system that corrects each normal direction of an image captured by another three cameras and projected on the basis of the projection transformation matrix so as to be in the same direction as the line direction .

Images