JP2005078313A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a sensation as if an opposite party seems to be really in front of a user. <P>SOLUTION: A detection part 3 acquires a three-dimensional information C<SB>a</SB>of a real space 41 in front of a display screen of a display part 8 where an image is displayed, and provides it to an image creation part 10. A reception part 11 obtains the three-dimensional information C<SB>b</SB>of the real space 42 and the image information D<SB>b</SB>acquired by photographing the real space 42 from a transmission part 27 comprising an information processing apparatus 22, and provides them to the image creation part 10. The image creation part 10 creates an image reflecting influence on the real space 42 by the real space 41 based on the three-dimensional information C<SB>a</SB>provided from the detection part 3, and the three-dimensional information C<SB>b</SB>and the image information D<SB>b</SB>acquired by photographing the real space 42 provided from the reception part 11 if it is assumed that the real space 42 is behind the display screen of the display part 8. For example, it is applied to a video conference system or the like performing communication with transmission of the images. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly, to an image processing apparatus, an image processing method, and a program that can display a more realistic image, for example.

  For example, in a conventional image dialogue apparatus typified by a video conference system or the like, an image of a transmission-side user imaged by a transmission-side imaging apparatus is transmitted to a communication partner via a communication line. Bidirectional communication is performed in which an image of a certain user is received via a communication line and an image of a user who is the received communication partner is displayed on a display device. According to such an image interaction apparatus, users can talk (converse) while looking at each other, so that even when the other party is far away, they can enjoy a sense of realism that they are close to themselves.

  As a means for providing a sense of presence or a feeling of sharedness that makes a communication partner feel close, for example, a video conference terminal device that projects an image of a communication partner in a life size using a projector or the like has been proposed. (For example, refer to Patent Document 1).

  There has also been proposed a means for providing a sense of realism or shared feeling by matching the line of sight with the communication partner. As such means, for example, a videophone device (for example, refer to Patent Document 2) in which the direction of the camera and the display screen are matched using a half mirror, or a light transmission state and a light confusion state are controlled. An image display / imaging device that displays and captures images in time series using a screen and a projector that can be displayed (see, for example, Patent Document 3), and a hologram screen and a projector that can display and image simultaneously. There is a display device with an imaging function (see, for example, Patent Document 4).

Furthermore, a system has been proposed that realizes eye-gaze matching even if the person moves by measuring the person viewing the image and moving the camera of the communication partner in accordance with the measured position information of the person ( For example, see Patent Document 5).
JP 2000-244484 A Japanese Patent Laid-Open No. 61-65683 JP-A-4-11485 JP-A-9-168141 JP 2000-83228 A

  As described above, it is possible to improve a sense of reality when communicating through a communication line by projecting the communication partner in life size or matching the line of sight with the communication partner. However, for example, there is a high demand for enjoying a higher sense of presence, such as a feeling that the communication partner is actually in front of you.

  The present invention has been made in view of such a situation, and can provide a higher sense of presence such that a communication partner is actually in front of you.

  An image processing apparatus according to the present invention includes a first acquisition unit that acquires first three-dimensional information of a first real space in front of a display screen on which an image is displayed, and a first position that is different from the first real space. Second acquisition means for acquiring the second three-dimensional information of the second real space and the first image obtained by imaging the second real space, and the first tertiary acquired by the first acquisition means When it is assumed that the second real space is located behind the display screen based on the original information and the second three-dimensional information and the first image acquired by the second acquisition unit, the first Generating means for generating a second image reflecting the influence of the real space on the second real space.

  The image processing method of the present invention includes a first acquisition step of acquiring first three-dimensional information of a first real space in front of a display screen on which an image is displayed, and a first position at a position different from the first real space. A second acquisition step of acquiring the second three-dimensional information of the second real space and a first image obtained by imaging the second real space, and the first tertiary acquired by the first acquisition step When it is assumed that the second real space is located behind the display screen based on the original information and the second three-dimensional information and the first image acquired by the second acquisition unit, the first Generating a second image reflecting the influence of the real space on the second real space.

  The program of the present invention includes the first three-dimensional information of the first real space in front of the display screen on which the image is displayed, and the second three-dimensional of the second real space at a position different from the first real space. When it is assumed that the second real space is located behind the display screen based on the information and the first image obtained by imaging the second real space, the first real space is the second It includes a generation step of generating a second image reflecting the influence on the real space.

  According to the image processing device, the image processing method, and the program of the present invention, the first three-dimensional information of the first real space in front of the display screen on which the image is displayed, and the position different from the first real space. When it is assumed that the second real space is located behind the display screen based on the second three-dimensional information of the second real space and the first image obtained by imaging the second real space In addition, a second image reflecting the influence of the first real space on the second real space is generated.

  According to the present invention, it is possible to provide a more realistic image.

  Embodiments of the present invention will be described below. The correspondence between the invention described in this specification and the embodiments is illustrated as follows. Although there is an embodiment which is described in the specification but is not described here as corresponding to the invention, it does not mean that the embodiment corresponds to the invention. It doesn't mean not. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. Absent.

  Further, this description does not mean that all the inventions described in the specification are claimed. In other words, this description is an invention described in the specification and is not claimed in this application, that is, an invention that will be filed in the future, filed by amendment, filed by amendment, or added The existence of is not denied.

  The image processing apparatus according to claim 1 is a first acquisition unit that acquires first three-dimensional information of a first real space in front of a display screen on which an image is displayed (for example, the detection unit 3 in FIG. 1). Or the receiving unit 31) in FIG. 10 and the second 3D information of the second real space at a position different from the first real space and the first image obtained by imaging the second real space. Second acquisition means (for example, the reception unit 11 in FIG. 1 or the detection unit 23 and the imaging camera 26 in FIG. 10), the first three-dimensional information acquired by the first acquisition means, and the second When it is assumed that the second real space is located behind the display screen based on the second three-dimensional information and the first image acquired by the acquisition means, the first real space is the second Generation means for generating a second image reflecting the influence on the real space (for example, the image generation unit in FIG. 1) 0, or characterized in that it comprises a drawing image generator 30 of 10) and.

  The image processing apparatus according to claim 2 is a display control unit (for example, the display control unit of FIG. 1) that displays the second image generated by the generation unit (for example, the image generation unit 10 of FIG. 1) on the display screen. 9) is further provided.

  The image processing apparatus according to claim 5, wherein the predetermined light source (for example, the light source 13 in FIG. 1) is an actual light source that exists in the first real space, and detection means (for example, an actual light source) The detector 5) of FIG. 1 is further provided.

  In the image processing apparatus according to claim 6, the first acquisition unit (for example, the detection unit 3 in FIG. 1) acquires the first three-dimensional information by detecting the first three-dimensional information from the first real space. The two acquisition means (for example, the receiving unit 11 in FIG. 1) acquire the second three-dimensional information and the first image by receiving them from another apparatus (for example, the information processing apparatus 22 in FIG. 1). It is characterized by that.

  In the image processing apparatus according to claim 7, the first acquisition unit (for example, the reception unit 31 in FIG. 10) uses the first three-dimensional information as another apparatus (for example, the information processing apparatus 2 in FIG. 10). The second acquisition means (for example, the detection unit 23 and the detector 25 in FIG. 10) acquires the second three-dimensional information by detecting it from the second real space. At the same time, the first image is obtained by imaging the second real space.

  The image processing method according to claim 8 is a first acquisition step of acquiring first three-dimensional information of the first real space in front of the display screen on which the image is displayed (for example, step S1 in FIG. 2). And a second acquisition step of acquiring the second three-dimensional information of the second real space at a position different from the first real space and the first image obtained by imaging the second real space (for example, Display screen based on step S2) in FIG. 2, the first three-dimensional information acquired by the first acquisition step, the second three-dimensional information acquired by the second acquisition means, and the first image. Generation step for generating a second image reflecting the influence of the first real space on the second real space when it is assumed that the second real space is located behind (for example, FIG. 2). Step S4).

  The program according to claim 9 is the first three-dimensional information of the first real space in front of the display screen on which the image is displayed, and the second real space second at a position different from the first real space. When it is assumed that the second real space is located behind the display screen based on the three-dimensional information and the first image obtained by imaging the second real space, the first real space is A generation step (for example, step S4 in FIG. 2) for generating a second image reflecting the influence on the second real space is included.

  Embodiments of the present invention will be described below. FIG. 1 is a block diagram showing a configuration example of a first embodiment of a communication system to which the present invention is applied.

  The network 40 connects the information processing apparatus 2 and the information processing apparatus 22, and bidirectionally transmits image information, audio information, position information such as three-dimensional information described later, and other data. As the network 40, for example, a local area network, the Internet, or other wired or wireless transmission media can be adopted.

  The user 1 is a user who operates the information processing apparatus 2, and, for example, communicates with the user 21 who operates the information processing apparatus 22 that exists far away via the network 40.

  The information processing apparatus 2 includes a detection unit 3, an imaging camera 6, a transmission unit 7, a display unit 8, a display control unit 9, an image generation unit 10, a reception unit 11, an information storage unit 12, and a voice processing unit (not shown). It is composed of

The detection unit 3 includes a camera 4-1, a camera 4-2, and a detector 5. The camera 4-1 and the camera 4-2 image the real space 41 in front of the display screen of the display unit 8 and supply an image signal of the real space 41 obtained by the imaging to the detector 5. Here, the user 1 and the light source 13 exist in the real space 41. The detector 5 performs predetermined processing by, for example, a stereo method (details will be described later) on the image signals supplied from the cameras 4-1 and 4-2, and the contour and size of the user 1. It is acquired by detecting the three-dimensional shape such as the three-dimensional information C _a which is information representing the position. Further, the detector 5 supplies the acquired three-dimensional information C _a to the transmission unit 7 and the image generation unit 10. Incidentally, the detector 5, X _a axis defined camera 4-1 as a reference, Y _a-axis, and the coordinate system formed by Z _a-axis (hereinafter, appropriately referred to as an imaging coordinate system) three-dimensional represented by Information C _a is detected.

As with the camera 4-1 and the camera 4-2, the imaging camera 6 acquires an image signal including the user 1 and the background surrounding the user 1 by imaging the real space 41. Further, the imaging camera 6, an image signal obtained by imaging the real space 41, and supplies this to the transmission unit 7 as the image information D _a. The imaging camera 6 is attached close to the upper end portion of the display unit 8. Furthermore, as the imaging camera 6, for example, a television camera can be employed.

The transmission unit 7 includes a network interface circuit, a transmission encoder, etc. (not shown). The transmission unit 7 is supplied from the three-dimensional information C _a of the user 1 supplied from the detector 5 constituting the detection unit 3, the image information D _a supplied from the imaging camera 6, and further supplied from an audio processing unit (not shown). The audio information thus transmitted is transmitted to the information processing apparatus 22 (the receiving unit 31 constituting the information processing apparatus 22) via the network 40. Note that the transmission unit 7 converts, for example, an image signal, an audio signal, a three-dimensional information signal, and the like into a transmission form corresponding to the network 40 and sends the converted signal to the network 40.

  The display unit 8 displays the image supplied from the display control unit 9 on the display screen. The display unit 8 can be constituted by, for example, a television monitor constituting a television receiver, that is, a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), a home theater system screen, or the like. .

  The display control unit 9 performs predetermined processing on the image signal supplied from the image generation unit 10 and supplies the image signal to the display unit 8 for display.

The image generation unit 10 includes the three-dimensional information C _a supplied from the detector 5 constituting the detection unit 3, the three-dimensional information C _b of the real space 42 in which the user 21 exists supplied from the reception unit 11, and the usage. The real space 42 is positioned behind the display screen of the display unit 8 by using image information D _b such as a background surrounding the user 21 and the user 21, and light source information supplied from the information storage unit 12 as necessary. The real space 41 has an influence on the real space 42 when it is assumed that the real space 42 in which the image displayed on the display screen is obtained exists behind the display screen. , That is, for example, an image signal in which the shadow of the user 1 is reflected in the image information D _b is generated and supplied to the display control unit 9.

The receiving unit 11 includes, for example, a network interface circuit and a decoder. The receiving unit 11 receives image information D _b as an image signal, audio signal, and three-dimensional information C _b as a position signal transmitted from the information processing apparatus 22 (the transmitting unit 27 constituting the information processing device 22) via the network 40. Are restored from the transmission form corresponding to the network 40 to the original form (format), and supplied to the image generation unit 10 and further to an audio processing unit (not shown).

  The information storage unit 12 stores light source information indicating the position of the light source 13 existing in the real space 41, and the information storage unit 12 supplies the stored light source information to the image generation unit 10. When there are a plurality of light sources in the real space 41, the information storage unit 12 can store light source information for some or all of the plurality of light sources. In addition, a virtual light source that is a virtual light source may be set in advance in the real space 41, and light source information of the virtual light source may be stored in the information storage unit 12.

  Here, in the information processing apparatus 2, a sound processing unit (not shown) includes, for example, an amplifier, a speaker, a microphone, and the like. The voice processing unit performs necessary processing on the voice uttered by the user, and causes the transmission unit 7 to transmit the information to the information processing apparatus 22 via the network 40. Further, the voice processing unit performs necessary processing on the voice of the user 21 received by the receiving unit 11 from the information processing apparatus 22 via the network 40, and outputs the processing from the speaker. If necessary, the sound processing unit can be provided with a processing unit that performs a process of canceling a phenomenon called echo that occurs when the microphone picks up the sound of the speaker.

  Note that the display unit 8 and the imaging camera 6 are preferably arranged as close to each other as possible in order to minimize the deviation between the user's line of sight and the user's line of sight projected on the display unit 8. Moreover, about the display part 8 and the imaging camera 6, by using the half mirror and matching the direction of the lens surface of the lens of the imaging camera 6 and the direction of the display screen of the display part 8, The deviation from the line of sight of the user 21 displayed on the display unit 8 can be reduced (Patent Document 2). Further, the display unit 8 and the imaging camera 6 employ a configuration in which a screen and a projector capable of controlling the light transmission state and the light confusion state are used, and an image can be displayed and imaged in time series. (Patent Document 3). Moreover, as the display part 8 and the imaging camera 6, the structure which can perform a display and an imaging simultaneously using a hologram screen and a projector is employable (patent document 4).

  The light source 13 exists in the real space 41 and emits light that illuminates the inside of the real space 41.

  In FIG. 1, the user 21, the information processing device 22, the light source 33, and the real space 42 on the communication partner side of the user 1 correspond to the user 1, the information processing device 2, the light source 13, and the real space 41, respectively. . The information processing device 22 includes a detection unit 3, a camera 4-1, a camera 4-2, a detection unit 5, an imaging camera 6, a transmission unit 7, a display unit 8, a display control unit 9, and an image generator that constitute the information processing device 2. Unit 10, receiver 11, and information storage unit 12, which are the same as detection unit 23, camera 24-1, camera 24-2, detection unit 25, imaging camera 26, transmission unit 27, display unit 28, and display control unit 29, respectively. , An image generation unit 30, a reception unit 31, and an information storage unit 32.

Therefore, like the detection unit 3, the detection unit 23 acquires the three-dimensional information of the real space 42 in front of the display screen of the display unit 28. The three-dimensional information of the real space 42 is converted into the three-dimensional information C _b. It describes. The imaging camera 26, like the imaging camera 6 images the real space 42, but outputs the image information, the image information, referred to as image information D _b.

  FIG. 2 is a flowchart for explaining the operation of the information processing apparatus 2 of FIG. Note that the information processing apparatus 22 performs the same processing as the information processing apparatus 2.

  The process of FIG. 2 is started when a communication link is established between the information processing apparatus 2 and the information processing apparatus 22.

The imaging camera 6 and each of the camera 4-1 and the camera 4-2 constituting the detection unit 3 start imaging of the real space 41, and obtain images of the user 1 and the background surrounding the user 1 obtained by the imaging. Is output. Images output from the camera 4-1 and the camera 4-2 are supplied to the detector 5. Image capturing camera 6 outputs is supplied to the transmitter 7 as image information D _a, via the network 40, is transmitted to the information processing apparatus 22.

In step S < _b > 1, the detector 5 generates the three-dimensional information Ca of the real space 41 from the images supplied from the camera 4-1 and the camera 4-2, and sends them to the transmission unit 7 and the image generation unit 10. Then, the process proceeds to step S2. Incidentally, the three-dimensional information C _a, which is supplied to the transmitting section 7 from the detecting unit 5, via the network 40, is transmitted to the information processing apparatus 22.

In step S2, the reception unit 11 receives the three-dimensional information C _b and the image information D _b from the transmission unit 27 constituting the information processing apparatus 22 transmitted via the network 40, and supplies to the image generator 10 Then, the process proceeds to step S3. That is, the information processing apparatus 22 performs the same processing as the information processing apparatus 2, and the three-dimensional information C _b and the image information D _b are transmitted from the information processing apparatus 22 to the information processing apparatus 2 via the network 40. since coming, the receiving unit 11 in step S2, receives the three-dimensional information C _b and the image information D _b.

  In step S3, the information storage unit 12 supplies the stored light source information of the light source 13 to the image generation unit 10, and proceeds to step S4.

In step S4, the image generation unit 10 receives the three-dimensional information C _a supplied from the detector 5 in step S1, the three-dimensional information C _b supplied from the reception unit 11 in step S2, and the image information D _b , and the step Based on the light source information supplied from the information storage unit 12 in S3, the effect of the real space 41 on the real space 42 is reflected when it is assumed that the real space 42 is located behind the display screen of the display unit 8. A shadow generation process for generating an image signal reflecting the shadow of the user 1 generated by the light source 13, for example, the shadow of the user 1 is performed. Details of the shadow generation process in step S4 will be described later.

  In step S5, the image generation unit 10 supplies the image signal generated in step S4 to the display control unit 9, and the process proceeds to step S6.

  In step S6, the display control unit 9 supplies the image signal supplied from the image generation unit 10 to the display unit 8, displays the image signal on the display screen, and proceeds to step S7.

  In step S <b> 7, the transmission unit 7 or the reception unit 11 determines whether the communication link with the information processing apparatus 22 has been disconnected. If it is determined in step S7 that the communication link is not disconnected, the process returns to step S1 and the same processing is repeated thereafter. If it is determined in step S7 that the communication link has been disconnected, the process ends.

  In the process of FIG. 2, the processes of steps S1 to S3 may be processed in time series or may be processed in parallel.

  Next, a stereo method used for detection of three-dimensional information by the detector 5 in FIG. 1 will be described. The stereo method associates pixels between a plurality of images obtained by photographing the same object with two or more directions (different line-of-sight directions), so that parallax information between corresponding pixels or a target from the camera The distance to the object and the shape of the object are obtained.

  That is, for example, the camera 4-1 and the camera 4-2 in FIG. 1 are referred to as a reference camera 4-1 and a detection camera 4-2, respectively, and the reference camera 4-1 and the detection camera 4-2 are used as objects. , A reference camera image including a projection image of the object is obtained from the reference camera 4-1, and a detection camera image including a projection image of the object is also obtained from the detection camera 4-2. Now, as shown in FIG. 3, if a certain point P on the object is displayed in both the reference camera image and the detected camera image, the position on the reference camera image where the point P is displayed, The parallax between the reference camera 4-1 and the detection camera 4-2 can be obtained from the position on the detection camera image, that is, the corresponding point, and the three-dimensional of the point P can be obtained using the principle of triangulation. A position in space (three-dimensional position) can be obtained.

  Therefore, in the stereo method, it is first necessary to detect corresponding points. As the detection method, for example, there is an area-based matching method using an epipolar line.

That is, as shown in FIG. 3, in the reference camera 4-1, the point P on the object is on a straight line L connecting the point P and the optical center (lens center) O _{1 of the} reference camera 4-1. is projected intersection n _a of the imaging surface S ₁ of the base camera 4-1.

Further, in the detection camera 4-2, the point P of the object is an imaging surface of the detection camera 4-2 on a straight line connecting the point P and the optical center (lens center) O ₂ of the detection camera 4-2. It is projected to the intersection n _b and S _2.

In this case, the straight line L is an intersection line L ₂ between the plane passing through the three points of the optical centers O ₁ and O ₂ and n _b (or point P) and the imaging surface S _{2 on} which the detection camera image is formed. projected onto the imaging surface S _2. Point P is a point on the straight line L, thus, the imaging surface S _2, the n _b point obtained through projection of the point P, and lies on the straight line L ₂ obtained by projecting the straight line L, this straight line L _2, epipolar Called the line. That is, there is a possibility that the corresponding point n _b of the point n _a exists on the epipolar line L _2. Therefore, the search for the corresponding point n _b may be performed on the epipolar line L ₂ .

Here, when the epipolar line, for example, can be considered for each pixel constituting the reference camera image formed on the imaging surface S _1, a positional relationship between the reference camera 4-1 and the detection camera 4-2 Known For example, an epipolar line existing for each pixel can be obtained.

Detection of corresponding points n _b from a point on the epipolar line L ₂ is, for example, can be carried out by the following area based matching.

That is, in area-based matching, as shown on the left of FIG. 4, a reference block that is a small rectangular block, for example, having a point n _a on the reference camera image as _a center (for example, an intersection of diagonal lines) As shown in the right side of FIG. 4, a detection block that is a small block having the same size as the reference block centered on a certain point on the epipolar line L ₂ projected on the detection camera image is extracted. , Extracted from the detected camera image.

Here, on the right side of FIG. 4, six points n _{b1 to} n _b6 are provided on the epipolar line L ₂ as points serving as the center of the detection block. The six points n _{b1 to} n _b6 are points that divide the straight line L in the three-dimensional space shown in FIG. 3 at predetermined constant distances such as 1 m, that is, the distance from the reference camera 4-1 is 1 m, 2m, 3m, 4m, 5 m, respectively point 6 m, a point obtained by projecting the imaging surface S ₂ of the detection cameras 4-2, therefore, the distance from the base camera 4-1 1m, 2m, 3m, 4m, 5m , 6 m (estimated distance), respectively (in this case, it can be said that the distance resolution is 1 m).

In area-based matching, detection blocks centered on the points n _{b1 to} n _b6 provided on the epipolar line L ₂ are extracted from the detection camera image, and the correlation between each detection block and the reference block is Calculation is performed using a predetermined evaluation function. Then, the point n _a center correlation highest detection block and the reference block centered on the point n _a is obtained as the corresponding point of the point n _a.

  Here, as the evaluation function (evaluation value) f (x, y), for example, the following equation can be adopted.

However, in the formula (1), (x, y ) represents the coordinates of the center point n _a reference block, (x', y ') represent the coordinates of the center point of the detection block. Further, Y _A (x + i, y + j) represents the luminance of the pixel at the point (x + i, y + j) on the reference camera image, and Y _B (x ′ + i, y ′ + j) represents the point (x on the detected camera image). '+ I, y' + j) represents the luminance of the pixel. W represents a reference block and a detection block, and i, jεW represents a point (x + i, y + j) or a point (x ′ + i, y ′ + j) at a point in the reference block or detection block, respectively. Represents something.

  Note that the evaluation value f (x, y) represented by the expression (1) is a predetermined point (x, y) on the reference camera image and a point of interest (x ′, y) on the detected camera image. The point (x ′, y ′) on the detected camera image that minimizes the evaluation value f (x, y) becomes smaller as the correlation with the '′) increases. , Y).

That is, for example, the point n _b1 provided on the epipolar line L ₂ when a function having a smaller value as the correlation is higher is used as the evaluation function, for example, as shown in Expression (1). _Assume that an evaluation value (value of an evaluation function) as shown in FIG. 5 is obtained for each of _{nb6 to nb6} . Here, in FIG. 5, the epipolar line L ₂ corresponding to each distance number (with the abscissa indicating the distance number corresponding to the distance from the reference camera 4-1 previously assigned to each of the points n _{b1 to} n _b6. The evaluation values for the upper points n _{b1 to} n _b6 ) are shown.

If the evaluation curve made by the evaluation value as shown in FIG. 5 is obtained, a point on the epipolar line L ₂ the evaluation value corresponding to the smallest (high correlation) distance number 3, at point n _a It is detected as a corresponding point. In FIG. 5, interpolation is performed using a value near the minimum value among the evaluation values (indicated by ● in FIG. 5) obtained for each of the points corresponding to the distance numbers 1 to 6, and the evaluation value is more It is also possible to obtain a point that becomes smaller (a point corresponding to 3.3 m indicated by x in FIG. 5) and detect that point as a final corresponding point.

In FIG. 4, as described above, the points that divide the straight line L connecting the point on the object in the three-dimensional space and the optical center O ₁ of the reference camera 4-1 at predetermined equal distances are detected by the detection camera 4-4. Although point obtained through projection 2 of the imaging surface S ₂ is set, this setting can be carried out, for example, during calibration of the reference camera 4-1 and the detection camera 4-2 (calibration method is particularly Not limited). Then, such setting is performed for each epipolar line existing for each pixel constituting the imaging surface S ₁ of the reference camera 4-1, up to a point set on the epipolar line (hereinafter referred to as a set point as appropriate). If a distance number / distance table that associates the distance number corresponding to the distance (distance from the reference camera 4-1) and the distance from the reference camera 4-1 is created in advance, the set point that becomes the corresponding point is determined. By detecting and converting the distance number corresponding to the set point with reference to the distance number / distance table, the distance from the reference camera 4-1 (the estimated value of the distance to the point on the object) is immediately ). In other words, the distance can be obtained directly from the corresponding points.

On the other hand, for the points n _a on the reference camera image, by detecting the corresponding points n _b on the detection camera image can be obtained parallax (disparity information) between the two points n _a and n _b. Further, if the positional relationship between the reference camera 4-1 and the detection camera 4-2 is known, the parallax between the two points n _a and n _b, the principle of triangulation, can determine the distance to the object it can. The distance can be calculated from the parallax by performing a predetermined calculation, but the parallax / distance table for adding the parallax ζ and the distance is calculated for all the pixels on the reference camera image. If prepared in advance, the distance from the reference camera 4-1 can be immediately obtained by detecting the corresponding point, obtaining the parallax, and referring to the parallax / distance table. Note that the same parallax / distance table may be used for all pixels depending on the camera installation conditions (positional relationship). Furthermore, the image can be transformed so that the same parallax / distance table can be used for all pixels.

  Here, the parallax and the distance to the object have a one-to-one correspondence. Therefore, obtaining the parallax and obtaining the distance to the object are equivalent to each other.

  When the distance is directly obtained from the corresponding point, the set point on the epipolar line corresponding to the far point in the three-dimensional space has a smaller interval between the adjacent set points. Therefore, the accuracy of the obtained distance deteriorates at the set point where the interval between the adjacent set points is smaller than the distance between the pixels of the detected camera image. Further, when obtaining the distance from the parallax, the parallax does not change much at a close position in the three-dimensional space (a position close to the reference camera 4-1). Since the highest accuracy of parallax is basically in units of pixels, when obtaining a distance from parallax, the accuracy of the distance deteriorates when obtaining a distance to a close position in the three-dimensional space. However, in any case, the accuracy problem can be solved by setting the set point and detecting the corresponding point in sub-pixel units smaller than the pixel.

Further, the detection of the corresponding point, to use a reference block and a detection block comprised of a plurality of pixels such blocks is to reduce the effects of noise, the characteristics of the pattern of pixels around the pixel (point) n _a of the reference camera image When, by utilizing the correlation between the characteristic of the pattern of pixels around the detected corresponding points on the camera image (pixels) n _b, it is for the sake of certainty of detection of the corresponding point, in particular, the change For a small number of reference camera images and detection camera images, the certainty of detection of corresponding points can be improved by increasing the block size due to the correlation of the images.

  Further, in the above-described case, as an evaluation function for evaluating the correlation between the reference block and the detection block, the pixels constituting the reference block and the detection blocks corresponding to the respective pixels represented by Expression (1) The sum of the absolute values of the pixel value differences of the pixels constituting the pixel is used, but other evaluation functions include the sum of squares of the pixel value differences, normalized cross correlation, etc. Can be used.

  FIG. 6 is a block diagram illustrating a configuration example of the image generation unit 10 of FIG. The image generation unit 30 in FIG. 1 is configured in the same manner as the image generation unit 10.

  The image generation unit 10 includes a three-dimensional information expansion unit 61, a pixel selection unit 62, a pixel position determination unit 63, a shadow information generation unit 64, and a pixel value setting unit 65.

The three-dimensional information expansion unit 61 receives the three-dimensional information C _a supplied from the detector 5 constituting the detection unit 3 and the three-dimensional information C _{b of the} user 21 supplied from the receiving unit 11. The three-dimensional information development unit 61 uses the three-dimensional information C _a supplied from the detector 5 constituting the detection unit 3 and the three-dimensional information C _b of the user 21 supplied from the receiving unit 11, for example, as shown in FIG. The display screen of the display unit 8 is an XY plane, and is expanded into a coordinate system (hereinafter referred to as a display coordinate system as appropriate) having a direction perpendicular to the display screen as the Z axis, and expanded into the display coordinate system. The three-dimensional information C _a and the three-dimensional information C _b are supplied to the pixel position determination unit 63 and the shadow information generation unit 64, respectively. In other words, the three-dimensional information C _a supplied to the image generation unit 10 is represented by coordinates of an imaging coordinate system with the camera 4-1 as a reference, and the three-dimensional information C supplied to the image generation unit 10. _{Since b} is expressed in the coordinates of the imaging coordinate system with the camera 24-1 as a reference, the three-dimensional information expansion unit 61 displays the three-dimensional information C _a and the three-dimensional information C _b both as display coordinates. Convert to the system coordinates.

The pixel selection unit 62, the image information D _b of the user 21 from the receiving unit 11 is supplied. Pixel selection unit 62 selects a pixel of interest in each frame of the image information D _b of the supplied user 21 from the receiving unit 11 supplies information representing the target pixel to the pixel position determination unit 63.

The pixel position determination unit 63 corresponds to the target pixel based on the three-dimensional information C _b expanded in the display coordinate system supplied from the three-dimensional information expansion unit 61 and the target pixel supplied from the pixel selection unit 62. A predetermined position on the display coordinate system (hereinafter referred to as a corresponding position as appropriate) is obtained and supplied to the shadow information generation unit 64.

The shadow information generation unit 64 is based on the three-dimensional information C _a supplied from the three-dimensional information development unit 61, the corresponding position supplied from the pixel position determination unit 63, and the light source information supplied from the information storage unit 12. Assuming that the real space 42 is located behind the display screen 8, the influence of the real space 41 on the real space 42, that is, the light emitted from the light source 13 is behind the display screen of the display unit 8. The shadow of the user 1 that occurs in the real space 42 assumed to be located is estimated, and the shadow information that is the shadow information is supplied to the pixel value setting unit 65.

The pixel value setting unit 65, the image information D _b of the user 21 from the receiving unit 11, and shadow information from the shadow information generation unit 64 is supplied. Pixel value setting unit 65, the image information D _b of the user 21 supplied from the receiving unit 11, based on the shadow data supplied from the shadow information generation unit 64, an image that reflects the shadow information into image information D _b Is supplied to the display control unit 9.

  Here, when it is assumed that the real space 42 is located behind the display screen of the display unit 8, the real space 41 and the real space 42 assumed to be located behind the display screen are configured. The space to be used is hereinafter referred to as real / virtual space as appropriate.

  FIG. 7 is a schematic diagram showing a real / virtual space.

The upper left part of FIG. 7 is defined camera 4-1 as a reference, shows a X _a axis, Y _a-axis, and Z _a imaging coordinate system comprising at axis A and the real space 41. Similarly, the lower left part of FIG. 7 shows X _b axis defined camera 24-1 as a reference, Y _b axis, and the and the real space 42 Z _b imaging coordinate system comprising at axis B. Three-dimensional information C _a real space 41 obtained by the detector 3, for example, the coordinates or the like in the imaging coordinate system A of each point on the user 1 existing in the real space 41. Further, the three-dimensional information C _b of the real space 42 obtained by the detection unit 23 is, for example, coordinates in the imaging coordinate system B of each point on the user 21 existing in the real space 42.

  7 shows the real space 42 shown in the upper right of FIG. 7 projected on the image displayed on the display screen of the display unit 8 in the real space 41 shown in the upper left of FIG. In this state, the real / virtual space composed of the real space 42 and the real space 41 is shown assuming that the display screen of the display unit 8 is located behind the display screen.

  The real / virtual space is a space in which the real space 41 and the real space 42 are virtually connected with the display screen of the display unit 8 as a boundary surface. Therefore, if the real / virtual space is a real space, for example, the shadow of the user 1 generated by the light source 13 existing in the real space 41 faces the real space 41 with respect to the display screen of the display unit 8. It can be projected onto the real space 42 at the position.

Therefore, the image generation unit 10 generates an image in which the shadow of the user 1 is projected on the image information D _b of the user 21 based on the three-dimensional information C _a , the three-dimensional information C _b , and the light source information of the light source 13. To do. Thereby, the image of the user 21 onto which the shadow of the user 1 is projected is displayed on the display unit 8. Similarly, the image generation unit 30 constituting the information processing unit 22 uses the three-dimensional information C _a , the three-dimensional information C _b , and the light source information of the light source 33 as the user 21 to the image information D _a of the user 1. An image in which the shadow of is projected is generated. Thereby, the image of the user 1 onto which the shadow of the user 21 is projected is displayed on the display unit 28.

  Therefore, the user 1 displays an image similar to an image that would appear visually if the user 21 who is the communication partner is actually behind the display screen of the display unit 8. The user 21 can also observe a similar image on the display screen of the display unit 28. As a result, the user 1 and the user 21 can communicate with each other while using the display screens of the display unit 8 and the display unit 28 as windows and feeling as if they are talking through the windows. it can.

  Next, a method of generating an image by the image generation unit 10 will be described with reference to FIG. Note that the image generation unit 30 also generates an image in the same manner as in the image generation unit 10.

  8 includes a real space 41 and the real space 42 when it is assumed that the real space 42 is located behind the display screen of the display unit 8 as in the right diagram of FIG. A real / virtual space is schematically shown. That is, FIG. 8 shows a real / virtual space configured by virtually arranging the display screens of the display unit 8 and the display unit 28 back to back.

The points on the real / virtual space in FIG. 8 can be represented by a display coordinate system including the X axis, the Y axis, and the Z axis with reference to the display screen of the display unit 8 (or the display unit 28). Therefore, the image generating unit 10, the three-dimensional information C _a real space 41 represented by the image acquisition coordinate system A defined camera 4-1 as reference, the three-dimensional information C _a represented by the display coordinate system Perform coordinate transformation (development). Likewise, the image generating unit 10, the three-dimensional information C _b of the real space 42 represented by the image acquisition coordinate system B as defined camera 24-1 as a reference is also three-dimensional information C _b represented in the display coordinate system Convert coordinates to.

  Note that the positional relationship between the camera 4-1 and the display unit 8 (the positional relationship between the camera 24-1 and the display unit 28) can be obtained in advance, and the coordinate conversion described above is performed based on this positional relationship. It can be carried out. If the display screens of the display unit 8 and the display unit 28 are different in size, normalization is performed in advance.

  The image generation unit 10 further obtains the position of the light source 13 on the display coordinate system from the light source information stored in the information storage unit 12.

The selection, the image generation unit 10 sequentially the frames constituting the image information D _b of the supplied user 21 from the receiving unit 11, a frame of interest, further the pixels constituting the frame of interest, successively, as the pixel of interest To do.

Here, the i-th pixel in the frame of interest, for example, in raster scan order is represented as IB _i .

Now, assuming that the pixel IB _i is selected as the target pixel, the pixel generation unit 10 determines the position (point) PB _i in the real space 42 projected onto the target pixel IB _i based on the three-dimensional information C _b . Find the coordinates of the display coordinate system.

Here, the position (point) PB _i is the corresponding position corresponding to the target pixel IB _i described above.

Thereafter, the image generation unit 10 obtains a straight line L _i that connects the corresponding position PB _i and the position of the light source 13 obtained from the light source information. Further, the image generation unit 10 determines whether or not the straight line L _i intersects the user 1 in the real space 41 based on the three-dimensional information C _a . When the straight line L _i intersects the user 1 in the real space 41, the image generation unit 10 assumes that the pixel of interest IB _i is a pixel that is affected by the shadow of the user 1, and the pixel value of the pixel of interest IB _i To a value that has a shadow effect. On the other hand, when the straight line L _i does not intersect with the user 1 in the real space 41, the image generation unit 10 assumes that the pixel of interest IB _i is a pixel that is not affected by the shadow of the user 1 and sets the pixel value thereof. Leave as it is.

  FIG. 9 is a flowchart illustrating the process of step S4 in FIG. 2, that is, the shadow generation process performed by the image generation unit 10 (or the image generation unit 30) in FIG.

In step S < _b > 11, the three-dimensional information expansion unit 61 expands the three-dimensional information Ca of the real space 41 supplied from the detector 5 from the imaging coordinate system A to the display coordinate system and is supplied from the reception unit 11. that the three-dimensional information C _b of the real space 42, to expand to that of the display coordinate system from the image coordinate system B. Further, the three-dimensional information expansion unit 61 supplies the three-dimensional information C _a represented in the display coordinate system to the shadow information generation unit 64 and also the three-dimensional information C _b represented in the display coordinate system. The process proceeds to step S12 from step S11.

In step S < _b > 12, the pixel selection unit 62 initializes a variable i for counting the pixels constituting the frame of interest to 1 with the frame of the image information Db of the user 21 supplied from the reception unit 11 as the frame of interest. The process proceeds to step S13.

In step S13, the pixel selection unit 62 selects the i-th pixel in the raster scan order among the pixels constituting the target frame as the target pixel IB _i , and supplies the selected pixel to the pixel position determination unit 63. Proceed to

In step S14, the pixel position determination unit 63, based on the three-dimensional information C _b from the three-dimensional information expanding section 61, corresponding to the pixel of interest IB _i were chosen supplied by the pixel selection unit 62, the real / virtual space The corresponding position PB _i (FIG. 8) on the display coordinate system is obtained and supplied to the shadow information generation unit 64, and the process proceeds to step S15.

In step S <b> 15, the shadow information generation unit 64 obtains the light source position on the display coordinate system of the light source 13 existing in the real space 41 based on the light source information supplied from the information storage unit 12. Further, the shadow information generation unit 64 obtains a straight line L _i (FIG. 8) connecting the light source position and the corresponding position PB _i from the pixel position determination unit 63, and proceeds from step S15 to step S16. In step S < _b > 16, the shadow information generation unit 64 determines that the line L _i is based on the three-dimensional information C _a from the three-dimensional information development unit 61 and the user 1 and other real objects existing in the real space 41. Determine if they intersect.

In step S <b> 16, when the shadow information generation unit 64 determines that the straight line L _i intersects the user 1 or other real object, the shadow information generation unit 64 displays the shadow information indicating that the target pixel IB _i is affected by the shadow, as the pixel value setting unit 65. To proceed to step S17.

Here, when the straight line L _i intersects with a real object existing in the user 1 or other real space 41, the light from the light source 13 is blocked by the real object existing in the user 1 or other real space 41. 1 This means that a shadow of a real object existing in the other real space 41 can be created.

In step S17, when the shadow information indicating that the pixel value setting unit 65 is affected by the shadow is supplied from the shadow information generation unit 64, the pixel value setting unit 65 configures the image information Db of the _target frame supplied from the reception unit 11. The pixel value of the target pixel IB _i to be changed is changed. That is, for example, when the pixel value of the target pixel IB _i is represented by an RGB (Red Green Blue) value, the pixel value setting unit 65 reduces the RGB value and decreases the pixel value of the target pixel IB _i. Then, the process proceeds from step S17 to step S19 as a new pixel value of the target pixel IB _i . Thus, by reducing the pixel value of the target pixel IB _i , an image darkened by the influence of the shadow is expressed. When the line L _i intersects with the user 1 and other real objects, when the intersecting position is the periphery of the user 1 and other real objects, the amount of reduction in the pixel value is reduced and the area around the shadow Processing that blurs the part can be employed.

On the other hand, if the shadow information generation unit 64 determines in step S16 that the straight line L _i does not intersect with the real object existing in the user 1 or other real space 41, the target pixel IB _i is not affected by the shadow. Is supplied to the pixel value setting unit 65, and the process proceeds to step S18. In step S18, when the shadow information indicating that it is not affected by the shadow is supplied from the shadow information generation unit 64, the pixel value setting unit 65 sets the pixel value of the pixel of interest IB _i as a new pixel value as it is, step S19. Proceed to Note that an image that is not affected by the shadow is expressed by leaving the pixel value of the target pixel IB _i as it is.

  In step S19, the pixel selection unit 62 determines whether or not the variable i represents the final pixel of the frame of interest, that is, whether or not the variable i is equal to the number of pixels constituting the frame of interest. If it is determined in step S19 that the variable i is not equal to the number of pixels constituting the target frame, the process proceeds to step S20, and the pixel selection unit 62 increments the variable i by 1, returns to step S13, and so on. The process is repeated. On the other hand, if it is determined in step S19 that the variable i is equal to the number of pixels constituting the target frame, and if an image of the target frame composed of new pixel values is obtained, the process returns to step S5 in FIG. Proceed to

  Note that the processes in steps S13 to S18 described above are performed independently for each pixel, and are not performed by sequentially incrementing the variable i, but are performed in parallel for all pixels (or some pixels). It is also possible to perform processing automatically.

  In step S5 in FIG. 2, the pixel of the target frame configured with the new pixel value obtained as described above is supplied from the image generation unit 10 to the display control unit 9.

  From the above, the user 1 (user 21) displays an image in which his / her own shadow is reflected in the environment of the user (user 1) as the communication partner on the display screen of the display unit 8 (display unit 28). Can be observed. As a result, the user 1 (user 21) is as if the display screen of the display unit 8 (display unit 28) is a “window” between the user 21 (user 1) who is the communication partner. It is possible to communicate while feeling a sense of reality as if the communication partner is actually in front of you.

  FIG. 10 is a block diagram showing a configuration example of a second embodiment of a communication system to which the present invention is applied. In the figure, portions corresponding to those in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate.

  The communication system of FIG. 10 is basically configured similarly to the communication system of FIG.

  However, in the embodiment of FIG. 1, an image to be displayed on the display unit 8 of the information processing device 2 is generated by the image generation unit 10 of the information processing device 2 and is displayed on the display unit 28 of the information processing device 22. 10 is generated by the image generation unit 30 of the information processing device 22, but in the embodiment of FIG. 10, the image displayed on the display unit 8 of the information processing device 2 is processed by the image processing of the information processing device 22. An image generated by the apparatus 30 and transmitted from the information processing apparatus 22 to the information processing apparatus 2, and an image to be displayed on the display unit 28 of the information processing apparatus 22 is generated by the image generation unit 10 of the information processing apparatus 2. 2 to the information processing device 22.

In other words, in FIG. 10, an image in which the shadow of the user 1 is reflected on the image of the user 21 (hereinafter referred to as a shadow image E _{1 as} appropriate) is generated by the image generation unit 30 instead of the image generation unit 10. Further, an image in which the shadow of the user 21 is reflected on the image of the user 1 (hereinafter referred to as a shadow image E _{21 as} appropriate) is generated not by the image generation unit 30 but by the image generation unit 10.

  In the embodiment of FIG. 10, the information processing apparatus 2 displays the image transmitted from the information processing apparatus 22 as it is on the display unit 8, and the information processing apparatus 22 is transmitted from the information processing apparatus 2. The coming image is displayed on the display unit 28 as it is.

Therefore, in FIG. 10, an imaging camera 6 constituting the information processing apparatus 2 supplies the image signal obtained by imaging the real space 41, the image generation unit 10 as the image information D _a.

The transmission unit 7 includes the three-dimensional information C _a of the user 1 supplied from the detector 5 constituting the detection unit 3, the light source information stored in the information storage unit 12, and the shadow image E supplied from the image generation unit 10. ₂₁ is transmitted to the receiving unit 31 constituting the information processing apparatus 22 via the network 40.

The display control unit 9 performs predetermined processing on the shadow image E ₁ supplied from the reception unit 11 and transmitted from the information processing device 22 and supplies the shadow image E ₁ to the display unit 8 for display.

The image generating unit 10, the three-dimensional information C _a user 1 from the detector 5 constituting the detecting unit 3, the image information D _a from the imaging camera 6, the receiving unit 11, is transmitted from the information processing apparatus 22 The three-dimensional information _Cb and the light source information of the light source 33 are supplied. The image generating unit 10 includes the three-dimensional information C _a of the user 1 supplied from the detector 5, the image information D _a supplied from the imaging camera 6, the three-dimensional information C _b supplied from the receiving unit 11, and the light source 33. the light source information was used to generate the shadow image E ₂₁ the user 1 is crowded reflects the shadow of the user 21 to the image information D _a displayed, and supplies this to the transmission unit 7.

The reception unit 11 displays the three-dimensional information C _b of the real space 42, the light source information of the light source 33, and the user 21 transmitted from the transmission unit 27 constituting the information processing apparatus 22 via the network 40. information D _a shadow of the user 1 receives such shadow image E ₁ to crowded were transferred to. Further, the reception unit 11 supplies the three-dimensional information C _b and the light source information of the light source 33 to the image generation unit 10 and the shadow image E ₁ to the display control unit 9, respectively.

  10, the user 21, the information processing device 22, the light source 33, and the real space 42 correspond to the user 1, the information processing device 2, the light source 13, and the real space 41, respectively, as in the case of FIG. . The information processing device 22 includes a detection unit 3, a camera 4-1, a camera 4-2, a detection unit 5, an imaging camera 6, a transmission unit 7, a display unit 8, a display control unit 9, and an image generator that constitute the information processing device 2. Unit 10, receiver 11, and information storage unit 12, which are the same as detection unit 23, camera 24-1, camera 24-2, detection unit 25, imaging camera 26, transmission unit 27, display unit 28, and display control unit 29, respectively. , An image generation unit 30, a reception unit 31, and an information storage unit 32.

As described above, in FIG. 10, the shadow image E ₁ is generated not by the image generation unit 10 but by the image generation unit 30, transmitted from the information processing device 22 to the information processing device 2, and displayed on the information processing device 2. The Similarly, the shadow image E ₂₁ is generated not by the image generation unit 30 but by the image generation unit 10, transmitted from the information processing apparatus 2 to the information processing apparatus 22, and displayed on the information processing apparatus 22.

Now, when attention is paid to, for example, the image generation unit 30 that generates the shadow image E ₁ of the image generation unit 10 and the image generation unit 30, the image generation unit 30 uses the information to generate the shadow image E _1. the three-dimensional information C _b and the image information D _b other obtained by the processing apparatus 22, (as obtained) obtained in the information processing apparatus 2 is required source information of the three-dimensional information C _a and the light source 13. In order to display the shadow image E ₁ generated by the image generation unit 30 of the information processing device 22 on the display unit 8 of the information processing device 2, the shadow image E ₁ is transferred from the information processing device 22 to the information processing device 2. It is necessary to send to.

For this reason, the information processing device 22 receives the three-dimensional information C _a and the light source information of the light source 13 from the information processing device 2, generates a shadow image E _1, and performs image transmission processing to transmit to the information processing device 2. Is called.

On the other hand, the information processing apparatus 2 transmits the three-dimensional information _Ca and the light source information of the light source 13 to the information processing apparatus 22, receives the shadow image E ₁ transmitted from the information processing apparatus 22, and displays it on the display unit 8. An image receiving process to be displayed is performed.

Note that the image transmission processing by the information processing device 22 and the image reception processing by the information processing device 2 are performed in the information processing device 2 in order to display the shadow image E ₁ . Therefore, in order to display the shadow image E ₂₁ in the information processing apparatus 22, it is necessary to perform image transmission processing in the information processing apparatus 2 and perform image reception processing in the information processing apparatus 22. That is, in order to display the shadow image E ₁ in the information processing apparatus 2 and to display the shadow image E ₂₁ in the information processing apparatus 22, both the information processing apparatus 2 and the information processing apparatus 22 perform image reception processing. Image transmission processing is performed.

  FIG. 11 is a flowchart illustrating an image transmission process performed by the information processing apparatus 2 and the information processing apparatus 22 in FIG. Here, the image transmission process performed by the information processing apparatus 22 will be described.

  The image transmission process in FIG. 11 is started when a communication link is established between the information processing apparatus 2 and the information processing apparatus 22.

In step S <b> 31, the reception unit 31 receives the three-dimensional information C _a of the real space 41 transmitted from another device, that is, the transmission unit 7 constituting the information processing device 2, and the light source 13 via the network 40. Obtained by receiving the light source information, supplied to the image generation unit 30, and proceeds to step S32.

In step S <b> 32, the camera 24-1 and the camera 24-2 image the real space 42 and supply an image obtained as a result to the detector 25. Further, in step S32, the detector 25, a camera 24-1 and from the image supplied from the camera 24-2 each obtained by detecting the three-dimensional information C _b of the real space 42, the transmission unit 27 and the image The data is supplied to the generation unit 30, and the process proceeds to step S33. The three-dimensional information C _b supplied from the detection unit 25 to the transmission unit 27 is transmitted to the information processing apparatus 2 via the network 40.

In step S33, the imaging camera 26 images the real space 42, thereby acquiring an image such as a background surrounding the user 21 and the user 21, and supplies the image 21 as image information _Db to the image generation unit 30. Proceed to step S34.

In step S34, the image generation unit 30 includes the three-dimensional information C _a and the light source information of the light source 13 supplied from the reception unit 31 in step S31, the three-dimensional information C _b supplied from the detector 25 in step S32, and the step S33 on the basis of the image information D _b supplied from the imaging camera 26, when the real space 42 behind the display screen of the display unit 8 constituting the information processing apparatus 2 is assumed to be located, the real space 41 The process for generating an image reflecting the influence on the real space 42, that is, the shadow generation process for generating an image signal reflecting the shadow of the user 1 generated by the light source 13, for example, has been described with reference to FIGS. To do so. Furthermore, the image generation unit 30 supplies the shadow image E ₁ that is the generated image signal to the transmission unit 27, and the process proceeds to step S35. The details of the shadow generation process in step S34 are the same as those described in FIG.

In step S <b> 35, the transmission unit 27 configures the information processing apparatus 2 via the network 40 with the shadow image E ₁ that is an image signal reflecting the shadow of the user 1 supplied from the image generation unit 30. The data is transmitted to the receiving unit 11, and the process proceeds to step S36.

  In step S <b> 36, the transmission unit 27 or the reception unit 31 determines whether the communication link with the information processing apparatus 2 has been disconnected. If it is determined in step S36 that the communication link is not disconnected, the process returns to step S31, and the same processing is repeated thereafter. If it is determined in step S36 that the communication link has been disconnected, the process ends.

  In the image transmission process of FIG. 11, the processes of steps S31 to S33 may be processed in time series or may be processed in parallel.

  FIG. 12 is a flowchart illustrating image reception processing performed by the information processing apparatus 2 and the information processing apparatus 22 in FIG. Here, the image receiving process performed by the information processing apparatus 2 will be described.

  The image reception process in FIG. 12 is started when a communication link is established between the information processing apparatus 2 and the information processing apparatus 22.

The imaging camera 6 and each of the camera 4-1 and the camera 4-2 constituting the detection unit 3 start imaging of the real space 41, and identify the user 1 and the user 1 in the real space 41 obtained by the imaging. Output the surrounding background image. Images output from the camera 4-1 and the camera 4-2 are supplied to the detector 5. Image capturing camera 6 outputs is supplied to the image generating unit 10 as the image information D _a.

In step S41, the detector 5 from the camera 4-1 and the camera 4-2 image supplied from each detect three-dimensional information C _a in the real space 41, and supplies to the transmission unit 7, the step S42 move on.

  In step S <b> 42, the information storage unit 12 supplies the light source information of the light source 13 to the transmission unit 7.

In step S < _b > 43, the transmission unit 7 configures the information processing device 22 via the network 40 using the three-dimensional information Ca supplied from the detector 5 and the light source information of the light source 13 supplied from the information storage unit 12. and transmitted to the receiving unit 31, the process proceeds to step S44, the receiving unit 11 determines whether a shadow image E ₁ has been transmitted from the information processing apparatus 22.

In step S44, if the shadow image E ₁ is determined not to have been transmitted from the information processing apparatus 22 returns to step S44, and repeats similar processing. On the other hand, when it is determined in step S44 that the shadow image E ₁ has been transmitted from the information processing apparatus 22, that is, in step S35 of FIG. 11, the transmission unit 27 of the information processing apparatus 22 transmits the shadow image E ₁ . If, the process proceeds to step S45, the receiving unit 11 receives a shadow image E ₁ transmitted from the transmitter 27 of information processing device 22, and supplies the display control unit 9, the process proceeds to step S46.

In step S46, the display control unit 9, the receiving unit 11 is displayed is supplied to the display unit 8 performs a predetermined process on the shadow image E ₁ supplied from, the process proceeds to step S47.

  In step S <b> 47, the transmission unit 7 or the reception unit 11 determines whether the communication link with the information processing device 22 has been disconnected. If it is determined in step S47 that the communication link is not disconnected, the process returns to step S41, and the same processing is repeated thereafter. If it is determined in step S41 that the communication link has been disconnected, the process ends.

  With reference to FIG. 13, the image transmission process of FIG. 11 by the information processing apparatus 22 and the image reception process of FIG. 12 by the information processing apparatus 2 will be further described.

In step S41, the detector 5 of the information processing apparatus 2 from the camera 4-1 and the camera 4-2 image supplied from each detect three-dimensional information C _a real space 41, and supplied to the transmission section 7 Then, the process proceeds to step S42.

  In step S <b> 42, the information storage unit 12 supplies the light source information of the light source 13 to the transmission unit 7.

In step S < _b > 43, the transmission unit 7 configures the information processing apparatus 22 via the network 40 using the three-dimensional information C _a supplied from the detector 5 and the light source information of the light source 13 supplied from the information storage unit 12. Transmit to the receiver 31. Then, the process proceeds to step S44, the receiving unit 11 determines whether a shadow image E ₁ has been transmitted from the information processing apparatus 22.

On the other hand, in step S31, the reception unit 31 of the information processing device 22 receives the three-dimensional information _Ca and the light source information of the light source 13 that the transmission unit 7 of the information processing device 2 transmits in step S43, and the image generation unit. 30.

The detection unit 23 of the information processing apparatus 22, in step S32, the image supplied from the camera 24-1 and the camera 24-2, detects three-dimensional information C _b of the real space 42, the image generator 30 Supply.

Further, the imaging camera 26 of the information processing apparatus 22 supplies in step S33, the image information D _b obtained real space 42 by capturing the image generation unit 30.

In step S34, the image generation unit 30 of the information processing device 22 receives the three-dimensional information C _a supplied from the reception unit 31, the light source information of the light source 13, the three-dimensional information C _b supplied from the detector 25, and based on the image information D _b supplied from the imaging camera 26, it generates the shadow image E _1, and supplies to the transmission unit 27, the process proceeds to step S35.

Then, the transmission unit 27, in step S35, sends a shadow image E ₁ supplied from the image generating unit 30 to the receiving unit 11 constituting the information processing apparatus 2, the process proceeds to step S36.

In response to this, the reception unit 11 of the information processing device 2 determines in step S44 that the shadow image E ₁ has been transmitted from the information processing device 22, proceeds to step S45, and receives the shadow image E ₁ . It is supplied to the display control unit 9.

Then, in step S46, the display control unit 9 of the information processing apparatus 2 is supplied to a display unit 8 performs a predetermined process on the shadow image E ₁ supplied to the display from the receiving unit 11, the process proceeds to step S47.

  In step S36 and step S47, it is determined whether or not the communication link between the information processing device 2 and the information processing device 22 has been disconnected. If it is determined that the communication link has not been disconnected, the process proceeds to step S31 and step S41. Returning, the information processing apparatus 2 and the information processing apparatus 22 repeat the same processing. If it is determined in step S36 and step S47 that the communication link is disconnected, the information processing device 2 and the information processing device 22 end the processing.

Also in the embodiment of FIG. 10, the shadow image E ₁ is displayed on the display unit 8 and the shadow image E ₂₁ is displayed on the display unit 28, so that the communication partner is actually in front of you. You can communicate with each other while feeling a feeling.

  In addition, the image generation unit 10 and the image generation unit 30 do not require a special sensor or the like, and can generate an image full of realism by simply performing image processing, that is, with an inexpensive and simple configuration. .

  The series of processes described above can be executed by hardware, but can also be executed by software.

  When the above-described series of processing is executed by software, the image generation unit 10 and the image generation unit 30 can be configured as, for example, a computer as illustrated in FIG.

  That is, FIG. 14 shows a configuration example of the information processing apparatus 2 (or information processing apparatus 22) configured as a computer base.

  In FIG. 14, a CPU (Central Processing Unit) 71 executes various processes according to a program stored in a ROM (Read Only Memory) 72 or a program loaded from a storage unit 78 to a RAM (Random Access Memory) 73. To do.

  The RAM 73 also appropriately stores data necessary for the CPU 71 to execute various processes.

  The CPU 71, ROM 72, and RAM 73 are connected to each other via a bus 74. An input / output interface 75 is also connected to the bus 74.

  The input / output interface 75 is connected to an input unit 76 such as a keyboard and a mouse, an output unit 77 composed of a display, a storage unit 78 composed of a hard disk, and a communication unit 79.

  A drive 82 is connected to the input / output interface 75 as necessary, and a magnetic disk 83, an optical disk 84, a magneto-optical disk 85, or a semiconductor memory 86 is appropriately mounted, and a computer program read from them is required. Is installed in the storage unit 78 according to the above.

  When a series of processing is executed by software, a program constituting the software is installed from the program storage medium in the computer of FIG.

  As shown in FIG. 14, a program storage medium for storing a program installed in a computer and ready to be executed by the computer includes a magnetic disk 83 (including a floppy disk), an optical disk 84 (CD-ROM (Compact Disk- Package media, such as Read Only Memory (DVD) (including Digital Versatile Disk), magneto-optical disk 85 (including MD (Mini-Disk) (registered trademark)), or semiconductor memory 86, or programs are temporarily stored Alternatively, it is configured by a ROM 72 that is permanently stored, a hard disk that constitutes the storage unit 78, or the like. The program is stored in the program storage medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via an interface such as a router or a modem as necessary.

  Further, a camera 80 and a detection unit 81 are connected to the input / output interface 75.

  14, the output unit 77, the storage unit 78, the communication unit 79, the camera 80, and the detection unit 81 are the display unit 8, the information storage unit 12, the transmission unit 7 and the reception unit 11, the imaging camera 6, and the detection unit of FIG. 3 respectively.

  The detection unit 81 includes a camera 81-1, a camera 81-2, and a detector 81-3 that are the same as the camera 4-1, the camera 4-2, and the detector 5 that constitute the detection unit 3, respectively.

  The CPU 71 executes processing performed by the image generation unit 10 by executing a program installed in the storage unit 78.

  In the present specification, the step of describing the program stored in the program storage medium is not limited to the processing performed in time series according to the described order, but is not necessarily performed in time series. Or the process performed separately is also included.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

  In addition to the configuration described above, the detection unit 3 may be configured to detect 3D information using an image supplied from a single camera capable of detecting 3D information, for example. That is, for example, the angle of view that is the range that can be captured by the lens, the degree of aberration that the parallel rays incident on the lens are not collected at one point, the color is blurred, the image is distorted, etc. If it can be obtained, for example, the distance from the size of the subject such as the head of the imaged person to the point where the subject is located can be estimated, and further, the relative position shown in the captured image can be estimated. From the position of the subject, the direction of the point where the subject is actually located can be estimated. If the distance and direction of the imaged subject can be obtained, the position of the subject in the three-dimensional space can be obtained.

  Furthermore, as a configuration of the detection unit 3, for example, a configuration in which the position of an object is measured by triangulation by irradiating a laser can be employed.

  The imaging camera 6 can also be used as one of the cameras 4-1 and 4-2 constituting the detection unit 3.

  In the information processing apparatus 2, light source information of a virtual light source can be employed as light source information used for the shadow generation process, instead of the light source information of the actual light source 13. In the present embodiment, the light source information of the light source 13 is stored in the information storage unit 12 in advance, but the light source information of the light source 13 can be detected.

  That is, in the detector 5, when the light source 13 is captured in the images captured by the cameras 4-1 and 4-2, the three-dimensional information of the light source 13 is detected, and the three-dimensional information is used as the light source information. The image generation unit 10 can be supplied. Further, in the detector 5, when the light source 13 is not captured in the images captured by the camera 4-1 and the camera 4-2, for example, it is projected around the user 1 by the light of the light source 13. The three-dimensional information such as the position of the light source 13 can be estimated from the shadow or the like, and the three-dimensional information can be supplied to the image generation unit 10 as the light source information.

  The present invention can be applied to, for example, a teleconferencing system, a videophone system, and other tools for communicating while exchanging images.

It is a block diagram which shows the structural example of 1st Embodiment of the communication system to which this invention is applied. 3 is a flowchart for explaining the operation of the information processing apparatus 2 in FIG. 1. It is a figure explaining a stereo method. It is a figure explaining a stereo method. It is a figure which shows the relationship between distance and an evaluation value. It is a block diagram which shows the structural example of the image generation part 10 of FIG. It is a schematic diagram explaining real / virtual space. It is a figure explaining the process of the image generation part. It is a flowchart explaining the shadow production | generation process of the image generation part 10 (image generation part 30). It is a figure which shows the structural example of 2nd Embodiment of the communication system to which this invention is applied. It is a figure explaining the image transmission process of the information processing apparatus 22 of FIG. It is a figure explaining the image reception process of the information processing apparatus 2 of FIG. It is a figure explaining the correspondence of the image transmission process of FIG. 11, and the image reception process of FIG. And FIG. 11 is a block diagram illustrating a hardware configuration example of a computer that implements the information processing apparatus 2.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 User, 2 Image processing apparatus, 3 Detection part, 4-1 Camera, 4-2 Camera, 5 Detector, 6 Imaging camera, 7 Transmission part, 8 Display part, 9 Display control part, 10 Image generation part, 11 Receiving unit, 12 information storage unit, 13 light source, 21 user, 22 information processing device 23 detection unit, 24-1 camera, 24-2 camera, 25 detector, 26 imaging camera, 27 transmission unit, 28 display unit, 29 Display control unit, 30 image generation unit, 31 reception unit, 32 information storage unit, 33 light source, 40 network, 41 real space, 42 real space, 61 three-dimensional information expansion unit, 62 pixel selection unit, 63 pixel position determination unit, 64 shadow information generation unit, 65 pixel value setting unit, 71 CPU, 72 ROM, 73 RAM, 74 bus, 75 input / output Force interface, 76 input unit, 77 output unit, 78 storage unit, 79 communication unit, 80 camera, 81 detection unit, 81-1 camera, 81-2 camera, 81-3 detector, 82 drive, 83 magnetic disk, 84 Optical disk, 85 magneto-optical disk, 86 semiconductor memory

Claims (9)

First acquisition means for acquiring first three-dimensional information of a first real space in front of a display screen on which an image is displayed;
Second acquisition means for acquiring second three-dimensional information of the second real space at a position different from the first real space and a first image obtained by imaging the second real space;
Based on the first three-dimensional information acquired by the first acquisition means and the second three-dimensional information acquired by the second acquisition means and the first image, the rear of the display screen And generating means for generating a second image reflecting the influence of the first real space on the second real space when it is assumed that the second real space is located at An image processing apparatus. The image processing apparatus according to claim 1, further comprising display control means for displaying the second image generated by the generation means on the display screen. The influence of the first real space on the second real space is
The image processing apparatus according to claim 1, wherein the image processing apparatus is a shadow of a real object existing in the first real space generated by a predetermined light source. The image processing apparatus according to claim 3, wherein the predetermined light source is a virtual light source. The predetermined light source is an actual light source existing in the first real space;
The image processing apparatus according to claim 3, further comprising detection means for detecting the actual light source. The first acquisition means acquires the first three-dimensional information by detecting the first three-dimensional information from the first real space,
The image processing apparatus according to claim 1, wherein the second acquisition unit acquires the second three-dimensional information and the first image from another apparatus. The first acquisition means acquires the first three-dimensional information by receiving it from another device,
The second acquisition means acquires the second three-dimensional information by detecting the second real space from the second real space, and captures the first image by imaging the second real space. The image processing apparatus according to claim 1, wherein the image processing apparatus acquires the image processing apparatus. A first acquisition step of acquiring first three-dimensional information of a first real space in front of a display screen on which an image is displayed;
A second acquisition step of acquiring second three-dimensional information of the second real space at a position different from the first real space and a first image obtained by imaging the second real space;
Based on the first three-dimensional information acquired by the first acquisition step, the second three-dimensional information acquired by the second acquisition means, and the first image, the rear of the display screen Generating a second image reflecting the influence of the first real space on the second real space, assuming that the second real space is located at An image processing method characterized by the above. The first three-dimensional information of the first real space in front of the display screen on which the image is displayed, the second three-dimensional information of the second real space at a position different from the first real space, and the first When it is assumed that the second real space is located behind the display screen based on the first image obtained by imaging the second real space, the first real space is the second A computer-readable program comprising a generation step of generating a second image reflecting the effect on the real space.

Images