JP2006129472A - Image processing apparatus and image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus and an image processing method whereby a user easily grasps its own positional relation. <P>SOLUTION: An outside dome screen 42 is arranged in a way of surrounding the user and displays an immersion image such as a scene around an object. On the other hand, an inside dome screen 42 is arranged at the inside of the outside dome screen 42 and displays a perspective image of the object sensed by a visual sense of the user when the user views the object from its viewpoint. In this case, the user easily grasps its own positional relation by the perspective image and enjoys the presence by means of the immersion image. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly, to an image processing apparatus, an image processing method, and a program that allow a user to easily grasp their positional relationship.

  As a conventional display device that presents a realistic image to a user, for example, there is a display device using an immersive screen. In a display device using an immersive screen, when the user is in a certain virtual space, an image that is visible to the user (hereinafter referred to as an immersive image) is displayed, so the user can view the immersive image. By doing so, it is possible to enjoy a sense of realism as if they were actually in the virtual space provided by the immersive image.

  By the way, in the display device using the immersive screen, as described above, when the user is in the virtual space, an image that is visually reflected is displayed, so that the user himself / herself is provided with the immersive image. However, it is sometimes difficult to grasp the relative position.

  On the other hand, for example, in a car racing game in a so-called video game, a bird's-eye view of a car operated by a user viewed from a certain viewpoint is displayed. In this case, the user can easily grasp the position of the automobile operated by the user, and can easily operate the automobile.

  However, when the bird's-eye view image is displayed, the immersive feeling and the realistic sensation are impaired as compared with the case where the immersive image is displayed.

  The present invention has been made in view of such a situation, and makes it easy for a user to grasp his / her positional relationship and to display a realistic image.

  The image processing apparatus according to the present invention includes: an image selection unit that selects a viewpoint image when a predetermined subject is viewed from a user's viewpoint based on the user's viewpoint from images captured from a plurality of positions; And a viewpoint image converting means for converting a viewpoint image into a light-emitting image based on the viewpoint of the user and the shape of the display means.

  The image processing method of the present invention includes an image selection step of selecting a viewpoint image when a predetermined subject is viewed from the user's viewpoint based on the user's viewpoint from images captured from each of a plurality of positions; A viewpoint image conversion step of converting a viewpoint image into a light-emitting image based on the viewpoint of the user and the shape of the display means is provided.

  The program of the present invention includes an image selection step of selecting a viewpoint image when a predetermined subject is viewed from the user's viewpoint based on the user's viewpoint from images captured from each of a plurality of positions; A viewpoint image conversion step of converting the viewpoint image into a light emission image based on the viewpoint and the shape of the display means is provided.

  In the image processing device, the image processing method, and the program according to the present invention, a viewpoint image when a predetermined subject is viewed from the user's viewpoint based on the user's viewpoint from images captured from a plurality of positions. Is selected, and the viewpoint image is converted into a light-emitting image based on the viewpoint of the user and the shape of the display means.

  As described above, according to the present invention, the user can easily grasp his / her positional relationship.

  FIG. 1 shows an imaging / display system to which the present invention is applied (a system is a logical collection of a plurality of devices, regardless of whether or not each component device is in the same housing). 2 shows an example of the configuration of an embodiment.

  The imaging system 1 captures a subject and an omnidirectional image of the subject, and records the resulting image data on, for example, an optical disk, a magnetic disk, a magnetic tape, a semiconductor memory, or other recording medium 3, or For example, the transmission is performed via a telephone line, a satellite line, a CATV (Cable Television) network, the Internet, or another transmission medium 4.

  The display system 2 reproduces image data from the recording medium 3 or receives and displays image data transmitted via the transmission medium 4.

  FIG. 2 shows a configuration example of the imaging system 1 of FIG.

  The wide-angle imaging unit 11 includes a wide-angle camera (video camera) that captures the periphery of the subject at a wide angle, and supplies wide-angle image data (moving image data) captured by the wide-angle camera to the multiplexer 13. Here, as the wide-angle camera constituting the wide-angle imaging unit 11, for example, as described later, an omnidirectional camera that captures the entire azimuth of the subject can be adopted, and accordingly, the wide-angle output from the wide-angle image unit 11. The image data is an image such as a scenery around the subject.

  The enclosure imaging unit 12 includes a plurality of cameras (video cameras) arranged so as to surround the periphery of the subject in the horizontal direction and the vertical direction, and receives image data (moving image data) captured by the plurality of cameras. To the multiplexer 13. Here, the enclosure imaging unit 12 outputs image data obtained by imaging the subject from the camera position (camera position) from each of the cameras arranged at a plurality of positions. These image data are hereinafter collectively referred to as enclosed image data as appropriate.

  The multiplexer 13 multiplexes the wide-angle image data supplied from the wide-angle imaging unit 11 and the enclosed image data supplied from the enclosed imaging unit 12, and supplies multiplexed image data obtained as a result to the recording / transmitting unit 14. To do.

  Here, the multiplexer 13 multiplexes the same frame of the wide-angle image data and the enclosed image data in association with each other. The multiplexer 13 is also configured to multiplex each frame of image data captured from a plurality of camera positions as enclosed image data in association with the camera position. It is assumed that the camera position of each camera constituting the enclosure imaging unit 12 is known in advance. In addition, as the camera position, for example, coordinates representing a position on a three-dimensional coordinate system with a certain position of the subject as an origin can be employed.

  The recording / transmission unit 14 records the multiplexed image data supplied from the multiplexer 13 on the recording medium 3 (FIG. 1) or transmits it via the transmission medium 4.

  Next, FIG. 3 shows a configuration example of the wide-angle imaging unit 11 of FIG.

  The wide-angle imaging unit 11 includes an omnidirectional camera 21. The omnidirectional camera 21 ideally receives a light beam incident on the omnidirectional camera 21 through a hemispherical surface as shown in FIG. 3 from the plane perpendicular to the optical axis to the imaging direction side. The image data corresponding to is output.

  Here, as the omnidirectional camera 21, for example, a camera with a fisheye lens, a camera with a hyperboloidal mirror, or the like can be adopted. In the present embodiment, a camera with a fisheye lens is adopted as the omnidirectional camera 21.

  The omnidirectional camera 21 is installed so as to capture an omnidirectional landscape around the subject. Therefore, for example, if the subject is a soccer game being performed at a stadium, the omnidirectional camera 21 is installed, for example, in the center of the stadium. The scene of the stand where the spectators are watching is imaged.

  Next, FIG. 4 shows a configuration example of the enclosure imaging unit 12 of FIG.

The enclosure imaging unit 12 includes a plurality of enclosure cameras 22 _k (k = 1, 2,...) Arranged around the subject so as to surround the subject. Images of the subject viewed from a number of viewpoints are captured. Therefore, if the subject is, for example, a soccer game being performed in a stadium, as described above, a large number of enclosure cameras 22 _k constituting the enclosure imaging unit 12 are soccer fields provided in the stadium. Thus, the surrounding camera 22 _k captures an image of a soccer game pattern on the soccer field.

The enclosure camera 22 _k may be arranged only in the longitude direction (horizontal direction) of the subject in a certain horizontal plane, or may be arranged in any direction (horizontal or vertical direction) of the longitude and latitude of the subject. . When the enclosure camera 22 _k is arranged only in the longitude direction of the subject within a certain horizontal plane, only the left and right parallax methods described later can be adopted. However, the enclosure camera 22 _k can be arranged in any direction of the longitude and latitude of the subject. In this case, it is possible to employ either the left / right parallax method or the upper / lower / left / right parallax method described later.

Next, FIG. 5 shows an arrangement relationship between the omnidirectional camera 21 (FIG. 3) and the enclosure camera 22 _k (FIG. 4).

The enclosure camera 22 _k is, for example, arranged on the spherical surface of a hemisphere surrounding the subject at equal longitude and equal latitude intervals, and images the subject with the position (camera position) of the enclosure camera 22 _k as a viewpoint.

The omnidirectional camera 21 is arranged, for example, at the center of the hemisphere where the enclosure camera 22 _k is arranged, and from the omnidirectional area around the subject, that is, from the center of the hemisphere where the enclosure camera 22 _k is arranged, on the spherical surface of the hemisphere. The direction of each point is imaged.

Here, in the embodiment of FIG. 5, the omnidirectional camera 21 is arranged such that its optical axis faces the north polar direction (latitude is 90 degrees) of the hemisphere in which the enclosing camera 22 _k is arranged. . Further, enclosure camera 22 _k, the optical axis is disposed toward the center direction of the hemisphere enclosure camera 22 _k is arranged.

The arrangement method of the omnidirectional camera 21 and the surrounding-camera 22 _k is not limited to the arrangement shown in FIG.

In the imaging system 1 shown in FIG. 2, the omnidirectional camera 21 (FIG. 3) captures an omnidirectional image of the subject in the wide-angle imaging unit 11, and omnidirectional image data (wide-angle image data) obtained as a result is obtained. To the multiplexer 13. At the same time, in the enclosure imaging unit 12, the enclosure camera 22 _k (FIG. 4) images the image of the subject viewed from the camera position, and supplies the enclosure image data obtained as a result to the multiplexer 13.

  The multiplexer 13 multiplexes the omnidirectional image data supplied from the wide-angle imaging unit 11 and the enclosed image data supplied from the enclosed imaging unit 12, and records / transmits multiplexed image data obtained as a result of the multiplexing. To the unit 14.

  The recording / transmission unit 14 records the multiplexed image data supplied from the multiplexer 13 on the recording medium 3 (FIG. 1) or transmits it via the transmission medium 4.

  Therefore, in the imaging system 1, when the subject is, for example, a soccer game being performed in a stadium, the scenery of a stand where a spectator watching a soccer game is present or a soccer ball that is greatly kicked up Are taken as omnidirectional image data, and an image of the ground of the soccer game or a player who is playing from various viewpoints is taken as enclosed image data.

  Next, FIG. 6 shows a configuration example of the display system 2 of FIG.

  The reproduction / reception unit 31 reproduces multiplexed image data from the recording medium 3 (FIG. 1), or receives multiplexed image data transmitted via the transmission medium 4 (FIG. 1), and multiplexes the multiplexed image data. The image data is supplied to the demultiplexer 32.

  The demultiplexer 32 separates the multiplexed image data supplied from the reproduction / reception unit 31 into omnidirectional image data and enclosed image data for each frame, and supplies the omnidirectional image data to the wide-angle image processing unit 33. At the same time, the enclosed image data is supplied to the enclosed image processing unit 35.

  The wide-angle image processing unit 33 processes the omnidirectional image data supplied from the demultiplexer 32 and supplies the image data obtained as a result of the processing to the wide-angle image display unit 34.

  The wide-angle image display unit 34 displays the immersive image that provides the user with an immersive feeling by displaying the image data supplied from the wide-angle image processing unit 33.

  The enclosure image processing unit 35 processes the enclosure image data supplied from the demultiplexer 32 according to the user's viewpoint supplied from the viewpoint detection unit 37, and the image data when the subject is viewed from the user's viewpoint. It is generated and supplied to the enclosed image display unit 36.

  The enclosure image display unit 36 displays the image data supplied from the enclosure image processing unit 35, thereby displaying a bird's-eye image when the subject is viewed from a predetermined viewpoint such as the viewpoint of the user.

  The viewpoint detection unit 37 detects the viewpoint of the user and supplies it to the enclosed image processing unit 35. Here, as the viewpoint detection unit 37, for example, a magnetic sensor as a three-dimensional position sensor manufactured by Polhemus may be employed. In this case, the viewpoint detection unit 37, which is a magnetic sensor, is mounted on the user's head, for example. In addition, as the viewpoint detection unit 37, it is also possible to employ a means that can input a viewpoint (change) such as a joystick or a trackball. That is, the viewpoint detection unit 37 does not need to be a means for detecting the actual user viewpoint, and may be a means capable of inputting a virtual user viewpoint.

  Next, FIG. 7 shows a configuration example of the wide-angle image display unit 34 of FIG.

  The wide-angle image display unit 34 includes an outer omnidirectional projector 41 and an outer dome screen 42.

  The outer omnidirectional projector 41 is a projector capable of emitting light rays at a wide angle, and includes, for example, a general projector and a fish-eye lens as an optical system that widens the light rays emitted from the projector.

  The outer dome screen 42 has, for example, a hemispherical dome shape and is disposed so as to surround the user. Further, the hemispherical dome as the outer dome screen 42 has such a size that it can move around to some extent even if a plurality of users enter the hemispherical dome, and its diameter is, for example, about 5 m.

  For example, a coating material that scatters light is applied to the inner (or outer) surface of the outer dome screen 42. An outer omnidirectional projector 41 is arranged inside the outer dome screen 42 so that light rays are emitted toward the inner surface.

  That is, the outer omnidirectional projector 41 is disposed such that, for example, its optical center coincides with the center of the hemispheric dome as the outer dome screen 42 and its optical axis is orthogonal to the horizontal plane. Then, the outer omnidirectional projector 41 emits a light beam corresponding to the image data supplied from the wide-angle image processing unit 33 through the fisheye lens, so that the light beam inside the outer dome screen 42 which is a hemispheric dome. It injects toward the whole surface.

  Light rays emitted from the outer omnidirectional projector 41 are received and scattered at each point on the inner surface of the outer dome screen 42 which is a hemispherical dome. Thus, on the inner surface of the outer dome screen 42 covering the user, an immersive image corresponding to the light beam emitted by the outer omnidirectional projector 41, that is, an omnidirectional image of the subject imaged by the imaging system 1 is displayed. The

  The shape of the outer dome screen 42 is not limited to the hemispherical dome shape as long as it surrounds the user, and a cylindrical shape or the like can also be adopted.

  Next, FIG. 8 shows a configuration example of the enclosed image display unit 36 of FIG.

  The enclosed image display unit 35 includes an inner omnidirectional projector 43 and an inner dome screen 44.

  The inner omnidirectional projector 43 is a projector that can emit a light beam at a wide angle similarly to the outer omnidirectional projector 41 of FIG. 7. For example, a general projector and a light beam emitted from the projector have a wide angle. It consists of a fisheye lens as an optical system.

  The inner dome screen 44 has, for example, a hemispherical dome shape, and a paint that scatters light, for example, is applied to the inner (or outer) surface thereof. An inner omnidirectional projector 43 is arranged inside the inner dome screen 44 so that light rays are emitted toward the inner surface.

  That is, for example, the inner omnidirectional projector 43 is arranged such that its optical center coincides with the center of the hemispheric dome as the inner dome screen 44 and its optical axis is orthogonal to the horizontal plane. The inner omnidirectional projector 43 emits a light beam corresponding to the image data supplied from the wide-angle image processing unit 33 through the fisheye lens, so that the light beam on the inner side of the inner dome screen 44 which is a hemispheric dome. It injects toward the whole surface.

  Light rays emitted from the inner omnidirectional projector 43 are received and scattered at each point on the inner surface of the inner dome screen 44 which is a hemispherical dome. The inner dome screen 44 itself is made of a transparent member, and light emitted from the inner omnidirectional projector 43 is scattered on the inner surface of the inner dome screen 44. A bird's-eye view image corresponding to the light beam emitted by 43, that is, an image obtained by viewing the subject imaged by the imaging system 1 from a certain viewpoint is displayed in a so-called rear-projected form.

  Here, the shape of the inner dome screen 44 is a hemispherical dome that is similar to the shape of the outer dome screen 42 in FIG. 7, but the shape of the inner dome screen 44 is similar to the shape of the outer dome screen 42. Need not be. That is, the shape of the inner dome screen 44 can be, for example, a cylindrical shape or a polygonal column shape independently of the shape of the outer dome screen 42.

  Next, FIG. 9 shows the positional relationship between the outer dome screen 42 (FIG. 7) and the inner dome screen 44.

  As described above, the outer dome screen 42 is disposed so as to surround the user. The inner dome screen 44 is disposed inside the outer dome screen 42.

  Therefore, a user inside the outer dome screen 42 and outside the inner dome screen 44 is displayed on the immersive image displayed on the inner surface of the outer dome screen 42 and on the outer surface of the inner dome screen 44. A bird's-eye view image can be viewed simultaneously. As a result, the user can enjoy the immersive feeling provided by the immersive image and can easily grasp his / her position from the bird's-eye view image.

  That is, for example, an immersive image such as a scene of a soccer stadium stand is displayed on the inner surface of the outer dome screen 42, and a soccer field is displayed on the outer surface of the inner dome screen 44 from the viewpoint of the user. Since the state of the game when viewed is displayed, the user can actually feel a sense of reality as if watching a soccer game at a soccer field.

  Next, as shown in FIG. 10, not only one person but also a plurality of users can enter the inside of the outer dome screen 42. Therefore, according to the display system 2, a plurality of users can be connected to the outer dome screen 42. The immersive image displayed on the screen 42 and the bird's-eye view image displayed on the inner dome screen 44 can be viewed simultaneously.

  Then, for example, as shown in FIG. 11, when there are two users A and B inside the outer dome screen 42 and the users A and B are at different positions, the inner dome screen In 44, for the user A, a bird's-eye image when the subject is viewed from the viewpoint of the user A is displayed, and for the user B, a bird's-eye image when the subject is viewed from the viewpoint of the user A is displayed. Is displayed.

  That is, in the display system 2, a bird's-eye image when the subject is viewed from the user's viewpoint is displayed on the inner dome screen 44 to the user.

  Therefore, the user can view an image (bird's-eye view image) when viewing the subject from the viewpoint, and has the freedom to select the viewpoint.

  Here, the position of the viewpoint that can be selected by the user can be limited to a position in a horizontal plane at a certain height inside the outer dome screen 42 as shown in FIG. 12, for example. Instead, for example, as shown in FIG. 13, the position may be an arbitrary position inside the outer dome screen 42. As shown in FIG. 12, the method of limiting the viewpoint to a position within a certain horizontal plane can be called a left-right parallax method because the user can move the viewpoint only in the left-right direction. Further, as shown in FIG. 13, a method that can set the viewpoint to an arbitrary position can be referred to as an up / down / left / right parallax method because the user can move the viewpoint in any direction, up, down, left, and right.

  Next, FIG. 14 shows a configuration example of the wide-angle image processing unit 33 of FIG.

  The omnidirectional image data for each frame output from the demultiplexer 32 (FIG. 6) is supplied to the frame memory 51. The frame memory 51 stores the omnidirectional image data supplied from the demultiplexer 32 in units of frames. The stored omnidirectional image data is read in frame units and supplied to the wide-angle image conversion unit 52. Note that the frame memory 51 has a plurality of memory banks. By switching banks, the omnidirectional image data can be stored and read out simultaneously.

  The wide-angle image conversion unit 52 converts the omnidirectional image data supplied from the frame memory 51 into rectangular latitude / longitude image data on a plane defined by the latitude direction and the longitude direction under the control of the controller 56, This is supplied to the angle of view correction unit 53.

  The angle-of-view correction unit 53 controls the latitude and longitude image data supplied from the wide-angle image conversion unit 52 based on the optical characteristics of the omnidirectional projector 41 for the outside of the wide-angle image display unit 34 (FIG. 7) under the control of the controller 56. The latitude / longitude image data after the correction is supplied to the latitude / longitude image conversion unit 54.

  The latitude / longitude image conversion unit 54 converts the latitude / longitude image data supplied from the angle-of-view correction unit 53 into omnidirectional image data and supplies the omnidirectional image data to the frame memory 55 under the control of the controller 56.

  The frame memory 55 stores the omnidirectional image data supplied from the latitude / longitude image conversion unit 54 in units of frames, reads out the stored omnidirectional image data in units of frames, and the wide angle image display unit 34 (FIG. 7). The omnidirectional projector 41 is supplied. The frame memory 55 also has a plurality of memory banks, like the frame memory 51. By switching the banks, the omnidirectional image data can be stored and read out simultaneously. .

  The controller 56 controls the wide-angle image conversion unit 52, the view angle correction unit 53, and the latitude / longitude image conversion unit 54 based on the storage state of the omnidirectional image data in the frame memory 51.

  Next, processing of the wide-angle image conversion unit 52, the angle-of-view correction unit 53, and the latitude / longitude image conversion unit 54 of FIG. 14 will be described with reference to FIGS.

  The omnidirectional image data stored in the frame memory 51, that is, the omnidirectional image data captured by the omnidirectional camera 21 constituting the wide angle imaging unit 11 of FIG.

  Now, as shown in FIG. 15, a three-dimensional coordinate system in which the optical axis of the omnidirectional camera 21 is the z axis and the imaging surface (light receiving surface) of the omnidirectional camera 21 is the xy plane is a wide angle image imaging coordinate system. In this wide-angle image capturing system, the relationship between the light beam incident on the omnidirectional camera 21 and the omnidirectional image data captured by the omnidirectional camera 21 receiving the light beam is considered.

  The light beam L incident on the omnidirectional camera 21 is refracted by the fisheye lens and is incident on the imaging surface of the omnidirectional camera 21. That is, the fisheye lens refracts the light beam L in a plane including the z-axis and the light beam L of the wide-angle image imaging coordinate system and makes the light beam L incident on the xy plane. Therefore, in the omnidirectional camera 21, for example, image data of a circular shape or a donut shape as shown by hatching in FIG. 16 is captured as omnidirectional image data.

  Here, in the wide-angle imaging coordinate system, as shown in FIG. 15, the angle with the x-axis on the xy plane is called longitude θ, and the angle with the z-axis is called latitude φ, and the longitude is θ. Assuming that the direction of latitude φ is the direction (θ, φ), the light ray L incident on the omnidirectional camera 21 from the direction (θ, φ) Refracts according to φ.

  Therefore, the light ray L incident from the direction (θ, φ) is expressed as a light ray L (θ, φ), and a point on the xy plane where the light ray L (θ, φ) is projected is represented by Q (x, y). ), The longitude θ and the latitude φ are expressed by the following equations using the x and y coordinates of the point Q.

・・・（１）

... (1)

・・・（２）

... (2)

However, in equation (2), the function f ₁ () is a function determined by the optical characteristics of the fisheye lens of the omnidirectional camera 21. Incidentally, the inverse function of the function f ₁ (), expressed as f ₁ ^-1 (), Equation (2) ^{is, √ (x 2 + y 2} ) = f 1 - can be represented as 1 (theta). Inverse function f ₁ ^-1 (θ) represents an optical property of a fisheye lens, the inverse function f ₁ ^-1 (θ) is, fisheye lens represented by [the focal length of the fisheye lens] × theta is equidistant projection (Equidistance projection) method.

  From Equations (1) and (2), a certain point Q (x, y) on the omnidirectional image captured by the omnidirectional camera 21 is a longitude representing the direction of the light beam L (θ, φ) incident on the point. It can be expressed by θ and latitude φ.

That is, the angle formed by the straight line connecting the point Q (x, y) and the origin O and the x axis is equal to the longitude θ of the light ray L. The distance √ (x ² + y ² ) between the point Q (x, y) and the origin O depends on the latitude φ of the light ray L. The longitude θ and the latitude φ are independent values.

  Therefore, for example, when considering a latitude / longitude coordinate system as a two-dimensional coordinate system having the longitude θ as the horizontal axis and the longitude φ as the vertical axis, the pixel Q (x, y) constituting the omnidirectional image data is Projection can be performed on the point Q ′ (θ, φ) on the latitude-longitude coordinate system.

  By projecting the omnidirectional image data to the latitude-longitude coordinate system, as shown in FIG. 17, the donut-shaped omnidirectional image data is rectangular image data defined by longitude θ and latitude φ, that is, latitude. Converted to longitude image data.

  As described above, the wide-angle image conversion unit 52 in FIG. 14 converts the omnidirectional image data into the latitude and longitude image data.

  Next, the optical characteristics of the fisheye lens of the omnidirectional camera 21 of the wide-angle imaging unit 11 (FIG. 3) that captures the omnidirectional image and the outside of the wide-angle image display unit 34 (FIG. 7) that emits light rays corresponding to the omnidirectional image. If the optical characteristics of the fisheye lens of the omnidirectional projector 41 are the same, the omnidirectional image data obtained by the omnidirectional camera 21 can be supplied to the outer omnidirectional projector 41 as it is. Since a light beam having a direction opposite to that of the light beam incident on the omnidirectional camera 21 is emitted from the outer dome screen 42 (FIG. 7) for receiving the light beam from the outer omnidirectional projector 41, theoretically Displays the same image as the landscape imaged by the omnidirectional camera 21.

  However, the optical characteristics of the fisheye lenses of the omnidirectional camera 21 (FIG. 3) and the outer omnidirectional projector 41 (FIG. 7) are not necessarily the same and may be different.

  Here, between the point Q (x, y) on the imaging surface where the light beam L (θ, φ) incident on the omnidirectional camera 21 arrives, and the longitude θ and the latitude φ, equations (1) and (1) There is a relationship shown in 2).

The longitude θ expressed by the equation (1) is not affected by the optical characteristics of the fisheye lens, but the latitude φ expressed by the equation (2) is obtained using the function f ₁ (). f ₁ () inverse function function f ₁ ^-1 of (), as described above, since representing optical characteristics of the fish-eye lens affected by the optical characteristics of the fish-eye lens.

  Therefore, when the optical characteristics of the fish-eye lens of the omnidirectional camera 21 (FIG. 3) and the omnidirectional projector 41 (FIG. 7) are different, the image displayed on the outer dome screen 42 (FIG. 7) is displayed. In order to obtain the same image as the landscape imaged in step 1, the azimuth image data captured by the omnidirectional camera 21 needs to be corrected in the latitude φ direction.

  Since the latitude φ direction of the omnidirectional image data is the vertical axis direction of the latitude / longitude image data shown in FIG. 17, the angle-of-view correction unit 53 in FIG. 14 receives the latitude / longitude image data supplied from the wide-angle image conversion unit 52. Is corrected to a latitude φ ′ by the following equation, for example.

・・・（３）

... (3)

  However, in Expression (3), α is a value determined by the optical characteristics of the fish-eye lens of the omnidirectional camera 21 (FIG. 3) and the outer omnidirectional projector 41 (FIG. 7).

  Therefore, if the pixel constituting the latitude / longitude image data supplied from the wide-angle image conversion unit 52 is expressed as Q ′ (θ, φ), the view angle correction unit 53 uses the pixel Q ′ (θ, φ) as the pixel. It is corrected to Q ′ (θ, φ ′).

  Note that, as shown in FIG. 17, the latitude φ represents the vertical axis direction of the latitude / longitude image data, and therefore corresponds to the vertical angle of view of the latitude / longitude image data. Therefore, the correction of the latitude φ corresponds to reducing or increasing the vertical field angle of the latitude and longitude image data.

  Even if rectangular latitude and longitude image data is supplied to the outer omnidirectional projector 41 (FIG. 7) as it is, the omnidirectional image captured by the omnidirectional camera 21 (FIG. 3) is displayed on the outer dome screen 42. Since it is not displayed, the latitude / longitude image conversion unit 54 in FIG. 14 again converts the latitude / longitude image data in which the latitude φ is corrected by the angle-of-view correction unit 53 into a donut shape (or a circular shape) as shown in FIG. ) Omnidirectional image data. The omnidirectional image data obtained by the latitude / longitude image conversion unit 54 is based on the optical characteristics of the fish-eye lens of the omnidirectional camera 21 (FIG. 3) and the outer omnidirectional projector 41 (FIG. 7). The corrected omnidirectional image data is supplied to the outer omnidirectional projector 41 (FIG. 7), so that the omnidirectional image captured by the omnidirectional camera 21 (FIG. 3) is displayed on the outer dome screen 42. Will be displayed.

  Next, processing (wide-angle image processing) performed by the wide-angle image processing unit 33 in FIG. 14 will be described with reference to the flowchart in FIG.

  The controller 56 monitors the frame memory 51, and starts wide-angle image processing when the frame memory 51 starts storing omnidirectional image data.

  That is, in the wide-angle image processing, first, in step S1, the controller 56 sets, for example, 1 as an initial value to the variable i for counting the number of frames, and proceeds to step S2. In step S2, the frame memory 51 reads the omnidirectional image data of the i-th frame stored therein, supplies it to the wide-angle image conversion unit 52, and proceeds to step S3.

  In step S3, the wide-angle image conversion unit 52 obtains the longitude θ from the equation (1) for each pixel Q (x, y) constituting the omnidirectional image data supplied from the frame memory 51, and proceeds to step S4. . In step S4, the wide-angle image conversion unit 52 determines the latitude φ for each pixel Q (x, y) constituting the omnidirectional image data supplied from the frame memory 51 using equation (2), and proceeds to step S5. .

  In step S5, the wide-angle image conversion unit 52 determines each pixel Q (x, y) constituting the omnidirectional image data supplied from the frame memory 51 as the longitude θ obtained for the pixel Q (x, y). By arranging at a point (θ, φ) on a latitude / longitude coordinate system represented by latitude φ, as shown in FIG. 17, the donut-shaped omnidirectional image data is converted into rectangular latitude / longitude image data. To do.

  When the wide-angle image conversion unit 52 obtains the latitude / longitude image data as described above in step S5, the wide-angle image conversion unit 52 supplies the latitude / longitude image data to the angle-of-view correction unit 53 and proceeds to step S6.

  In step S6, the angle-of-view correction unit 53 corrects the latitude φ direction of the latitude / longitude image data supplied from the wide-angle image conversion unit 52 to the latitude φ ′ according to the equation (3), and the corrected latitude φ 'And the latitude / longitude image data represented by the longitude θ is supplied to the latitude / longitude image conversion unit 54, and the process proceeds to step S7.

  In step S <b> 7, the latitude / longitude image conversion unit 54 sets the latitude / longitude image data supplied from the angle-of-view correction unit 53 on the xy plane of the wide-angle image imaging coordinate system for each pixel Q ′ (θ, φ ′). The x coordinate is obtained according to the equation (4), and the process proceeds to step S8.

  In step S8, the latitude / longitude image conversion unit 54 applies the pixel Q ′ (θ, φ ′) constituting the latitude / longitude image data supplied from the angle-of-view correction unit 53 on the xy plane of the wide-angle image imaging coordinate system. The y coordinate is obtained according to the equation (5), and the process proceeds to step S9.

・・・（４）

... (4)

・・・（５）

... (5)

Here, the function f ₂ (φ ′) in the equations (4) and (5) is a function representing the optical characteristics of the outer omnidirectional projector 41 (FIG. 7) (depending on the projection method of the fisheye lens of the outer omnidirectional projector 41). Function). By solving equations (1) and (2) for x and y, equations similar to equations (4) and (5) can be obtained.

  In step S <b> 9, the latitude / longitude image conversion unit 54 replaces each pixel Q ′ (θ, φ ′) constituting the latitude / longitude image data supplied from the view angle correction unit 53 with the pixel Q ′ (θ, φ ′). Is arranged at a point (x, y) on the xy plane of the wide-angle image capturing coordinate system represented by the x-coordinate and y-coordinate obtained for the rectangular latitude-longitude image data, and the donut-shaped omnidirectional image This omnidirectional image data is converted into data, and is supplied to the frame memory 55 and stored as emission omnidirectional image data which is image data to be emitted by the outer omnidirectional projector 41 (FIG. 7).

  As described above, the light emission omnidirectional image data stored in the frame memory 55 is sequentially read out and supplied to the wide-angle image display unit 34 (FIG. 6). In the wide-angle image display unit 34, the outer omnidirectional projector 41 (FIG. 7) emits a light beam corresponding to the light emission omnidirectional image data, and this light beam is received by the outer dome screen 42 (FIG. 7). Thus, on the outer dome screen 42, an omnidirectional landscape image of the subject imaged by the omnidirectional camera 21 (FIG. 3) is displayed as an immersive image.

  Thereafter, the process proceeds to step S10, and the controller 51 refers to the frame memory 51 to determine whether or not the i-th frame is the last frame. When it is determined in step S10 that the i-th frame is not the last frame, that is, when the omnidirectional image data of the frame after the i-th frame is stored in the frame memory 51, the process proceeds to step S11, and the controller In step 56, the variable i is incremented by 1, and the process returns to step S2. Thereafter, the same processing is repeated.

  If it is determined in step S10 that the i-th frame is the last frame, that is, if the omnidirectional image data of the frame after the i-th frame is not stored in the frame memory 51, the wide-angle image processing is performed. Exit.

  If the optical characteristics of the omnidirectional camera 21 (FIG. 3) and the omnidirectional projector 41 (FIG. 7) are the same, the omnidirectional image data output from the demultiplexer 32 (FIG. 6) is as shown in FIG. The wide-angle image processing unit 33 can be directly supplied to the wide-angle image display unit 34 as emission omnidirectional image data.

  Next, an outline of the processing of the enclosed image processing unit 35 in FIG. 6 will be described.

  In the enclosed image display unit 36 (FIG. 6), as described above, on the inner dome screen 44 (FIG. 8), an image of the subject viewed from the user's viewpoint is displayed to the user. Further, when there are a plurality of users, on the inner dome screen 44 (FIG. 8), an image of the subject viewed from the user's viewpoint is displayed for each of the plurality of users.

  Therefore, in the inner dome screen 44, as shown in FIG. 19, the inner dome screen 44 is temporarily transparent, and the inside dome screen 44 becomes an object to be imaged by the enclosure imaging unit 12 (FIG. 2). If the subject is present, the same image as the subject and the image shown in each user's vision is displayed.

  That is, in the embodiment of FIG. 19, there are two users A and B, but on the inner dome screen 44, the subject from the viewpoint of the user A is at a portion that intersects with a straight line connecting the center of the inner dome screen 44 and the viewpoint of the user A. The image when viewing is displayed. Further, on the inner dome screen 44, an image when the subject is viewed from the viewpoint of the user B is displayed at a portion intersecting with a straight line connecting the center thereof and the viewpoint of the user B.

  For this reason, in the enclosed display unit 36 of FIG. 8, for example, as shown in FIG. 20, the omnidirectional projector 43 has an image of the subject viewed from the viewpoint of each user on a display surface such as a liquid crystal panel (not shown). (Hereinafter referred to as a viewpoint image as appropriate) and emits a light beam corresponding to the viewpoint image through the fisheye lens, whereby the image of the subject viewed from the viewpoint of each user on the inner dome screen 44 is Displayed to each user.

  Therefore, the enclosed image processing unit 35 in FIG. 6 uses the enclosed image data supplied from the demultiplexer 32, and for each viewpoint supplied from the viewpoint detection unit 37, an image when the subject is viewed from the viewpoint (viewpoint image). ), And by arranging the viewpoint image for each viewpoint at a position corresponding to the viewpoint, the emission image corresponding to the light beam emitted from the inner omnidirectional projector 43 (FIG. 8), that is, the inner side A light emission image (hereinafter also referred to as a display image) to be displayed on the display surface of the omnidirectional projector 43 is generated.

  By the way, as shown in FIG. 21, for example, when the users A and B are separated from each other, and the viewpoints of the users A and B are also separated, Even when an image of the subject viewed from the viewpoint (viewpoint image) is displayed, the user A can see only the viewpoint image for the viewpoint of the user A, and the user B can also view the viewpoint for the viewpoint of the user B. Only the image can be seen.

  That is, the user A cannot see the viewpoint image regarding the viewpoint of the user B, or even if it can be viewed, it can only be viewed to the extent that the viewing of the viewpoint image regarding the viewpoint of the user A is not hindered. Similarly, the viewpoint image of the viewpoint of the user A is not visible for the user B, or even if it is visible, the viewpoint image for the viewpoint of the user B is not disturbed.

  On the other hand, for example, as shown in FIG. 22, when the users A and B are close to each other, the viewpoints are close to each other, and the viewpoint images for the users A and B displayed on the inner dome screen 44 overlap each other. Sometimes. Further, even when the viewpoint images for the users A and B displayed on the inner dome screen 44 do not overlap each other, the viewpoint images for the users A and B are displayed at close positions on the inner dome screen 44. Therefore, the viewpoint image regarding the viewpoint of the user B prevents the viewing of the viewpoint image regarding the viewpoint of the user A by the user A, and the viewpoint image regarding the viewpoint of the user A is the viewpoint image regarding the viewpoint of the user B. The viewing by the user B is hindered.

  Therefore, when the distance between the viewpoints of the users A and B is short, the enclosed image processing unit 35 sets a viewpoint common to the users A and B at a position close to both of the viewpoints of the users A and B. An image (viewpoint image) in which the subject is viewed from the viewpoint is generated. In this case, on the inner dome screen 44, as shown in FIG. 22, an image of the subject viewed from a common viewpoint is displayed for both the users A and B. Can be prevented from being hindered.

  FIG. 23 shows a configuration example of the enclosed image processing unit 35 of FIG. 6 that performs the above-described processing (enclosed image processing).

  The viewpoint buffer 61 is supplied with the viewpoint of the user output from the viewpoint detection unit 37 (FIG. 6), and the viewpoint buffer 61 stores the viewpoint supplied from the viewpoint detection unit 37. When there are a plurality of users, the viewpoint detection unit 37 in FIG. 6 detects the viewpoints of the plurality of users and supplies the viewpoints to the viewpoint buffer 61. In this case, the viewpoint buffer 61 stores all the viewpoints of a plurality of users supplied from the viewpoint detection unit 37.

  The viewpoint processing unit 62 reads the user's viewpoint stored in the viewpoint buffer 61 and supplies the viewpoint to the viewpoint storage unit 63 while performing predetermined processing on the viewpoint as necessary. The viewpoint storage unit 63 stores the viewpoint supplied from the viewpoint processing unit 62. The viewpoint direction calculation unit 64 obtains the viewpoint direction (viewpoint direction) stored in the viewpoint storage unit 63 and supplies it to the viewpoint image conversion unit 67.

  The frame memory 65 stores the enclosed image data supplied from the demultiplexer 32 (FIG. 6) in units of frames, reads the stored enclosed image data, and supplies it to the image selection unit 66. The frame memory 65 is also configured in the same manner as the frame memory 51 of FIG. 14, so that the frame memory 65 can simultaneously store and read the enclosed image data.

The image selection unit 66 selects image data necessary for generating a viewpoint image for the viewpoint stored in the viewpoint storage unit 63 from the enclosed image data supplied from the frame memory 65. That is, the enclosure image data is a set of image data captured by a plurality of enclosure cameras 22 ₁ , 22 ₂ , 22 ₃ ,... Arranged at a plurality of camera positions, as shown in FIG. Now, assuming that the image data captured by the enclosure camera 22 _k constituting the enclosure image data is referred to as camera image data, the image selection unit 66 uses the enclosure image data based on the viewpoint stored in the viewpoint storage unit 63. Select camera image data from.

  Then, the image selection unit 66 processes the camera image data selected from the enclosed image data based on the viewpoint stored in the viewpoint storage unit 63 as it is, or processes it as viewpoint image data for the viewpoint as a viewpoint image conversion unit. 67.

  The viewpoint image conversion unit 67 uses the viewpoint image data supplied from the image selection unit 66 based on the viewpoint direction supplied from the viewpoint direction calculation unit 64 and the shape of the inner dome screen 44 (FIG. 8). Display image data to be displayed on the display screen of the omnidirectional projector 43 (FIG. 8) is generated and supplied to the frame memory 68.

  The above-described image selection unit 66 and viewpoint image conversion unit 67 generate display image data for generating display image data from enclosed image data based on the viewpoint of the user and the shape of the inner dome screen 44 (FIG. 8). Part 70 is configured.

  The frame memory 68 stores the display image data supplied from the viewpoint image conversion unit 67 and supplies the stored display image data to the inner omnidirectional projector 43 (FIG. 8). Here, the frame memory 68 is also configured in the same manner as the frame memory 51 of FIG. 14, so that the frame memory 68 can simultaneously store and read display image data. .

  The controller 69 controls the viewpoint processing unit 62, the viewpoint direction calculation unit 64, the image selection unit 66, and the viewpoint image conversion unit 67 while referring to the storage status of the enclosed image data in the frame memory 65.

  Next, processing of the viewpoint direction calculation unit 64 in FIG. 23 will be described with reference to FIG.

  The viewpoint direction calculation unit 64 obtains longitude and latitude representing the direction of the viewpoint (viewpoint direction) stored in the viewpoint storage unit 63.

  That is, a three-dimensional coordinate system in which the optical axis of the inner omnidirectional projector 43 is the z axis and the display surface of the inner omnidirectional projector 43 is the xy plane is now referred to as an enclosed image display coordinate system. The viewpoint direction calculation unit 64 obtains, as the viewpoint direction, the direction of viewing the viewpoint V (x, y, z) at a certain position from the origin O in the enclosed image display coordinate system.

  Therefore, in the enclosed image display coordinate system, as shown in FIG. 24, the angle formed with the x axis on the xy plane is referred to as longitude θ, and the angle formed with the z axis is referred to as latitude φ, and the longitude is θ. If the direction of latitude φ is represented as direction (θ, φ), longitude θ and latitude φ representing the viewpoint direction (θ, φ) of viewpoint V (x, y, z) are respectively expressed by the following equations: Can be obtained.

・・・（６）

... (6)

・・・（７）

... (7)

  The viewpoint direction calculation unit 64 obtains the viewpoint direction (θ, φ) according to the equations (6) and (7) and supplies it to the viewpoint image conversion unit 67.

  Next, processing of the display image generation unit 70 in FIG. 23 will be described with reference to FIGS.

  To display a viewpoint image in which the subject is viewed from the user's viewpoint on the inner dome screen 44 (FIG. 8), simply display the viewpoint direction of the user's viewpoint as shown in FIG. 25, for example. Camera image data captured from the viewpoint direction (θ, φ) is displayed as a viewpoint image in a predetermined range on the inner dome screen 44 centering on the intersection U between (θ, φ) and the inner dome screen 44. do it.

  Accordingly, a viewpoint image that is camera image data captured from the viewpoint direction (θ, φ) is now represented as a viewpoint image (θ, φ), and the viewpoint direction (θ, φ) of the user's viewpoint and the inner dome screen 44. Assuming that a predetermined range on the inner dome screen 44 centering on the intersection U with the display area (θ, φ), the viewpoint image (θ, φ) is displayed in the display area (θ, φ). It is necessary to write the viewpoint image (θ, φ) at a position on the display image to be displayed on the display surface of the inner omnidirectional projector 43 (FIG. 8) so that it can be displayed.

  23, the viewpoint image (θ, φ) as camera image data captured from the viewpoint direction (θ, φ) of the viewpoint stored in the viewpoint storage unit 63 in the image selection unit 66 is displayed. ) And the viewpoint image conversion unit 67 arranges the viewpoint image (θ, φ) in the display image so that it can be displayed in the display area (θ, φ).

Now, for example, a two-dimensional coordinate system for the display surface of the inner omnidirectional projector 43 and the xy plane, as it is referred to as a display surface coordinate system, as shown in FIG. 26, the x and y coordinates, x _s representing the coordinates and y _s coordinates. A two-dimensional coordinate system in which the viewpoint image (θ, φ) is the xy plane is referred to as a viewpoint image coordinate system. As shown in FIG. 26, the origin O _s of the display surface coordinate system is the intersection of the optical axis of the inner omnidirectional projector 43 and the display surface, and the origin O of the viewpoint image coordinate system is a rectangular viewpoint image. It is assumed that the center is (θ, φ) (the intersection of diagonal lines of the rectangular viewpoint image (θ, φ)).

In order to display the viewpoint image (θ, φ) in the display area (θ, φ), first, as shown in FIG. 26, the viewpoint image (θ, φ) is set to the origin O _{s of the} display surface coordinate system. Therefore, it is necessary to arrange at a position separated by a distance g (φ) corresponding to the latitude φ. Further, in order to display the viewpoint image (θ, φ) in the display area (θ, φ), secondly, as shown in FIG. 27, the viewpoint image (θ, φ) is rotated counterclockwise at an angle. It is necessary to rotate by π / 2 + θ.

Accordingly, the viewpoint image conversion unit 67 first calculates the coordinates O (x _sO , y _sO ) in the display plane coordinate system of the origin O of the viewpoint image coordinate system of the visual image (θ, φ) according to the equation (8). Ask.

・・・（８）

... (8)

  Here, in the equation (8), the function g (φ) is a function representing the optical characteristics of the inner omnidirectional projector 43 (FIG. 8) (a function determined by the projection method of the fisheye lens of the inner omnidirectional projector 43). .

Further, the viewpoint image conversion unit 67 represents the coordinates in the viewpoint image coordinate system of the pixels Q constituting the visual image (θ, φ) as Q (x _Q , y _Q ) and the coordinates in the display surface coordinate system. , Q (x _sQ , y _sQ ), the coordinates Q (x _sQ , y _sQ ) in the display surface coordinate system of the pixel Q constituting the visual image (θ, φ) according to the equation (9). Ask for.

・・・（９）

... (9)

Then, the viewpoint image conversion unit 67 converts the pixel value of each pixel Q (x _Q , y _Q ) constituting the visual image (θ, φ) to the coordinates (x _sQ , y _sQ ) represented by the equation (9). The visual image (θ, φ) is converted into a display image by writing it at a position on the frame memory 68 corresponding to.

  Next, the processing (enclosed image processing) of the enclosed image processing unit 35 of FIG. 23 will be described with reference to the flowcharts of FIGS.

  The controller 69 monitors the frame memory 65. When the frame memory 65 starts storing the enclosed image data, the enclosed image processing is started.

  That is, in the enclosed image processing, first, in step S21, the controller 69 sets, for example, 1 as an initial value to the variable i for counting the number of frames, and proceeds to step S22.

  The process of setting 1 to the variable i in step S21 and the process of incrementing the variable i of step S53 described later are the process of setting 1 to the variable i in step S1 of FIG. This is performed in synchronization with the process of incrementing the variable i. That is, the wide-angle image processing in the wide-angle image processing unit 33 and the enclosed image processing in the enclosed image processing unit 35 are performed synchronously for the same frame. As a result, the immersive image and the bird's-eye image of the same frame are displayed synchronously on the wide-angle image display unit 34 and the enclosed image display unit 36.

  In step S22, the controller 69 sets, for example, 1 as an initial value to the variable n for counting the number of users, and the process proceeds to step S23. In step S <b> 23, the viewpoint processing unit 62 reads the viewpoint of the user #n from the viewpoint buffer 61 and registers it in the viewpoint storage unit 63.

  That is, the viewpoint detection unit 37 (FIG. 6) detects the viewpoints of all the users in the outer screen dome 42 (FIG. 7), for example, in the frame period, and the stored contents of the viewpoint buffer 61 according to the detected latest viewpoint. In step S23, the viewpoint processing unit 62 reads the viewpoint of the user #n stored in the viewpoint buffer 61 at the timing of the i-th frame and registers it in the viewpoint storage unit 63. Note that the viewpoint of the user #n can be stored in the viewpoint storage unit 63 together with an ID (Identification) representing the user #n, for example.

  After the viewpoint processing unit 62 registers the viewpoint of the user #n in the viewpoint storage unit 63, the process proceeds to step S24, and the controller 69 determines the total number of users whose variable n is in the outer screen dome 42 (FIG. 7). Determine if it is equal to N. If it is determined in step S23 that the variable n is not equal to N, the process proceeds to step S25, where the controller 69 increments the variable n by 1, returns to step S23, and thereafter repeats the same processing.

  If it is determined in step S24 that the variable n is equal to N, that is, if the viewpoints of all the users in the outer screen dome 42 (FIG. 7) are stored in the viewpoint storage unit 63, the process proceeds to step S26. Proceeding, the controller 69 sets the total number N of users in the outer screen dome 42 (FIG. 7) as an initial value to the variable M representing the total number of viewpoints to be processed by the viewpoint selection unit 66, and the step Proceed to S27.

  In step S27, the controller 69 sets, for example, 1 as the initial value to the variable m for counting the viewpoint, and the near-field viewpoint as the initial value is set in the flag flag indicating the presence / absence of the near-field viewpoint described later. For example, 0 is set, indicating no, and the process proceeds to step S28.

  In step S <b> 28, the viewpoint processing unit 62, with respect to the m-th viewpoint among the M viewpoints stored in the viewpoint storage unit 63, distances between the other M−1 viewpoints (inter-viewpoint distance). And a viewpoint whose distance between the viewpoints is equal to or less than (or less than) a predetermined threshold TH1 is detected as a short-distance viewpoint for the m-th viewpoint. In step S29, the viewpoint processing unit 62 determines whether there is a short-distance viewpoint for the m-th viewpoint.

  If it is determined in step S29 that there is no short-distance viewpoint for the m-th viewpoint, that is, viewpoints whose inter-viewpoint distance from the m-th viewpoint is equal to or less than the threshold TH1 are stored (registered) in the viewpoint storage unit 63. If not, steps S30 to S34 are skipped and the process proceeds to step S35.

  If it is determined in step S29 that there is a short-distance viewpoint for the m-th viewpoint, that is, a viewpoint (short-distance viewpoint) whose inter-viewpoint distance from the m-th viewpoint is equal to or less than a threshold TH1 is a viewpoint storage unit. If it is stored (registered) in 63, the process proceeds to step S30, and the controller 69 sets, for example, 1 indicating that there is a short-distance viewpoint in the flag flag, and proceeds to step S31.

  Here, the process of step S30 can be skipped when 1 is already set in the flag flag.

  In step S31, the viewpoint processing unit 62 detects the number of viewpoints stored in the viewpoint storage unit 63 as the short-distance viewpoint for the m-th viewpoint, and further, the short-distance viewpoint and the m-th viewpoint are detected. Detect the center of gravity of the viewpoint.

  In step S32, the viewpoint processing unit 62 registers (writes) the center of gravity obtained in step S31 as the mth viewpoint in the viewpoint storage unit 63 in an overwritten manner, and then proceeds to step S33. In step S33, the viewpoint processing unit 62 deletes the viewpoint stored in the viewpoint storage unit 63 as a short-distance viewpoint for the m-th viewpoint, and proceeds to step S34.

  In step S34, the controller 69 subtracts the number of short-distance viewpoints detected in step S31 from the variable M representing the total number of viewpoints to be processed by the viewpoint selection unit 66, and sets the subtraction value to the variable M. Set a new value and proceed to step S35. That is, in step S33, the short-distance viewpoints among the viewpoints stored in the viewpoint storage unit 63 are deleted. Therefore, in step S34, the value of the variable M is decreased by the number of deleted short-distance viewpoints. .

  Therefore, according to the processing of steps S28 to S34 surrounded by a dotted line in FIG. 28, one or more viewpoints close to the mth viewpoint are detected, and these one or more viewpoints and the mth viewpoint are It is integrated into the position of the center of gravity of the viewpoint.

  In step S <b> 35, the controller 69 determines whether the variable m is equal to the variable M, that is, the number of viewpoints M stored in the viewpoint storage unit 63. If it is determined in step S35 that the variable m is not equal to the variable M, the process proceeds to step S36, where the controller 69 increments the variable m by 1, returns to step S28, and the same processing is repeated thereafter.

  When it is determined in step S35 that the variable m is equal to the variable M, the process proceeds to step S37, and the controller 69 determines whether or not the flag flag is equal to 1. If it is determined in step S37 that the flag flag is equal to 1, that is, viewpoints stored in the viewpoint storage unit 63 include viewpoints integrated in the loop processing of steps S28 to S36 performed previously. If the viewpoints may still be integrated, the process returns to step S27, and the same processing is repeated thereafter.

  On the other hand, if it is determined in step S37 that the flag flag is not equal to 1, that is, the flag flag remains 0 set in step S27, and is in the viewpoint stored in the viewpoint storage unit 63. If there is no object whose distance from another viewpoint is equal to or less than the threshold value TH1, the process proceeds to step S41 in FIG.

In step S41, the controller 69 sets, for example, 1 as an initial value to the variable m for counting the viewpoint, and the process proceeds to step S42. In step S <b> 42, the image selection unit 66 calculates the distance between the mth viewpoint stored in the viewpoint storage unit 66 and the camera viewpoint of each enclosed camera 22 _k constituting the enclosure imaging unit 12 as a camera distance.

Here, the camera viewpoint of the enclosure camera 22 _k is obtained by normalizing (dividing) the camera position multiplexed in the enclosure image data by a predetermined normalization coefficient β. As the normalization coefficient β, for example, a value obtained by dividing the distance between the farthest viewpoint that can be taken by the user in the enclosed image display coordinate system and the origin by the distance between the enclosed camera 22 _k farthest from the subject and the subject is used. be able to.

After the processing of step S42, the process proceeds to step S43, and the image selection unit 66 obtains the minimum value _Dmin among the camera distances obtained for the mth viewpoint in step S42 as the minimum camera distance, and proceeds to step S44. . In step S44, the image selection unit 66 determines whether or not the minimum camera distance _Dmin for the mth viewpoint is equal to or less than (or less than) a predetermined threshold value TH2.

If it is determined in step S44 that the minimum camera distance _Dmin for the mth viewpoint is equal to or smaller than the predetermined threshold TH2, that is, the mth viewpoint and the camera viewpoint that gives the minimum camera distance _Dmin are close to each other. Accordingly, when the camera viewpoint can be regarded as the m-th viewpoint, the process proceeds to step S45, and the image selection unit 66 gives the minimum camera distance D _min from the enclosed image data stored in the frame memory 65. the camera image of the i-th frame that is captured by the enclosure camera 22 _k of the camera viewpoint is selected as the viewpoint image for the m-th viewpoint, and supplies the viewpoint image converting unit 67, the process proceeds to step S49.

If it is determined in step S44 that the minimum camera distance _Dmin for the mth viewpoint is not less than or equal to the predetermined threshold TH2, that is, the mth viewpoint is close to the camera viewpoint that gives the minimum camera distance _Dmin. However, if the camera viewpoint cannot be regarded as the m-th viewpoint, the process proceeds to step S47, and the image selection unit 66 sets the second camera distance obtained for the m-th viewpoint. The small frame is detected as the second camera distance D ₂ , and the i-th frame captured by the camera camera 22 _k of the camera viewpoint that gives the minimum camera distance D _min from the enclosed image data stored in the frame memory 65. and the camera image, second camera image of the i-th frame that is captured by the enclosure camera 22 k _'of the camera viewpoint to provide the second camera distance D ₂ Tsunoka To select the La image. Further, in step S47, the image selecting section 66, the camera viewpoint to provide a camera image of the i-th frame that is captured by the enclosure camera 22 _k of the camera viewpoint which gives the minimum camera distance D _min, the second camera distance D ₂ The camera image of the i-th frame captured by the enclosure camera 22 _{k ′} is combined at a ratio D ₂ : D _min between the second camera distance D ₂ and the minimum camera distance D _min to generate a combined image.

That is, the c th pixel constituting the camera image of the i-th frame captured by the minimum camera distance D _min enclosure camera 22 _k of the camera viewpoint giving, with represented by p _c, gives the second camera distance D ₂ Assuming that the c-th pixel constituting the camera image of the i-th frame captured by the camera 22 _{k ′} captured by the camera viewpoint is represented by q _c , the image selection unit 66 sets the pixels _pc and q _c to the formula ( D ₂ × p _c + D _min × q _c ) / (D ₂ + D _min ) is combined to generate a combined pixel, and a combined image composed of such combined pixels is generated.

  Here, the composite image is generated using two camera images, but the composite image can be generated using three or more other camera images.

  Thereafter, the process proceeds to step S48, and the image selection unit 66 selects the composite image generated in step S47 as the viewpoint image for the mth viewpoint, supplies the selected image to the viewpoint image conversion unit 67, and proceeds to step S49.

  In step S49, the viewpoint image for the mth viewpoint is converted into a display image.

  That is, in step S 49, the viewpoint direction calculation unit 64 calculates the viewpoint direction (θ, φ) of the mth viewpoint stored in the viewpoint storage unit 63 and supplies it to the viewpoint image conversion unit 67. The viewpoint image conversion unit 67 is based on the viewpoint direction (θ, φ) of the mth viewpoint supplied from the viewpoint direction calculation unit 64, and each pixel of the viewpoint image for the mth viewpoint supplied from the image selection unit 66. The position of the display image that should be placed is calculated by equations (8) and (9). Then, the viewpoint image conversion unit 67 arranges (writes) the pixels of the viewpoint image for the mth viewpoint at the position (address) on the frame memory 68 corresponding to the position of the display image thus obtained. The process proceeds to step S50.

  Here, the light corresponding to the display image written in the frame memory 68 as described above is emitted from the inner omnidirectional projector 43, so that the inner dome screen 44 is viewed from the mth viewpoint. In this case, an image when the subject is viewed from the viewpoint is displayed.

  In step S <b> 50, the controller 69 determines whether the variable m is equal to the total number M of viewpoints stored in the viewpoint storage unit 63. If it is determined in step S50 that the variable m is not equal to M, the process proceeds to step S51, where the controller 69 increments the variable m by 1, returns to step S42, and thereafter repeats the same processing.

  If it is determined in step S50 that the variable m is equal to M, that is, viewpoint images for all viewpoints stored in the viewpoint storage unit 63 are converted into display images and written to the frame memory 68. In this case, the inner omnidirectional projector 43 (FIG. 8) uses the display image stored in the frame memory 68 as the image of the i-th frame and emits light corresponding to the image. As a result, when viewed from each of the first to Mth viewpoints, the bird's-eye view when viewing the subject from the viewpoint is displayed at a position corresponding to each viewpoint on the inner dome screen 44.

  Thereafter, the process proceeds to step S52, and the controller 69 refers to the frame memory 65 to determine whether or not the i-th frame is the last frame. If it is determined in step S52 that the i-th frame is not the last frame, that is, if frame image data of a frame after the i-th frame is stored in the frame memory 65, the process proceeds to step S53, and the controller 69 Increments the variable i by 1 and returns to step S22 in FIG. 28, and thereafter the same processing is repeated.

  If it is determined in step S52 that the i-th frame is the last frame, that is, if no frame image data of a frame after the i-th frame is stored in the frame memory 65, frame image processing is performed. finish.

  As described above, in the display system 2, the immersive image of the i-th frame is displayed on the inner surface of the outer dome screen 42 and the bird's-eye image of the i-th frame is displayed on the outer surface of the inner dome screen 44. Therefore, the user can enjoy the immersive feeling provided by the immersive image and can easily grasp his / her position from the bird's-eye view image.

Therefore, when the imaging / display system of FIG. 1 is used for imaging and displaying a sports program, for example, the enclosing camera 22 _k (FIG. 4) of the imaging system 1 is set so as to surround the stadium where the sport is performed. By arranging a large number of images continuously and arranging the omnidirectional camera 21 (FIG. 3) in the center of the stadium to capture an image, the display system 2 can display an image full of realism.

  In addition, since the bird's-eye view image corresponding to the user's viewpoint is displayed on the inner dome screen 44 of the display system 2 at a position corresponding to the user's viewpoint, the user moves the viewpoint of the bird's-eye image presented to the user. It varies depending on the situation. Therefore, the user can perform so-called peeping, and can view an image of a subject hidden from a certain viewpoint by moving the viewpoint. Furthermore, it is possible to present a bird's-eye view of a subject viewed from the viewpoint of each user to a plurality of users.

  Further, according to the display system 2, for example, distant fish and animals are displayed on the outer dome screen 42 (FIG. 7), and nearby fish and animals are displayed on the inner dome screen 44 (FIG. 8). In this way, a natural observation facility such as an aquarium or a zoo can be virtually realized.

  Further, in the display system 2, it is possible to display not an actually captured image but an artificially created image such as computer graphics. That is, the display system 2 can display an image of outer space created as computer graphics, an image of a micro structure such as a molecule, or the like. In this case, it is possible to provide an interface that allows the user to intuitively understand a structure that cannot be visually recognized normally.

  The display system 2 can also be applied to, for example, so-called video game image display.

  Next, a series of processing by the wide-angle image processing unit 33 and the enclosed image processing unit 35 in FIG. 6 can be performed by hardware or can be performed by software. When a series of processing is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

  Therefore, FIG. 30 illustrates a configuration example of an embodiment of a computer in which a program for executing the series of processes described above is installed.

  The program can be recorded in advance in a hard disk 105 or a ROM 103 as a recording medium built in the computer.

  Alternatively, the program is stored temporarily on a removable recording medium 111 such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored permanently (recorded). Such a removable recording medium 111 can be provided as so-called package software.

  The program is installed in the computer from the removable recording medium 111 as described above, or transferred from the download site to the computer wirelessly via a digital satellite broadcasting artificial satellite, LAN (Local Area Network), The program can be transferred to a computer via a network such as the Internet, and the computer can receive the program transferred in this way by the communication unit 108 and install it in the built-in hard disk 105.

  The computer includes a CPU (Central Processing Unit) 102. An input / output interface 110 is connected to the CPU 102 via the bus 101, and the CPU 102 operates an input unit 107 including a keyboard, a mouse, a microphone, and the like by the user via the input / output interface 110. When a command is input as a result, the program stored in a ROM (Read Only Memory) 103 is executed accordingly. Alternatively, the CPU 102 also transfers from a program stored in the hard disk 105, a program transferred from a satellite or a network, received by the communication unit 108 and installed in the hard disk 105, or a removable recording medium 111 attached to the drive 109. The program read and installed in the hard disk 105 is loaded into a RAM (Random Access Memory) 104 and executed. Thus, the CPU 102 performs processing according to the above-described flowchart or processing performed by the configuration of the above-described block diagram. Then, the CPU 102 outputs the processing result from the output unit 106 configured with an LCD (Liquid Crystal Display), a speaker, or the like, for example, via the input / output interface 110, or from the communication unit 108 as necessary. Transmission and further recording on the hard disk 105 are performed.

  Here, in this specification, the processing steps for describing a program for causing a computer to perform various types of processing do not necessarily have to be processed in time series according to the order described in the flowchart, but in parallel or individually. This includes processing to be executed (for example, parallel processing or processing by an object).

  Further, the program may be processed by a single computer, or may be processed in a distributed manner by a plurality of computers. Furthermore, the program may be transferred to a remote computer and executed.

  In the display system 2, it is possible to display an image recorded on the recording medium 3 (FIG. 1) offline, so that an image transmitted via the transmission medium 4 is displayed in real time. It is also possible.

Further, as described above, the inner omnidirectional projector 43 (FIG. 8) emits a light beam corresponding to the display image, and the light beam is projected back onto the inner dome screen 44 (FIG. 8). Is displayed, the left and right are reversed in the viewpoint image. In order to prevent the left-right reversal of the viewpoint image displayed on the inner dome screen 44, for example, it is necessary to invert the left and right of the camera image that is the source of the viewpoint image in advance. For example, the image selection unit 66 (FIG. 23) of the enclosure image processing unit 35 can perform the processing. Alternatively, the left / right inversion of the camera image can be optically performed when the camera image is captured by the enclosure camera 22 _k (FIG. 4) of the imaging system 1, for example.

  Furthermore, in the present embodiment, the inner omnidirectional projector 43 (FIG. 8) emits a light beam corresponding to the display image, and the light beam is projected back onto the inner dome screen 44 (FIG. 8). In this case, it is necessary to generate a display image in consideration of the shape of the inner dome screen 44 (FIG. 8). That is, since the viewpoint image has a rectangular shape, the rectangular viewpoint image is arranged at a predetermined position of the display image, and the light corresponding to the display image is emitted from the inner omnidirectional projector 43. Thus, when the viewpoint image is displayed on the inner dome screen 44, the rectangular viewpoint image is projected onto the spherical surface of the hemispheric inner dome screen 44, so that the viewpoint image displayed on the inner dome screen 44 is distorted. It will be. Therefore, in the viewpoint image conversion unit 67 (FIG. 23), the viewpoint image arranged in the display image based on the shape of the inner dome screen 44 so that the viewpoint image is displayed on the inner dome screen 44 in a rectangular shape without distortion. Need to be converted.

  However, in the present embodiment, the radius of the inner dome screen 44 (FIG. 8) is large to some extent, and the range in which the viewpoint image is displayed can be regarded as a plane. In this case, conversion of the viewpoint image arranged in the display image based on the shape of the inner dome screen 44 may or may not be performed.

  In the present embodiment, only one enclosure imaging unit 12 (FIG. 2) is provided in the imaging system 1, but a plurality of enclosure imaging units 12 can be provided. In this case, the plurality of enclosure imaging units 12 capture a plurality of subjects that are spatially or temporally separated, and the display system 2 processes the images of the plurality of subjects as a camera image. It is possible to do so.

  Furthermore, the wide-angle image processing unit 33, the enclosed image processing unit 35, and the viewpoint detection unit 37 of FIG. 6 can be provided in the imaging system 1 (FIG. 2) instead of being provided in the display system 1.

It is a figure which shows the structural example of one Embodiment of the imaging / display system to which this invention is applied. 1 is a block diagram illustrating a configuration example of an imaging system 1. FIG. 2 is a perspective view illustrating a configuration example of a wide-angle imaging unit 11. FIG. 4 is a perspective view illustrating a configuration example of an enclosure imaging unit 12. FIG. It is a diagram illustrating the positional relationship of the omnidirectional camera 21 and the surrounding-camera 22 _k. 3 is a block diagram illustrating a configuration example of a display system 2. FIG. 4 is a perspective view illustrating a configuration example of a wide-angle image display unit 34. FIG. 5 is a perspective view illustrating a configuration example of an enclosed image display unit 36. FIG. It is a figure explaining the positional relationship of the outer side dome screen 42 and the inner side dome screen 44. FIG. It is a figure which shows a mode that several users are viewing an image. It is a figure which shows a mode that the image of the to-be-photographed object seen from each user's viewpoint is shown to several users. It is a figure which shows a left-right parallax system. It is a figure which shows an up-down-left-right parallax system. 3 is a block diagram illustrating a configuration example of a wide-angle image processing unit 33. FIG. It is a figure which shows a wide-angle image imaging coordinate system. It is a figure which shows an omnidirectional image. It is a figure which shows an omnidirectional image and the latitude longitude image. It is a flowchart explaining wide angle image processing. It is a figure explaining the image displayed on the inner dome screen. It is a figure which shows the image displayed on the display surface of the omnidirectional projector 43 for inner side, and the image displayed on the inner dome screen 44. FIG. It is a figure which shows a mode that the viewpoint of the users A and B is separated. It is a figure which shows the state from which the viewpoint of the users A and B is near. 3 is a block diagram illustrating a configuration example of a surrounding image processing unit 35. FIG. It is a figure explaining the process of the viewpoint direction calculation part. It is a figure explaining the process of the display image generation part. It is a figure explaining the process of the display image generation part. It is a figure explaining the process of the display image generation part. It is a flowchart explaining a surrounding image process. It is a flowchart explaining a surrounding image process. It is a block diagram which shows the structural example of one Embodiment of the computer to which this invention is applied.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Imaging system, 2 Display system, 3 Recording medium, 4 Transmission medium, 11 Wide-angle imaging part, 12 Enclosure imaging part, 13 Multiplexer, 14 Recording / transmission part, 21 Omnidirectional camera, 22k Enclosure camera, 31 Playback / reception part, 32 Demultiplexer, 33 Wide-angle image processing unit, 34 Wide-angle image display unit, 35 Enclosed image processing unit, 36 Enclosed image display unit, 37 Viewpoint detection unit, 41 Outside omnidirectional projector, 42 Outside dome screen, 43 Inside omnidirectional Projector, 44 inner dome screen, 51 frame memory, 52 wide angle image conversion unit, 53 angle of view correction unit, 54 latitude / longitude image conversion unit, 55 frame memory, 56 controller, 61 viewpoint buffer, 62 viewpoint processing unit, 63 viewpoint description Unit, 64 viewpoint direction calculation unit, 65 frame memory, 66 image selection unit, 67 viewpoint image conversion unit, 68 frame memory, 69 controller, 70 display image generation unit, 101 bus, 102 CPU, 103 ROM, 104 RAM, 105 Hard disk, 106 output unit, 107 input unit, 108 communication unit, 109 drive, 110 input / output interface, 111 removable recording medium

Claims (10)

In an image processing device that generates a light emission image corresponding to a light beam emitted from a light emitting unit that emits a light beam toward a display unit that displays an image by processing images picked up from a plurality of positions.
Image selecting means for selecting a viewpoint image when a predetermined subject is viewed from the user's viewpoint based on the user's viewpoint from images captured from each of the plurality of positions;
An image processing apparatus comprising: viewpoint image conversion means for converting the viewpoint image into the light-emitting image based on the viewpoint of the user and the shape of the display means. The image processing apparatus according to claim 1, further comprising a viewpoint detection unit that detects a viewpoint of the user. The image processing apparatus according to claim 1, further comprising the plurality of imaging units. The image selection means selects the viewpoint image for each of a plurality of user viewpoints,
The viewpoint image conversion means converts the viewpoint image for each of the viewpoints of the plurality of users to generate one emission image,
The display means displays an image when a predetermined subject is viewed from each of the viewpoints of the plurality of users by displaying the light emission image corresponding to the light beam emitted by the light emission means. The image processing apparatus according to 1. And further comprising an integration means for integrating the viewpoints of a plurality of users into one viewpoint,
The image selecting unit selects the viewpoint image for each of the viewpoints of the plurality of users from images captured by the plurality of imaging units, and the viewpoint for the one viewpoint is selected from the viewpoint images. Generate an image,
The image processing apparatus according to claim 1, wherein the viewpoint image conversion unit converts the viewpoint image for the one viewpoint into the light emission image. The image selection unit weights and adds the viewpoint images for the viewpoints of the plurality of users according to the distance between each of the viewpoints of the plurality of users and the one viewpoint. The image processing apparatus according to claim 5, wherein the viewpoint image is generated. The image processing apparatus according to claim 5, wherein the integration unit integrates the viewpoints of a plurality of users into a position of the center of gravity of the viewpoints of the plurality of users. A viewpoint distance calculating means for calculating a viewpoint distance between the viewpoints of the plurality of users;
The image processing apparatus according to claim 5, wherein the integration unit integrates the viewpoints of the plurality of users whose viewpoint distance is equal to or less than a predetermined distance into one viewpoint. In an image processing method for generating a light emission image corresponding to a light beam to be emitted to a light emitting unit that emits a light beam toward a display unit that displays an image by processing images picked up from a plurality of positions,
An image selection step of selecting a viewpoint image when a predetermined subject is viewed from the user's viewpoint based on the user's viewpoint from images captured from each of the plurality of positions;
An image processing method comprising: a viewpoint image conversion step of converting the viewpoint image into the light-emitting image based on the viewpoint of the user and the shape of the display means. In a program for causing a computer to perform image processing for generating a light emission image corresponding to a light beam emitted from a light emitting unit that emits a light beam toward a display unit that displays an image by processing images picked up from a plurality of positions ,
An image selection step of selecting a viewpoint image when a predetermined subject is viewed from the user's viewpoint based on the user's viewpoint from images captured from each of the plurality of positions;
A program comprising: a viewpoint image conversion step of converting the viewpoint image into the light-emitting image based on the viewpoint of the user and the shape of the display means.

Images