JP2006352539A - Wide-field video system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enhance presence by presenting video information also to peripheral parts of a visual field in addition to presenting a video image to the center part of a human visual field. <P>SOLUTION: At a stage of photographing the video image to be presented at the center part of the visual field, video images to be presented at the peripheral parts of the visual field are simultaneously acquired. Namely, a scene is photographed from various positions and directions by a camera array at a stage of photographing the video image, the video image to be presented to the center part of the visual field and the video image to be presented at the peripheral part of the visual field are simultaneously synthesized using optional viewpoint video technology to generate a wide-field video image. As an example, an object 133 is captured at the center by turning a virtual camera to a valid visual field 127 when a video image to be presented to a video display device is generated. A valid field video image to be presented to the valid visual field 127 is obtained by synthesizing a video image in the virtual camera 135 by the optional video technology from video images of many cameras 134. Similarly, peripheral video images are obtained by turning the virtual camera 135 to induced visual fields 128, 129 and auxiliary visual fields 130, 131. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a wide-field video system, and more particularly to a video data generation technique in a video display system that enhances the sense of reality by presenting wide-field video information.

  According to the research by Hatada et al. (Non-patent Document 1), as shown in FIG. 1, the human visual field is classified into a discrimination visual field 101, an effective visual field 102, a guidance visual field 103, and an auxiliary visual field 104 according to the function of the visual function. The discrimination visual field 101 is a range in which high-density information such as graphic identification can be accurately received. The effective visual field 102 is a range in which natural information can be received only by eye movement although the discrimination ability is lower than the discrimination visual field 101. The guidance visual field 103 has only a recognition ability to the extent that the present stimulus is present and can be easily identified, but has a range influencing when judging overall external information. The auxiliary visual field 104 is a range in which only the presence of a stimulus can be determined.

  The current high-definition television is designed to present an image (this image is referred to as an effective visual field image in this specification) in a range covering the effective visual field 102 among the above. That is, no image is presented in the guiding visual field 103 and the auxiliary visual field 104. By presenting a video (in this specification, the video presented in the guidance visual field 103 or the auxiliary visual field 104 in this specification is also referred to as the peripheral video) in the peripheral part of the visual field such as the guidance visual field 103 or the auxiliary visual field 104 It can be expected to increase.

As the peripheral video, for example, a video having the same resolution as the effective visual field video, a video with a lower resolution, or an image using illumination as an ultra-low resolution video is considered. Examples of using the illumination in this way include, for example, Patent Document 1 and a technology called Ambright installed in a Philips television. Note that in this specification, video information with a reduced resolution in order to be presented in the auxiliary visual field or the guidance visual field using the lighting device is referred to as a simple peripheral video. An image covering the entire human visual field, which is a combination of the effective visual field video and the peripheral video or the simple peripheral video, is referred to as a wide-field video.
JP 2000-173783 A Toyohiko Hatada, Haruo Sakata, Hideo Kusaka: “Directional sensation induction effect by screen size-Basic experiment of presence by a large screen-” Television Society Journal, Vol. 33, no. 5, pp. 407-413 (1979) Kitahara, Sato, Ota, “3D image display with motion parallax reproducible using multi-eye stereo method-Generation and evaluation of display image-”, Television Society Journal, Vol. 50, no. 9, (1996) Primm, Fujii, Tanimoto: "Real-time system for free viewpoint TV", IEICE Technical Report, IE2002-120 (2002) Higashi, Fujii, Tanimoto: “Detection of viewer's head for free-viewpoint television”, Information Processing Society of Japan Report, 2003-HI-104 (5) (2003)

  However, in Patent Document 1 and Philips Ambright, only an effective visual field image is assumed as an input signal, and an illumination control signal is estimated and generated from the effective visual field image. Therefore, there is a difference from an actual simple peripheral image, and it is difficult to say that effective lighting can be presented to increase the sense of reality. At the stage of capturing an effective visual field image, it is sufficient to simultaneously acquire image information necessary for generating a peripheral image or a simple peripheral image, but such a system does not exist.

  In order to solve the above-mentioned problems, the present invention simultaneously acquires video information necessary for generating a peripheral video or a simple peripheral video at the stage of taking an effective visual field video, It is possible to present a simple peripheral video.

  By the way, there is what is called arbitrary viewpoint video technology. This is to shoot a scene using a number of cameras and combine and process the video data to synthesize the video as if it were shot with a virtual camera at a point where no camera originally existed. Technology. Using this arbitrary viewpoint video technology, it is a feature of the present invention that a peripheral video or a simple peripheral video is generated simultaneously with an effective visual field video, that is, a wide-field video is generated.

  In order to solve the above-described problem, the first technical means of the present invention includes a video generation unit that synthesizes a wide-field video composed of one or more video data from a video shot by one or more cameras. It is a feature.

  According to a second technical means, in the first technical means, as a method of synthesizing a wide-field video composed of one or more video data in the video generation unit, a virtual camera that originally does not have a camera is generated from the video by one or more cameras. It uses an arbitrary viewpoint video technology that can synthesize video at various camera positions.

  According to a third technical means, in the second technical means, the video generation unit generates an arbitrary viewpoint video from a viewpoint separated by an interval between the left and right eyes, thereby generating a stereoscopically wide-view video. It is a feature.

  The fourth technical means includes a sensor capable of detecting the viewer's head position in the second or third technical means, changes the virtual camera position following the movement of the viewer's head, A wide-field image including one or more video data at a camera position is generated.

  According to a fifth technical means, in any one of the second to fourth technical means, the broadcasting station sends all video information photographed by the one or more cameras to the viewer side, and the viewer side It is characterized in that it receives video information and uses it to generate a wide-field video by an arbitrary viewpoint video generation technique.

  The sixth technical means is any one of the second to fourth technical means for generating a wide-field video from video information photographed by one or more cameras on the broadcasting station side by an arbitrary viewpoint video generation technique. It is characterized by sending visual field images to viewers.

  According to a seventh technical means, in any one of the second to fourth technical means, the video content producer side records all video information photographed by one or more cameras on a recording medium, and the viewer side It is characterized in that video information is reproduced from a recording medium and a wide-field video is generated using an arbitrary viewpoint video generation technique using the video information.

  According to an eighth technical means, in any one of the second to fourth technical means, a video field producer generates a wide-field video from video information photographed by one or more cameras using an arbitrary viewpoint video generation technique. Thus, the image is recorded on a recording medium, and a wide-field video is reproduced from the recording medium on the viewer side.

  A ninth technical means is any one of the first to eighth technical means, comprising one or more display units, and presenting a wide-field image on the display unit.

  A tenth technical means is the same as the ninth technical means, wherein a display device having a plurality of pixels is used as the display unit.

  The eleventh technical means is the ninth technical means characterized in that an illumination device is used as the display unit.

  A twelfth technical means is characterized in that, in the ninth technical means, a display device having a plurality of pixels and a lighting device are mixedly used as the display unit.

  A thirteenth technical means is that, in the tenth technical means or the twelfth technical means, the video generation unit occupies the viewer's visual field when viewed from the resolution of each of the one or more display units or the viewer's viewpoint position. The video data is generated according to the area.

  The fourteenth technical means is the information from the optical sensor that enables the video generation unit to acquire information on the luminance, saturation, and hue of an angular region in a certain direction in any one of the first to thirteenth technical means. And a wide-field video using the video information captured by one or more cameras.

According to the present invention, the following effects can be obtained.
By using the arbitrary viewpoint video technology, it is possible to generate a wide-field video from video data from the camera array, and to provide a video with a high sense of presence to the viewer.
Arbitrary viewpoint video technology allows video to be synthesized for a field of view of any size and direction, so that video can be generated in accordance with the field of view covered by each of one or more display units that make up a wide-field video display device. Is possible. Therefore, any configuration of a wide-field video display device can be used.

  Furthermore, according to the present invention, two wide-field images from a viewpoint separated by the distance between the left and right eyes can be generated by an arbitrary viewpoint image technology, so that it is possible to provide a wide-field image that can be stereoscopically viewed. .

  In addition, according to the present invention, it is also possible to generate a wide-field image by changing the viewpoint position so as to follow the movement of the head position in combination with a sensor capable of measuring the viewer's head position. This makes it possible to reproduce so-called moving parallax, in which the image changes as the head moves, resulting in a more realistic image. In addition, it can be combined with a method for generating a stereoscopically viewable wide-field image, and an image with higher presence can be provided.

  In addition, when the present invention is applied to broadcasting and a configuration is adopted in which arbitrary viewpoint video generation processing is performed in the receiving device, the viewpoint of the wide-field video can be freely changed on the viewer side, so that it is more interactive and more realistic. You can see a high picture.

  In addition, when the present invention is applied to broadcasting and a configuration in which arbitrary viewpoint video generation processing is performed in advance before transmission, it is not necessary to send all the images of the camera array to the viewer side, and a wide-field video display device is configured. Since it is only necessary to send as many videos as the number of one or more display units, the amount of transmission data can be reduced.

  In addition, when the present invention is applied to package media and a configuration is adopted in which arbitrary viewpoint video generation processing is performed in the playback device, the viewpoint of the wide-field video can be freely changed on the viewer side, so that it is more interactive and more realistic. You can see high-quality images.

  In addition, when the present invention is applied to a package medium and an arbitrary viewpoint video generation process is performed in advance before recording the content, it is not necessary to record all the images of the camera array on the package medium. Since it suffices to record as many videos as the number of one or more display units constituting the recording unit, the amount of recording data can be reduced.

  Depending on the number and arrangement of cameras in the camera array, if the video information necessary to generate the effective visual field image and the peripheral image or the simple peripheral image cannot be obtained, the entire camera can be By using an omnidirectional camera that can shoot video and an optical sensor that can acquire brightness, saturation, and hue information in an angle area in a certain direction, an effective visual field image and peripheral video or simple peripheral video can be generated. It can be used as auxiliary information for generating a wide-field image with no gaps.

Examples of the present invention will be described below.
First, some examples of the wide-field video system in the present invention will be shown. Each of these is a system that presents video information with a wide field of view so as to cover the viewer's field of view, but this is an example in which the size, number, type, and arrangement of the display units are different. In this specification, an apparatus for presenting video information is widely referred to as a “display unit”, and among them, a device having a plurality of display pixels is referred to as a “video display apparatus”. On the other hand, an “illumination device” that can control the hue, saturation, and luminance of illumination light is regarded as a display unit that displays a single pixel. The lighting device will be described in detail later.

  FIG. 2 shows a first example of a wide-field video system according to the present invention. It shows a state where a room in which three flat image display devices are installed is viewed from the rear. Here, a video display device 105, a video display device 106, and a video display device 107 are installed in a room, and a state in which a viewer 108 sitting on a chair is viewing it is shown. The video display device 105 is installed on the wall surface in front of the viewer 108, the video display device 106 is installed on the left wall surface of the viewer 108, and the video display device 107 is installed on the right wall surface of the viewer 108. These video display devices 105 to 107 are realized by projection by a projector, for example.

  FIG. 3 is a top view of the first example of the wide-field video system. A state in which the video display device 105, the video display device 106, and the video display device 107 are installed so as to cover the visual field of the viewer 108 is shown.

  FIG. 4 shows a second example of the wide-field video system in the present invention. It shows a state where a room in which five flat image display devices are installed is viewed from the rear. Here, a video display device 109, a video display device 110, a video display device 111, a video display device 112, and a video display device 113 are installed in a room, and are viewed by a viewer 108 sitting on a chair. The situation is shown. The video display device 109 is on the wall in front of the viewer 108, the video display device 110 is diagonally forward left of the viewer 108, the video display device 111 is diagonally forward right of the viewer 108, and the video display device 112 is viewer 108. The video display device 113 is installed on the right wall surface of the viewer 108 on the left wall surface. These video display devices 109 to 113 are realized by, for example, a cathode ray tube television, a liquid crystal television, a plasma television, a rear projection television, a projection screen by a projector, or the like.

  FIG. 5 is a top view of the second example of the wide-field video system. A state in which the video display device 109 to the video display device 113 are installed so as to cover the visual field of the viewer 108 is shown.

  FIG. 6 shows a third example of the wide-field video system in the present invention. A state in which a room where an image display device 114 using an arched screen shaped like a part of a cylinder is installed is viewed from the rear is shown. The video display device 114 is installed in the room, and the viewer 108 who is sitting on the chair looks at it. The video display device 114 is realized by projection by a plurality of projectors, for example.

  FIG. 7 is a top view of the third example of the wide-field video system. A state in which a video display device 114 using an arched screen is installed so as to cover the visual field of the viewer 108 is shown.

  FIG. 8 shows a fourth example of the wide-field video system in the present invention. A state in which a room in which a video display device 115 using a dome-shaped screen shaped like a part of a spherical surface is installed is viewed from the rear is shown. A video display device 115 is installed in the room, and the viewer 108 sitting on a chair looks at it. The video display device 115 is realized by projection by a plurality of projectors, for example.

  FIG. 9 is a top view of the fourth example of the wide-field video system. A state in which a video display device 115 using a dome type screen is installed so as to cover the visual field of the viewer 108 is shown.

  FIG. 10 shows a fifth example of the wide-field video system in the present invention. In addition to a flat image display device, a view of a room where an illumination device is installed is shown from the rear. A video display device 116, a lighting device 117, a lighting device 118, a lighting device 119, a lighting device 120, a lighting device 121, a lighting device 122, and a lighting device 123 are installed in the room. Note that the viewer is not shown in FIG. This is because the video display device 116 is hidden by the viewer's drawing. The video display device 116 is realized by, for example, a cathode ray tube television, a liquid crystal television, a plasma television, a rear projection television, a projection screen by a projector, or the like. The illuminating devices 117 to 123 are realized by an illuminating device using, for example, RGB three-color LEDs.

  FIG. 11 is a top view of the fifth example of the wide-field video system. It is shown that the video display device 116 is installed in front of the viewer 108 and the lighting devices 117 to 123 are installed so as to cover the guidance field and auxiliary field of view of the viewer 108.

  Here, the illumination devices 117 to 123 are used as a very low-resolution video display device, and are controlled according to the input video signal. FIG. 10 shows an example in which the lower wall surface of each lighting device is illuminated, but this is merely an example. Various other lighting devices such as those that are installed on the lower side and illuminate the upper wall surface of the lighting device, those that illuminate the wall surface around the lighting device, and those that illuminate the lighting device itself can be considered. Or may be used in combination. Further, the arrangement of the illumination device is an example, and the present invention is not limited to this. For example, since the visual resolution of the human being is higher in the guided visual field than in the auxiliary visual field, the number of lighting devices installed in the guiding visual field may be larger than the number of lighting devices installed in the auxiliary visual field. In addition, for ease of explanation, the field of view is divided into several parts in the horizontal direction and illuminated. However, it is also possible to divide the field of view in the vertical direction and install an illumination device for each. These lighting devices may use various types of lighting devices such as those that can control the hue, saturation, and luminance of illumination light in addition to those that can simply change the luminance of the illumination.

Thus, in the present invention, not only a video display device such as a liquid crystal television but also a lighting device is a video display device, and is treated as a device having different displayable video resolutions.
Further, as described above, among the human visual field, the guidance visual field and the auxiliary visual field have low discrimination ability, and there is only a recognition ability that allows the presence of a stimulus and simple identification. Therefore, in the peripheral part of the visual field, the effect of enhancing the sense of reality can be expected only by presenting the color stimulus by the lighting device.

  FIG. 12 shows a sixth example of the wide-field video system in the present invention. A video display device 109, a video display device 110, a video display device 111, a lighting device 122, and a lighting device 123 are installed in the room, and the viewer 108 sitting on a chair is shown. Yes. This example is a combination of the second example shown in FIG. 4 and the fifth example shown in FIG. The video display device 109 is on the front wall of the viewer 108, the video display device 110 is on the left front of the viewer 108, the video display device 111 is on the front right of the viewer 108, and the lighting device 122 is on the viewer 108. The lighting device 123 is installed on the left wall surface of the viewer 108 on the left wall surface.

  FIG. 13 is a top view of the sixth example of the wide-field video system. The video display devices 109 to 111 are installed in front of the viewer 108 or in front of the left and right, and the lighting devices 122 to 123 are installed in the peripheral portion of the viewer's 108 field of view. The second example shown in FIG. 5 and the fifth example shown in FIG. 11 are combined. As described above, the video display device and the lighting device may be mixedly used.

  In the above example shown in FIG. 2 to FIG. 13, the display unit is not installed because the back side of the viewer 108 does not enter the field of view, but on the back side in consideration of the viewer 108 looking back. Of course, a display unit may be installed.

  As described above, even if it is a wide-field video system, a combination of video display devices and lighting devices of various types, shapes, and arrangements can be considered as specific enemies.

Next, video data to be presented in the above wide-field video system and its photographing device will be described.
The video currently provided by package media such as TV broadcasts and DVDs is assumed to be displayed on one rectangular screen and presented to the viewer's effective field of view. It is. This does not include surrounding video or simple peripheral video. Therefore, in order to obtain video data to be presented in the wide-field video system shown in FIGS. 2 to 13, a peripheral image or a simple peripheral video is generated at the stage of capturing an effective visual field video using a different shooting means. Therefore, it is possible to simultaneously acquire necessary video information and present a correct peripheral video or a simple peripheral video in the video information presentation device. In the present invention, an arbitrary viewpoint video technique is used for this purpose.

  First, the arbitrary viewpoint video technology will be described. Arbitrary viewpoint video technology means that a scene is shot using a large number of cameras (camera arrays), and the video data is integrated and processed to place a virtual camera at a point where no camera originally exists. It is a technology that synthesizes video as if it were taken. An example of a camera array used in the arbitrary viewpoint video technology is schematically shown in FIG. A field 124 such as a sport is arranged in such a manner that a large number of cameras 125 surround the field 124 from various directions. Using the arbitrary viewpoint video technology, it is possible to synthesize the video of the virtual camera 126 from the video of many cameras 125.

For example, Non-Patent Document 2 (Kitahara, Sato, Ota, “3-D image display capable of reproducing motion parallax using multi-eye stereo method—generation and evaluation of display image”, Television Society Journal, Vol. 50, No. .9, (1996)), an image obtained by restoring an object shape model from an image of a camera array using a so-called computer vision technique such as three-dimensional shape estimation and modeling, and observing the model from an arbitrary viewpoint. Is generated by a three-dimensional CG technique to realize an arbitrary viewpoint video.
In a study described in Non-Patent Document 3 (Prim, Fujii, Tanimoto: “Real-time system for free viewpoint television”, IEICE Technical Report, IE2002-120 (2002)), a multidimensional image space called a light space is used. By interpolating between sparse data obtained by projecting images from a large number of cameras, and extracting appropriate data from a dense space, an image from an unphotographed viewpoint is generated.
By using this arbitrary viewpoint video technology, it is possible to generate a peripheral video or a simple peripheral video simultaneously with an effective visual field video.

  A method for generating an effective visual field image, a peripheral image, or a simple peripheral image using the arbitrary viewpoint image technology will be described in more detail below.

  FIG. 15 is a top view of the second example of the wide-field video system shown in FIG. Here, an effective visual field 127, a guiding visual field (left) 128, a guiding visual field (right) 129, an auxiliary visual field (left) 130, and an auxiliary visual field (right) 131 are shown in an overlapping manner, one for each visual field. Video display devices are compatible. The out-of-view 132 indicates a range outside the view on the back side of the viewer 108. Note that the arrangement of the video display devices in FIG. 15 is merely an example, and for ease of explanation, each video display device is installed so as to match the visual field classification in FIG. Actually, as shown in FIG. 15, it is not always necessary to install the video display device according to the visual field classification according to FIG.

  FIGS. 16, 17, and 18 are views showing the relationship between the subject, the camera array, and the virtual camera in the wide-field video system as viewed from above. A large number of cameras 134 are arranged so as to surround the subject 133. A number of cameras A to P are assigned to a large number of cameras 134. In addition, an effective visual field 127, a guiding visual field (left) 128, a guiding visual field (right) 129, an auxiliary visual field (left) 130, and an auxiliary visual field (right) 131 are shown in an overlapping manner. The out-of-view 132 indicates a range outside the view on the viewer's back side. Using the video from these cameras 134, the video from the virtual camera 135 is synthesized. The arrangement of the numerous cameras 134 is an example for explanation, and is not limited to this arrangement in practice. In addition, since it is a top view, it is illustrated as being arranged on a plane, but in practice it is of course possible to dispose the camera in a three-dimensional manner in the height direction.

  FIG. 16 shows a case where an image in the direction of the effective visual field 127, that is, an image to be presented on the image display device 109 in FIG. 15 is generated. The virtual camera 135 faces the effective visual field 127 and captures the subject 133 as a center. An effective visual field image to be presented in the effective visual field 127 is obtained by synthesizing the video image of the virtual camera 135 from the video images of a large number of cameras 134 using an arbitrary viewpoint video technology.

  FIG. 17 shows a case where an image in the direction of the guidance visual field (right) 129, that is, an image presented on the image display device 111 in FIG. 15 is generated. The virtual camera 135 faces the guidance visual field (right) 129. By synthesizing the video from the virtual camera 135 from the images of a large number of cameras 134 using the arbitrary viewpoint video technology, a peripheral video to be presented in the guidance visual field (right) 129 is obtained. The peripheral video presented in the guidance field (left) 128 can be obtained by performing the same processing.

  FIG. 18 shows a case where a video in the direction of the auxiliary visual field (right) 131, that is, a video to be presented on the video display device 113 in FIG. 15 is generated. The virtual camera 135 faces the auxiliary visual field (right) 131. By synthesizing the video from the virtual camera 135 by using the arbitrary viewpoint video technology from the videos of the many cameras 134, a peripheral video to be presented in the auxiliary visual field (right) 131 is obtained. The peripheral video presented in the auxiliary visual field (left) 130 can be obtained by performing the same processing.

  The effective visual field image and the peripheral image thus obtained are displayed on the image display devices 109 to 113 in FIG.

  FIG. 19 is a top view of the first example of the wide-field video system shown in FIG. 3 above, in which the state of the visual field division for generating the effective visual field video and the peripheral video is overlapped. The field of view is divided into three regions: a field of view (front) 136, a field of view (left) 137, and a field of view (right) 138 in accordance with the arrangement of the display unit. For each field of view, it is possible to generate an effective field image and a peripheral image by performing the same processing as the method described with reference to FIGS.

  FIG. 20 is a top view of the third example of the wide-field video system shown in FIG. 7 above, in which the state of the visual field division for generating the effective visual field video and the peripheral video is overlapped. In this case, since the display unit is a single curved screen, the entire field of view is handled as a single field of the field of view (entire surface) 139. It is possible to generate an effective visual field image and a peripheral image by performing the same processing as that described in FIGS. 16 to 18 on the visual field. This is equivalent to a process of combining so-called panoramic images.

  FIG. 21 is a top view of the fifth example of the wide-field video system shown in FIG. 11 above, in which the state of the visual field division for generating the effective visual field video and the peripheral video is superimposed. The visual field is divided into a visual field (1) 140 to a visual field (7) 146 in accordance with the arrangement of the video display device 116 and the lighting devices 117 to 123. The video to be displayed on the video display device 116 can be synthesized by performing the same processing as the method described with reference to FIGS. In addition, images to be presented to the lighting devices 117 to 123 are obtained by combining the peripheral images for the field of view corresponding to the respective lighting devices by the same processing as the method described with reference to FIGS. The average value of each of RGB of all the pixels is obtained, and this is obtained as a simple peripheral image for each lighting device. That is, in this specification, the lighting device is regarded as a display unit having only one display pixel, and an image obtained by reducing the resolution of the peripheral video to one pixel becomes a simple peripheral video.

  FIG. 22 is a top view of the sixth example of the wide-field video system shown in FIG. 13 above, in which the state of the visual field division for generating the effective visual field video and the peripheral video is superimposed. The visual field is divided into a visual field (8) 147 to a visual field (12) 151 in accordance with the arrangement of the video display devices 109 to 111 and the lighting devices 122 to 123. The video to be displayed on the video display devices 109 to 111 can be synthesized by performing the same processing as the method described with reference to FIGS. Further, the video to be presented to the lighting devices 122 to 123 can be synthesized by performing the same processing as the method described in FIG. As described above, even in an environment in which a video display device and a lighting device are mixed as a display unit, it is possible to generate an effective visual field video, a peripheral video, or a simple peripheral video according to each display unit by using arbitrary viewpoint video technology. It is possible to respond flexibly.

  In the description with reference to FIGS. 15 to 22 described above, the viewing environment is described based on the top view, so that the division of the visual field is shown only on the horizontal plane. However, in reality, the video display environment is a three-dimensional space. The display unit is arranged in a three-dimensional manner, and the visual field is divided and processed in a three-dimensional manner. The method of the present invention is of course applicable to processing in such a three-dimensional space.

  Further, since the out-of-view 132 is an area that does not normally enter the viewer's 108 view, it is possible to omit the presentation of the peripheral video or the simple peripheral video. However, since the viewer enters the field of view when the viewer looks back, a display unit may be installed in the region outside the field of view 132 to generate and present a peripheral video or a simple peripheral video.

  Also, depending on the number and arrangement of cameras in the camera array, there may be a case where video information necessary for generating an effective visual field video and a peripheral video or a simple peripheral video cannot be obtained. In order to cope with such cases, information about the brightness, saturation, and hue of an angle area in a certain direction is acquired, as well as an all-around camera that can shoot all-around video with a single camera using a reflector. It is also conceivable to use an optical sensor as possible. These all-around cameras and sensors cannot acquire high-resolution video information, but can be used as auxiliary information for generating an effective visual field video and a peripheral video or a simple peripheral video.

  Further, if the method of the present invention is used, video from the viewpoint position of the right eye and video from the viewpoint position of the left eye are generated by arbitrary viewpoint video technology, and each of them is viewed by the left and right eyes. It is also possible to present stereoscopic images in a wide-field image system.

  Further, by using the method of the present invention, it is possible to reproduce the moving parallax in the wide-field video system as described above. When a human is actually looking at something, moving the human head changes the positional relationship between the human eye and the subject, so the appearance changes. This is called moving parallax. If the viewer's head position is detected by a sensor so that moving parallax is reproduced when generating an effective visual field image, and the image is generated by changing the virtual camera position using that information, it is more realistic. High picture. For example, in Non-Patent Document 4 (Higashi, Fujii, Tanimoto: “Detection of viewer's head for free-viewpoint television”, Information Processing Society of Japan Report, 2003-HI-104 (5) (2003)) An arbitrary viewpoint video system that generates an effective visual field image in which the virtual camera position is changed following the left-right movement of the person's head has been studied. According to the technique of the present invention, not only an effective visual field image but also a peripheral image or a simple peripheral image can be generated by changing the virtual camera position following the movement of the viewer's head.

  This will be specifically described with reference to FIG. The arrangement of the video display devices 109 to 113 in FIG. 23 is the same as that in FIG. However, the viewer's 108 head position moves to the left from the viewer's 108 head position in FIG. In accordance with this, the manner of division of the field of view also changes, which is the field of view (13) 152 to field of view (17) 156. The sensor 157 is for detecting the head position of the viewer 108. Various means are conceivable, such as a camera installed and the viewer 108 photographed and detected from the image, or a magnetic sensor or ultrasonic sensor attached to the viewer's 108 head to detect the position. In the present invention, the means is not particularly limited. In addition, the position and shape of the sensor are not limited to those shown in FIG. The field of view is divided into fields of view (13) 152 to fields of view (17) 156 according to the information on the head position of the viewer 108 obtained by the sensor 157, and an effective field of view image, a peripheral image, or a simple peripheral image is arbitrarily viewed from each point of view. Generated and presented by technology. In this way, not only the effective visual field video but also the peripheral video or the simple peripheral video can be converted into the video reflecting the movement parallax, and the video is more realistic.

  Up to this point, the description has been made mainly on the wide-field video display unit and the video generation method to be presented there in the present invention. Below, the whole system is explained.

  FIG. 24 is a block diagram illustrating a first system configuration example of a wide-field video system based on an arbitrary viewpoint video technology. This is an example when the present invention is applied to broadcasting. The wide-field video system includes multi-view video data 201, data transmission means 202, a reception device 203, an arbitrary viewpoint video generation unit 204, and a display unit 205. The multi-view video data 201 is a plurality of video data shot using a camera array at a broadcasting station. All the multi-view video data 201 is delivered as it is to the viewer by the data transmission means 202. At this time, it is also conceivable to send the multi-view video data 201 after compressing it by some means. The receiving device 203 is installed at the viewer's place, and receives the multi-view video data 201 sent by the transmission means 202. The receiving device 203 includes an arbitrary viewpoint video generation unit 204. The arbitrary viewpoint video generation unit 204 processes the received multi-view video data 201 to generate an effective visual field video or a peripheral video or a simple peripheral video, and outputs the generated video to the display unit 205.

  In the first example, since all the multi-view video data 201 is transmitted, there is a disadvantage that the amount of data to be transmitted becomes enormous. On the other hand, since the viewer generates the video by the arbitrary viewpoint video technology, there is an advantage that the viewer can freely select the viewpoint and reflect it on the video.

  FIG. 25 is a block diagram illustrating a second system configuration example of the wide-field video system based on the arbitrary viewpoint video technology. This is another example when the present invention is applied to broadcasting. The wide-field video system includes multi-view video data 201, data transmission means 202, arbitrary viewpoint video generation unit 204, reception device 206, and display unit 205. The multi-view video data 201 is a plurality of video data shot using a camera array at a broadcasting station. The multi-view video data 201 is processed by the arbitrary viewpoint video generation unit 204 to generate an effective visual field video, a peripheral video, or a simple peripheral video. Up to this point, the broadcasting station processes first. The generated effective visual field video, peripheral video, or simple peripheral video is sent to the viewer by the data transmission means 202. At this time, the generated effective visual field image, peripheral image, or simple peripheral image may be sent after being compressed by some means. The receiving device 206 is installed at a viewer's place, receives an effective visual field image or a peripheral image or a simple peripheral image sent by the transmission unit 202, and outputs the image to the display unit 205.

  In the second example, since the effective visual field image, the peripheral image, or the simple peripheral image is generated and transmitted in advance on the broadcasting station side, it is not necessary to transmit all the multi-view video data 201, and the amount of data to be transmitted can be reduced. There is an advantage that you can. On the other hand, there is a disadvantage that the viewer only sees the video generated on the broadcasting station side and cannot select the viewpoint freely.

  In the first system configuration example and the second system configuration example of the wide-field video system based on the arbitrary viewpoint video technology described above, the data transmission unit 202 may be, for example, a unit that uses radio waves such as a terrestrial wave or a broadcasting satellite, or a wired system. And means using a network such as an optical fiber are conceivable. The present invention is not particularly limited to the means. In addition, as described above, one or more display units 205 are provided, which may be a video display device or an illumination device. Further, although described as an example in the case of application to broadcasting, the same system can be applied to streaming broadcasting via a network.

  26 and 27 are block diagrams illustrating a third system configuration example of the wide-field video system based on the arbitrary viewpoint video technology. This is an example when the present invention is applied to a package medium.

  FIG. 26 is a block diagram when data is recorded on a package medium. The wide-field video system includes multi-view video data 201 and a recording medium 207. Multi-view video data 201 is a plurality of video data shot by a content creator using a camera array. All the multi-view video data 201 is recorded on the recording medium 207 as it is. At this time, it is conceivable that the multi-view video data 201 is recorded after being compressed by some means.

  FIG. 27 is a block diagram for reproducing data recorded on a package medium. The wide-field video system includes a recording medium 207, a playback device 208, an arbitrary viewpoint video generation unit 204, and a display unit 205. The playback device 208 reads the multi-view video data 201 recorded on the recording medium 207. The arbitrary viewpoint video generation unit 204 processes the read multi-view video data 201 to generate an effective visual field video, a peripheral video, or a simple peripheral video, and outputs the generated video to the display unit 205.

  In the third example, since all the multi-view video data 201 is recorded on the recording medium 207, there is a disadvantage that the capacity required for recording becomes enormous. On the other hand, since the viewer generates the video by the arbitrary viewpoint video technology, there is an advantage that the viewer can freely select the viewpoint and reflect it on the video.

  28 and 29 are block diagrams showing a fourth system configuration example of the wide-field video system based on the arbitrary viewpoint video technology. This is another example when the present invention is applied to a package medium.

  FIG. 28 is a block diagram when data is recorded on a package medium. The wide-field video system includes multi-view video data 201, an arbitrary viewpoint video generation unit 204, and a recording medium 209. The multi-view video data 201 is a plurality of video data shot by a content producer using a camera array. The multi-view video data 201 is processed by the arbitrary viewpoint video generation unit 204 to generate an effective visual field video, a peripheral video, or a simple peripheral video. The generated effective visual field video, peripheral video, or simple peripheral video is recorded on the recording medium 209. At this time, the generated effective visual field image, peripheral image, or simple peripheral image may be recorded after being compressed by some means.

  FIG. 29 is a block diagram for reproducing data recorded on a package medium. The wide-field video system includes a recording medium 209, a playback device 210, and a display unit 205. The playback apparatus 210 reads out the effective visual field video, the peripheral video, or the simple peripheral video recorded on the recording medium 209 and outputs the read video to the display unit 205.

  In this example, since an effective visual field image, a peripheral image, or a simple peripheral image is generated on the content producer side and then recorded on the package medium, there is an advantage that the amount of data to be recorded can be reduced. On the other hand, the viewer only sees the video generated on the content producer side, and there is a disadvantage that the viewer cannot freely select a viewpoint.

  In the first system configuration example of the wide-field video system based on the arbitrary viewpoint video technology described in FIG. 24 and the third system configuration example of the wide-field video system based on the arbitrary viewpoint video technology described in FIGS. The viewer-side receiving device 203 or the playback device 208 generates an effective visual field image, a peripheral image, or a simple peripheral image. For this reason, even if the configuration of the display unit 205 is different or the arrangement is different for each viewer, an effective visual field image, a peripheral image, or a simple peripheral image is generated in accordance with the environment of each viewer. It has the advantage of not being affected by environmental differences.

  Conversely, the second system configuration example of the wide-field video system based on the arbitrary viewpoint video technology described in FIG. 25 and the fourth system configuration example of the wide-field video system based on the arbitrary viewpoint video technology described in FIGS. 28 and 29. Then, an effective visual field video, a peripheral video, or a simple peripheral video is generated on the broadcast station side or the content producer side. At this time, an effective visual field image, a peripheral image, or a simple peripheral image cannot be generated unless any device configuration or arrangement is assumed for the display unit 205.

  One solution is to define some standard device configuration and placement. In this specification, this is called a standard wide-field video display environment. The broadcast station side or content producer side generates an effective visual field image, a peripheral image, or a simple peripheral image in accordance with a standard wide-field image display environment. On the viewer side, the display unit is arranged according to the standard wide-field video display environment. In this way, any viewer can play a wide-field video under the same conditions.

  However, the environment on the viewer side is different, and it is not always possible to prepare and install a display unit according to the standard wide-field video display environment. A solution to this problem is shown in another application filed by the applicant of the present application and will not be discussed here.

  The recording medium 207 or the recording medium 209 in FIGS. 26 to 29 is a so-called semiconductor such as a so-called optical medium such as a CD, a DVD, a Blu-ray Disc, or an HD-DVD, an SD card, or a compact flash (registered trademark) card. A magnetic recording medium such as a memory or HDD can be considered. This patent can be used regardless of the type of recording medium.

It is a figure which shows the classification | category of a visual field. It is the figure which looked at the 1st example of the wide-field video system from back. 1 is a top view of a first example of a wide-field video system. FIG. It is the figure which looked at the 2nd example of the wide visual field image system from back. It is a top view of the 2nd example of a wide visual field image system. It is the figure which looked at the 3rd example of a wide visual field picture system from back. It is a top view of the 3rd example of a wide field image system. It is the figure which looked at the 4th example of the wide-field video system from back. It is a top view of the 4th example of a wide field image system. It is the figure which looked at the 5th example of the wide-field video system from back. It is a top view of the 5th example of a wide field image system. It is the figure which looked at the 6th example of the wide-field video system from back. It is a top view of the 6th example of a wide field image system. It is a figure which shows an example of the camera array of arbitrary viewpoint video technologies. It is a figure which shows the division | segmentation of the visual field in the 2nd example of a wide visual field video system. It is a figure which shows the relationship between the to-be-photographed object, camera array, and virtual camera in a wide visual field video system. It is another figure which shows the relationship between the to-be-photographed object, camera array, and virtual camera in a wide visual field video system. It is another figure which shows the relationship between the to-be-photographed object, camera array, and virtual camera in a wide visual field video system. It is a figure which shows the mode of the division | segmentation of the visual field in the 1st example of a wide visual field video system. It is a figure which shows the mode of the division | segmentation of the visual field in the 3rd example of a wide visual field video system. It is a figure which shows the mode of the division | segmentation of the visual field in the 5th example of a wide visual field video system. It is a figure which shows the mode of the division | segmentation of the visual field in the 6th example of a wide visual field video system. It is a figure which shows the division | segmentation of a visual field when the viewer's head position moves in the 2nd example of a wide visual field video system. It is a block diagram which shows the 1st system configuration example of the wide visual field video system by arbitrary viewpoint video technologies. It is a block diagram which shows the 2nd system configuration example of the wide visual field video system by arbitrary viewpoint video technologies. It is a block diagram (recording side) which shows the 3rd system configuration example of the wide visual field video system by arbitrary viewpoint video technology. It is a block diagram (reproduction | regeneration side) which shows the 3rd system configuration example of the wide visual field video system by arbitrary viewpoint video technologies. It is a block diagram (recording side) which shows the 4th system configuration example of the wide visual field video system by arbitrary viewpoint video technology. It is a block diagram (reproduction | regeneration side) which shows the 4th system structural example of the wide view video system by arbitrary viewpoint video technologies.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 ... Discrimination visual field, 102 ... Effective visual field, 103 ... Guidance visual field, 104 ... Auxiliary visual field, 105 ... Video display device, 106 ... Video display device, 107 ... Video display device, 108 ... Viewer, 109 ... Video display device, 110 ... Video display device, 111 ... Video display device, 112 ... Video display device, 113 ... Video display device, 114 ... Video display device, 115 ... Video display device, 116 ... Video display device, 117 ... Illumination device, 118 ... Illumination device DESCRIPTION OF SYMBOLS 119 ... Illuminating device, 120 ... Illuminating device, 121 ... Illuminating device, 122 ... Illuminating device, 123 ... Illuminating device, 124 ... Field, 125 ... Camera, 126 ... Virtual camera, 127 ... Effective visual field, 128 ... Guidance visual field (left ) 129 ... guided field of view (right), 130 ... auxiliary field of view (left), 131 ... auxiliary field of view (right), 132 ... out of field, 133 ... subject, 134 ... Mela (AP), 135 ... Virtual camera, 136 ... Field of view (front), 137 ... Field of view (left), 138 ... Field of view (right), 139 ... Field of view (entire), 140 ... Field of view (1), 141 ... Field of view (2), 142 ... Field of view (3), 143 ... Field of view (4), 144 ... Field of view (5), 145 ... Field of view (6), 146 ... Field of view (7), 147 ... Field of view (8), 148 ... Field of view ( 9), 149 ... Field of view (10), 150 ... Field of view (11), 151 ... Field of view (12), 152 ... Field of view (13), 153 ... Field of view (14), 154 ... Field of view (15), 155 ... Field of view (16 ) ... 156 (field of view), 157 ... sensor, 201 ... multi-view video data, 202 ... data transmission means, 203 ... receiving device, 204 ... arbitrary viewpoint video generating unit, 205 ... display unit, 206 ... receiving device, 207 ... recording medium, 208 ... reproducing device, 209 ... Recording medium, 210 ... playback device.

Claims (14)

  A wide-field video system comprising: a video generation unit that synthesizes a wide-field video composed of one or more video data from video shot by one or more cameras.   As a method for synthesizing a wide-field video composed of the one or more video data in the video generation unit, an arbitrary viewpoint capable of synthesizing a video at a virtual camera position where no camera originally exists from the video by the one or more cameras 2. The wide-field video system according to claim 1, wherein video technology is used.   The wide-view video system according to claim 2, wherein the video generation unit generates a wide-view video that can be stereoscopically viewed by generating an arbitrary viewpoint video from a viewpoint separated by an interval between the left and right eyes.   A sensor capable of detecting the viewer's head position is provided, the virtual camera position is changed following the movement of the viewer's head, and a wide-field image composed of one or more video data at the virtual camera position is generated. The wide-field video system according to claim 2 or 3, wherein   In a broadcasting system, all the video information photographed by the one or more cameras on the broadcasting station side is sent to the viewer side, the video information is received on the viewer side, and a wide field of view is generated using an arbitrary viewpoint video generation technique using the video information. 5. The wide-field image system according to claim 2, wherein the image is generated.   3. The broadcast system according to claim 2, wherein a wide-field video is generated from video information photographed by the one or more cameras on the broadcast station side by an arbitrary viewpoint video generation technique, and the wide-field video is sent to a viewer. 5. A wide-field video system according to any one of items 1 to 4.   In a video recording / playback system using a recording medium, the video content producer records all video information captured by the one or more cameras on the recording medium, and the viewer plays the video information from the recording medium. 5. A wide-field video system according to claim 2, wherein a wide-field video is generated by an arbitrary viewpoint video generation technique using the same.   In a video recording / playback system using a recording medium, a video field producer generates a wide-field video from video information captured by the one or more cameras using an arbitrary viewpoint video generation technique, and records the video on a recording medium. 5. The wide-field video system according to claim 2, wherein the wide-field video is reproduced from the recording medium on the side.   9. The wide-field image system according to claim 1, further comprising one or more display units, wherein the wide-field image is presented on the display unit.   The wide-field video system according to claim 9, wherein a display device having a plurality of pixels is used as the display unit.   The wide-field video system according to claim 9, wherein an illumination device is used as the display unit.   The wide-field video system according to claim 9, wherein a display device having a plurality of pixels and an illumination device are used in combination as the display unit.   The video generation unit generates video data according to an area occupied by the display unit in the viewer's visual field when viewed from the resolution of each of the one or more display units and the viewpoint position of the viewer. Or the wide-field image system according to 12.   The video generation unit uses information from an optical sensor that can acquire luminance, saturation, and hue information in a certain angle region in a certain direction and video information captured by the one or more cameras, and uses a wide-field video. The wide-view video system according to claim 1, wherein the wide-view video system is generated.

Images