JP2007025111A - System, method and program for presenting video image, and recording medium thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system and a method for presenting a video image which can easily reproduce an actual surrounding video image or surrounding light illumination to an effective field of view image on the reproducing side. <P>SOLUTION: On the generating side of video information data, when the effective field of view video image is photographed, information such as panoramic image data required for generating the surrounding video image or the surrounding light illumination is simultaneously acquired as well and on the reproducing side of the video information data, the video image or the illumination is presented also to surrounding areas other than an effective field of view. The generated video information data are transmitted to a receiver by a data transmitting means and reproduced by a display means 105 for presenting the effective field of view image and a display means 106 for presenting the surrounding image or the surrounding light illumination. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a video presentation system, a video presentation method, a video presentation program, and a recording medium for reproducing a video captured in an environment having a wide visual field with a realistic sensation with a wide visual field.

  According to the research by Hatada et al. (See Non-Patent Document 1), the human visual field is classified into a discrimination visual field 101, an effective visual field 102, a guidance visual field 103, and an auxiliary visual field 104 as shown in FIG. The discrimination visual field 101 is a range in which high-density information such as graphic identification can be accurately received. The effective visual field 102 is a range in which natural information can be received only by eye movement although the discrimination ability is lower than the discrimination visual field 101. The guidance visual field 103 has only a recognition ability to the extent that the present stimulus is present and can be easily identified, but has a range influencing when judging overall external information. The auxiliary visual field 104 is a range in which only the presence of a stimulus can be determined.

  The current high-definition television is designed to present an image (in the present invention, this image is referred to as an effective visual field image) in a range covering the effective visual field 102 among the above. That is, the guidance visual field 103 and the auxiliary visual field 104 are not considered. However, it is expected that a sense of reality is further enhanced by presenting the image or ambient light illumination in the guidance visual field 103 and the auxiliary visual field 104.

  However, if an image or ambient light illumination is to be presented in peripheral areas such as the guide visual field 103 and the auxiliary visual field 104, the video display device and the illumination device become very large. Introducing such a device in a viewing environment such as a home is actually difficult both in terms of cost and installation space.

  Here, considering that the auxiliary visual field 103 and the guiding visual field 104 have low recognition ability of human visual functions, the resolution of the auxiliary visual field 103 and the guiding visual field 104 is lower than that of the video presented in the effective visual field 102. It is conceivable to present video information. For example, in the auxiliary visual field 103 and the guidance visual field 104, even by presenting the video information and illumination information using an illumination device capable of controlling the hue, saturation, and luminance based on the video information with reduced resolution, It can be expected to improve the presence (see, for example, Patent Document 1).

In addition, for generating a video with a wide field of view, a technique capable of simultaneously capturing images in all directions (for example, see Patent Information 2) is also known, and a video playback system with a wide field of view is desired.
JP 2000-173783 A JP-A-6-295333 Toyohiko Hatada, Haruo Sakata, Hideo Kusaka, “Directional sensation induction effect by screen size-Basic experiment of realism by large screen”, Journal of Television Society, Vol. 33, no. 5, pp. 407-413 (1979)

As a system for controlling illumination in conjunction with an effective visual field image, there is the above-mentioned Patent Document 1 or a product (for example, an illumination device called Philips, manufactured by Philips). However, in these video presentation systems, an illumination control signal is estimated and generated from information of an effective visual field video as an input signal. For this reason, there is a difference from actual ambient video or ambient light illumination, and it is difficult to say that effective illumination for increasing the sense of reality can be presented.
The present invention has been made in view of the above circumstances, and an object thereof is to provide an image presentation system and an image presentation method that can easily reproduce an actual peripheral image or ambient light illumination for an effective visual field image on the reproduction side.

  In order to solve the above-mentioned problems, the present invention also provides information such as panoramic image data necessary for generating a peripheral video or ambient light illumination when shooting an effective visual field video on the video information data generation side. At the same time, on the reproduction side of the video information data, video or illumination is presented in the peripheral area in addition to the effective visual field. The generated video information data is transmitted to the receiving device by the data transmission means, and is reproduced by the display means for presenting the effective visual field image and the display means for presenting the peripheral image or the ambient light illumination.

  An image presentation system according to the present invention for the above includes a first display means for presenting a main video and a second display means for presenting information subordinate to the main video, and a video photographed by one or more cameras. The video data presented on the first display means and the display data presented on the second display means are generated from the information. Preferably, the second display means is arranged around the first display means, the first display means is arranged in the effective visual field, and the second display means is arranged in the guidance visual field or auxiliary visual field.

  In addition, the image presentation system according to the present invention obtains information by a sensor that can obtain luminance, saturation, hue, and chromaticity information in a predetermined angle region in addition to video information from one or more cameras. The video data presented on the first display means and the display data presented on the second display means may be generated. Furthermore, the display data presented on the second display means may be generated using an omnidirectional visual system technique.

  Furthermore, the image presentation system according to the present invention includes an ambient light illumination generation unit or an ambient image generation unit for generating display data to be presented on the second display means on either the transmission side or the reception side. The resolutions of the video data presented on the first display means and the display data presented on the second display means can be the same or different, and if they are different, the display presented on the second display means The resolution of the data is lower than that of the video data presented on the first display means.

  According to the present invention, when capturing an effective visual field video, video information such as panoramic image data required to generate a peripheral video or ambient light illumination is also acquired at the same time. It is possible to reproduce a state close to the surrounding image or the ambient light illumination at the time of photographing in the guiding field or auxiliary field of view. Therefore, it is possible to view an image with an effective field of view with a sense of reality close to the actual state.

  Embodiments of the present invention will be described with reference to the drawings. FIG. 1 shows an example of a video viewing environment according to the present invention. The image display device 105 and the illumination device 106 are installed in the room, and the viewer 107 is viewing it. The image display device 105 is a first display means in the present invention, and various display means such as a cathode ray tube television, a liquid crystal television, a plasma television, a rear projection television, and a projection screen by a projector can be used. The lighting device 106 is the second display means in the present invention, and can control the hue, saturation, luminance, etc. of the illumination light. Various lighting means such as LED lighting, fluorescent lamps, incandescent lamps can be used. Can be used.

  Note that the number, arrangement, and types of the large number of lighting devices 106 illustrated in FIG. 1 are examples, and for example, a display device capable of displaying an image may be used as the lighting device. In the example of FIG. 1, the lighting device 106 is not installed on the back side of the viewer 107 because the viewer 107 does not enter the field of view, but the lighting device 106 is also installed on the back side because the viewer 107 may look back. You may make it do. Further, in FIG. 1, a sound reproducing device such as a speaker is not clearly shown, but they may be present.

  Next, video information, video data, and display data for presentation in the video viewing environment will be described. Currently, it is assumed that video information provided by package media such as TV broadcasts and DVDs is displayed on one rectangular screen and presented to the viewer's effective field of view. This does not include information about the video around the screen, lighting, and the like. In the present invention, the video data and other display data presented in the video viewing environment shown in FIG. 1 are subordinate to the main video in addition to the main video presented in an effective field of view using a shooting method different from the conventional one. It is to simultaneously acquire information related to lighting and images in the peripheral area and reproduce the information in a viewing environment as shown in FIG.

  In addition to the above-mentioned effective visual field image, as a technique for acquiring information related to ambient light illumination and surrounding images in the surrounding area, for example, an omnidirectional image is taken by an omnidirectional camera disclosed in Patent Document 2. There is a technique. This omnidirectional camera can shoot an image of 360 degrees around it, and creates a 360-degree panorama converted image or a perspective projection converted image similar to a normal camera by performing arithmetic processing inside the computer. Can do. Here, the perspective projection conversion image shows a perspective when viewing something from the same point as a person's viewpoint (the far one is small and the near one is large). An image to be expressed. In Patent Document 2, a reflecting mirror having a hyperboloid shape of one of the two leaf hyperboloids, and a hyperboloid having the other hyperboloid shape of the two leaf hyperboloids, the center being the other hyperboloid And an image input device including a lens arranged at the focal point of the lens.

  FIG. 2 is a diagram illustrating an example of an omnidirectional visual system using the omnidirectional camera. Arbitrary point p projected on the image plane 5 and an arbitrary line on the straight line connecting the focal point Om of the hyperboloidal mirror 3 and the point P ′ where the extended line of the straight line connecting the center Oc of the camera lens intersects the hyperboloidal mirror 3 There is a real point P. As a result, it is possible to capture an image reflected on the hyperboloid mirror 3 and create a panorama conversion image or a perspective projection conversion image centered on the focal point Om of the hyperboloid mirror 3 based on the input image.

The panorama converted image is obtained by panorama converting an input image obtained by rotating 360 degrees on the image surface about a straight line Om-Oc centered on the center Oc of the camera lens, and projecting the image into a cylindrical shape. Represents. FIG. 3 is a diagram showing a region selection method in the polar coordinate system of the panorama converted image. For example, when extracting angular range of θ _h start~θ _h end from the panoramic transformed image, and thus to extract the hatched region in FIG.

  By using this omnidirectional visual system method, it is possible to acquire the main image and image information around the omnidirectional subordinate to the image. At this time, the effective visual field image which is the main image is displayed on, for example, the image display device 105 in FIG. 1, and the ambient light illumination of the subordinate image is used as information for controlling the illumination device 106 in FIG. As the ambient light illumination data, for example, the surrounding image acquired by the method as described above is converted into information such as luminance, saturation, hue, chromaticity, etc. in a predetermined angle region and displayed on the lighting device 106. Can do. In this case, the peripheral image in which the resolution is lowered may be used.

  FIG. 4 is a diagram showing a first example when the present invention is applied to a video broadcasting system. The video information 201 including the panorama converted image is transmitted by the data transmission unit 202 and received by the receiving device 203. On the receiving device 203 side, an ambient light illumination generation unit 204 is provided, which processes the received video information 201 and presents it on the image display device 105 and display data presented on the illumination device 106 as ambient light illumination. Is generated. Note that the video information 201 is a main video taken at a broadcasting station or the like as a transmission source, and information subordinate to the main video, for example, panorama converted image data by an omnidirectional visual system technique. In addition, information on luminance, saturation, hue, and chromaticity of a predetermined angle region acquired by a sensor or the like may be added to this.

  In the example of FIG. 4, all video information 201 including panorama converted image data by the omnidirectional visual system technique is transmitted as it is to the receiving device 203 installed at the viewer's place by the data transmission means 202. As the data transmission means 202, for example, a method using radio waves such as terrestrial waves and broadcasting satellites, and a method using a network such as an optical fiber are conceivable. The video information received on the receiving side is generated by, for example, generating video data that is the main video to be presented to the image display device 105 as the first display means by the ambient light illumination generation unit 204 provided in the reception device 203. In addition, display data dependent on the main video presented to the lighting device 106 as the second display means is generated.

  In the example of FIG. 4, since all information such as video and panoramic image data is transmitted as video information 201, the amount of data to be transmitted becomes enormous. However, since ambient light illumination can be generated on the reception side, there is an advantage that it is easy to reproduce the illumination environment on the photographing side according to the illumination arrangement and illumination information on the reception side such as a viewer.

  FIG. 5 shows a second example when the present invention is applied to a video broadcasting system. Video information 201 including video captured on the transmission side such as a broadcasting station and panoramic image data is processed by an ambient light illumination generation unit 204 arranged on the transmission side. In the ambient light illumination generation unit 204, in addition to the video data that is the main image presented on the image display device 105 that is the first display means, the ambient light illumination that is presented on the illumination device 106 that is the second display means. Display data to generate.

  The data generated and processed by the ambient light illumination generation unit 204 is transmitted by the data transmission unit 202 to the receiving device 205 installed at the viewer. As the data transmission means 202 in this case, for example, a method using radio waves such as terrestrial waves or broadcasting satellites, or a method using a network such as an optical fiber can be considered. The receiving device 205 receives video data and display data such as ambient light illumination sent by the transmission unit 202. The received video data is presented on the image display device 105 as the first display means, and the received display data such as ambient light illumination is presented by controlling the illumination device 106 as the second display means.

  In the example of FIG. 5, since video data and display data such as ambient light illumination are generated on the transmission side and then transmitted, there is an advantage that the amount of data to be transmitted can be reduced. On the other hand, it may be difficult to reproduce the illumination environment on the photographing side according to the illumination arrangement and illumination information on the receiving side such as a viewer.

Next, a specific example of ambient light illumination using the display data described above will be described. In the example described with reference to FIG. 4, ambient light illumination data is generated in the ambient light illumination generation unit 204 provided in the viewer-side receiving device 203. For this reason, even if the arrangement of the illumination device is different for each viewer on the receiving side, ambient light illumination data can be generated accordingly.
FIG. 6 is a diagram for explaining an example of a method for generating ambient light illumination data in this case. A plurality of illumination devices 106 are denoted by reference numerals A to P. Here, reference numerals 120 to 131 denote visual field regions when the viewer 107 is centered. For example, E and F of the plurality of lighting devices 106 are arranged in the area of the visual field 121 on the right front side.

FIG. 7 shows a procedure for generating ambient light illumination data to be presented by the lighting devices E and F, for example. First, at step S301, the angle theta _h start corresponding to the visual field 121, with no reads theta _h end The, even in the panoramic converted image shown in FIG. 3, the region of θ _h start~θ _h end corresponding to the angle Extract converted image data. Here, θ _h start indicates the start angle of the visual field 121, and θ _h end indicates the end angle of the visual field 121. Next, in step S302, average values of RGB (three primary colors of red, green, and blue light) of each pixel are obtained from the converted image data extracted as described above, and this is obtained as ambient light illumination. Data. In the last step S303, the illumination devices E and F are controlled according to the RGB values of the ambient light illumination data. In this way, it is possible to generate ambient light illumination data and control the illumination devices E and F.

  In the above example, the same ambient light illumination is presented to the lighting devices E and F, but different ambient light illuminations can be presented to the lighting devices E and F. As shown in FIG. 6, the visual field 121 may be divided into a visual field including the illumination device E and a visual field including the illumination device F, and ambient light illumination may be generated for each. Thus, by dividing the field of view according to the arrangement of the illumination devices and generating the respective ambient light illumination, it is possible to cope with various arrangements of the illumination devices. In addition, since the example of FIG. 6 is a view of the viewing environment as viewed from above, the field of view only shows the division on the horizontal plane, but actually the field of view is divided in the vertical direction as well. It is also possible to deal with lighting devices with various arrangements.

  In the example described with reference to FIG. 5, ambient light illumination data is generated on the broadcast station side or content producer side. At this time, the ambient light illumination data cannot be generated unless some arrangement of the illumination device is assumed. One solution is to determine a standard lighting arrangement (in the present invention, this is called a standard lighting arrangement), and the ambient light data is generated on the broadcast station side or content producer side according to the standard lighting arrangement. . On the viewer side, the lighting device is arranged according to the standard lighting arrangement.

  However, the environment on the viewer side is different, and lighting cannot always be installed according to the standard lighting arrangement. As an example of the arrangement of the illumination device on the viewer side, an example in which the illumination device is applied to a home theater set is shown in FIG. When installing a home theater set in the home, it may not be possible to install the lighting according to the standard lighting arrangement. As a solution to this problem, the ambient light illumination generated on the broadcasting station side or content producer side on the transmitting side is a solution. It is conceivable that the data is interpolated and output in accordance with the assumed arrangement of the illumination device on the viewer side.

  An example of this interpolation method will be described with reference to FIG. FIG. 9 is a diagram showing an example of region division in ambient light illumination data generation on the broadcast station side or the video content producer side. The surrounding area centering on the camera 115 capable of photographing the surroundings is divided into, for example, 12 areas to be illumination areas (0) to (11). Then, ambient light illumination data is generated in each of these 12 illumination areas based on the panorama converted image.

  On the other hand, it is assumed that the viewing environment on the receiving side such as the viewer is the image display device 105, the illumination device 106, and the viewer 107 as shown in FIG. Returning to FIG. 6, when the illumination areas (0) to (11) generated on the transmission side in FIG. 9 are overlapped and the illumination device 106 (A to P) is controlled to perform ambient light illumination, the following is performed. Become.

Illumination device A: Outputs ambient light illumination of illumination area (9) Illumination device B: Outputs ambient light illumination of illumination area (10) Illumination device C: Approximately 40% of illumination area (10) and illumination area (11) Since about 60% is included, the ambient light illumination of each illumination area is mixed at a ratio of 4 to 6 and output. Illumination device D: Outputs ambient light illumination of the illumination area (11) Illumination device E: Illumination area Outputs ambient light illumination of (1) Illumination device F: Since approximately 60% of illumination area (1) and approximately 40% of illumination area (2) are included, 6 pairs of ambient light illumination in each illumination area 4 mixed and output Illumination device G: Output ambient light illumination of illumination area (2) Illumination device H: Output ambient light illumination of illumination area (3) Illumination device I: Ambient light of illumination area (4) Illumination output Illumination device J: About 40% of the illumination area (4) and about 60% of the illumination area (5) are included. The ambient light illumination of each illumination area is mixed at a ratio of 4 to 6 and output. Lighting device K: Outputs ambient light illumination of the illumination region (5). Lighting device L: Outputs ambient light illumination of the illumination region (7). Illumination device M: Since about 60% of the illumination area (7) and about 40% of the illumination area (8) are included, the ambient light illumination of each illumination area is mixed at a ratio of 6 to 4 and output. Device N: Output ambient light illumination of illumination area (8) Illumination device O: Output ambient light illumination of illumination area (0) Illumination device P: Output ambient light illumination of illumination area (6)

  As described above, it is possible to output ambient light illumination by appropriately interpolating according to the difference in environment such as the arrangement and size of the illumination device on the reception side. In the example shown in FIG. 9, the illumination area (0) to the illumination area (11) are divided into 12 areas, but the number of divisions and the size of each of the divided areas are not limited to this. In addition, although the two-dimensional illumination area is divided on the horizontal plane as shown in the top view, the actual viewing environment is a three-dimensional space, and the illumination devices are also arranged three-dimensionally. Therefore, it is possible to perform simple peripheral video interpolation processing in a three-dimensional space in accordance with the arrangement of the illumination devices, as well as dividing the illumination area.

  FIGS. 10 and 11 are diagrams for explaining other embodiments, and are different in that the ambient light generation unit 204 of FIGS. That is, in the above-described embodiment, the example in which the ambient light illumination is reproduced on the viewer side is shown, but the panoramic image data is displayed on the ambient image display device as the ambient image. At this time, the illumination device 106 (A to P) as the second display means in FIG. 6 is replaced with the surrounding image display device 105 ′ (A ′ to P ′) to display the surrounding image. In this case, the resolution of the video data presented on the first display means and the video data (display data) presented on the second display means may be the same or different. In the case of different resolutions, for example, the resolution of the surrounding image displayed by the second display unit is set lower than the resolution of the effective visual field image displayed by the first display unit.

  Note that the video presentation method according to the present invention can be recorded on a computer-readable recording medium in which a program to be executed by a computer is recorded. As a result, it is possible to provide a portable recording medium on which a program for realizing the video presentation method is recorded.

  In the present embodiment, the recording medium may be a program medium, such as a memory (not shown) such as a ROM (Read Only Memory) because processing is performed by a microcomputer. Further, although not shown, a program reading device may be provided as an external storage device, and the program medium may be read by inserting a recording medium therein.

  In any case, the stored program may be configured to be accessed and executed by the microprocessor, or in any case, the program is read and the read program is illustrated in the microcomputer. The program may be downloaded to a non-program storage area and the program may be executed. It is assumed that this download program is stored in the main device in advance.

  Here, the program medium is a recording medium configured to be separable from the main body, such as a tape system such as a magnetic tape or a cassette tape, a magnetic disk such as a floppy (registered trademark) disk or a hard disk, a CD-ROM, an MO, Disk system IC cards (including memory cards) for optical disks such as MD and DVD, card systems such as optical cards, or mask ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash ROM It may be a medium that carries a fixed program including a semiconductor memory.

It is a figure which shows an example of the image viewing environment in this invention. It is a figure which shows an example of the omnidirectional visual system by an omnidirectional camera. It is a figure for demonstrating the area | region selection method in the polar coordinate system of a panorama conversion image. In this invention, it is a figure which shows the example which presents ambient light illumination. It is a figure which shows the other example which presents ambient light illumination in this invention. It is the figure which looked at an example of the image viewing environment in the present invention from the upper surface. It is a figure which shows the procedure which produces | generates ambient light illumination data for showing with the illuminating device of a video viewing environment. It is a figure which shows the example which applies an illuminating device to a home theater set. It is a figure which shows an example of the area division | segmentation in the surrounding light illumination production | generation by the image | video provision side. In this invention, it is a figure which shows the example which shows a surrounding image. In this invention, it is a figure which shows the other example which shows a surrounding image. It is a figure explaining that a human visual field is classified into each visual field by the function of visual function.

Explanation of symbols

105, 105 '... image display device, 106 ... lighting device, 107 ... viewer, 201 ... video information, 202 ... data transmission means, 203 ... receiving device, 204 ... ambient light illumination generator, 205 ... receiving device, 206 ... Ambient image generation unit.

Claims (14)

  First display means for presenting a main video and second display means for presenting information subordinate to the main video are presented to the first display means from video information photographed by one or more cameras. An image presentation system for generating image data and display data to be presented on the second display means.   2. The video presentation system according to claim 1, wherein the second display means is arranged around the first display means.   The video display system according to claim 2, wherein the first display unit is arranged in an effective visual field.   4. The video presentation system according to claim 3, wherein the second display means is arranged in a guidance visual field or an auxiliary visual field.   5. The video presentation system according to claim 1, wherein the display data presented on the second display means is generated using an omnidirectional visual system technique.   A sensor that can acquire information on luminance, saturation, hue, and chromaticity in a predetermined angle region is used in combination, and information by the sensor and video information by the one or more cameras are generated in combination. The video presentation system according to any one of claims 1 to 5.   The ambient light illumination generation unit for generating display data to be presented on the second display means is provided on either the transmission side or the reception side. The video presentation system described in 1.   The surrounding image generation part for producing | generating the display data shown to a said 2nd display means is provided in either the transmission side or the receiving side, The any one of Claims 1-5 characterized by the above-mentioned. The video presentation system described.   9. The video according to claim 1, wherein the resolution of the video data presented on the first display means and the display data presented on the second display means are the same. Presentation system.   9. The video presentation system according to claim 1, wherein the resolution of the video data presented on the first display means is different from the resolution of the display data presented on the second display means. .   The video presentation system according to claim 9, wherein the display data presented on the second display means has a lower resolution than the video data presented on the first display means.   A main video is presented on the first display means, information subordinate to the main video is presented on the second display means, and video presented on the first display means from video information photographed by one or more cameras A method for presenting video and display data to be presented on the second display means.   A program for causing a computer to execute the video presentation method according to claim 12.   A computer-readable recording medium on which the program according to claim 13 is recorded.

Images