JP2003235058A - View point free type image display apparatus and method, storage medium, computer program, broadcast system, and attached information management apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To attach an advertisement corresponding to a set view point to a video point free type video image in the case of displaying the video image at an optional field angle. <P>SOLUTION: A user can dynamically select a viewed field angle when viewing or reproducing video image contents and an advertisement depending on the viewed field angle is prepared in advance for the view point free type video image. When the user or a viewer designates a viewed field angle to the view point free type video image, the view point free type video image is displayed at the designated field angle accordingly and the corresponding advertisement is superimposingly displayed, and thus the opportunity of increasing the displayed advertisement can be increased. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a viewpoint free-form image display apparatus and method, a storage medium, and a computer, which display a viewpoint free-form image which a user can freely switch between viewpoints during playback / display. BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a program, a broadcasting system, and an additional information management device and method, and in particular, an omnidirectional camera including a plurality of cameras installed so that one point in space captures an image of the surrounding area as a viewpoint The present invention relates to a viewpoint free-form image display apparatus and method relating to a viewpoint free-form image formed by connecting images captured by, a storage medium, a computer program, a broadcasting system, and an additional information management apparatus and method.

More specifically, the present invention relates to a viewpoint free-form image display device and method, a storage medium, a computer program, a broadcasting system, and an addition device for displaying and outputting an advertisement and other additional information to a viewpoint free-form image. BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information management device and method, and more particularly, a viewpoint free-form image display device and method, a storage medium, a computer program, a broadcasting system, and additional information for displaying additional information such as advertisements according to a set viewpoint. The present invention relates to a management device and method.

[0003]

2. Description of the Related Art An omnidirectional camera is known as a device for providing a landscape image of the surroundings of a user. This type of ambient camera system is configured by installing a plurality of cameras so that an image of the surroundings is captured with a certain point in space as a viewpoint. Surround camera consisting of multiple cameras
The system or omnidirectional camera system captures an image of a region far wider than the field of view of each camera by using a single wide-angle camera by successfully connecting the boundaries between the images captured by the adjacent cameras. Image processing is performed to generate such an image.

If a lens having a wide viewing angle is used, the camera can pick up a wide range of images, but the resolution is reduced by that much and it becomes difficult to see the details. On the other hand, the ambient camera system can provide a wide range of captured images with high resolution.

On the other hand, along with the development of information and communication technology, there is a demand for distributing video images taken by cameras to remote locations through various distribution media.

On the other hand, after transmitting an omnidirectional image composed of a plurality of camera images on one channel, an arbitrary angle of view can be selected on the receiver side of the image so that a viewpoint free form, that is, a user ( Or, the viewer can enjoy the video from the favorite viewpoint.

[0007] By the way, in many cases, advertisement information is added to or inserted in the form of a banner advertisement, an advertisement slot, etc. in broadcast contents and other video contents. Generally, the advertising fees obtained from advertisers are the main source of income for broadcasters and content providers, and the problems related to the free and low prices of services such as broadcasting and content provision, and ultimately the motivation for content creation. Is.

[0008] In a broadcasting system such as a conventional digital BS broadcast that can simultaneously receive a data broadcast and a normal broadcast, data can be added and sent in the form of a banner advertisement or the like.

On the other hand, in the case of the viewpoint free-form broadcasting described above, it is possible to freely switch the viewing angle of view by the user or the viewer side, so that an advertising effect corresponding to the viewing angle of view is produced. Opportunity spreads.

However, when the viewpoint free-form video is broadcast as it is in the conventional data broadcasting, the additional data cannot be changed by switching the viewing angle of view, so that the business opportunity such as advertisement can be fully utilized. I can't.

[0011]

SUMMARY OF THE INVENTION An object of the present invention is to provide an excellent viewpoint freedom, which is capable of suitably displaying a viewpoint free-form image which the user can freely switch between viewpoints during playback / display. To provide an image display device and method, a storage medium, a computer program, a broadcasting system, and an additional information management device and method.

It is a further object of the present invention to provide an excellent viewpoint free-form image display device and method, a storage medium, a computer program, and a broadcast, which can display and output an advertisement and other additional information to a viewpoint free-form image. A system, and an additional information management device and method are provided.

It is a further object of the present invention to provide an excellent viewpoint free-form image display device and method, a storage medium, a computer program, and a broadcast that can display additional information such as advertisements according to a set viewpoint. A system, and an additional information management device and method are provided.

[0014]

The present invention has been made in view of the above problems, and a first aspect thereof is a viewpoint free-form image display device or method for displaying a viewpoint free-form image. The viewpoint free-form image input means or step for inputting the viewpoint free-form image, the angle-of-view specifying means or step for specifying the viewing angle of the viewpoint-free-form image, and the angle-of-view specifying means or step Additional information acquisition means or step for selectively acquiring additional information according to the specified angle of view, and cutting out the angle of view specified by the angle of view specifying means or step from the viewpoint free-form image and the selected addition And a viewing image display means or step for displaying the information together with information, and a viewpoint free-form image display device or method.

In the viewpoint free-form image display device or method according to the first aspect of the present invention, the user can dynamically switch the viewing angle when viewing or playing back the video content, and the viewpoint free-form video can be displayed. Is prepared in advance with additional information according to the viewing angle. The additional information referred to here is information content that is displayed by being superimposed on the viewed image, such as a banner advertisement.

Since the viewpoint or free-view image can provide a viewing mode in which the user or the viewer arbitrarily specifies the angle of view, that is, the viewing place, new effects can be expected as entertainment and amusement. On the other hand, there is a problem of where to place the additional information on the display screen when the user changes the viewing location. Also, the advertising effect of the advertising information may differ for each viewing location, and it is considered necessary to reconsider the selection of additional information to be used as the viewing angle of view moves.

According to the viewpoint free-form image display apparatus or method according to the first aspect of the present invention, when the user specifies the view angle of the viewpoint free-form image to be viewed, the view angle specified by the user is used. The free-viewpoint image is displayed, and additional information corresponding to the angle of view is dynamically selected and displayed on the viewed image in a superimposed manner. Therefore, it is possible to increase the chances of increasing the degree of exposure of the advertisement, and it is possible to selectively use the advertisement information having a higher advertisement effect depending on the viewing place.

Here, the viewpoint free-form image display device or method according to the first aspect of the present invention further comprises additional information management means or steps for accumulating and managing the additional information for each viewing location regarding the viewpoint free-form image. May be. In such a case, the additional information acquisition means or step may acquire additional information corresponding to the angle of view designated by the angle of view designation means in the viewpoint free-form image from the accumulated additional information. it can.

Further, the additional information may be provided on one or more additional information providing servers provided on an external network such as the Internet. For example, a wide area information search system such as WWW is used to provide the additional information content. In such a case, the additional information acquisition means or step can search for and download the required additional information on the external network in the form of an HTTP request or the like.

Further, in the viewpoint free-form image display device or method according to the first aspect of the present invention, in response to the additional information being acquired by the additional information acquiring means or step,
Alternatively, it may further include a charging means or step for charging an information usage fee for each additional information used in response to the additional information being displayed by the viewing image display means or step.

As a result, it becomes easy to construct a profit model of an advertisement fee for viewpoint free-form video, and it is possible to contribute to the development of the viewpoint free-form video distribution business.

A second aspect of the present invention is a storage in which computer software physically written in a computer-readable format is described so as to execute a process of displaying a viewpoint free-form image on a computer system. The medium is a medium, and the computer software includes a viewpoint free-form image input step of inputting a viewpoint free-form image, an angle-of-view specifying step of specifying a view-angle of view of the viewpoint free-form image, and the angle-of-view specifying step. Additional information acquisition step of selectively acquiring additional information according to the angle of view specified by, and the angle of view specified by the angle of view specifying step is cut out from the viewpoint free-form image and displayed together with the selected additional information. And a viewing image display step for performing the same.

The storage medium according to the second aspect of the present invention is, for example, a medium for providing computer software in a computer-readable format to a general-purpose computer system capable of executing various program codes. Such a medium is, for example, a DVD (Digital Vers
atile Disc), CD (Compact Disc), FD (Flexible Disc)
Disk), MO (Magneto-Optical disc), and other removable storage media. Alternatively, it is technically possible to provide computer software to a specific computer system via a transmission medium such as a network (whether the network is wireless or wired).

Further, the storage medium according to the second aspect of the present invention is a computer system provided with a predetermined computer.
A computer for realizing software functions
It defines a structural or functional collaborative relationship between software and a storage medium. In other words, the second aspect of the present invention
By installing the predetermined computer software in the computer system via the storage medium according to the first aspect, the cooperative action is exerted on the computer system, and the viewpoint free-form image according to the first aspect of the present invention. The same effect as that of the display device or the method thereof can be obtained.

A third aspect of the present invention is a computer program written so as to execute a process of displaying a viewpoint free-form image on a computer system, and inputting a viewpoint free-form image. View-free-form image input step, view-angle specifying step for specifying a view-angle of view of viewpoint-free-form image, and addition for selectively obtaining additional information according to the view-angle specified by the view-angle specifying step A computer program comprising: an information acquisition step; and a viewing image display step of cutting out the view angle specified by the view angle specifying step from the viewpoint free-form image and displaying the view information together with the selected additional information. Is.

A computer according to the third aspect of the present invention
The program defines a computer program written in a computer-readable format so as to realize a predetermined process on a computer system. In other words, by installing the computer program according to the third aspect of the present invention in the computer system, a cooperative action is exerted on the computer system, and the viewpoint free-form according to the first aspect of the present invention. The same effects as those of the image display device or the method thereof can be obtained.

A fourth aspect of the present invention is a broadcasting system for viewpoint free-form video, wherein the viewpoint free-form video to be broadcast can specify a viewing angle of view on the receiver side. The broadcasting system is characterized in that the added information is different for each viewing location in the frame.

The broadcasting system according to the fourth aspect of the present invention can distribute free-viewpoint video. The viewpoint free-form video is a viewing mode in which the user or the viewer arbitrarily designates the angle of view, that is, the viewing location, and while a new amusement effect can be expected, it is displayed when the viewing location is switched on the user side. There is a problem where to place the additional information on the screen. Further, the advertising effect of the advertising information may differ for each viewing location, and it is necessary to reconsider the selection of the additional information to be used as the viewing angle of view moves.

According to the broadcasting system of the fourth aspect of the present invention, when the user specifies the viewing angle of view of the viewpoint free-form image, the viewpoint free-form video is displayed according to the specified angle of view. At the same time, the additional information corresponding to the angle of view is dynamically selected and the content is distributed together with the viewed image. Therefore, it is possible to increase the chances of increasing the degree of exposure of the advertisement, and it is possible to selectively use the advertisement information having a higher advertisement effect depending on the viewing place.

A fifth aspect of the present invention is a storage medium storing a viewpoint free-form image, wherein the stored viewpoint free-form image can freely select a viewing angle. The storage medium is characterized in that additional information is set according to a viewing location in the video frame.

The storage medium according to the fifth aspect of the present invention can distribute free-viewpoint images. The viewpoint free-form video is a viewing form in which the user or the viewer arbitrarily designates the angle of view, that is, the viewing place. While a new amusement effect can be expected, the viewing place is switched by the user who plays the content. Sometimes, there is the problem of where to place the additional information on the display screen. Further, the advertising effect of the advertising information may differ for each viewing location, and it is necessary to reconsider the selection of the additional information to be used as the viewing angle of view moves.

According to the storage medium of the fifth aspect of the present invention, when the user specifies the viewing angle of view of the viewpoint free-form image, the viewpoint free-form image is displayed at the specified angle of view. At the same time, the additional information corresponding to the angle of view can be dynamically selected, and the content can be reproduced so as to be superimposed and displayed on the viewed image. Therefore, it is possible to increase the chances of increasing the degree of exposure of the advertisement, and it is possible to selectively use the advertisement information having a higher advertisement effect depending on the viewing place.

A sixth aspect of the present invention is an additional information management apparatus or method for managing additional information used when displaying a viewing image, wherein the viewing image arbitrarily designates a viewing angle. From the additional information storage means, a possible viewpoint free-form image, and additional information storage means or step for storing additional information in association with a viewing place regarding the viewpoint free-form image, and additional information corresponding to a designated angle of view. An additional information management device or method comprising: additional information reading means or step for taking out.

Since the viewpoint or free-view image can provide a viewing mode in which the user or the viewer arbitrarily specifies the angle of view, that is, the viewing location, a new amusement effect is expected, but the viewing location is set on the user side. There is a problem of where to place the additional information on the display screen when switching. Further, the advertising effect of the advertising information may differ for each viewing location, and it is necessary to reconsider the selection of the additional information to be used as the viewing angle of view moves.

According to the additional information management device or method of the sixth aspect of the present invention, it is possible to manage the additional information corresponding to each viewing location of the viewpoint free-form image. Therefore, when the user specifies the viewing angle of view in the viewpoint free-form image, the viewpoint free-form video is displayed according to the specified angle of view, and the additional information corresponding to the angle of view is dynamically selected. Then, the content is distributed together with the viewed image.
As a result, it is possible to increase the opportunity to increase the degree of exposure of the advertisement, and it is possible to selectively use the advertisement information having a higher advertisement effect depending on the viewing place.

Further objects, features and advantages of the present invention are as follows.
It will be apparent from the embodiments of the present invention described later and the more detailed description based on the accompanying drawings.

[0037]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below in detail with reference to the drawings.

First, as an example of the viewpoint free-form video, input of 360-degree video by an omnidirectional camera of the type that simultaneously shoots omnidirectional video around a certain point on the space as a viewpoint and its image processing will be described. To do. As will be described later, on the receiver user side, a video frame of an arbitrary viewpoint can be cut out and viewed from the omnidirectional video.

FIG. 1 shows the configuration of an omnidirectional camera 10 of the type applied to the present embodiment, which simultaneously shoots 360-degree omnidirectional images of a certain point in space as a viewpoint.

As shown in the figure, the omnidirectional camera 10 is
Eight cameras 1 arranged in a predetermined circumferential direction at substantially equal intervals
1A to 11H, and a reflecting portion 12 in which eight plane mirrors arranged in the line-of-sight direction of each camera 11A are arranged on an octagonal pyramid. Here, each of the cameras 11A to 11H
Of the cameras 11A to 11H so that their projection centers coincide with each other near the center of the omnidirectional camera 10 and that the line-of-sight directions of the cameras 11A to 11H are spaced at a predetermined angle on one horizontal plane.
A to 11H are installed.

By photographing the surrounding scene reflected by the corresponding plane mirror by each camera 11A, the omnidirectional camera 10 as a whole can photograph a mirror image of a 360-degree image in the horizontal direction.

By connecting a VTR to each of the cameras 11A ..., a total of eight video (and audio) streams can be recorded. Further, by capturing the video stream recorded by each camera 11A ... into the computer as computer data (bitmap file) via the video capture card,
The camera images of two books are pasted together and one horizontal direction 360
It is possible to generate an omnidirectional image of a degree.

At the time when each camera image recorded by the VTR is captured as a bit map file by the computer, one video stream has 1
The video of the frame is in SD (Standard Definition) format, and is represented as a still image composed of 720 × 480 pixels.

In the configuration of the omnidirectional camera 10 shown in FIG.
The images picked up by each of the cameras 11A ... Are recorded in a rotated form. Relationship between camera angles of view and 8 cameras 11A ...
For the convenience of matching the viewpoints, the processings such as 90 degree rotation and mirror inversion are performed before the eight video streams are combined, as shown in FIG.

In FIG. 3, 90 for each M frame of the video material data taken by the eight cameras 11A to 11H.
A processing procedure for performing processing such as degree rotation and mirror inversion is shown in the form of a flowchart. Such a processing procedure is realized, for example, by executing a predetermined control program on a computer that captures video frames of the cameras 11A to 11H. However,
n and m are variables introduced for counting the number of cameras and frames, respectively.

First, the initial value 1 is substituted for n and m (steps S1 and S2). Then, capturing is performed on the video data of the mth frame of the nth camera (step S3).

Next, lens distortion correction is performed on the captured bitmap video frame based on the distortion parameter of the nth camera (step S).
4).

Then, as shown in FIG. 2, the lens distortion-corrected bitmap image frame is rotated by 90 degrees (step S5), and after mirror inversion (step S6), the data is stored in the hard disk device or the like. Save (step S7).

Then, m is incremented by 1 (step S8), and if the frame number m does not reach the maximum frame number M (step S9), the process returns to step S3 to process the above-mentioned video frame data. Repeat the process.

On the other hand, when the number of processing frames for the nth camera reaches the predetermined value M, n is incremented by 1 (step S10), and if n has not reached 8 (step S11). ), Returning to step S2, the above-described processing of the video frame data is repeatedly executed for the video frame of the next camera.

Also, all eight cameras 11A-11H
When the data processing is completed, the data is saved (step S12), and the entire processing routine ends.

After processing the video frame data for each camera 11A ... As shown in FIG.
8 synchronized video frames of A ... Are pasted together as shown in FIG. 4 to generate one frame. At this time, as the configuration of the omnidirectional camera 10, each of the cameras 11A to 1A
By arranging the respective video frames on the side surface of the cylinder in the order in which 1H is installed (see FIG. 1), the combined video frames are captured by the omnidirectional camera 10.
An omnidirectional image of 0 degrees can be displayed.

When such an omnidirectional video frame is distributed by terrestrial broadcasting, satellite broadcasting, cable / television broadcasting, high-definition broadcasting, etc., on the receiver side, the viewing user specifies from among the received omnidirectional video frames. By cutting out and displaying the angle of view, the original camera 11
It is possible to seamlessly watch an arbitrary angle of view from an omnidirectional scene without being aware of whether the image was shot in A ...

For example, in the omnidirectional video frame shown in FIG. 4, when the viewing angle F1 is designated by the viewing user, the video frame corresponding to this is cut out and displayed on the receiver, but the cameras 11A and 11B are displayed. It is not necessary for the viewing user to be aware of the fact that the video frames captured in (3) are connected. Similarly, even when the angle of view is moved to F2, the video frame corresponding to this is cut out and displayed on the receiver.
It is not necessary for the viewing user to be aware that it is made by connecting the video frames captured in C and 11D.

When a viewpoint free-form video frame such as an omnidirectional video is distributed by terrestrial broadcasting, satellite broadcasting, cable television broadcasting, high-definition broadcasting, etc., the video frame has a huge amount of data. HD
(High Definition) -MPEG2 (Moving Picture Ex)
Image compression is performed using the perts Group 2) method. At this time, it is necessary to make the aspect ratio closer to the image compression method because of the request to match the omnidirectional video with the size of HD-MPEG2. For this reason, as shown in FIG. 4, an omnidirectional image originally formed by arranging the image frames of the cameras 11A to 11H on the cylindrical side surface is divided into two equal parts, and the latter half frame division part is divided into the first half. It is arranged below the frame division.

As a result of performing such frame division and rearrangement, the video frames of the cameras 11A and 11H and the video frames of the cameras D and E are originally continuous, but data is distributed. On the omnidirectional video frame, they are arranged at distant places as shown in FIG.

In such a case, when the angle of view of the omnidirectional video image is moved on the receiver side, the area extending over the video frame of the camera H and the camera A or the area extending over the video frame of the camera D and the camera E. It is difficult to switch seamlessly.

Therefore, in this embodiment, when dividing and rearranging the omnidirectional image frame having a cylindrical side surface, the image frames of the cameras adjacent to one of the divided edges are additionally attached. That is, in the vicinity of both edges of the omnidirectional video, the video frames of the pasted cameras are rearranged so that they overlap.

In FIG. 5, when dividing and rearranging the omnidirectional video frame having a cylindrical side surface, the video frames of the cameras adjacent to one of the divided edges are additionally attached. , An omnidirectional video frame for distribution is shown.

In the example shown in the figure, the data is finally HD.
In order to match the size of the MPEG2 system, including the fact that the aspect ratio is the same, the head of the omnidirectional video frame division portion in the first half is adjacent to the camera 11H adjacent to the camera 11A and the camera 11H. Camera 11
G video frames are sequentially pasted together, and the camera 11E is placed at the beginning of the omnidirectional video frame division part in the latter half.
The video frames of the camera 11D adjacent to the camera 11D and the video frame of the camera 11C adjacent to the camera 11D are sequentially attached.

Therefore, in the illustrated omnidirectional video frame, the video frames of the cameras 11C and 11D are overlapped in the first half and the second half, and the cameras 11G and 11D are also present.
The video frames of H are duplicated in the first half and the second half.

In the present embodiment, in order to deliver the omnidirectional video frame as shown in FIG. 5, the data size is further reduced, and dummy data is added to add H
It is adapted to the size of D-MPEG (1920 × 1080 pixels) (see FIG. 6), and one frame is used as one bitmap file to generate a serial number file group of bitmaps.

The data of the bitmap serial number file of the format shown in FIG. 6 is rendered on a computer,
The signal is converted into an HD-SDI (Serial Digital Interface) output signal and encoded and compressed by an MPEG2 hardware encoder to generate HD-MPEG2 format transport stream data.

FIG. 7 shows each camera 11A that has been captured and processed according to the processing procedure shown in FIG.
The processing procedure for generating the bitmap serial number file data (omnidirectional video frame for distribution) in the format as shown in FIG. 6 from the video frames of ˜11H is shown in the form of a flowchart. This processing procedure is realized, for example, in the form of executing a predetermined control program on a computer that captures the video frames of the cameras 11A to 11H.

First, the initial value 1 is substituted for the variable n (step S21). Then, the image data of the m-th frame of the n-th camera is extracted from the data stored after 90-degree rotation and mirror inversion processing (see step S12 in FIG. 3) (step S22).

Next, m of the nth camera taken out
The video data of the frame is pasted on the side surface of the cylinder in the three-dimensional coordinate system (step S23). This is realized by performing a plane curved surface conversion.

Further, the image data pasted on the side surface of the cylinder is arranged according to the camera direction (step S2).
4).

Then, n is incremented by 1 (step S25), and if n has not reached 8 (step S26), the process returns to step S22 and the above-mentioned video frame of the video frame of the next camera is added. The pasting process on the side surface of the cylinder is repeatedly executed.

All eight cameras 11A to 11H
When the process of attaching the video frame to the side surface of the cylinder is completed, the radius of the cylinder and other parameters are calculated while adjusting the consistency of the adjacent images (step S).
27).

Then, the side surface of the cylinder is cut and converted into a rectangular plane (step S28).

Further, this rectangular plane is divided at half in the horizontal axis direction (camera arrangement direction), and the right half thereof is arranged below the left half (step S29).

Then, the lower right quarter area of the rectangular plane arranged in step S29 is duplicated to be placed at the left end of the upper left portion, and the upper right quarter area of the rectangular plane is duplicated. , Located at the left end of the lower left (step S
30).

Next, the rectangular plane including the overlapping portion at the left end is reduced to a size of 1920 × 948 pixels so as to fit the size of the HD-MPEG system (step S31).

Next, dummy data is added to the lower part of the rectangular plane reduced to a size of 1920 × 948 pixels, and 1920 × 1080 compliant with the HD-MPEG system.
The pixel data is used (step S32).

Finally, the various geometric transformation parameters from step S22 to S32 are saved (step S3
3), the entire processing routine ends.

FIG. 8 shows, in the form of a flow chart, a processing procedure for rendering an omnidirectional video frame made into a bitmap serial number file according to the processing procedure shown in FIG. This processing procedure is, for example, performed by each camera 1
It is realized by executing a predetermined control program on a computer that captures video frames 1A to 11H.

First, the initial value 1 is assigned to the variables m and n (steps S41 and S42).

Next, from the bitmap serial number file, n
The video data of the m-th frame for the second camera is taken out (step S43).

Then, n is incremented by 1 (step S44), and if n has not reached 8 (step S45), the process returns to step S42 to take out the video data of the mth frame of the next camera. .

All eight cameras 11A to 11H
When the extraction of the video data of the m-th frame is completed, the extracted video data is geometrically converted using the parameters stored in step S33 of FIG. 7, and the data is saved. (Step S46).

Then, m is incremented by 1 (step S47), and the frame number m is the maximum frame number M.
If not reached (step S48), step S
Returning to step 42, the geometric conversion process for the above-mentioned video data is repeated.

When the geometric conversion processing has been completed for all video data, the entire processing routine is completed.

An omnidirectional video frame for distribution generated according to the processing procedure shown in FIG. 7 (see FIG. 6).
As a bitmap serial number file, further rendered on a computer according to the processing procedure as shown in FIG. 8, converted into a signal of HD-SDI (Serial Digital Interface) output, and encoded by an MPEG2 hardware encoder. It is possible to generate the HD-MPEG2 format transport stream data by performing the compression compression.

On the side of the receiver that receives such a transport stream, it is possible to perform free-viewpoint reception, as will be described later. That is, it is possible to seamlessly cut out and display an arbitrary angle of view designated by the viewing user from the omnidirectional video frame. The viewing user does not need to be aware of the video frame of which camera of the omnidirectional camera 10 the displayed video is.

In order to broadcast an omnidirectional video frame as described above and receive free viewpoint from the receiver side, it is necessary to send data for determining that this data is an omnidirectional video.

In existing satellite broadcasting, PSI (Program Service In) which defines the type of data broadcasted, etc.
formation) data, a flag or the like indicating that the broadcast program is a 360-degree omnidirectional video is added there, and the receiver side discriminates this flag to move the cursor key (described later). You can activate and allow free viewpoint viewing. (A part of the transmission control signal standardized by ISO / IEC13818-1 is called PSI. PSI data consists of four types of tables (PAT, CAT, PMT, NIT) and is transmitted in section format.)

Further, when performing data broadcasting, BML (Broadcast markup Languag) as data broadcasting content is added to the viewpoint free-form video content as described above.
e) Content is added.

The viewpoint free-form video can provide a viewing mode in which the viewing location is arbitrarily specified by the user or the viewer side. On the other hand, when the viewing location is switched by the user side, the viewing screen is displayed. There is a problem of where to place the additional information. Further, the advertising effect of the advertising information may differ for each viewing location, and it is necessary to reconsider the selection of the additional information to be used as the viewing angle of view moves. You can expect new amusement effects.

Therefore, in this embodiment, different viewpoint advertisements are prepared for the viewpoint free-form video depending on the viewing angle. Further, the advertisement information is composed of a advertisement table such as a banner advertisement and a lookup table describing the correspondence between the start point coordinates and / or the end point coordinates of the angle of view and the banner advertisement to be used. Then, when distributing the viewpoint free-form video by satellite broadcasting or the like, such advertisement information can be transmitted at the same time as the body of the viewpoint free-form video as BML content for data broadcasting.

On the user side receiving the video contents,
When you specify the viewing angle of view from the viewpoint free-form image, the viewpoint free-form image is displayed according to the specified angle of view, and additional information corresponding to the angle of view is dynamically selected.
It is superimposed and displayed on the viewed image. Therefore, it is possible to increase the chances of increasing the degree of exposure of the advertisement, and it is possible to selectively use the advertisement information having a higher advertisement effect depending on the viewing place.

FIG. 9 schematically shows how different banner advertisements are set for each area of the viewpoint free-form video frame.

In the example shown in the figure, the viewpoint free-form video frame is divided into four areas, and the banner advertisement 1, the banner advertisement 2, the banner advertisement 3 and the banner advertisement 4 are divided for each of the areas 1 to 4. Are set respectively.

In such a case, on the receiver side of the viewpoint free-form broadcasting, when the viewing angle of view is designated in the area 1, the banner advertisement 1 is superimposed and displayed, and the viewing angle of view is designated in the area 2. When the viewing angle of view is designated in the area 3, the banner advertisement 3 is superimposed and displayed, and when the viewing angle of view is designated in the area 4, the banner advertisement 4 is displayed. It will be displayed superimposed.

Next, a viewpoint free-form video broadcasting system for distributing and viewing the viewpoint free-form video will be described.

FIG. 10 schematically shows the configuration on the transmission (broadcasting station) side of the viewpoint free-form video broadcasting system according to the embodiment of the present invention. As shown in the figure, the transmission side is composed of an omnidirectional camera system 10 that acquires an omnidirectional image as a viewpoint free-form image, and a transmitter 110 arranged in a broadcasting station or the like.

The camera system 10 for taking an omnidirectional video image at a predetermined location includes a plurality of cameras 11A ... As already described with reference to FIG. further,
A microphone for inputting voice at the shooting site may be provided. When the camera system 10 shoots an omnidirectional image from a certain point in space, each camera 11
A ... Are arranged at substantially equal intervals in a predetermined circumferential direction.

A VTR is connected to each of the cameras 11A, 11B, ...
Streams can be recorded concurrently.

Various computer processes can be performed by video-capturing these recorded video streams through the switch 111 and loading them into the computer 112 as computer data (bitmap file). it can. For example, each camera image is captured, data processing such as 90-degree rotation and mirror inversion explained with reference to the flowchart shown in FIG. 3, and bitmap serial number explained with reference to the flowchart shown in FIG. The viewpoint free-form video data for distribution can be generated through processing such as generation of file data and rendering of the video frame described with reference to the flowchart shown in FIG.

The viewpoint free-form video data output by the computer 112 is a video frame of 1920 × 1080 pixel size which can be handled by the HD-MPEG system. The subsequent MPEG encoder 113 encodes the video stream by the HD-MPEG method.

The PSI input device 114 sets a flag indicating that the video stream is a viewpoint free-form omnidirectional video. Further, the BML input device 115 inputs the data broadcasting content which is described in the BML format and is edited in advance. The BML content includes a script describing a moving rule and a scaling rule of a display area of a viewpoint free-form video. In addition, the BML content may include advertisement information regarding viewpoint free-form video. As described above with reference to FIG. 9, for example, the advertisement information is divided into a plurality of areas in the free-viewpoint video frame, and a different banner advertisement is set for each area, which is in the BML script format. It has been described.

The multiplexer (MUX) 116 is an HD
-MPEG encoded video stream and PSI
Data and BML contents are multiplexed. The paid content is scrambled by the scrambler 117 in order to prevent it from being viewed without permission by interception or the like.

Then, the transmission section (TX) 118 modulates the scrambled video stream, performs A / D conversion, and up-converts the baseband signal into a satellite wave, and then transmits the satellite wave. The video stream reaches the receiver (set top box: STB) 150 via the broadcasting satellite 170.

The key issuing device 119 also issues Key data for descrambling the scrambled video stream to restore it to its original state. Key data is distributed in advance to a subscriber (subscriber) who receives and views pay contents by using a telephone line, a network, or a satellite.

FIG. 11 schematically shows the structure of the receiving side of the viewpoint free-form video broadcasting system according to the embodiment of the present invention.

On the receiver 150 side, the received satellite wave is down-converted into a baseband signal in the receiving section (RX) 151, A / D converted and demodulated,
It is temporarily held in the frame buffer 152. Receiver 1
On the side of 50, first, by checking the PSI data, it can be confirmed that the received video frame is a viewpoint free-form video.

On the other hand, the subscriber receives the Key data issued by the Key data issuing device 119 via the telephone line, the network or the like, and the IC card 162
Etc. The descrambler 153 takes out the Key data from the IC card 162 and stores it in the buffer 1
The scrambled data stored in 52 is descrambled to restore the video stream. Further, the subsequent demultiplexer 154 separates this into an MPEG stream and BML content.

The MPEG decoder 155 MPEG-decodes the MPEG stream, reproduces the original viewpoint free-form video frame, and temporarily stores it in the frame buffer 156.

Further, the BML engine 159 processes BML contents. For example, BML for data broadcasting
The content is analyzed and the resulting display data is superimposed on the viewpoint free-form video frame in the frame buffer 156. It also analyzes the moving rules and scaling rules of viewpoint free-form video written as scripts. In addition, the BML engine 159 uses the scroller 15
The BML script is executed according to the viewing angle of view designated by 8.

In this embodiment, the advertisement information is included in the received BML content. The viewpoint free-form video frame is divided into a plurality of areas, and a different banner advertisement is set for each area. The BML engine 159
The advertisement information extracted from the BML content is stored in the advertisement information management unit 160. Further, as shown in FIG. 9, it is described as a BML script which advertisement information is used according to the position of the view angle designated on the viewpoint free video frame. BML engine is such a BM
The L script is executed to dynamically determine the advertisement information to be used according to the specified view angle.

The scroller 158 is a user interface for the user or viewer to specify the viewing angle of view to be cut out from the viewpoint free-form video and displayed.
Further, it is possible to instruct enlargement / reduction (ZOOM) of a display image through the scroller 158.

The cut-out driver 157 cuts out a viewable angle of view from the viewpoint free-form video frame stored in the frame buffer 156, and outputs it to the television monitor for display. First, the video at the default camera position is cut out and displayed on the TV monitor. When the user inputs a viewing angle of view specification via the scroller 158, the BML engine 159 outputs the BML.
The specified angle of view is cut out from the viewpoint free-form image in the frame buffer 156 according to the movement rule and the enlargement / reduction rule obtained by executing the script.
Display output on the monitor.

Advertising information to be used differs depending on the viewing angle of view designated by the user input through the scroller 158 (FIG. 9). The advertisement information selection unit 161 selects the advertisement information (banner advertisement) corresponding to the currently specified viewing angle of view from the advertisement information management unit 160 according to the execution result of the BML script by the BML engine 159,
The data is output to the cutout driver 157. The cutout driver 157 superimposes the received advertisement information on the viewing angle of view cut out from the frame buffer 156, and the
Display output on the monitor. Therefore, it is possible to increase the chances of increasing the degree of exposure of advertisements, and it is possible to selectively use advertisement information having a higher advertisement effect depending on the viewing location.

The selection result by the advertisement information selection unit 161 is
It corresponds to an advertisement information exposure history indicating how much each advertisement information is viewed or exposed. The exposure history of such advertisement information is important data in order to accurately and fairly determine the advertisement fee to be charged to each advertiser. The advertisement information selection unit 161 may sequentially record the advertisement information exposure history in, for example, the IC card 162. Then, the advertisement information exposure history for each user written in the IC card 162 is transferred to a broadcasting station or another predetermined institution via an upper link such as a telephone line or a network so that each advertisement information can be obtained. The viewing / exposure history of may be aggregated at one place for centralized management.

In the viewpoint free-form video distribution system according to this embodiment, the additional information corresponding to the angle of view specified by the receiving user is dynamically selected. In addition, it is possible to selectively use the advertising information having a higher advertising effect depending on the viewing place.
Further, it is expected that the advertising information used according to the designation of the angle of view is frequently switched, but the advertising fee can be calculated according to the usage frequency of each information information. This makes it easier to construct a profit model of an advertisement fee related to viewpoint free-form video, and can contribute to the development of the viewpoint free-form video distribution business.

FIG. 12 shows, in the form of a flowchart, a processing procedure for displaying / viewing a viewpoint free-form image performed by the receiver 150. The operation of the receiver 150 will be described below with reference to this flowchart.

When the broadcast data is received, first, the flag data of the PSI data is extracted (step S51),
It is determined whether the received data is a viewpoint free-form video stream (step S52).

If the received data is not the viewpoint free-form video stream, it is MPEG-decoded as normal broadcast data, and the decoded video stream is subjected to normal display processing on the TV monitor screen (step S59).

On the other hand, if it is determined that the received data is a viewpoint free-form video stream (step S5)
2) First, the image at the default camera position is cut out and displayed on the television monitor (step S53). Next, acceptance of the operation for moving the viewing angle of view via the scroller 158 is started (step S54).

When a user input is made to the scroller 158, the cutout driver 57 is interrupted. In the subsequent decision block S55, the scroller 15
It is determined whether or not the viewing angle of view of the viewpoint free-form video frame, that is, the cutout display area has been moved by a user input via 8.

While there is no movement of the viewing angle of view, the viewing angle of view of the coordinates already specified is cut out from the viewpoint free-form video frame and displayed and output on the television monitor (step S5).
9) Then, the process returns to step S54 to wait until the next operation of moving the viewing angle of view.

If the viewing angle of view has moved, then the advertisement information selecting process corresponding to the specified viewing angle of view is executed (step S56). The selection process of the advertisement information can be described in, for example, a BML script format, and details thereof will be described later.

After the advertisement information selection processing is performed, the selected advertisement information is stored in the frame buffer 15
It is superposed on the viewing angle of view cut out from No. 6 and displayed on the television monitor (step S57). Then, the process returns to step S54 and waits until the next operation of moving the viewing angle of view.

FIG. 13 shows step S5 shown in FIG.
6 shows a procedure of a process of selecting advertisement information according to the viewing angle of view in 6 in the form of a flowchart. This processing procedure can be described in the form of a BML script, for example. Hereinafter, with reference to this flowchart, a process of selecting advertisement information according to a viewing angle of view will be described.

First, the drawing start coordinates (x, y) of the viewing angle designated by the scroller 158 are acquired (step S60).

Next, in the decision block S61, the drawing start coordinates (x, y) are 640 ≦ x <1 in the viewpoint free-form video frame consisting of 1920 × 1080 pixels.
It is determined whether 280 and 0 ≦ y <474 is satisfied.

If the determination result in step S61 is affirmative, the banner advertisement 1 assigned to the area is taken out from the advertisement information management section 160 (step S6).
2), the entire processing routine ends.

On the other hand, when the result of the determination in step S61 is negative, in the subsequent determination block 63, the drawing start coordinate (x, y) is 1280 ≦ x <1920 within the viewpoint free-form video frame. And 0 ≦ y <4
74, or 0 ≦ x <640 and 474 ≦ y <9
It is determined whether 48 is satisfied.

If the determination result in step S63 is affirmative, the banner advertisement 2 assigned to the area is taken out from the advertisement information management section 160 (step S6).
4) The entire processing routine is ended.

On the other hand, when the result of the determination in step S63 is negative, the drawing start coordinate (x, y) is 640≤x <1280 in the viewpoint free-form video frame in the subsequent decision block 65. And 474 ≦ y <
It is determined whether 948 is satisfied.

If the determination result in step S65 is affirmative, the banner advertisement 2 assigned to the area is taken out from the advertisement information management section 160 (step S6).
6) The entire processing routine ends.

On the other hand, if the result of the determination in step S65 is negative, then in the following decision block 67, the drawing start coordinates (x, y) are 1280≤x <1920 within the viewpoint free-form video frame. And 474 ≦ y
<948, or 0 ≦ x <640 and 0 ≦ y <4
It is determined whether or not 74 is satisfied.

If the determination result in step S67 is affirmative, the banner advertisement 2 assigned to the area is taken out from the advertisement information management section 160 (step S6).
8) Then, the entire processing routine ends.

On the other hand, if the determination result in step S67 is negative, error processing is performed (step S69), and then the entire processing routine ends.

In short, the receiver 150 side demultiplexes the stream data sent thereto and executes the BML script according to the viewpoint movement performed via the scroller 158. As a result, the display of additional information such as a banner advertisement can be dynamically switched according to the viewing angle of view. In the BML script, since each advertisement information is defined in a form that depends on the location of the specified angle of view, for example, Z such as enlargement or reduction within the defined coordinate area.
Even when the zooming operation is performed, the same advertisement information can be displayed.

[Supplement] The present invention has been described in detail with reference to the specific embodiment. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the scope of the present invention.

In the present specification, the present invention has been described by taking as an example a broadcasting system that transmits and receives omnidirectional video in a viewpoint free form, but the gist of the present invention is not limited to these application examples. Absent. That is, in addition to the above-described embodiment, the present invention can be applied to the case of similarly performing viewpoint free-form viewing by utilizing the mechanism of cutting out the viewpoint from the normal high-definition video on the receiver side.

In short, the present invention has been disclosed in the form of exemplification, and the contents described in this specification should not be construed in a limited manner. To determine the gist of the present invention,
The claims section mentioned at the beginning should be taken into consideration.

[0138]

As described above in detail, according to the present invention,
Superior viewpoint free-form image display apparatus and method, storage medium, computer program, and broadcast capable of suitably displaying a viewpoint free-form image that the user can freely switch between viewpoints during playback / display A system and an additional information management device and method can be provided.

Further, according to the present invention, an excellent viewpoint free-form image display device and method, a storage medium, a computer program, capable of displaying and outputting an advertisement and other additional information to a viewpoint free-form image, It is possible to provide a broadcasting system, and an additional information management device and method.

Further, according to the present invention, an excellent viewpoint free-form image display device and method, a storage medium, a computer program, which can display processing of additional information such as advertisement according to a set viewpoint, It is possible to provide a broadcasting system, and an additional information management device and method.

In the viewpoint free-form video system to which the present invention is applied, the contents of the data broadcast can be changed according to the viewpoint specified by the user or the viewer, and the opportunity to increase the exposure of the advertisement can be increased. Also,
By defining the data that depends on the cutout coordinate area,
Not only the advertisement but also the additional information for the subject can be attached and sent.

[Brief description of drawings]

FIG. 1 is a 360 point in which one point in space is designated.
FIG. 1 is a diagram showing a configuration of an omnidirectional camera 10 of a type that simultaneously shoots an omnidirectional image of a degree.

[Fig. 2] Fig. 2 is a diagram showing a state of performing a process of rotating a video image by 90 degrees and a mirror inversion process, which is performed before combining eight video streams.

FIG. 3 is a flowchart showing a processing procedure for performing processing such as 90 ° rotation and mirror inversion on each M frame of video material data captured by eight cameras 11A to 11H.

FIG. 4 is a diagram showing a state where eight synchronized video frames of each camera 11A ... Are pasted together according to the arrangement of the cameras 11A ... to generate one frame.

FIG. 5 is an omni-direction for distribution configured such that, when dividing and rearranging an omnidirectional video frame having a cylindrical side surface, video frames of a camera adjacent to one of the divided edges are additionally attached. It is the figure which showed the example of a video frame.

FIG. 6 shows a state in which the data size of an omnidirectional video frame as shown in FIG. 5 is reduced and dummy data is added to fit the size of HD-MPEG (1920 × 1080 pixels). It is a figure.

7 is a flowchart showing a processing procedure for generating bitmap serial number file data (distribution omnidirectional video frame) in a format as shown in FIG. 6 from video frames of the cameras 11A to 11H.

FIG. 8 is a flowchart showing a processing procedure for rendering an omnidirectional video frame made into a bitmap serial number file according to the processing procedure shown in FIG. 7.

FIG. 9 schematically illustrates how different banner advertisements are set for each area of a viewpoint free-form video frame.

FIG. 10 is a diagram schematically showing a configuration on the transmission (broadcasting station) side of the viewpoint free-form video broadcasting system according to the embodiment of the present invention.

FIG. 11 is a diagram schematically showing the configuration of the receiving side of the viewpoint free-form video broadcasting system according to the embodiment of the present invention.

FIG. 12 is a flowchart showing a processing procedure for performing display / viewing processing of a viewpoint free-form video performed in the receiver 150.

FIG. 13 is a flowchart showing a procedure of a process of selecting advertisement information according to a viewing angle of view.

[Explanation of symbols]

10 ... Omnidirectional camera system 11 ... Camera 12 ... Reflector 110 ... Transmitter 111 ... Switcher, 112 ... Computer 113 ... MPEG encoder, 114 ... PSI input device 115 ... BML input device, 116 ... Multiplexer 117 ... Scrambler, 118 ... Transmission unit 119 ... Key data issuing device 150 ... Receiver 151 ... Reception unit, 152 ... Frame buffer 153 ... Descrambler, 154 ... Demultiplexer 155 ... MPEG decoder, 156 ... Frame buffer 157 ... Cutout driver, 158 ... Scroller 159 ... BML engine 160 ... Advertisement information management unit, 161 ... Advertisement information selection unit

Claims (18)

[Claims] 1. A viewpoint free-form image display device for displaying a viewpoint free-form image, wherein a viewpoint free-form image input means for inputting a viewpoint free-form image and a viewing angle of view of the viewpoint free-form image are designated. Angle of view specifying means, additional information acquisition means for selectively acquiring additional information according to the angle of view specified by the angle of view specifying means, and the angle of view specified by the angle of view specifying means A viewpoint free-form image display device, comprising: a viewing image display unit that cuts out an image and displays it together with the selected additional information. 2. The viewpoint free-form image display device according to claim 1, wherein the additional information is advertisement information associated with each viewing location of the viewed image. 3. An additional information management unit that manages additional information for each viewing location regarding the viewpoint free-form image, wherein the additional information acquisition unit is an image designated by the angle-of-view designating unit in the viewpoint free-form image. 2. The viewpoint free-form image display device according to claim 1, wherein the additional information corresponding to the corner is acquired from the additional information management means. 4. The additional information management means is composed of one or more additional information providing servers arranged on an external network, and the additional information acquisition means is the view angle designating means of the viewpoint free-form image. 4. The viewpoint free-form image display device according to claim 3, wherein additional information corresponding to a designated angle of view is searched for on the external network. 5. Each additional information used, in response to the additional information being acquired by the additional information acquiring means or in response to the additional information being displayed by the viewing image display means. 2. The viewpoint free-form image display device according to claim 1, further comprising a charging means for charging the information usage fee. 6. A viewpoint free-form image display method for displaying a viewpoint free-form image, comprising: a viewpoint free-form image input step of inputting a viewpoint free-form image; Angle of view specifying step, an additional information acquisition step of selectively acquiring additional information according to the angle of view specified by the angle of view specifying step, and the angle of view specified by the angle of view specifying step A viewpoint free-form image display method, comprising: a viewing image display step of cutting out from an image and displaying the additional information together with the selected additional information. 7. The viewpoint free-form image display method according to claim 6, wherein the additional information is advertisement information associated with each viewing location of the viewed image. 8. An additional information management step for managing additional information for each viewing location regarding the viewpoint free-form image, wherein in the additional information acquisition step, the image specified in the view angle specification step among the viewpoint free-form images. 7. The viewpoint free-form image display method according to claim 6, wherein the additional information corresponding to the corner is acquired from the additional information managing means. 9. The additional information is managed by one or more additional information providing servers arranged on an external network, and the additional information acquisition step is performed by the view angle specifying step of the viewpoint free-form image. The viewpoint free-form image display method according to claim 6, wherein the additional information corresponding to the designated angle of view is searched for on the external network. 10. Each of the additional information used in response to the additional information being acquired in the additional information acquiring step or in response to the additional information being displayed in the viewing image displaying step. 7. The viewpoint free-form image display method according to claim 6, further comprising a charging step of charging the information usage fee of. 11. A storage medium that physically stores computer software in a computer-readable format, which is written so as to execute a process of displaying a viewpoint free-form image on a computer system. Is based on the viewpoint free-form image input step of inputting the viewpoint free-form image, the angle-of-view specifying step of specifying the viewing angle of the viewpoint-freeform image, and the angle of view specified by the angle-of-view specifying step. An additional information acquisition step of selectively acquiring additional information; and a viewing image display step of cutting out the angle of view specified by the angle of view specifying step from the viewpoint free-form image and displaying it together with the selected additional information. A storage medium characterized by: 12. A computer program written to execute a process of displaying a viewpoint free-form image on a computer system, comprising: a viewpoint free-form image input step of inputting a viewpoint free-form image; An angle-of-view designation step of designating a viewing angle of the free-form image; an additional information acquisition step of selectively acquiring additional information according to the angle of view specified by the angle-of-view designation step; And a viewing image display step of cutting out the angle of view specified by the step from the viewpoint free-form image and displaying it together with the selected additional information. 13. A broadcasting system for viewpoint free-form video, wherein the viewpoint free-view video to be broadcast can specify a viewing angle of view on the receiver side, and is added to each viewing location in the video frame. A broadcasting system characterized by different information. 14. A storage medium storing a viewpoint free-form image, wherein the viewpoint free-form image stored can freely select an angle of view to be viewed and is dependent on a viewing place in the video frame. A storage medium having additional information set therein. 15. An additional information management device for managing additional information used when displaying a viewing image, wherein the viewing image is a viewpoint free-form image in which a viewing angle can be arbitrarily designated, Additional information storage means for storing additional information in association with a viewing location regarding a shape image, and additional information reading means for extracting additional information corresponding to a specified angle of view from the additional information storage means are provided. And an additional information management device. 16. The additional information management device according to claim 15, wherein the additional information is advertisement information associated with each viewing location of the viewed image. 17. A method of managing additional information for managing additional information used when displaying a viewing image, wherein the viewing image is a viewpoint free-form image in which a viewing angle can be arbitrarily designated, An additional information management method, comprising: an additional information storage step of storing additional information in association with a viewing location related to a shape image; and an additional information reading step of extracting additional information corresponding to a designated angle of view. . 18. The additional information management method according to claim 17, wherein the additional information is advertisement information associated with each viewing location of the viewed image.