JP2012048639A - Free viewpoint video forming device, and method and program for displaying advertisement in free visual point video system - Google Patents

Free viewpoint video forming device, and method and program for displaying advertisement in free visual point video system Download PDF

Info

Publication number
JP2012048639A
JP2012048639A JP2010192412A JP2010192412A JP2012048639A JP 2012048639 A JP2012048639 A JP 2012048639A JP 2010192412 A JP2010192412 A JP 2010192412A JP 2010192412 A JP2010192412 A JP 2010192412A JP 2012048639 A JP2012048639 A JP 2012048639A
Authority
JP
Japan
Prior art keywords
advertisement
video
area
free viewpoint
free
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
JP2010192412A
Other languages
Japanese (ja)
Other versions
JP5567942B2 (en
Inventor
Akio Ishikawa
彰夫 石川
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
KDDI Corp
Original Assignee
KDDI Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by KDDI Corp filed Critical KDDI Corp
Priority to JP2010192412A priority Critical patent/JP5567942B2/en
Publication of JP2012048639A publication Critical patent/JP2012048639A/en
Application granted granted Critical
Publication of JP5567942B2 publication Critical patent/JP5567942B2/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Abstract

PROBLEM TO BE SOLVED: To provide a free viewpoint video forming device capable of always displaying an advertisement at a position easily recognized by a user, without obstructing a subject of a video to be provided, even if the user specifies any viewpoint.SOLUTION: A free viewpoint video forming device includes: retention means for retaining data of a subject and a background to generate a free viewpoint video; generation means for generating a video from the received viewpoint with the use of the data retained by the retention means; determination means for determining an area into which a desired advertisement is to be inserted from the background area of the video generated by the generation means; and composition means for generating a video in which the desired advertisement is attached to the area into which the advertisement is to be inserted in the video generated by the generation means.

Description

本発明は、利用者が指定した視点における映像を提供する自由視点映像表示技術に関し、より詳しくは、利用者に提供する映像内に広告を表示する自由視点映像システムに関する。   The present invention relates to a free viewpoint video display technique for providing video at a viewpoint designated by a user, and more particularly, to a free viewpoint video system for displaying an advertisement in a video provided to a user.

例えば、特許文献１には、あらかじめ複数のカメラにより撮影したデータに基づき、利用者が指定した任意の視点からの映像を提供する自由視点映像システムのための構成が開示されている。   For example, Patent Document 1 discloses a configuration for a free viewpoint video system that provides video from an arbitrary viewpoint specified by a user based on data captured by a plurality of cameras in advance.

特許文献１に記載の構成を用いれば、例えば、サッカーといったスポーツの試合を複数のカメラにより撮影することで、利用者に対して、利用者があたかもフィールド内にいるかの様な映像を提供することが可能になる。   If the configuration described in Patent Document 1 is used, for example, a game of sports such as soccer is photographed by a plurality of cameras, thereby providing the user with an image as if the user is in the field. Is possible.

特開２００８−０１５７５６号公報JP 2008-015756 A

これら自由視点映像システムにおいて、その利用者に対して商品やサービスの宣伝を行うために、自由視点映像内の何らかの対象物上に広告を表示することを考える。例えば、広告を表示する対象物をあらかじめ固定的に決定しておくものとすると、利用者が指定する視点からは広告が見えない位置、例えば、広告が被写体の影となったり、生成する映像の範囲外となったりすることが生じる。また、広告は被写体を妨害しない様に表示することが必要である。   In these free viewpoint video systems, in order to advertise products and services to the user, it is considered that an advertisement is displayed on some object in the free viewpoint video. For example, if the target for displaying the advertisement is fixedly determined in advance, the position where the advertisement cannot be seen from the viewpoint specified by the user, for example, the advertisement is a shadow of the subject, or the generated video Or out of range. Also, it is necessary to display advertisements so as not to disturb the subject.

したがって、本発明は利用者が任意の視点を指定したとしても、提供する映像の被写体を妨害することなく、かつ、利用者が認識し易い位置に常に広告を表示する方法、自由視点映像生成装置及びプログラムを提供することを目的とする。   Therefore, the present invention provides a method for displaying an advertisement at a position where the user can easily recognize without obstructing the subject of the provided video even when the user designates an arbitrary viewpoint, and a free viewpoint video generation device And to provide a program.

本発明における自由視点映像生成装置によれば、
自由視点映像を生成するための被写体及び背景のデータを保持する保持手段と、保持手段に保持されているデータを使用して、受信した視点から見た映像を生成する生成手段と、生成手段が生成した映像の背景の領域から所望の広告を挿入する領域を決定する決定手段と、生成手段が生成した映像内の前記広告を挿入する領域に前記所望の広告を貼り付けた映像を生成する合成手段とを備えていることを特徴とする。 According to the free viewpoint video generation device of the present invention,
Holding means for holding subject and background data for generating a free viewpoint video, generating means for generating video viewed from a received viewpoint using data held in the holding means, and generating means, A determining unit that determines a region in which a desired advertisement is inserted from a background region of the generated video, and a composition that generates a video in which the desired advertisement is pasted in the region in which the advertisement is inserted in the video generated by the generating unit Means.

本発明における自由視点映像生成装置と利用者端末とを含む自由視点映像システムにおいて、利用者端末に提供する自由視点映像に広告を挿入する方法によれば、
利用者端末が自由視点映像生成装置に視点を通知するステップと、自由視点映像生成装置が、通知された視点から見た映像を生成するステップと、自由視点映像生成装置が、生成した映像の背景の領域から所望の広告を挿入する領域を決定するステップと、自由視点映像生成装置が、生成した映像内の前記広告を挿入する領域にある対象物上に前記所望の広告を貼り付けた映像を生成するステップと、自由視点映像生成装置が、前記広告を貼り付けた映像を利用者端末に送信するステップとを備えていることを特徴とする。 In the free viewpoint video system including the free viewpoint video generation device and the user terminal according to the present invention, according to the method of inserting an advertisement into the free viewpoint video provided to the user terminal,
A step in which the user terminal notifies the viewpoint to the free viewpoint video generation device, a step in which the free viewpoint video generation device generates a video viewed from the notified viewpoint, and a background of the video generated by the free viewpoint video generation device. Determining a region in which a desired advertisement is to be inserted from the region, and a free-viewpoint video generation device that displays a video in which the desired advertisement is pasted on an object in the region in which the advertisement is inserted in the generated video The free viewpoint video generation device includes a step of generating, and a step of transmitting the video with the advertisement pasted to a user terminal.

本発明の他の実施形態によると、
自由視点映像生成装置は、背景を示す画素の位置情報に基づき背景内の１つ以上の連続面を判定し、前記広告を挿入する領域については、複数の連続面を跨らない様に選択することも好ましい。 According to another embodiment of the invention,
The free viewpoint video generation device determines one or more continuous surfaces in the background based on position information of pixels indicating the background, and selects an area in which the advertisement is inserted so as not to straddle a plurality of continuous surfaces. It is also preferable.

また、本発明の他の実施形態によると、
各連続面内の所定の誤差以下で平面に近似できる領域から、前記広告を挿入する領域を選択することも好ましい。 Also, according to another embodiment of the present invention,
It is also preferable to select an area into which the advertisement is inserted from an area that can be approximated to a plane within a predetermined error within each continuous plane.

さらに、本発明の他の実施形態によると、
前記近似した平面の法線方向の、前記受信した視点の方向に対する角度差が所定値以下の領域から、前記広告を挿入する領域を選択することも好ましい。 Furthermore, according to another embodiment of the present invention,
It is also preferable to select a region where the advertisement is inserted from a region where an angle difference between a normal direction of the approximate plane and a direction of the received viewpoint is a predetermined value or less.

本発明におけるプログラムによれば、
前記自由視点映像生成装置としてコンピュータを機能させることを特徴とする。 According to the program of the present invention,
A computer is caused to function as the free viewpoint video generation device.

本発明によれば、利用者が視点を自由に操作しても、視野内の見易い位置と向きに目的の広告が表示されるよう調整されるため、目的の広告の宣伝効率が高まるという効果を得ることができる。   According to the present invention, even if the user freely operates the viewpoint, the target advertisement is adjusted to be displayed at an easy-to-see position and orientation within the field of view, so that the advertising efficiency of the target advertisement is improved. Obtainable.

本発明による自由視点映像生成装置の構成図である。1 is a configuration diagram of a free viewpoint video generation device according to the present invention. 本発明による自由視点映像システムの構成図である。1 is a configuration diagram of a free viewpoint video system according to the present invention. FIG. 自由視点映像生成部が生成した自由視点映像を示す図である。It is a figure which shows the free viewpoint image | video which the free viewpoint image | video production | generation part produced | generated. 本発明による広告表示方法では起こりえない広告の表示形態を説明する図である。It is a figure explaining the display form of the advertisement which cannot occur with the advertisement display method by this invention. 広告挿入領域の選択を説明する図である。It is a figure explaining selection of an advertisement insertion field.

本発明を実施するための形態について、以下では図面を用いて詳細に説明する。なお、以下の説明において、視点とは位置及び方向にて指定される値である。また、本発明において、例えば、サッカーの試合における選手、ボール、ゴール等、自由視点映像の利用者が注目する対象を被写体と呼び、壁や空等、それ以外の対象を背景と呼ぶものとする。   EMBODIMENT OF THE INVENTION The form for implementing this invention is demonstrated in detail below using drawing. In the following description, the viewpoint is a value specified by position and direction. Also, in the present invention, for example, a target that a user of a free viewpoint video, such as a player, a ball, or a goal in a soccer game, is referred to as a subject, and another target such as a wall or sky is referred to as a background. .

図２は、本発明による自由視点映像システムの構成図である。図２において、利用者が、利用者端末２０を操作して、ネットワーク経由で自由視点映像生成装置１０に視点及び視野を通知すると、自由視点映像生成装置１０は、受信した視点から見た、受信した視野に含まれる映像を生成し、生成した映像を表示するためのデータを含む合成映像信号を利用者端末２０に送信する。利用者端末２０は、合成映像信号に基づき映像をディスプレイに表示する。なお、視野、つまり、視点の方向を中心とした表示領域の大きさについては、利用者が指定するのではなく、システムで固定の値としても良い。   FIG. 2 is a block diagram of a free viewpoint video system according to the present invention. In FIG. 2, when the user operates the user terminal 20 to notify the free viewpoint video generation apparatus 10 of the viewpoint and the visual field via the network, the free viewpoint video generation apparatus 10 receives the received view from the viewpoint. The video included in the visual field is generated, and a composite video signal including data for displaying the generated video is transmitted to the user terminal 20. The user terminal 20 displays the video on the display based on the composite video signal. Note that the field of view, that is, the size of the display area centered on the direction of the viewpoint, may not be specified by the user but may be a fixed value in the system.

図１は、本発明による自由視点映像生成装置１０の構成図である。図１に示す様に、自由視点映像生成装置１０は、自由視点映像生成データ保持部１と、自由視点映像生成部２と、広告情報保持部３と、広告領域決定部４と、合成部５と、送信符号化部６とを備えている。   FIG. 1 is a configuration diagram of a free viewpoint video generation apparatus 10 according to the present invention. As shown in FIG. 1, the free viewpoint video generation device 10 includes a free viewpoint video generation data holding unit 1, a free viewpoint video generation unit 2, an advertisement information holding unit 3, an advertisement area determination unit 4, and a synthesis unit 5. And a transmission encoding unit 6.

自由視点映像生成データ保持部１は、複数のカメラにより撮影した映像を基にあらかじめ生成した自由視点映像を生成するためのデータを保持している。これらデータの生成方法は、例えば、特許文献１に記載されている方法といった、公知の種々の方法を利用できる。なお、自由視点映像を生成するためのデータは、被写体の映像を生成するためのデータと、背景を生成するためのデータとが区別可能となっている。   The free viewpoint video generation data holding unit 1 holds data for generating a free viewpoint video generated in advance based on videos taken by a plurality of cameras. As these data generation methods, for example, various known methods such as the method described in Patent Document 1 can be used. The data for generating the free viewpoint video can be distinguished from the data for generating the video of the subject and the data for generating the background.

広告情報保持部３は、自由視点映像中に追加する広告を示す情報、つまり、広告として表示する領域の形状、表示する文字列、図形、それらの色等を特定する情報を保持している。なお、広告として表示する領域の大きさは固定であっても、自由視点映像内の広告として使用できる大きさに合わせて縮小又は拡大する形態であっても良い。   The advertisement information holding unit 3 holds information indicating the advertisement to be added to the free viewpoint video, that is, information specifying the shape of the area to be displayed as the advertisement, the character string to be displayed, the figure, the color thereof, and the like. The size of the area to be displayed as the advertisement may be fixed, or may be reduced or enlarged in accordance with the size that can be used as the advertisement in the free viewpoint video.

自由視点映像生成部２は、利用者端末２０から受信する視点及び視野情報に基づき、受信した視点の位置及び方向を中心とし、受信した視野で特定される範囲の映像を生成する。なお、自由視点映像生成部２における自由視点映像の生成も公知の種々の方法を利用できる。なお、自由視点映像を生成するためには、指定された視点と、背景及び各被写体の位置関係を認識することが必要である。つまり、自由視点映像生成部２は、生成した映像の背景及び各被写体の位置を判定しており、これら位置情報、言い換えると、生成した映像の各画素の位置情報と、各画素が被写体に対応するものであるか背景に対応するものであるかを示す情報を広告領域決定部４に出力する。   Based on the viewpoint and field of view information received from the user terminal 20, the free viewpoint image generation unit 2 generates an image in a range specified by the received field of view centered on the position and direction of the received viewpoint. Various methods known in the art can also be used to generate a free viewpoint video in the free viewpoint video generation unit 2. In order to generate a free viewpoint video, it is necessary to recognize the designated viewpoint, the background, and the positional relationship between each subject. In other words, the free viewpoint video generation unit 2 determines the background of the generated video and the position of each subject, and in other words, the position information of each pixel of the generated video and each pixel corresponds to the subject. The information indicating whether it is to be performed or to correspond to the background is output to the advertisement area determination unit 4.

広告領域決定部４は、後述する方法にて広告を挿入する領域を決定して合成部５に通知し、合成部５は、自由視点映像生成部２から入力される自由視点映像と広告情報保持部３から入力される広告を合成する。より具体的には、入力される自由視点映像内の広告領域決定部４から通知される領域を、広告情報保持部３から入力される広告に置き換えた映像を生成する。なお、自由視点映像に挿入する広告は、その位置にある対象物に貼り付ける様に挿入を行う。つまり、自由視点映像内の視点の方向に対して垂直な面に広告を挿入すると決定した場合、挿入される広告も視点の方向に垂直となるが、視点の方向に対してＸ°の角度を持つ面に広告を挿入すると決定した場合、挿入される広告も視点の方向に対してＸ°の傾きを持つことになる。また、曲面に対して広告を挿入すると決定した場合、広告も同じ曲面で表示を行う。最後に送信符号化部６は、合成部５が生成した映像を符号化して合成映像信号生成して利用者端末２０に送信する。   The advertisement area determination unit 4 determines an area into which an advertisement is to be inserted by a method to be described later, and notifies the synthesis unit 5. The synthesis unit 5 holds the free viewpoint video and advertisement information input from the free viewpoint video generation unit 2. The advertisement inputted from the part 3 is synthesized. More specifically, a video in which an area notified from the advertisement area determination unit 4 in the input free viewpoint video is replaced with an advertisement input from the advertisement information holding unit 3 is generated. The advertisement to be inserted into the free viewpoint video is inserted so as to be pasted on the object at that position. In other words, if it is decided to insert an advertisement in a plane perpendicular to the viewpoint direction in the free viewpoint video, the inserted advertisement is also perpendicular to the viewpoint direction, but the angle of X ° with respect to the viewpoint direction is set. When it is determined that an advertisement is to be inserted into the holding surface, the inserted advertisement also has an inclination of X ° with respect to the direction of the viewpoint. In addition, when it is determined to insert an advertisement into a curved surface, the advertisement is also displayed on the same curved surface. Finally, the transmission encoding unit 6 encodes the video generated by the synthesis unit 5 to generate a synthesized video signal and transmits it to the user terminal 20.

続いて、広告領域決定部４における広告挿入位置の決定方法について説明する。図３は、自由視点映像生成部２が生成した自由視点映像を示す図である。なお、図３において、符号５０及び５１は被写体である人を表しており、符号６０、６１及び６２は、それぞれ、背景である壁、建物及び道路を表しており、符号５０、５１、６０、６１及び６２以外の領域を示す符号６３は、背景である空を表しているものとする。なお、建物６１は、壁６０より遠い位置にあるものとする。   Next, a method for determining the advertisement insertion position in the advertisement area determination unit 4 will be described. FIG. 3 is a diagram illustrating the free viewpoint video generated by the free viewpoint video generation unit 2. In FIG. 3, reference numerals 50 and 51 represent persons as subjects, reference numerals 60, 61, and 62 represent background walls, buildings, and roads, respectively, and reference numerals 50, 51, 60, Reference numeral 63 indicating a region other than 61 and 62 represents the sky as the background. It is assumed that the building 61 is located far from the wall 60.

広告領域決定部４は、自由視点映像生成部２より各画素が被写体に対応するのか否かと、その画素が表す対象物の位置情報を受け取るが、まず、被写体以外の領域、つまり、背景の領域を広告挿入可能領域とする。つまり、図３においては、被写体５０及び５１以外の領域を広告挿入可能領域とする。   The advertisement area determination unit 4 receives from the free viewpoint video generation unit 2 whether or not each pixel corresponds to the subject and the position information of the object represented by the pixel. First, an area other than the subject, that is, a background area Is an advertisement insertion possible area. That is, in FIG. 3, an area other than the subjects 50 and 51 is set as an advertisement insertable area.

続いて、広告領域決定部４は、広告挿入可能領域から、広告を挿入可能な大きさの連続面を画素の位置情報に基づき探索し、任意の連続面を広告挿入領域に決定する。広告が連続面より広い場合は、広告を連続面に収まるように縮小、或いは、広告の一部分のみ連続面に貼り付け旗状にする。したがって、図４に示す様に、広告７１は、建物６１と空６３の領域の両方に貼り付けられることはなく、建物６１又は空６３の領域内でのみ貼り付けられることになる。同様に、広告７２は、図４に示す様に、壁６０と建物６１の２つの領域に跨ることはなく、壁６０又は建物６１の領域内のみで貼りついている様に挿入されることになる。   Subsequently, the advertisement area determination unit 4 searches the advertisement insertable area for a continuous surface having a size capable of inserting an advertisement based on the pixel position information, and determines an arbitrary continuous surface as the advertisement insertion area. When the advertisement is wider than the continuous surface, the advertisement is reduced to fit on the continuous surface, or only a part of the advertisement is pasted on the continuous surface to form a flag. Therefore, as shown in FIG. 4, the advertisement 71 is not pasted in both the building 61 and the sky 63 area, and is pasted only in the building 61 or the sky 63 area. Similarly, as shown in FIG. 4, the advertisement 72 does not straddle the two regions of the wall 60 and the building 61, and is inserted so as to stick only within the region of the wall 60 or the building 61. .

なお、広告領域決定部４は、広告挿入領域を選択するに当たり、以下の基準を用いることが好ましい。まず、凹凸の激しい表面に広告を挿入すると、その広告が読みにくくなるため、広告領域決定部４は、広告挿入可能領域内の平坦である領域から広告挿入領域を決定する。具体的には、ある領域を平面で近似したときに、近似した平面と元の面との誤差の総和が所定値以下となる領域から広告挿入領域を決定する。   The advertisement area determination unit 4 preferably uses the following criteria when selecting the advertisement insertion area. First, when an advertisement is inserted on a surface with extremely uneven surfaces, the advertisement becomes difficult to read. Therefore, the advertisement area determination unit 4 determines an advertisement insertion area from a flat area within the advertisement insertion possible area. Specifically, when an area is approximated by a plane, an advertisement insertion area is determined from an area where the sum of errors between the approximate plane and the original surface is a predetermined value or less.

また、広告領域決定部４は、広告挿入可能領域内のある領域の法線方向と、視点の方向との角度を求め、この角度が所定の値以下である領域から広告挿入領域を決定する。これは、広告を視点の方向に対して正対させて広告を見やすくするためである。なお、ある領域が曲面である場合、その曲面を誤差が最小となる平面で近似して法線を求める。   Further, the advertisement area determination unit 4 obtains an angle between the normal direction of a certain area in the advertisement insertion possible area and the direction of the viewpoint, and determines the advertisement insertion area from an area where the angle is equal to or less than a predetermined value. This is to make the advertisement easier to see by making the advertisement face to face. When a certain region is a curved surface, the normal is obtained by approximating the curved surface with a plane that minimizes the error.

例えば、図３においては、壁６０、建物６１を表す面は視点の方向に対して垂直であり、したがって、壁６０、建物６１の法線方向と視点の方向との角度は零である。また、空６３に対応する画素の位置情報は無限遠であるため、その法線方向と視点の方向との角度も零となる。これに対して、道路６２を示す面の法線方向は、視点の方向に対して、ほぼ直角となる。したがって、広告領域決定部２は、広告挿入領域を道路６２内から選択することは行わず、壁６０、建物６１及び空６３を示す領域内から選択する。   For example, in FIG. 3, the planes representing the wall 60 and the building 61 are perpendicular to the direction of the viewpoint, and therefore the angle between the normal direction of the wall 60 and the building 61 and the direction of the viewpoint is zero. Further, since the position information of the pixel corresponding to the sky 63 is infinite, the angle between the normal direction and the viewpoint direction is also zero. On the other hand, the normal direction of the surface indicating the road 62 is substantially perpendicular to the viewpoint direction. Therefore, the advertisement area determination unit 2 does not select the advertisement insertion area from the road 62 but selects the area from the area showing the wall 60, the building 61, and the sky 63.

また、図５に示す様に、ある背景に属する対象物８０の面が曲面の場合、近似した平面の法線方向より、符号８２で示す領域が、符号８１で示す領域より好ましい領域となる。   As shown in FIG. 5, when the surface of the object 80 belonging to a certain background is a curved surface, the region indicated by reference numeral 82 is more preferable than the region indicated by reference numeral 81 from the approximate normal direction of the plane.

また、広告領域決定部２は、自由視点映像のより中央に位置する連続面から広告挿入位置を決定することが好ましい。これは、自由視点映像の利用者に対してより広告を認識させやすくするためである。   Moreover, it is preferable that the advertisement area determination unit 2 determines the advertisement insertion position from a continuous plane located at the center of the free viewpoint video. This is to make it easier for the user of the free viewpoint video to recognize the advertisement.

また、視点の移動に対しては、挿入した広告が自由視点映像の範囲外となるまで同じ位置に挿入しておく形態であっても良いが、例えば、図３の壁６０内のある領域に広告を挿入している場合等においては、視点が移動するに伴い広告の挿入位置を壁６０内において移動させて、常に、映像の中央付近に表示させる形態であっても良い。なお、いずれの場合においても広告が自由視点映像の範囲外となる場合、被写体の影になる場合、広告が被写体を妨害する状態となった場合には、広告挿入領域を再決定する。   In addition, for the movement of the viewpoint, the inserted advertisement may be inserted at the same position until it is outside the range of the free viewpoint video. For example, in a certain area within the wall 60 in FIG. For example, when an advertisement is inserted, the advertisement insertion position may be moved in the wall 60 as the viewpoint moves to always display near the center of the video. In any case, when the advertisement is outside the range of the free viewpoint video, when the advertisement becomes a shadow of the subject, or when the advertisement is in a state of obstructing the subject, the advertisement insertion area is determined again.

なお、上述した実施形態においては自由視点映像装置１０において合成映像を生成して利用者端末２０に送信していたが、自由視点映像生成データ保持部１及び広告情報保持部３を有するサーバ装置が、利用者端末から受信した視点及び視野情報に基づき自由視点映像を生成するために必要なデータと広告情報を利用者端末２０に送信し、利用者端末２０にて自由視点映像の生成、広告挿入領域の決定及び合成処理を行う形態であっても良い。この場合、利用者端末２０が自由視点映像生成装置を組み込むことになる。つまり、利用者端末２０に図１の自由視点映像生成部２と、広告位置領域決定部４と、合成部５を設けることになる。   In the above-described embodiment, the free viewpoint video device 10 generates a composite video and transmits it to the user terminal 20. However, a server device including the free viewpoint video generation data holding unit 1 and the advertisement information holding unit 3 is provided. The data and advertisement information necessary for generating a free viewpoint video based on the viewpoint and view information received from the user terminal are transmitted to the user terminal 20, and the user terminal 20 generates the free viewpoint video and inserts the advertisement. It may be a form in which a region is determined and combined. In this case, the user terminal 20 incorporates a free viewpoint video generation device. That is, the user terminal 20 is provided with the free viewpoint video generation unit 2, the advertisement position area determination unit 4, and the synthesis unit 5 of FIG. 1.

なお、本発明による自由視点映像生成装置１０は、コンピュータを図１の各部として機能させるプログラムにより実現することができる。このコンピュータプログラムは、コンピュータが読み取り可能な記憶媒体に記憶されて、又は、ネットワーク経由で配布が可能なものである。さらに、本発明は、ハードウェア及びソフトウェアの組合せによっても実現可能である。   The free viewpoint video generation apparatus 10 according to the present invention can be realized by a program that causes a computer to function as each unit in FIG. This computer program is stored in a computer-readable storage medium or can be distributed via a network. Furthermore, the present invention can be realized by a combination of hardware and software.

１ 自由視点映像生成データ保持部
２ 自由視点映像生成部
３ 広告情報保持部
４ 広告位置領域決定部
５ 合成部
６ 送信符号化部
１０ 自由視点映像生成装置
２０ 利用者端末
５０、５１ 被写体
６０、６１、６２、６３ 背景
７１、７２ 広告
８０ 対象物
８１、８２ 領域 DESCRIPTION OF SYMBOLS 1 Free viewpoint video production | generation data holding | maintenance part 2 Free viewpoint video production | generation part 3 Advertising information holding | maintenance part 4 Advertising position area | region determination part 5 Composition part 6 Transmission encoding part 10 Free viewpoint video generation apparatus 20 User terminal 50, 51 Subject 60, 61 , 62, 63 Background 71, 72 Advertisement 80 Object 81, 82 Area

Claims (9)

自由視点映像を生成するための被写体及び背景のデータを保持する保持手段と、
保持手段に保持されているデータを使用して、受信した視点から見た映像を生成する生成手段と、
生成手段が生成した映像の背景の領域から所望の広告を挿入する領域を決定する決定手段と、
生成手段が生成した映像内の前記広告を挿入する領域に前記所望の広告を貼り付けた映像を生成する合成手段と、
を備えている自由視点映像生成装置。 Holding means for holding subject and background data for generating a free viewpoint video;
Generation means for generating video viewed from the received viewpoint using data held in the holding means;
Determining means for determining a region for inserting a desired advertisement from the background region of the video generated by the generating unit;
Combining means for generating a video in which the desired advertisement is pasted in an area in which the advertisement is inserted in the video generated by the generating means;
A free-viewpoint video generation device. 決定手段は、背景を示す画素の位置情報に基づき背景内の１つ以上の連続面を判定し、前記広告を挿入する領域については、複数の連続面を跨らない様に選択する、
請求項１に記載の装置。 The determining means determines one or more continuous surfaces in the background based on pixel position information indicating the background, and selects the region where the advertisement is inserted so as not to cross a plurality of continuous surfaces.
The apparatus of claim 1. 決定手段は、各連続面内の所定の誤差以下で平面に近似できる領域から、前記広告を挿入する領域を選択する、
請求項２に記載の装置。 The determining means selects an area into which the advertisement is inserted from an area that can be approximated to a plane within a predetermined error within each continuous plane.
The apparatus of claim 2. 決定手段は、前記近似した平面の法線方向の、前記受信した視点の方向に対する角度差が所定値以下の領域から、前記広告を挿入する領域を選択する、
請求項３に記載の装置。 The determining means selects an area in which the advertisement is inserted from an area in which an angle difference between a normal direction of the approximate plane and a direction of the received viewpoint is equal to or less than a predetermined value.
The apparatus of claim 3. 自由視点映像生成装置と利用者端末とを含む自由視点映像システムにおいて、利用者端末に提供する自由視点映像に広告を挿入する方法であって、
利用者端末が自由視点映像生成装置に視点を通知するステップと、
自由視点映像生成装置が、通知された視点から見た映像を生成するステップと、
自由視点映像生成装置が、生成した映像の背景の領域から所望の広告を挿入する領域を決定するステップと、
自由視点映像生成装置が、生成した映像内の前記広告を挿入する領域にある対象物上に前記所望の広告を貼り付けた映像を生成するステップと、
自由視点映像生成装置が、前記広告を貼り付けた映像を利用者端末に送信するステップと、
を備えている方法。 In a free viewpoint video system including a free viewpoint video generation device and a user terminal, a method for inserting an advertisement into a free viewpoint video provided to a user terminal,
A step in which the user terminal notifies the viewpoint to the free viewpoint video generation device;
A step of generating a video viewed from the notified viewpoint by the free viewpoint video generation device;
A step of the free viewpoint video generation device determining a region in which a desired advertisement is inserted from a background region of the generated video;
A free viewpoint video generation device generating a video in which the desired advertisement is pasted on an object in a region in which the advertisement is inserted in the generated video; and
A free-viewpoint video generation device that transmits a video pasted with the advertisement to a user terminal;
A method comprising: 自由視点映像生成装置は、通知された視点から見た映像の背景を示す画素の位置情報に基づき背景内の１つ以上の連続面を判定し、前記広告を挿入する領域については、複数の連続面を跨らない様に選択する、
請求項５に記載の方法。 The free viewpoint video generation device determines one or more continuous planes in the background based on pixel position information indicating the background of the video viewed from the notified viewpoint, and a plurality of continuous areas are inserted for the area into which the advertisement is inserted. Choose not to cross the surface,
The method of claim 5. 自由視点映像生成装置は、各連続面内の所定の誤差以下で平面に近似できる領域から、前記広告を挿入する領域を選択する、
請求項６に記載の方法。 The free viewpoint video generation device selects an area for inserting the advertisement from an area that can be approximated to a plane with a predetermined error within each continuous plane.
The method of claim 6. 自由視点映像生成装置は、前記近似した平面の法線方向の、前記通知された視点の方向に対する角度差が所定値以下の領域から、前記広告を挿入する領域を選択する、
請求項７に記載の方法。 The free viewpoint video generation device selects an area in which the advertisement is inserted from an area where an angle difference with respect to the direction of the notified viewpoint is equal to or less than a predetermined value with respect to the normal direction of the approximate plane.
The method of claim 7. 請求項１から４のいずれか１項に記載の自由視点映像生成装置としてコンピュータを機能させるプログラム。   The program which makes a computer function as a free viewpoint image | video production | generation apparatus of any one of Claim 1 to 4.
JP2010192412A 2010-08-30 2010-08-30 Free viewpoint video generation apparatus, method and program for displaying advertisement in free viewpoint video system Expired - Fee Related JP5567942B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2010192412A JP5567942B2 (en) 2010-08-30 2010-08-30 Free viewpoint video generation apparatus, method and program for displaying advertisement in free viewpoint video system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2010192412A JP5567942B2 (en) 2010-08-30 2010-08-30 Free viewpoint video generation apparatus, method and program for displaying advertisement in free viewpoint video system

Publications (2)

Publication Number Publication Date
JP2012048639A true JP2012048639A (en) 2012-03-08
JP5567942B2 JP5567942B2 (en) 2014-08-06

Family

ID=45903392

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2010192412A Expired - Fee Related JP5567942B2 (en) 2010-08-30 2010-08-30 Free viewpoint video generation apparatus, method and program for displaying advertisement in free viewpoint video system

Country Status (1)

Country Link
JP (1) JP5567942B2 (en)

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014041259A (en) * 2012-08-22 2014-03-06 Yahoo Japan Corp Advertisement distribution device, advertisement distribution method, and advertisement distribution program
JP2015204512A (en) * 2014-04-14 2015-11-16 パナソニックＩｐマネジメント株式会社 Information processing apparatus, information processing method, camera, reception device, and reception method
JP2016061885A (en) * 2014-09-17 2016-04-25 ヤフー株式会社 Advertisement display device, advertisement display method and advertisement display program
JP2017510167A (en) * 2014-02-07 2017-04-06 ソニー インタラクティブ エンタテインメント アメリカ リミテッド ライアビリテイ カンパニー Scheme for determining the location and timing of advertisements and other inserts in the media
WO2018051747A1 (en) * 2016-09-14 2018-03-22 キヤノン株式会社 Image processing device, image generating method, and program
JP2018049591A (en) * 2016-09-14 2018-03-29 キヤノン株式会社 Image processing apparatus, image generation method, and program
EP3367666A1 (en) 2017-02-28 2018-08-29 Canon Kabushiki Kaisha Image processing apparatus, image processing method, and program for inserting a virtual object in a virtual viewpoint image
JP2018180655A (en) * 2017-04-04 2018-11-15 キヤノン株式会社 Image processing device, image generation method, and program
JP2018180654A (en) * 2017-04-04 2018-11-15 キヤノン株式会社 Information processing device, image generation method, and program
JP2018537758A (en) * 2015-10-16 2018-12-20 フュージョン、インコーポレイテッド Expansion of multi-viewpoint image data including composite objects using IMU and image data
WO2019021375A1 (en) * 2017-07-25 2019-01-31 富士通株式会社 Video generation program, video generation method, and video generation device
US10271082B2 (en) 2014-04-14 2019-04-23 Panasonic Intellectual Property Management Co., Ltd. Video distribution method, video reception method, server, terminal apparatus, and video distribution system
JP2020101847A (en) * 2018-12-19 2020-07-02 キヤノン株式会社 Image file generator, method for generating image file, image generator, method for generating image, image generation system, and program
US10715839B2 (en) 2007-03-22 2020-07-14 Sony Interactive Entertainment LLC Scheme for determining the locations and timing of advertisements and other insertions in media
JP2020526125A (en) * 2017-06-27 2020-08-27 ピクセルロット エルティーディー．Ｐｉｘｅｌｌｏｔ Ｌｔｄ． Method and system for fusing user-specific content into video production
JP2020135222A (en) * 2019-02-15 2020-08-31 キヤノン株式会社 Image generation device, image generation method, and program
WO2021015035A1 (en) * 2019-07-19 2021-01-28 株式会社ソニー・インタラクティブエンタテインメント Image processing apparatus, image delivery system, and image processing method
WO2022181378A1 (en) 2021-02-26 2022-09-01 富士フイルム株式会社 Image processing device, image processing method, and program
WO2022181379A1 (en) * 2021-02-26 2022-09-01 富士フイルム株式会社 Image processing device, image processing method, and program
EP4254962A1 (en) 2022-03-31 2023-10-04 Canon Kabushiki Kaisha Image processing apparatus, control method and program

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10880533B2 (en) 2018-06-25 2020-12-29 Canon Kabushiki Kaisha Image generation apparatus, image generation method, and storage medium, for generating a virtual viewpoint image

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002077726A (en) * 2000-09-01 2002-03-15 Nippon Telegr & Teleph Corp <Ntt> System for supplying advertisement information in video and method for the same and recording medium with its program recorded
JP2002253847A (en) * 2001-02-28 2002-09-10 Konami Co Ltd Game advertisement charging system and program for family game or the like
JP2008015756A (en) * 2006-07-05 2008-01-24 Kddi Corp Method, device and program for generating free visual point image by local area division

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002077726A (en) * 2000-09-01 2002-03-15 Nippon Telegr & Teleph Corp <Ntt> System for supplying advertisement information in video and method for the same and recording medium with its program recorded
JP2002253847A (en) * 2001-02-28 2002-09-10 Konami Co Ltd Game advertisement charging system and program for family game or the like
JP2008015756A (en) * 2006-07-05 2008-01-24 Kddi Corp Method, device and program for generating free visual point image by local area division

Cited By (45)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10715839B2 (en) 2007-03-22 2020-07-14 Sony Interactive Entertainment LLC Scheme for determining the locations and timing of advertisements and other insertions in media
US8943535B2 (en) 2012-08-22 2015-01-27 Yahoo Japan Corporation Advertisement distribution apparatus and advertisement distribution method
JP2014041259A (en) * 2012-08-22 2014-03-06 Yahoo Japan Corp Advertisement distribution device, advertisement distribution method, and advertisement distribution program
JP2017510167A (en) * 2014-02-07 2017-04-06 ソニー インタラクティブ エンタテインメント アメリカ リミテッド ライアビリテイ カンパニー Scheme for determining the location and timing of advertisements and other inserts in the media
JP2021100263A (en) * 2014-02-07 2021-07-01 ソニー・インタラクティブエンタテインメント エルエルシー Scheme to determine location and timing of advertisements and other insert in media
JP2019110544A (en) * 2014-02-07 2019-07-04 ソニー インタラクティブ エンタテインメント アメリカ リミテッド ライアビリテイ カンパニー Scheme for determining locations and timing of advertisements and other insertions in media
US10271082B2 (en) 2014-04-14 2019-04-23 Panasonic Intellectual Property Management Co., Ltd. Video distribution method, video reception method, server, terminal apparatus, and video distribution system
JP2015204512A (en) * 2014-04-14 2015-11-16 パナソニックＩｐマネジメント株式会社 Information processing apparatus, information processing method, camera, reception device, and reception method
JP2016061885A (en) * 2014-09-17 2016-04-25 ヤフー株式会社 Advertisement display device, advertisement display method and advertisement display program
JP2018537758A (en) * 2015-10-16 2018-12-20 フュージョン、インコーポレイテッド Expansion of multi-viewpoint image data including composite objects using IMU and image data
US11089284B2 (en) 2016-09-14 2021-08-10 Canon Kabushiki Kaisha Image processing apparatus, image generating method, and storage medium
JP2020095729A (en) * 2016-09-14 2020-06-18 キヤノン株式会社 Image processing device, image processing method, and program
CN109716398B (en) * 2016-09-14 2023-11-17 佳能株式会社 Image processing apparatus, image generating method, and storage medium
JP2018049591A (en) * 2016-09-14 2018-03-29 キヤノン株式会社 Image processing apparatus, image generation method, and program
JP2019067427A (en) * 2016-09-14 2019-04-25 キヤノン株式会社 Image processing apparatus, image processing method, and program
CN109716398A (en) * 2016-09-14 2019-05-03 佳能株式会社 Image processing equipment, image generating method and program
US20190199997A1 (en) * 2016-09-14 2019-06-27 Canon Kabushiki Kaisha Image processing apparatus, image generating method, and storage medium
WO2018051747A1 (en) * 2016-09-14 2018-03-22 キヤノン株式会社 Image processing device, image generating method, and program
EP3514764A4 (en) * 2016-09-14 2020-03-18 C/o Canon Kabushiki Kaisha Image processing device, image generating method, and program
US10705678B2 (en) 2017-02-28 2020-07-07 Canon Kabushiki Kaisha Image processing apparatus, image processing method, and storage medium for generating a virtual viewpoint image
JP7086522B2 (en) 2017-02-28 2022-06-20 キヤノン株式会社 Image processing equipment, information processing methods and programs
EP3367666A1 (en) 2017-02-28 2018-08-29 Canon Kabushiki Kaisha Image processing apparatus, image processing method, and program for inserting a virtual object in a virtual viewpoint image
JP2018142164A (en) * 2017-02-28 2018-09-13 キヤノン株式会社 Image processing apparatus, image generating method, and program
JP2018180655A (en) * 2017-04-04 2018-11-15 キヤノン株式会社 Image processing device, image generation method, and program
JP2018180654A (en) * 2017-04-04 2018-11-15 キヤノン株式会社 Information processing device, image generation method, and program
JP7080586B2 (en) 2017-04-04 2022-06-06 キヤノン株式会社 Information processing equipment, information processing methods and programs
JP7013139B2 (en) 2017-04-04 2022-01-31 キヤノン株式会社 Image processing device, image generation method and program
JP2020526125A (en) * 2017-06-27 2020-08-27 ピクセルロット エルティーディー．Ｐｉｘｅｌｌｏｔ Ｌｔｄ． Method and system for fusing user-specific content into video production
JP7303754B2 (en) 2017-06-27 2023-07-05 ピクセルロット エルティーディー． Method and system for integrating user-specific content into video production
JPWO2019021375A1 (en) * 2017-07-25 2020-03-26 富士通株式会社 Video generation program, video generation method and video generation device
US10999535B2 (en) 2017-07-25 2021-05-04 Fujitsu Limited Recording medium, method, and information processing device for generating three-dimensional synthesized video of information related to subjects with background area based on free viewpoint video
WO2019021375A1 (en) * 2017-07-25 2019-01-31 富士通株式会社 Video generation program, video generation method, and video generation device
JP7010292B2 (en) 2017-07-25 2022-01-26 富士通株式会社 Video generation program, video generation method and video generation device
JP7297439B2 (en) 2018-12-19 2023-06-26 キヤノン株式会社 File generation device and file generation method, image generation device and image generation method, and program
US11436788B2 (en) 2018-12-19 2022-09-06 Canon Kabushiki Kaisha File generation apparatus, image generation apparatus, file generation method, and storage medium
JP2020101847A (en) * 2018-12-19 2020-07-02 キヤノン株式会社 Image file generator, method for generating image file, image generator, method for generating image, image generation system, and program
JP2020135222A (en) * 2019-02-15 2020-08-31 キヤノン株式会社 Image generation device, image generation method, and program
JP7349793B2 (en) 2019-02-15 2023-09-25 キヤノン株式会社 Image processing device, image processing method, and program
JP2021018575A (en) * 2019-07-19 2021-02-15 株式会社ソニー・インタラクティブエンタテインメント Image processing device, image distribution system, and image processing method
WO2021015035A1 (en) * 2019-07-19 2021-01-28 株式会社ソニー・インタラクティブエンタテインメント Image processing apparatus, image delivery system, and image processing method
JP7385385B2 (en) 2019-07-19 2023-11-22 株式会社ソニー・インタラクティブエンタテインメント Image distribution system and image distribution method
US11847715B2 (en) 2019-07-19 2023-12-19 Sony Interactive Entertainment Inc. Image processing apparatus, image distribution system, and image processing method
WO2022181378A1 (en) 2021-02-26 2022-09-01 富士フイルム株式会社 Image processing device, image processing method, and program
WO2022181379A1 (en) * 2021-02-26 2022-09-01 富士フイルム株式会社 Image processing device, image processing method, and program
EP4254962A1 (en) 2022-03-31 2023-10-04 Canon Kabushiki Kaisha Image processing apparatus, control method and program

Also Published As

Publication number Publication date
JP5567942B2 (en) 2014-08-06

Similar Documents

Publication Publication Date Title
JP5567942B2 (en) Free viewpoint video generation apparatus, method and program for displaying advertisement in free viewpoint video system
US9613463B2 (en) Augmented reality extrapolation techniques
EP3712839A1 (en) Image capturing device, image capturing system, image processing method, and carrier means
WO2012029576A1 (en) Mixed reality display system, image providing server, display apparatus, and display program
US20160321837A1 (en) Blending between street view and earth view
US20120105581A1 (en) 2d to 3d image and video conversion using gps and dsm
TW200912512A (en) Augmenting images for panoramic display
KR101932537B1 (en) Method and Apparatus for displaying the video on 3D map
JP2015187797A (en) Image data generation device and image data reproduction device
KR20110118727A (en) System and method of indicating transition between street level images
JP2014520298A (en) 2D image capture for augmented reality representation
WO2016167160A1 (en) Data generation device and reproduction device
JP2018033107A (en) Video distribution device and distribution method
KR20070104129A (en) Method and apparatus for generating 3d on screen display
JP7224894B2 (en) Information processing device, information processing method and program
CN109616033B (en) Exhibition hall interactive display system and method
WO2022237047A1 (en) Surface grid scanning and displaying method and system and apparatus
JP2004054890A (en) Method and device for image display, method and device for transmission, image display system, recording medium, and program
US11563928B2 (en) Image processing apparatus, image processing method, and storage medium
JP7296735B2 (en) Image processing device, image processing method and program
JP2019536388A (en) Television broadcasting system for generating extended images
KR20090132317A (en) Method and system for providing additional information based on video images
JP2005157610A (en) Image processor and image processing method
JP2005165283A (en) Map display device
KR101860215B1 (en) Content Display System and Method based on Projector Position

Legal Events

Date Code Title Description
A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20130304

RD04 Notification of resignation of power of attorney

Free format text: JAPANESE INTERMEDIATE CODE: A7424

Effective date: 20130408

RD03 Notification of appointment of power of attorney

Free format text: JAPANESE INTERMEDIATE CODE: A7423

Effective date: 20130520

RD04 Notification of resignation of power of attorney

Free format text: JAPANESE INTERMEDIATE CODE: A7424

Effective date: 20130521

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20140114

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20140122

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20140320

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20140611

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20140620

R150 Certificate of patent or registration of utility model

Ref document number: 5567942

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150

LAPS Cancellation because of no payment of annual fees