JP2004088474A - Video processing apparatus, video processing method, program and recording medium, and video processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video processing apparatus for composing, e.g., a video in which a viewer can recognize the wider space of a broadcasted video, and to provide a video processing method, a program and a recording medium, and a video processing system. <P>SOLUTION: The video processing apparatus 1 comprises an image acquiring section 2 to which television image signals are supplied; camera movement estimating section 3 which estimates the movement of a camera from an input image acquired by the section 2, and estimating a side drop to be estimated to stick out from an image frame; image storing memory 4 which stores the side drop; image composing section 5 which composes a plurality of protruding portions to create an image; image outputting section 6 which outputs the composite image obtained by composing the side drops; and control section 7 which controls the above sections. <P>COPYRIGHT: (C)2004,JPO

Description

【００３６】
【数２】

【００３７】
こうして、上記式（１），（２）を使用することにより、垂直ライン上の全ての画素の動きベクトルを求める必要がなくなり、処理が高速化する。
【００３８】
次に、本実施の形態の映像処理方法について説明する。図９は、ズーム補償する場合の映像処理方法を示すフローチャートである。図９に示すように、先ず、供給される映像信号の各フレーム画像内の動きベクトルを計算する。具体的には、Ｔ＝ｔ−１からＴ＝ｔにおける動きベクトルＶを算出する（ステップＳ１）。そして、算出した動きベクトルの方向及び大きさから、画面内で同様の動きであるか否かを判断し（ステップＳ２）、更に同様の動きでない場合は、放射状となっているか否か判断する（ステップＳ３）。
【００３９】
ステップＳ２で同様の動きベクトルが検出された場合、及びステップＳ３で放射状の動きが検出された場合は、次の処理に進む。即ち、得られた動きベクトルから次のフレームではみ出すはみ出し部分を推定する（ステップＳ４）。そして、推定したはみ出し部分、例えば画枠の左端からはみ出すと推定される各画素を抽出すると共に画像蓄積メモリ４に保存する（ステップＳ５）。
【００４０】
次に、画像蓄積メモリ４から過去に記憶されたはみ出し部分を読み込み（ステップＳ６）、新しく抽出したはみ出し部分を動きベクトルＶを外挿した位置に書き込む（ステップＳ７）。更に、過去の合成結果を動きベクトルＶを外挿した位置に書き込む（ステップＳ８）。そして、画素の不足及び重複部分があるか否かを検出し（ステップＳ９）、不足画素又は重複画素がある場合は補償処理を行い（ステップＳ１０）、合成した画像を出力して（ステップＳ１１）、処理を終了する。
【００４１】
ここで、ステップＳ１において、動きベクトルが同様の動きをしているか否かは、最頻値のベクトルの割合が閾値以上の場合、最頻値のベクトルをもつ画素が全画面に分布している場合、又は画像をブロック分割し、ブロック毎に最頻のベクトルを求め、それが全てのブロックで同じ場合等を検出することにより判定することができる。
【００４２】
また、ステップＳ３において、動きベクトルが沸き出しているか否かの判定は、画像をブロックに分割し、その位置に応じた向をベクトルがもっているか等により判定することができる。
【００４３】
更に、ステップＳ９において、補償処理は、同じ位置に複数の画素が配置された場合、即ち重複画素となった場合と、１画素も配置されない位置がある場合に行う。複数の画素が配置された場合には、これらを平均したものをその位置の新しい画素値とする。１画素も配置されない場合には、周辺の画素から補間する。
【００４４】
また、本実施の形態においては、ズーム補償しない場合及びズーム補償する場合においても、動きベクトルからはみ出し部分推定するものとして説明したが、現フレームより時間的に前の前フレーム画像を保存し、現フレーム画像との差分からはみ出し部分を抽出するようにしてもよい。
【００４５】
次に、このような映像処理装置により合成され創造された合成画像を表示するための映像処理システムについて説明する。本実施の形態の映像処理システムは、映像信号が供給され合成画像を合成する上述の図１に示す映像処理装置と、通常の映像又は創造された画像を表示する複数台の表示装置とから構成される。ここで、映像処理装置は、各表示装置毎に用意しても、１台の画像処理装置から創造した画像又は通常の映像を、各表示装置に供給するものとしてもよい。
【００４６】
以下、各表示装置毎に映像処理装置を設ける場合を分散型と称し、１台の映像処理装置により複数の表示装置に映像を供給する場合を統合型と称して説明する。図１０及び図１１は、本発明の実施の形態に係る分散型映像処理システムを示す夫々模式図及びブロック図である。また、図１２及び図１３は、総合型映像処理システムを示す夫々模式図及びブロック図である。
【００４７】
図１０及び図１１に示すように、分散型の映像処理システム１００は、各表示装置２０ａ〜２０ｃに一つずつ、入力映像から合成画像を合成する映像処理装置１０ａ〜１０ｃが接続されたものである。この分散型の場合には、各映像処理装置１０ａ〜１０ｃが独立に配置される。そのため、映像処理装置１０ａ〜１０ｃには、設定時に手動又は各映像処理装置１０ａ〜１０ｃ間で通信する等して自動で、その相対的な位置を検出して記憶する位置記憶部３０ａ〜３０ｃを各映像処理部１１ａ〜１１ｃの前段に設けるものとする。そして、この結果を使用して、各映像処理装置１０ａ〜１０ｃの映像処理部１１ａ〜１１ｃがどのような画像を作成するかを決定することができる。ここで、映像処理部１１ａ〜１１ｃは、上述の図１に示す映像処理装置１と同様の構成を有し、現フレームにおいて、これに時間的に連続する後フレームにおいてはみ出すと推定されるはみ出し部分を抽出し、このはみ出し部分を合成して合成画像を生成するものである。
【００４８】
このような映像処理システム１００においては、例えば、合成画像を表示する周辺表示装置を２台用意し、通常の映像を表示するメイン表示装置２０ｂの左右両側に配置する場合、左右の周辺表示装置２０ａ，２０ｃにて、夫々左端及び右端からはみ出すはみ出し部分から合成されて創造される合成画像を表示するよう設定することができる。
【００４９】
また、図１２及び図１３に示すように、統合型の映像処理システム２００は、映像信号から複数の合成画像を合成する映像処理部１１と、表示装置２０ａ〜２０ｃと、映像処理部１１と各表示装置２０ａ〜２０ｃに対応して設けられた出力ポート５０ａ〜５０ｃとから構成される。映像処理部１１は、上述の図１に示す映像処理装置１と同様に、現フレームにおいて、これに時間的に連続する後フレームでは画枠からはみ出すと推定されるはみ出し部分を抽出し、このはみ出し部分を合成して合成画像を生成するものであり、映像の動きベクトルの違い等により、複数の合成画像を生成し、この結果を各表示装置２０ａ〜２０ｃに表示するよう出力ポート５０ａ〜５０ｃに出力する。統合型の場合には、出力のポート５０ａ〜５０ｃをどの表示装置２０ａ〜２０ｃに接続するかによって、各表示装置２０ａ〜２０ｃで適切な表示を行うことができる。
【００５０】
このような本実施の形態においては、映像処理装置が放送された映像から、動きベクトルを検出し、カメラの操作を推定することにより、前フレームにおいて、これに時間的に連続する後フレームでは画枠からはみ出すと推定されるはみ出し部分を抽出し、このはみ出し部分を合成した合成画像を生成し、本来の映像とは別に本来の映像の周辺の画像を創造して複数の表示装置にて表示することにより、画像による表現可能な空間をより広げて認識することができる。
【００５１】
また、図１０及び図１２に示す例では、周辺表示装置２０ａ，２０ｃは、メイン表示装置２０ｂの左右に配置された２台としたが、上下方向等にも表示装置を配置すれば、更に広がりのある映像を視聴することができる。
【００５２】
また、上述の実施の形態では、ハードウェアの構成として説明したが、これに限定されるものではなく、任意の処理を、制御部にコンピュータプログラムを実行させることにより実現することも可能である。この場合、コンピュータプログラムは、記録媒体に記録して提供することも可能であり、また、インターネットその他の伝送媒体を介して伝送することにより提供することも可能である。
【００５３】
【発明の効果】
以上詳細に説明したように、本発明に係る映像処理装置及び方法は、フレーム単位の画像からなる映像における現フレーム画像においてその時間的に後の後フレーム画像の画枠からはみ出すはみ出し部分を該現フレーム画像から抽出し、抽出されたはみ出し部分を出力するため、現フレームより後の後フレームでは画枠からはみ出すはみ出し部分、即ち通常は画枠からはみ出して表示されなくなってしまうはみ出し部分を現フレームから抽出することで、カメラを動かして撮影された画像を元に、過去に放送され、現在において放送されていない部分を再現することができ、これを映像が表示される領域とは別領域に、通常の映像と共に表示すれば、より広がりのある映像を視聴することができる。
【図面の簡単な説明】
【図１】本発明の実施の形態に係る映像処理装置を示すブロック図である。
【図２】（ａ）乃至（ｄ）は、カメラ操作があった場合の動きベクトルを示す模式図である。
【図３】（ａ）及び（ｂ）は、夫々ズーム補償を行わない第１の合成方法及びズーム補償を行う第２の合成方法を示す模式図である。
【図４】はみ出し部分を周辺に配置された表示装置に表示する操作を説明する模式図である。
【図５】（ａ）は、合成画像を示す模式図であり、（ｂ）実際の映像を示す模式図である。
【図６】（ａ）及び（ｂ）は、夫々ズーム補償を行う場合及び行わない場合の結果を示す模式図である。
【図７】（ａ）及び（ｂ）は、夫々時刻Ｔ＝ｔ及び時刻Ｔ＝ｔ＋１における合成画像及び受信画像を示す模式図である。
【図８】垂直ライン上の画素を間引きして動きベクトルを求める方法を示す模式図である。
【図９】本発明の実施の形態における映像処理方法を示すフローチャートである。
【図１０】本発明の実施の形態に係る分散型映像処理システムを示す模式図である。
【図１１】本発明の実施の形態に係る分散型映像処理システムを示すブロック図である。
【図１２】本発明の実施の形態に係る総合型映像処理システムを示す模式図である。
【図１３】本発明の実施の形態に係る総合型映像処理システムを示すブロック図である。
【符号の説明】
１，１０ａ，１０ｂ，１０ｃ　映像処理装置、２　画像取得部、３　カメラ動き推定部、４　画像蓄積メモリ、５　画像合成部、６　画像出力部、７　制御部、２０ａ，２０ｂ，２０ｃ　表示装置、１１，１１ａ，１１ｂ，１１ｃ　映像処理部、１００，２００　映像処理システム、３０ａ，３０ｂ，３０ｃ　位置記憶部、５０Ａ，５０Ｂ，５０Ｃ　出力ポート、１００，２００　映像処理システム[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a video processing device, a video processing method, a program and a recording medium, and a video processing system for synthesizing an image from a television broadcast video or the like.
[0002]
[Prior art]
Conventionally, a large-screen image synthesizing apparatus for reconstructing a plurality of television image signals obtained by spatially dividing and photographing the same scene into a continuous large-screen image signal is disclosed in JP-A-10-32755. It has been disclosed.
[0003]
In the technology described in this publication, a television camera that continuously captures images while panning and tilting each partial image forming a large-screen image, a swing angle of the camera, and an angle of view of an image captured by the camera are defined. A camera data detection unit that detects each video field or video frame, and an image conversion that perspective-transforms a plurality of captured images captured for each video field or video frame based on the detected swing angle and angle of view. It comprises at least a circuit and a large screen memory for storing a plurality of perspective-transformed images at predetermined addresses determined by the perspective transformation.
[0004]
According to the technology described in the above publication configured as described above, when generating a large-screen image, it is not necessary to take geometrical matching between images that are in contact with each other, and the detection angle of the camera and the angle of view of the captured image are eliminated. A large screen image can be synthesized only by calculating the parameters of the perspective transformation from only the image transformation.
[0005]
[Problems to be solved by the invention]
However, in the technology described in the above publication, it is necessary that the camera data detection unit receives pan and tilt information from the camera platform and detects the camera swing angle and the angle of view of the captured image. It is not possible to compose a video for which information cannot be obtained. As described above, it is not possible to obtain information such as the camera swing angle and the angle of view of a captured image in a normal television broadcast image or the like. Therefore, if a large-screen image having a wide viewing angle can be synthesized from a normal image without using such information, a new viewing environment is obtained for the viewer, which is convenient.
[0006]
The present invention has been proposed in view of such a conventional situation, and for example, a video processing apparatus and a video processing method capable of synthesizing a video that can be recognized by expanding the space of a broadcast video. , A program and a recording medium, and a video processing system.
[0007]
[Means for Solving the Problems]
In order to achieve the above-described object, a video processing apparatus and method according to the present invention are provided with a video composed of images in frame units, and protrude from a current frame image in an image frame in a later frame image temporally later than the current frame image. A protruding portion is extracted, and the extracted protruding portion is output.
[0008]
In the present invention, it is possible to extract a portion of the current frame that protrudes from the subsequent frame, and output a protruding portion that normally protrudes from the image frame and is not displayed. If an image is displayed simultaneously with a normal image in another area different from the area to be displayed, a wider image can be viewed.
[0009]
In addition, a program according to the present invention causes a computer to execute the above-described video processing, and a recording medium according to the present invention stores such a program and is readable by a computer.
[0010]
An image processing system according to the present invention is provided with a plurality of display devices and an image composed of images in units of frames, and extracts, from a current frame image, a protruding portion that protrudes from an image frame in a later frame image temporally later than the current frame image. A video processing device having an extracting unit and an output unit for outputting the protruding portion extracted by the extracting unit, wherein the display device includes a main display device for displaying the video, and a display device for displaying the protruding portion. It is characterized by having the above peripheral display device.
[0011]
In the present invention, the protruding portion which normally protrudes from the image frame and is no longer displayed is extracted, and the protruding portion and the normal video can be displayed on separate display devices, so that the viewer can view the video more than the normal video. Spread images can be viewed using a plurality of display devices.
[0012]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. This embodiment, the present invention, based on the image taken by moving the camera, by reproducing the part that is broadcasted in the past and not currently broadcast, by using a plurality of display devices to display, The present invention is applied to a video processing device for displaying a video having a wide range.
[0013]
FIG. 1 is a block diagram showing a video processing device according to an embodiment of the present invention. As shown in FIG. 1, a video processing device 1 according to the present embodiment operates, for example, an image acquisition unit 2 to which a television image signal is supplied and a camera operation (movement) from an input image acquired by the image acquisition unit 2. , And a camera operation estimating unit 3 for estimating a protruding portion which is estimated to protrude from the image frame, an image storage memory 4 for storing the protruding portion, and an image synthesizing unit for synthesizing a plurality of protruding portions to create an image. 5, an image output unit 6 for outputting a synthesized image (creative image) obtained by synthesizing the protruding portions, and a control unit 7 for controlling these.
[0014]
In a video broadcast on television, a camera is moved to shoot a space in order to express the extent of the space. In the present embodiment, as described above, in a moving image, a protruding portion which is estimated to protrude from the image frame in a temporally continuous subsequent frame is extracted from the current frame and stored in the image storage memory 4. By saving, it is possible to use an image of a space that was displayed in the past and is not displayed at the present moment, and this protruding portion is used together with the currently being broadcasted image, for example, another display device provided separately. By displaying the image in another area, the viewer can perceive the image as a wider image than a normal image.
[0015]
In addition, using the images acquired in this way, it is possible to view a wide image or create a panoramic image using a plurality of display devices.
[0016]
Therefore, the camera operation estimating unit 3 detects a motion vector of the supplied input image, and from the input image, for example, a motion of a camera panned horizontally, a motion of a camera tilted vertically, or a zoom. Estimate the movement of the camera that has been zoomed in and out, from its motion vector. As a method of detecting a camera operation, for example, there is a gradient method, block matching, or the like. Here, the motion vector is detected even when the camera is almost stationary and only the subject moves, but in the present embodiment, when many of the motion vectors in the image are in the same direction, or By detecting only the case where the motion vectors in the image are radially distributed, it is possible to detect only the case where there is a camera operation, separately from the motion of the subject.
[0017]
FIGS. 2A to 2D are schematic diagrams showing motion vectors when a camera operation is performed. For example, as shown in FIGS. 2 (a) and 2 (b), the image when the camera operation is performed, not when only the subject moves, is similar in the same direction such as the horizontal direction and the vertical direction, respectively. 2 (c) and 2 (d), the focal point of the camera is zoomed in and zoomed out, respectively, to obtain a radial and similar size motion vector. May exist, for example. Specifically, in the case of FIGS. 2A and 2B, for example, it is detected that the ratio of the mode vector is equal to or greater than a predetermined threshold value, or an image having the mode vector is detected. Is distributed over the entire screen. In the case of FIG. 2C and FIG. 2D, the image is divided into blocks of a predetermined size, and it is determined whether or not the detected motion vector has a direction corresponding to the position. Can be detected.
[0018]
In the present embodiment, a natural image is taken as an example and the camera operation is estimated from a motion vector indicating the motion of the entire screen. However, the present invention is not limited to a natural image, and an artificial image such as an animation may be used. However, the present invention can be similarly applied by assuming a virtual camera.
[0019]
Returning to FIG. 1, a protruding portion which is estimated to protrude from a subsequent frame temporally continuous with the current frame is extracted, and the extracted protruding portion is stored in the image storage memory 4 as needed. Then, the image synthesizing unit 5 reads the image of the protruding portion stored in the image storage memory 4, generates a synthesized image by synthesizing the image, and sends it to a predetermined display device (not shown) via the image output unit 6. Supply.
[0020]
As a method for synthesizing the protruding portion, in the current frame, a protruding portion which is estimated to protrude from the subsequent frame is stored, and a first method of synthesizing the protruding portion and zoom compensation of the protruding portion in accordance with the detected motion vector are used. After processing, there is a second method of combining.
[0021]
FIGS. 3A and 3B are schematic diagrams showing a first combining method without performing zoom compensation and a second combining method with performing zoom compensation, respectively. FIGS. 3A and 3B show a received image obtained from an actual video broadcasted on a television on the right side, and a composite image obtained by combining a protruding portion on the left side. In the received image, time T = t−1. This shows a case where a pixel P1 having a motion vector V obtained from a frame image at time T = t is displayed as a composite image at time T = t + 1. The motion vector V has a horizontal component V1 and a vertical component V2, and the synthesized image is displayed by synthesizing a portion protruding from the left end of the image frame of the received image. As shown in FIG. 3A, in the first method that does not perform zoom compensation, the pixel P1 is displayed by being shifted by the horizontal component V1 on the composite image. On the other hand, as shown in FIG. 3B, in the second method of performing the zoom compensation, the pixel P1 is displayed by moving the vector V, that is, the horizontal component V1 and the vertical component V2, even on the composite image. You.
[0022]
FIG. 4 is a schematic diagram illustrating an operation of combining the protruding portions and displaying the protruding portions on a display device arranged in the periphery by the first method. In FIG. 4, for example, the display device is provided with a main display device 20b disposed at the center for displaying a television broadcast image, and peripheral display devices 20a and 20c disposed on the left and right thereof to display a composite image. Here is an example. As shown in FIG. 4A, in a T frame at time T displayed on the main display device 20b, first, a motion vector V of the entire screen is detected, and the motion of the camera is estimated from the motion vector V.
[0023]
When the movement of the camera is pan / tilt, the pixels of the protruding portion which is presumed to protrude from the image frame are extracted and stored in the image storage memory 4 and displayed on the peripheral display devices 20a and 20c. Here, the direction of the motion vector of the T frame image displayed on the main display device 20b is horizontal leftward, and the protruding image which is estimated to protrude from the left end of the image frame of the main display device 20b is extracted and arranged on the left side. An example of the display on the displayed peripheral display device 20a is shown. At time T, the protruding portion 21 extracted in the (T-1) frame is displayed on the peripheral display device 20a disposed on the left side.
[0024]
In this way, if the protruding image estimated to protrude to the left is displayed on the left peripheral display device 20a and the protruding image estimated to protrude to the right is displayed on the right peripheral display device 20c, the main display device 20b , And a composite image displayed on the peripheral display devices 20a and 20c looks more natural.
[0025]
Then, as shown in FIG. 4B, in the T + 1 frame at the time T + 1, the protruding portion 22 estimated in the T frame is displayed on the left display device 20a, so that it is combined with the already extracted protruding portion 21. Then, the composite image is displayed on the display device 20a. Then, in the main display device 20b, the protruding portion 23 which is presumed to protrude in the T + 2 frame is extracted. By repeating such processing, an image can be displayed not only on the central screen of the main display device 20b where a normal image is displayed, but also on the left screen and the right screen of the peripheral display devices 20a and 20b. With the three display devices, a wide image space can be expressed.
[0026]
FIG. 5A is a schematic diagram illustrating a synthesized image in which the protruding portion is synthesized, and FIG. 5B is a schematic diagram illustrating an actual video. As shown in FIGS. 5A and 5B, displaying the synthesized image together with the actual video makes it easier to recognize the spatial spread of the video.
[0027]
Next, a description will be given of a second combination method for performing zoom compensation, which is preferable when there is a zoom or diagonal movement as shown in FIG. In the case of pan, tilt, etc., the motion vector of the camera can be processed in only one direction, vertical or horizontal. However, when there is zoom or diagonal motion, the motion vector in two directions, vertical and horizontal, is calculated. To detect. Then, an appropriate position is estimated for each pixel, and the pixels are arranged. At this time, a pixel is created by performing an interpolation process from a peripheral image at a position where the pixel is not arranged. When a plurality of pixels are arranged at the same position on the composite image, an averaging process is performed.
[0028]
FIGS. 6A and 6B are schematic diagrams showing the results when zoom compensation is not performed and when zoom processing is performed, respectively. According to the present invention, a composite image to be displayed on a peripheral display device by synthesizing a protruding portion only from information of a received image (video) is generated. Here, a case will be described in which a synthesized image is synthesized from a received image in a case where the camera is zoomed in, that is, the camera is moved to the right while the focus is narrowed (the object in the screen is moved to the left). . 6A and 6B, the right side is a schematic diagram showing a received image at time T = t to t + 3.
[0029]
As shown in FIG. 6A, if zoom compensation is not performed, the size of the zoom of the protruding portion to be combined differs, so that the combination results in different sizes regardless of the same subject. Becomes discontinuous. On the other hand, as shown in FIG. 6B, the zoom compensation of the composite image is performed such that a portion protruding from a received image having a different zoom size in each frame has the same zoom size as the latest protruding portion. , The composite image becomes continuous.
[0030]
Next, a specific method of zoom compensation in the case of FIG. 6 described above will be described. FIGS. 7A and 7B are schematic diagrams showing a combined image and a received image obtained by combining the protruding portions at time T = t and time T = t + 1, respectively. Here, the pixels P1 and P2 on the same horizontal line on the composite image displayed on the peripheral display device are placed on the same horizontal line as the pixels P1 and P2 on the received image displayed on the main display device. A case will be described in which zoom compensation is performed using the end pixel P3 near the left end.
[0031]
First, as shown in FIG. 7A, the motion vector V of the end pixel P3 on the received image is obtained using the received image at T = t and the image at T = t−1. Based on the obtained motion vector V, as shown in FIG. 7B, a combined image at the time T = t + 1 is combined. That is, the pixels P1 and P2 on the composite image are located at positions shifted by the motion vector of the pixel P3 on the same horizontal line on the received video. In this way, all the pixels on the line connecting the pixel P1 and the pixel P2 on the composite image are arranged at positions shifted by the motion vector V of the pixel P3 calculated from the times T = t and T = t-1. I do. Further, the pixel P3 is estimated to protrude from the received image at the time T = t + 1 from the motion vector V detected at the time T, and is moved to the composite image based on the motion vector V.
[0032]
In this manner, on the received image, a pixel for detecting a motion vector for zoom compensation can be a pixel in a protruding portion estimated to protrude from the image frame. For example, when the camera moves vertically upward while zooming in the received image, that is, when the subject moves downward on the screen while being enlarged, the camera is arranged above the main display device that displays the received image. When a composite image is displayed on the peripheral display device, it can be a pixel near the upper end on the received image and on the same vertical line.
[0033]
In addition, for example, in FIG. 6, it is preferable to obtain the end pixels for calculating the motion vector with respect to all the pixels arranged in the vertical direction at the left end of the received image. .
[0034]
FIG. 8 is a schematic diagram showing a method of obtaining a motion vector by thinning out pixels on a vertical line. FIG. 8 shows four pixels P0 to P4 (not shown) arranged at a distance L on the leftmost vertical line on the received image. Here, the motion vectors V of the pixels P0 and P3 from the temporally continuous frames ₀ And V ₃ Is calculated and the motion vector V ₀ And V ₃ When only the known motion vectors are used, the motion vectors V of the pixels P1 and P2 are ₁ And V ₂ Is obtained as in the following equations (1) and (2).
[0035]
(Equation 1)

[0036]
(Equation 2)

[0037]
Thus, by using the above equations (1) and (2), it is not necessary to obtain the motion vectors of all the pixels on the vertical line, and the processing speeds up.
[0038]
Next, a video processing method according to the present embodiment will be described. FIG. 9 is a flowchart illustrating a video processing method when zoom compensation is performed. As shown in FIG. 9, first, a motion vector in each frame image of the supplied video signal is calculated. Specifically, a motion vector V at T = t from T = t-1 is calculated (step S1). Then, based on the calculated direction and magnitude of the motion vector, it is determined whether or not the motion is the same in the screen (step S2). If the motion is not the same, it is determined whether or not the motion is radial (step S2). Step S3).
[0039]
When a similar motion vector is detected in step S2, and when a radial motion is detected in step S3, the process proceeds to the next process. That is, the protruding portion that protrudes in the next frame is estimated from the obtained motion vector (step S4). Then, the estimated protruding portion, for example, each pixel presumed to protrude from the left end of the image frame is extracted and stored in the image storage memory 4 (step S5).
[0040]
Next, the protruding portion stored in the past is read from the image storage memory 4 (step S6), and the newly extracted protruding portion is written at the position where the motion vector V is extrapolated (step S7). Further, the past synthesis result is written at the position where the motion vector V is extrapolated (step S8). Then, it is detected whether or not there is a missing or overlapping portion of pixels (step S9). If there is a missing or overlapping pixel, compensation processing is performed (step S10), and a combined image is output (step S11). , And the process ends.
[0041]
Here, in step S1, whether or not the motion vectors are moving in the same manner is determined if the ratio of the mode vector is equal to or larger than the threshold, and the pixels having the mode vector are distributed over the entire screen. In this case, the determination can be made by dividing the image into blocks, obtaining the most frequent vector for each block, and detecting a case where the vector is the same for all the blocks.
[0042]
In step S3, the determination as to whether or not the motion vector is boiling can be made by dividing the image into blocks and determining whether the vector has a direction corresponding to the position.
[0043]
Further, in step S9, the compensation processing is performed when a plurality of pixels are arranged at the same position, that is, when there are overlapping pixels and when there is a position where no pixel is arranged. When a plurality of pixels are arranged, an average of these is used as a new pixel value at that position. If no pixel is arranged, interpolation is performed from surrounding pixels.
[0044]
Further, in the present embodiment, the description has been made assuming that a portion protruding from a motion vector is estimated in a case where zoom compensation is not performed and in a case where zoom compensation is performed. The protruding portion may be extracted from the difference from the frame image.
[0045]
Next, a video processing system for displaying a composite image created and synthesized by such a video processing device will be described. The video processing system according to the present embodiment includes the video processing device shown in FIG. 1 that supplies a video signal and synthesizes a composite image, and a plurality of display devices that display a normal video or a created image. Is done. Here, the video processing device may be prepared for each display device, or may supply an image created from one image processing device or a normal video to each display device.
[0046]
Hereinafter, a case in which a video processing device is provided for each display device will be referred to as a distributed type, and a case in which one video processing device supplies video to a plurality of display devices will be referred to as an integrated type. FIGS. 10 and 11 are a schematic diagram and a block diagram, respectively, showing a distributed video processing system according to an embodiment of the present invention. FIGS. 12 and 13 are a schematic diagram and a block diagram, respectively, showing the integrated video processing system.
[0047]
As shown in FIGS. 10 and 11, a distributed video processing system 100 is configured such that video processing devices 10 a to 10 c that combine a composite image from an input video are connected to each of the display devices 20 a to 20 c. is there. In the case of the distributed type, the video processing devices 10a to 10c are independently arranged. Therefore, the image processing apparatuses 10a to 10c are provided with position storage units 30a to 30c that detect and store their relative positions manually at the time of setting or automatically by communicating between the image processing apparatuses 10a to 10c. It is provided before each of the video processing units 11a to 11c. Using this result, it is possible to determine what kind of image the video processing units 11a to 11c of each of the video processing devices 10a to 10c create. Here, the video processing units 11a to 11c have the same configuration as that of the video processing device 1 shown in FIG. Is extracted, and the protruding portions are combined to generate a combined image.
[0048]
In such a video processing system 100, for example, when two peripheral display devices for displaying a composite image are prepared and arranged on both left and right sides of a main display device 20b for displaying a normal video, the left and right peripheral display devices 20a , 20c can be set so as to display a composite image created by being composited from portions protruding from the left end and the right end, respectively.
[0049]
As shown in FIGS. 12 and 13, the integrated video processing system 200 includes a video processing unit 11 that synthesizes a plurality of composite images from a video signal, display devices 20a to 20c, and a video processing unit 11. It comprises output ports 50a to 50c provided corresponding to the display devices 20a to 20c. The video processing unit 11 extracts, as in the video processing apparatus 1 shown in FIG. 1 described above, a protruding portion of the current frame which is estimated to protrude from the image frame in a subsequent frame temporally continuous with the current frame. A composite image is generated by synthesizing portions, and a plurality of composite images are generated due to a difference in a motion vector of a video and the like, and the results are output to output ports 50a to 50c so as to be displayed on each of display devices 20a to 20c. Output. In the case of the integrated type, an appropriate display can be performed on each of the display devices 20a to 20c depending on which display device 20a to 20c the output port 50a to 50c is connected to.
[0050]
In this embodiment, the video processing apparatus detects a motion vector from the broadcast video and estimates the operation of the camera, so that the image is processed in the previous frame and in the subsequent frame temporally continuous with the previous frame. The protruding portion that is presumed to protrude from the frame is extracted, a combined image is generated by combining the protruding portion, and an image around the original video is created separately from the original video and displayed on a plurality of display devices. Thereby, the space that can be represented by the image can be expanded and recognized.
[0051]
In addition, in the examples shown in FIGS. 10 and 12, the peripheral display devices 20a and 20c are two arranged on the left and right sides of the main display device 20b. You can watch the video with the presence.
[0052]
Further, in the above-described embodiment, the hardware configuration has been described. However, the present invention is not limited to this. Any process can be realized by causing the control unit to execute a computer program. In this case, the computer program can be provided by being recorded on a recording medium, or can be provided by being transmitted via the Internet or another transmission medium.
[0053]
【The invention's effect】
As described in detail above, the video processing apparatus and method according to the present invention can be used to display a portion of a current frame image in a video composed of images in frame units which protrudes from the image frame of a later frame image in time. To extract the extracted part from the frame image and output the extracted part, the part that protrudes from the image frame in the frame after the current frame, that is, the part that normally protrudes from the image frame and is not displayed, is extracted from the current frame. By extracting, based on the image taken by moving the camera, it is possible to reproduce the part that was broadcasted in the past and not currently broadcasted, and this was in a different area from the area where the video is displayed, If displayed together with a normal image, a wider image can be viewed.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a video processing device according to an embodiment of the present invention.
FIGS. 2A to 2D are schematic diagrams illustrating motion vectors when a camera operation is performed.
FIGS. 3A and 3B are schematic diagrams showing a first combining method without performing zoom compensation and a second combining method with performing zoom compensation, respectively.
FIG. 4 is a schematic diagram for explaining an operation of displaying a protruding portion on a display device arranged in the periphery.
FIG. 5A is a schematic diagram illustrating a composite image, and FIG. 5B is a schematic diagram illustrating an actual video.
FIGS. 6A and 6B are schematic diagrams illustrating results when zoom compensation is performed and when zoom compensation is not performed, respectively.
FIGS. 7A and 7B are schematic diagrams showing a composite image and a received image at time T = t and time T = t + 1, respectively.
FIG. 8 is a schematic diagram showing a method of obtaining a motion vector by thinning out pixels on a vertical line.
FIG. 9 is a flowchart illustrating a video processing method according to an embodiment of the present invention.
FIG. 10 is a schematic diagram showing a distributed video processing system according to an embodiment of the present invention.
FIG. 11 is a block diagram showing a distributed video processing system according to an embodiment of the present invention.
FIG. 12 is a schematic diagram showing an integrated video processing system according to an embodiment of the present invention.
FIG. 13 is a block diagram illustrating an integrated video processing system according to an embodiment of the present invention.
[Explanation of symbols]
1, 10a, 10b, 10c Video processing device, 2 image acquisition unit, 3 camera motion estimation unit, 4 image storage memory, 5 image synthesis unit, 6 image output unit, 7 control unit, 20a, 20b, 20c display device, 11 , 11a, 11b, 11c video processing unit, 100, 200 video processing system, 30a, 30b, 30c position storage unit, 50A, 50B, 50C output port, 100, 200 video processing system

Claims (15)

Extraction means for supplying a video composed of images in frame units and extracting, from the current frame image, a protruding portion that protrudes from the image frame in a temporally later frame image,
Output means for outputting the protruding portion extracted by the extraction means. A display unit for displaying the protruding image,
2. The image processing apparatus according to claim 1, wherein the display unit has another area different from an area where the image is displayed. The extracting means detects a movement of a camera from the video, and estimates the protruding portion from the motion of the camera. A storage means for extracting and storing the protruding portion estimated by the motion detecting means. And having
2. The video processing apparatus according to claim 1, wherein the output unit outputs the protruding portion stored in the storage unit. An image synthesizing unit that synthesizes a plurality of the protruding portions,
2. The video processing apparatus according to claim 1, wherein said output means outputs a composite image generated by said image composition means. The image synthesizing unit synthesizes the protruding portion and a past synthesized image each time a frame image is input,
5. The video processing apparatus according to claim 4, wherein the output unit outputs the composite image every time a frame image is input. The motion detecting means detects a motion vector from the preceding and following frame images,
5. The video processing apparatus according to claim 4, wherein said image synthesizing means synthesizes after compensating a past synthesis result and the protruding portion with the motion vector. An image consisting of images in frame units is supplied, and an extraction step of extracting, from the current frame image, a portion that protrudes from the image frame in a later frame image in time,
Outputting the protruding portion extracted by the extracting means. 8. The video processing method according to claim 7, further comprising a display step of displaying the protruding portion output in the output step on a display unit having another area different from an area where the video is displayed. The extraction step detects a movement of a camera from the video, and estimates a protruding portion from the motion of the camera.Extracts the protruding portion estimated in the motion detecting step and stores the extracted portion in a storage unit. And a step of storing
8. The video processing method according to claim 7, wherein the protruding portion stored in the storage unit is output in the output step. Having an image combining step of combining a plurality of the protruding portions,
8. The video processing method according to claim 7, wherein in the output step, a composite image generated in the image composition step is output. A program for causing a computer to execute a predetermined operation,
An image consisting of images in frame units is supplied, and an extraction step of extracting, from the current frame image, a portion that protrudes from the image frame in a later frame image in time,
An output step of outputting the protruding portion extracted by the extraction means. A computer-readable recording medium recorded with a program for causing a computer to execute a predetermined operation,
An image consisting of images in frame units is supplied, and an extraction step of extracting, from the current frame image, a portion that protrudes from the image frame in a later frame image in time,
An output step of outputting the protruding portion extracted by the extraction means. A recording medium on which a program is recorded. A plurality of display devices;
A video composed of images in frame units is supplied, and extraction means for extracting, from the current frame image, a portion protruding from the image frame in a later frame image in time, and outputting the protruding portion extracted by the extraction means And a video processing device having an output unit that performs
The video processing system according to claim 1, wherein the display device includes a main display device that displays the video, and one or more peripheral display devices that display the protruding portion. The extraction means of the video processing device detects a motion of a camera from the video, and a motion detection means for estimating the protruding portion from the motion of the camera, and extracts the protruding portion estimated by the motion detection means. And storage means for storing
14. The video processing system according to claim 13, wherein the output unit outputs the protruding portion stored in the storage unit to the peripheral display device. The video processing device includes image combining means for combining a plurality of the protruding portions,
14. The video processing system according to claim 13, wherein the output unit outputs the composite image generated by the image compositing unit to the peripheral display device.

Images