JP2011119926A - Video processing apparatus, video processing method and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video processing apparatus that improves a feeling of depth of a video only by image processing using neither an exclusive video display device nor special eyeglasses. <P>SOLUTION: The video processing apparatus 1 which performs processing for enhancing the feeling of depth of the video acquired by a video acquisition part 11 includes: a depth information acquisition part 12 which acquires depth information indicating distance in the depth direction of each of a plurality of video parts constituting the video; a video division part 13 which divides the video into a foreground video part and a background video part based on the depth information acquired by the depth information acquisition part 12 and the video; a frame object giving determination part 15a which determines whether a frame object for enhancing depth of the video should be given; and a video synthesis part 16 which synthesizes each video part with the frame object so that the frame object is superposed on the background video part, and the foreground video part is further superposed on the frame object when it is determined that the frame object should be given. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a video processing apparatus, a video processing method and a computer program for operating a computer as the video processing apparatus.

Various methods have been proposed for enhancing the stereoscopic effect or depth of a two-dimensional image displayed on a video display device such as a television or a mobile phone. For example, a stereoscopic viewing method using binocular parallax has been proposed as one of the methods for enhancing the stereoscopic effect or the sense of depth. The stereoscopic method is a method of giving a viewer an illusion by sending a left-eye parallax image and a right-eye parallax image to each of the viewer's left and right eyes, and giving a stereoscopic effect or a sense of depth to a two-dimensional plane. It is.
The method of sending the left-eye parallax image and the right-eye parallax image to each of the left and right eyes is synchronized with the video display device that alternately displays the left-eye parallax image and the right-eye parallax image and the switching frequency of each parallax image. Then, there is a method using glasses that switch the left and right shutters to block the optical path (for example, Patent Document 1).
Further, the left-eye parallax image and the right-eye parallax image are color-converted into a red image and a blue image, respectively, and by using a video display device that displays each color-converted image in a superimposed manner, and red and blue glasses, There is an anaglyph method that sends a red image and a blue image to each eye.
Furthermore, there is a method of sending a left-eye parallax image and a right-eye parallax image to the left and right eyes, respectively, by using a video display device that displays the left-eye parallax image and the right-eye parallax image with different polarizations and polarized glasses. There is (for example, Patent Document 2).

  On the other hand, in the field of painting, painting techniques such as a perspective method, a shadow method, a combination of forward color and backward color are used to enhance the stereoscopic effect or depth of the painting. Paintings created using these pictorial techniques are called trick art or Tromp-Rouille. Trick art uses the above-mentioned pictorial techniques to describe the background in a plane work and the overlapping relationship between each object, so that some of the two-dimensionally drawn objects are almost three-dimensional space in the real world. An illusion can be created as if it were protruding into the plane, giving a three-dimensional effect or depth to the two-dimensional work.

JP 60-7291 A JP-A-1-171390

  However, in the systems according to Patent Documents 1 and 2, it is necessary to prepare a dedicated video display device and special glasses. In addition, viewers are required to wear special glasses, and there is a problem that viewing methods are significantly limited.

  The present invention has been made in view of such circumstances, and a video processing apparatus and a video processing method capable of improving the sense of depth of video only by image processing without using a dedicated video display device and special glasses. Another object of the present invention is to provide a computer program for operating a computer as the video processing apparatus.

  A video processing apparatus according to the present invention is a video processing apparatus that performs a process of enhancing a sense of depth of an input video, and includes depth information indicating a distance in a depth direction of each of a plurality of video portions constituting the video. Depth information acquisition means to be acquired, video division means for dividing the video into a plurality of video portions having different distances in the depth direction based on the depth information acquired by the depth information acquisition means and the video, and the video division A depth-enhanced video for emphasizing the depth of the video is superimposed on the one video portion divided by the means, and further, the depth-enhanced video has another distance in the depth direction shorter than that of the one video portion. A video composition unit that synthesizes each video part and depth-enhanced video so that the video part is superimposed, and a depth-enhanced video is superimposed based on the other video parts divided by the video division unit. Hear a judging means for judging whether said image synthesis means, when said determination means determines that shalt be superimposed depth-enhancing image, characterized in that you have to be superimposed depth-enhancing image.

  The video processing apparatus according to the present invention includes means for specifying a video part corresponding to a video at an end of the input video, which is another video part divided by the video splitting unit, and the determination unit Is characterized in that when the video portion at the end of the input video is not included in the other video portions, it is determined that the depth-enhanced video should be superimposed.

  The video processing apparatus according to the present invention includes means for specifying a video part corresponding to a video at an end of the input video, which is another video part divided by the video splitting unit, and the determination unit If the video portion at the end of the input video is included in the other video portion, the depth-enhanced video is superimposed according to the size of the area of the video portion at the end included in the other video portion It is characterized in that it is determined whether or not to be performed.

  The video processing apparatus according to the present invention includes a generation unit that generates a depth-enhanced video having a luminance or color different from that of the video based on the luminance or color of the input video, and the video synthesis unit includes the generation unit. The depth-enhanced video generated in this way is synthesized.

  In the video processing apparatus according to the present invention, the generation unit may be configured such that the luminance or color is based on the luminance or color of one video portion and / or another video portion divided by the video dividing unit. A feature is that different depth-emphasized videos are generated.

  The video processing apparatus according to the present invention is configured so that a plurality of videos are input in chronological order, and from the video obtained by synthesizing the depth-enhanced video and the video not synthesized with the depth-enhanced video, A means for selecting an image obtained by synthesizing the depth-enhanced image is provided.

  The video processing apparatus according to the present invention is configured such that a plurality of videos are input in chronological order, and acquires moving direction information indicating the moving direction of the video portion between the videos input in chronological order. A moving direction information acquiring unit; and a generating unit that generates a depth-enhanced video having a shape corresponding to the moving direction information acquired by the moving direction information acquiring unit, wherein the video synthesizing unit is generated by the generating unit It is characterized in that the depth-enhanced video is synthesized.

  The video processing apparatus according to the present invention is configured such that a plurality of videos are input in chronological order, and acquires moving direction information indicating the moving direction of the video portion between the videos input in chronological order. A moving direction information acquisition unit is provided, and the generation unit generates a depth-enhanced video having a shape corresponding to the movement direction information acquired by the moving direction information acquisition unit.

  The video processing apparatus according to the present invention includes a storage unit that stores a predetermined three-dimensional image, and the generation unit acquires the three-dimensional image stored in the storage unit and the movement direction information acquisition unit. Rotation processing means for rotating the three-dimensional image so that the movement direction indicated by the movement direction information of one video portion has a predetermined positional relationship, and the three-dimensional image rotated by the rotation processing means is A depth-emphasized video having a two-dimensional shape obtained by projecting onto a two-dimensional plane is generated.

  A video processing method according to the present invention is a video processing method for performing a process of enhancing a sense of depth of an input video, and depth information indicating a distance in a depth direction of each of a plurality of video parts constituting the video. In order to divide the video into a plurality of video parts having different distances in the depth direction based on the acquired depth information and the video, and to emphasize the depth of the video based on the other video parts divided It is determined whether or not the depth-enhanced video should be superimposed on the video portion, and if it is determined that the depth-enhanced video should be superimposed, the depth-enhanced video for emphasizing the depth of the video on one divided video portion Each of the video portions and the depth-enhanced video is synthesized such that another video portion having a shorter distance in the depth direction than the one video portion is superimposed on the depth-enhanced video. That.

  A computer program according to the present invention is a computer program that causes a computer to execute processing for enhancing a sense of depth of an image, and indicates the distance in the depth direction of each of a plurality of image portions constituting the image. Dividing the video into a plurality of video parts having different distances in the depth direction based on the depth information and the video, and depth enhancement for emphasizing the depth of the video based on the other divided video parts Determining whether or not to superimpose the video on the video portion, and if it is determined that the depth-enhanced video should be superimposed, a depth-enhanced video for emphasizing the depth of the video is displayed on the divided video portion. In addition, each depth-enhanced video is superimposed on another video portion having a shorter distance in the depth direction than the one video portion. Characterized in that and a step of synthesizing an image portion and a depth-enhancing image.

In the present invention, the depth information indicating the distance in the depth direction of each of the plurality of video portions constituting the video is acquired, and based on the acquired depth information, the video is a plurality of videos having different distances in the depth direction. Divide into parts. Then, based on the other divided video parts, it is determined whether or not the depth-enhanced video should be superimposed on the video part. If it is determined that the depth-enhanced video should not be superimposed on the video part, the depth-enhanced video is not superimposed. When it is determined that the depth-enhanced video should be superimposed, a depth-enhanced video for emphasizing the depth of the video is superimposed on at least one video part, and further, the depth-enhanced video is more than the one video part. Each video portion and the depth-enhanced video are combined so that other video portions having a short distance in the depth direction are superimposed. The synthesized video is one video part, depth-enhanced video, and other video part superimposed in this order, so the depth of one video part and the other video part is depth-enhanced video. Is emphasized by.
Specifically, when a depth-enhanced video is superimposed on a part of one video part, the viewer recognizes that the depth-enhanced video is located on the near side of the one video part. In addition, when another video portion is superimposed on a part of the depth-enhanced video, the viewer recognizes that the other video portion is positioned on the near side of the depth-enhanced video portion. Therefore, it is possible to give the viewer a sense of depth that one video portion and another video portion are separated in the depth direction.
Note that the number of depth-enhanced videos is not limited to one, and each video part and depth-enhanced video are divided so that the video is divided into three or more video parts and the depth-enhanced video is inserted between each video part. The technical idea of synthesizing is also included in the present invention.

  In the present invention, when another video portion, that is, the foreground video portion includes the video portion at the end of the input video, it is determined that the depth-enhanced video should not be superimposed, and the input video If the video portion at the end of the image is not included, it is determined that the depth-enhanced video should be superimposed.

  In the present invention, whether or not the depth-enhanced video should be superimposed according to the size of the area of another video part, that is, the video part of the foreground located at the edge of the video. judge. For example, when the area of the video portion at the end is large, it is determined that the depth-enhanced video should not be superimposed. When the area of the video portion at the end is small, it is determined that the depth-enhanced video should be superimposed.

  In the present invention, the generation unit generates a depth-enhanced video having a luminance or color different from that of the video based on the luminance or color of the input video. Therefore, the depth-enhanced video and the video portion have different brightness or color. Therefore, it is possible to effectively enhance the depth of one video part and another video part.

  In the present invention, the generation means generates a depth-enhanced video having a luminance or color different from that of the video portion based on the luminance or color of one video portion and / or another video portion. Therefore, the depth-enhanced video and the video portion have different brightness or color. Therefore, it is possible to effectively enhance the depth of one video part and another video part.

  In the present invention, it is determined whether or not a depth-enhanced video is superimposed on each of a plurality of videos input in chronological order, and the depth-enhanced video is selectively superimposed to enhance the depth. Then, an image obtained by synthesizing the depth-enhanced video is selected from an image obtained by synthesizing the depth-enhanced video and an image in which the depth-enhanced video is not synthesized. Therefore, it is possible to view a moving image consisting of a video with an emphasized depth.

  In the present invention, the movement direction information acquisition means acquires movement direction information indicating the movement direction of the video portion between the videos input in chronological order. And a production | generation means produces | generates the depth emphasis video which has a shape according to the acquired moving direction information. That is, the generation unit generates a depth-enhanced video having a shape that can emphasize the movement of the video part.

In the present invention, the storage means stores a three-dimensional image that is the basis of the depth-enhanced video. The rotation processing unit rotates the three-dimensional image so that the three-dimensional image stored in the storage unit and the movement direction indicated by the movement direction information acquired by the movement direction information acquisition unit have a predetermined positional relationship. Let That is, the three-dimensional image rotates so as to face the moving direction of the video part. Then, the generation unit generates a depth-enhanced video having a two-dimensional shape obtained by projecting the rotated three-dimensional image onto a predetermined two-dimensional plane. Therefore, the synthesized depth-enhanced video has a shape that faces the moving direction of the video part. Therefore, the movement of the video part is emphasized.
The three-dimensional image means an image in a three-dimensional space, and the three-dimensional image includes a planar image as well as a three-dimensional image in the three-dimensional space.

  According to the present invention, it is possible to improve the sense of depth of video only by image processing without using a dedicated video display device and special glasses.

It is a block diagram which shows the example of 1 structure of the video processing apparatus which concerns on embodiment of this invention. It is explanatory drawing which shows an example of the image | video which the video acquisition part acquired. It is explanatory drawing which shows depth information notionally. It is explanatory drawing which shows a foreground image part and a background image part notionally. It is explanatory drawing which showed popping-out information notionally. It is explanatory drawing which showed notion | assignment determination method of the frame object notionally. It is explanatory drawing which shows an example of the other image | video which the image | video acquisition part acquired. It is explanatory drawing which shows notionally the depth information of the said other image | video. It is explanatory drawing which shows notionally the foreground image part and background image part which were obtained by dividing | segmenting the said other image | video. It is explanatory drawing which showed notion | assignment determination method of the frame object notionally. It is explanatory drawing which shows an original three-dimensional frame object notionally. It is explanatory drawing which shows notionally the shape determination method of a frame object. It is explanatory drawing which shows notionally the determination method of the brightness | luminance and color of a frame object. It is explanatory drawing which shows notionally the processing content of a video synthetic | combination part. It is explanatory drawing which shows notionally the moving image obtained by selectively providing a frame object with respect to the some image | video which comprises a moving image. It is a flowchart which shows the flow of the video processing method which a video processing apparatus implements. It is a flowchart which shows the flow of operation | movement of a frame object production | generation part. 10 is a block diagram illustrating a configuration example of a video processing apparatus according to Modification 1. FIG. It is explanatory drawing which shows notionally the moving image obtained by selectively arranging the image | video to which the frame object was provided. FIG. 10 is a block diagram illustrating a configuration example of a video processing apparatus according to Modification 2. 10 is a block diagram illustrating a configuration example of a video processing apparatus according to Modification 3. FIG. It is a schematic diagram which shows the curtain object which is an example of a depth emphasis image | video. It is explanatory drawing which shows notionally the shape determination method of the frame object which concerns on the modification 5. FIG. It is a block diagram which shows the video processing apparatus which concerns on the modification 6.

Hereinafter, the present invention will be described in detail with reference to the drawings illustrating embodiments thereof.
FIG. 1 is a block diagram showing a configuration example of a video processing apparatus 1 according to an embodiment of the present invention. The video processing apparatus 1 according to the present embodiment includes a video acquisition unit 11, a depth information acquisition unit 12, a video division unit 13, a pop-out information acquisition unit 14, a frame object generation unit 15, and a video synthesis unit 16.

<Video acquisition unit>
The video acquisition unit 11 acquires a video to be processed to improve the stereoscopic effect or the depth feeling, and outputs the acquired video to the video dividing unit 13. The video acquired by the video acquisition unit 11 may be either a still image or a moving image. A still image is composed of one frame of video, and a moving image is composed of multiple frames of video in time series order. The video may be compressed by a predetermined encoding method such as JPEG (Joint Photographic Experts Group), MPEG-2 (Moving Picture Expert Group phase 2) or the like, or may be uncompressed. good. When the encoded image is acquired, the image acquisition unit 11 obtains the acquired image by decoding the acquired image into, for example, an RGB format or a YUV format according to a predetermined encoding method. The video is output to the video dividing unit 13.
In the following, in this embodiment, for the sake of simplicity of explanation, a description will be given mainly of processing performed on a single frame image constituting a still image or a moving image. It is assumed that the same processing is performed on the video.

  FIG. 2 is an explanatory diagram illustrating an example of a video acquired by the video acquisition unit 11. The image F1 shown in FIG. 2 is data indicating the brightness and color of each of a plurality of pixels arranged two-dimensionally, and a plurality of objects having different distances in the depth direction, for example, birds, trees, sun, sky, clouds It is comprised from the image | video equivalent to subjects. Hereinafter, when the specific video shown in FIG. 2 is shown, it will be referred to as “video F1”, and any video acquired by the video acquisition unit 11 will be referred to as “video”. The distance in the depth direction refers to the distance between the subject related to the object and a predetermined position, for example, the position of the photographing apparatus used for capturing the video. Hereinafter, the distance is referred to as depth as appropriate.

<Depth information acquisition unit>
The depth information acquisition unit 12 acquires depth information indicating the depth of each of the plurality of objects constituting the video obtained from the video acquisition unit 11, and outputs the acquired depth information to the video division unit 13. In the present embodiment, the distance between the imaging device and each subject in the depth direction is measured at the time of imaging, and depth information having distance information obtained by measurement is input to the video processing device 1 separately from the video. Shall be.
In addition, what is necessary is just to measure the distance between an imaging device and each to-be-photographed object, for example applying a stereo method. Specifically, a common subject is imaged by two imaging units that are separated from each other, the parallax of the subject in the two images captured by each imaging unit is calculated, and the imaging device and the subject are calculated based on the principle of triangulation. Find the distance.
Further, the imaging apparatus is provided with an infrared irradiating unit for ranging to irradiate the subject with infrared rays and an infrared detecting unit for measuring the intensity of infrared rays reflected from the subject, and based on the intensity of infrared rays reflected from each subject. The distance between the imaging device and the subject may be obtained.

  FIG. 3 is an explanatory diagram conceptually showing the depth information. As shown in FIG. 3, an image having depth information associated with each of a plurality of objects constituting a video is referred to as a depth image G1. For example, the depth is indicated by numbers in ascending order of 1, 2,. Specifically, the depth image G1 is composed of a plurality of pixels similar to the input video, and any numerical value of 1 to 5 indicating the depth corresponding to each pixel constituting the input video Is assigned as the pixel value of each pixel of the depth image G1. Here, for simplicity of explanation, the depth information is shown in five stages. However, the depth information may be less than five or more than five, and of course, there is no problem even if the information is shown steplessly. .

<Video division unit>
The video dividing unit 13 divides the video acquired by the video acquiring unit 11, for example, F1, into a foreground video portion F11 and a background video portion F12 based on the depth information acquired by the depth information acquiring unit 12 (FIG. 4). reference). Then, the video dividing unit 13 outputs the divided foreground video part F11 and background video part F12 to the frame object generation unit 15 and the video composition unit 16. Specifically, the video dividing unit 13 compares the depth corresponding to each pixel of the acquired video with a predetermined threshold, and when the depth is less than the threshold, the pixel is set as a pixel of the foreground video portion F11. If the depth is greater than or equal to the threshold value, the pixel of the background video portion F12 is used. The threshold value is a constant stored in advance by the video dividing unit 13.
The variable indicating each pixel is n = 0, 1, 2,..., The variable indicating the foreground video portion F11 and the background video portion F12 is Px (n), the variable indicating the depth of each pixel is Depth (n), the threshold value Is Th1, Px (n) is represented by the following formulas (1) and (2).
Px (n) = background (Th1 <Depth (n)) (1)
Px (n) = Foreground (Th1 ≧ Depth (n)) (2)

  FIG. 4 is an explanatory diagram conceptually showing the foreground video portion F11 and the background video portion F12. When the threshold Th1 is 2, the video F1 shown in FIG. 2 is surrounded by the foreground video part F11 (solid line in FIG. 4A) based on the depth image G1 and the threshold Th1 = 2 shown in FIG. A white area) and a background video portion F12 (a white area surrounded by a solid line in FIG. 4B (an area excluding a gray area surrounded by a broken line)).

The threshold value Th1 has been described as a value recorded in advance in the video dividing unit 13, but may be a value that can be freely set by a viewer using the video processing device 1. Further, the threshold Th1 may be obtained by calculation. For example, the threshold Th1 is expressed by the following formula (3).
Th = (ΣDepth (n)) / (w * h) (3)
Where n is an integer of 0, 1, 2,..., W * h, h is the height of the image F1 (the number of pixels arranged in the vertical direction), and w is the width of the image F1 (the pixels arranged in the horizontal direction). Number).

<Jump Information Acquisition Unit>
The pop-out information acquisition unit 14 acquires pop-out information indicating the pop-out direction set for each object in the video F <b> 1 and outputs the acquired pop-out information to the frame object generation unit 15. Here, the pop-out direction is information indicating in which direction the pop-out feeling should be given when emphasizing the pop-out of each object in the video.

  FIG. 5 is an explanatory diagram conceptually showing pop-out information. For example, as shown in FIG. 5A, the pop-up information is a virtual image in the longitudinal direction (vertical direction) of the image F1 in the Y-axis, the horizontal direction (horizontal direction) in the X-axis, and the longitudinal direction perpendicular to the image plane. It is represented by a three-dimensional vector in a three-dimensional space with a simple axis as the Z axis. This pop-out information is specified for each object, as shown in FIG. In this embodiment, the pop-out information is handled as a normalized unit vector.

<Frame object generator>
The frame object generation unit 15 determines whether or not a frame object H3 (see FIG. 14) for emphasizing the depth of the video should be added to the input video, for example, the videos F1 and F2 (see FIGS. 2 and 7). A frame object addition determination unit 15a, a storage unit 15b that stores information on which the frame object H3 is based, a rotation processing unit 15c and a projection conversion unit for determining the shape of the frame object H3 based on the pop-out information 15d and a color determination unit 15e that determines the luminance and color of the frame object H3 based on the luminance and color of the foreground video portion F11 and the background video portion F12. Here, the frame object H3 is inserted between the foreground video portion F11 and the background video portion F12 to give a sense of relative distance between the foreground and the background, and to allow the viewer to perceive a stereoscopic effect and a sense of depth. It is an object for. In the present embodiment, a frame-shaped image surrounding the outer periphery of the image F1 is generated as the frame object H3. In particular, in this embodiment, the frame object generation unit 15 determines whether or not there is an effect of emphasizing popping out when the frame object H3 is given to the input images F1 and F2, and if it is effective. When the determination is made, a frame-like image surrounding the outer periphery of the image is generated as the frame object H3. Conversely, when it is determined that the effect of emphasizing the pop-out feeling is low, the frame object H3 is not generated.

The frame object addition determination unit 15a determines whether or not there is an effect of emphasizing popping out by adding a frame object H3 to the video F1 based on the input foreground video portion F11 and background video portion F12.
In the present embodiment, the input video F1 is divided into a foreground video portion F11 and a background video portion F12, and the frame object H3 and the foreground video portion F11 are appropriately superimposed on the background video portion F12, thereby two-dimensionally. An illusion is caused as if a part of the drawn object is popping out into the real world three-dimensional space, and the popping out feeling of the video F1 is emphasized. Here, in order to cause the viewer to have an illusion that a part of the object has jumped out into the three-dimensional space of the real world, the popping out object, that is, the foreground video part F11 is all depicted without being lost. It is desirable.
Therefore, the frame object provision determination unit 15a of the present embodiment determines whether or not the foreground video portion F11 is depicted without being missing, and determines whether or not to generate the frame object H3 based on the determination result. to decide. That is, when the foreground video portion F11 is depicted without being missing, the frame object provision determining unit 15a generates a frame object, and when the foreground video portion F11 is missing, the frame object H3 is not generated. Next, specific processing contents performed by the frame object provision determination unit 15a will be described.

  First, the frame object addition determination unit 15a acquires the foreground video part F11 and the background video part F12 (see FIG. 4) output from the video dividing unit 13. Then, the frame object addition determination unit 15a determines whether or not the foreground video portion F11 is in contact with the edge of the video, in other words, whether or not the foreground video portion F11 includes the video at the end of the input video F1. Determine whether. More specifically, the frame object addition determination unit 15a determines whether or not the pixels constituting the foreground video portion F11 are included in any of the upper side, the lower side, the left side, and the right side in the video F1.

FIG. 6 is an explanatory view conceptually showing the frame object assignment determination method. FIG. 6A is a conceptual diagram of the input video F1. The image F1 is composed of a plurality of pixels Px_in (x, y). However, x (0 ≦ x ≦ w) indicates a pixel position in the width direction (horizontal direction) of the image F1, and y (0 ≦ y ≦ h) indicates a pixel position in the height direction (vertical direction). Each pixel Px_in (x, y) has luminance information of the video F1.
FIG. 6B is a conceptual diagram of the foreground video portion F11. The foreground video portion F11 is composed of a plurality of pixels Px_fwd (x, y). Each pixel Px_fwd (x, y) has information indicating whether or not the pixel constitutes a foreground video, and has luminance information related to the foreground video when constituting the foreground video.

The frame object addition determination unit 15a uses the information on the pixel Px_fwd (x, y) that forms the foreground image portion F11, and the pixels Px_in (x = 1, y = 1) to Px_in () that form the upper side P01 of the image F1. The pixel Px_in (x, y) constituting the foreground image portion F11 is specified from x = w, y = 1), and the number of pixels of the foreground image portion F11 located on the upper side P01 of the image F1 is calculated. Similarly, the frame object addition determination unit 15a includes the pixels Px_in (x = 1, y = h) to Px_in (x = w, y = h) constituting the lower side of the video F1, and the pixel Px_in (x = x) constituting the left side. 1, y = 1) to Px_in (x = 1, y = h) and the foreground image portion F11 from the pixels Px_in (x = w, y = 1) to Px_in (x = w, y = h) constituting the right side The constituent pixels are specified, and the number of pixels of the foreground video portion F11 located on any of the upper side P01, the lower side, the left side, and the right side of the video F1 is calculated. Then, the frame object addition determination unit 15a determines that the frame object H3 should be added when the number of pixels of the foreground video portion F11 located on any of the upper side P01, the lower side, the left side, and the right side of the video F1 is zero. If the number of pixels is not 0, it is determined that the frame object H3 should not be added.
In the case of the video F1, the object included in the foreground video portion F11 is not in contact with any of the upper side P01, the lower side, the left side, and the right side of the video F1, that is, the upper side P01, the lower side, the left side, and the right side of the video F1. Since the number of pixels of the foreground video portion F11 located in any one is 0, it is determined that the frame object H3 should be added. In this case, the frame object generation unit 15 generates a frame object H3 according to a processing procedure described later, and outputs the generated frame object H3 to the video composition unit 16.

  FIG. 7 is an explanatory diagram showing an example of another video F2 acquired by the video acquisition unit, FIG. 8 is an explanatory diagram conceptually showing the depth information of the other video F2, and FIG. 9 is the other video F2. FIG. 10 is an explanatory diagram conceptually showing the frame object assignment determination method. FIG. 10 is a conceptual diagram showing the foreground video portion F21 and the background video portion F22 obtained by dividing. With the same procedure as described above, the video acquisition unit 11 acquires the video F2, outputs the acquired video F2 to the video division unit 13, and the depth information acquisition unit 12 acquires and acquires the depth image G2 of the video F2. The obtained depth image G2 is output to the video dividing unit 13. The video dividing unit 13 acquires the video F2 and the depth image G2, divides the video F2 into the foreground video portion F21 and the background video portion F22 based on the video F2 and the depth image G2, and the divided foreground video portion. F 21 and the background video portion F 22 are output to the frame object generation unit 15.

  The frame object addition determination unit 15a determines whether or not the frame object H3 should be added to the video F2 as well as the video F1. In the case of the video F2, since a part P02 of the foreground video portion F21 is in contact with the upper side of the input video F2, it is determined that the frame object H3 should not be added. In this case, the frame object generation unit 15 does not generate the frame object H3 and does not output the frame object H3 to the video composition unit 16.

Note that the frame object assignment determination method described above is an example. For example, when the number of pixels of the foreground video portions F11 and F21 located on any of the upper side P01, the lower side, the left side, and the right side of the video F1 is less than a predetermined threshold, it is determined that the frame object H3 should be added, and the pixel When the number is equal to or greater than a predetermined threshold, the frame object H3 may be determined not to be given.
In the above example, the case where the pixels constituting the foreground video portions F11 and F21 are specified from the pixels of the video portions at the ends of the videos F1 and F2 has been described. You may specify the pixel which comprises the edge part of image | video F1, F2 from the pixel which comprises F11,21.

  The storage unit 15b stores information that is the basis of the frame object H3 in advance. Specifically, a three-dimensional image in a three-dimensional space is stored. Hereinafter, the three-dimensional image is referred to as an original three-dimensional frame object H1 (see FIG. 11).

  FIG. 11 is an explanatory diagram conceptually showing the original three-dimensional frame object H1. The original three-dimensional frame object H1 has a rectangular frame shape that is centered at the origin in the three-dimensional space and substantially parallel to the XY plane. H2 represents a normal vector H2 of the original three-dimensional frame object H1.

First, the frame object generation unit 15 determines the shape of the frame object H3 based on the original three-dimensional frame object H1 and the pop-out information.
FIG. 12 is an explanatory diagram conceptually showing a method for determining the shape of the frame object H3. Here, as shown in FIG. 12A, it is assumed that an object F31 exists in the video F3 and its pop-out information is designated. Note that the video F3 is a simplified version of the video F1 in order to explain the method of generating the frame object H3. The shape of the frame object H3 is such that the original three-dimensional frame object H1 is rotated in accordance with the pop-out direction in the virtual three-dimensional space shown in FIG. , H21 (see FIG. 12) is projected onto the XY plane. This will be described in detail below.

First, an inclination vector that defines the inclination of the original three-dimensional frame object H1 is calculated. The inclination vector is expressed by the following equation (4).
(X1, y1, z1) = (a * x, b * y, c * z) (4)
However, (x1, y1, z1) is pop-out information, and a, b, c are constants (0 ≦ a, b, c ≦ 1.0) stored in advance by the frame object generation unit 15.

  Then, the rotation processing unit 15c rotates the original three-dimensional frame object H1 so that the normal vector H2 of the original three-dimensional frame object H1 matches the inclination vector (x1, y1, z1).

  Next, the projective conversion unit 15d converts the three-dimensional frame objects H11 and H21 after the rotation processing into a two-dimensional shape that is orthogonally projected onto the XY plane, and stores the two-dimensional shape as the shape of the frame object H3.

  For example, as shown in FIG. 12B, when the pop-out information of the object F31 is given by (0, 0, 1) and a = 1.0, b = 1.0, and c = 1.0, the slope The vector is (0,0,1). Then, the rotation processing unit 15c rotates the original three-dimensional frame object H1 so that the normal vector H2 of the original three-dimensional frame object H1 substantially matches the inclination vector (0, 0, 1). The final shape obtained by projecting the three-dimensional frame object H11 after the rotation processing onto the XY plane is the shape shown on the XY plane in FIG.

Also, as shown in FIG. 12C, the pop-out information of the object F31 is given by (x, 0, √ (1-x ² )), and a = 1.0, b = 1.0, c = 1. In the case of .0, the inclination vector is (x, 0, √ (1−x ² )). Then, the rotation processing unit 15c rotates the original three-dimensional frame object H1 so that the normal vector H2 of the original three-dimensional frame object H1 substantially coincides with the inclination vector (x, 0, √ (1-x ² )). The final shape obtained by projecting the three-dimensional frame object H21 after the rotation processing onto the XY plane has a shape as shown in the XY plane of FIG.

Next, the frame object generation unit 15 determines the luminance and color of the frame.
FIG. 13 is an explanatory diagram conceptually showing a method of determining the brightness and color of the frame object H3. The color determination unit 15e determines the color of the frame object H3 based on the luminance of the entire video, that is, the luminance of both the foreground video portion F11 and the background video portion F12. FIG. 13A is a video F4 acquired by the video acquisition unit 11 particularly at a certain point in time, and FIG. 13B is a luminance histogram of the video F4. The average value of the luminance of the video F4 is represented by f4. ing. The color determination unit 15e stores in advance the threshold value Th2, the color C1 of the frame object H3 when the average luminance f4 is equal to or greater than the threshold value Th2, and the color C2 of the frame object H3 when it is less than the threshold value Th2. . Note that the color C1 and the color C2 have different luminance. Since the average brightness f4 of the video F4 is equal to or greater than the threshold Th2, the color determination unit 15e determines C1 as the color of the frame object H3 as illustrated in FIG.

  Similarly, FIG. 13D is a video F5 acquired by the video acquisition unit 11 at another time point, and FIG. 13E is a luminance histogram of the video F5, and the average value of the luminance of the video F5 is represented by f5. ing. Since the average luminance f5 of the video F5 is less than the threshold Th2, the color determination unit 15e determines the color C2 as the color of the frame object H3 as shown in FIG.

The color of the frame object H3 is not particularly limited. However, it is preferable to select a color whose luminance is lower than the threshold Th2 when the average luminance is equal to or higher than the threshold Th2, and a color whose luminance is higher than the threshold Th2 when the average luminance is lower than the threshold Th2.
In addition, the constant d may be stored in advance in the color determination unit 15e, and the luminance of the frame object H3 may be determined by the following equations (5) and (6).
Luminance of frame object H3 = average luminance-d (average luminance ≧ threshold Th2) (5)
Luminance of frame object H3 = average luminance + d (average luminance <threshold Th2) (6)

  Further, a translucent frame object H3 may be generated based on the background video portion F12. When the frame object H3 is translucent, even if the background video portion F12 is obscured by the frame object H3, the viewer can know the content of the obscured background video portion F12. Therefore, it is possible to reduce the amount of information reduction of the video and to emphasize the depth of the video.

  Furthermore, the frame object H3 may be arranged as an object imitating a frame, a window frame, a television frame, or the like.

  Furthermore, although the example in which the colors C1 and C2 of the frame object H3 are determined based on the luminance of the images F4 and F5 has been described, the color of the frame object H3 is determined based on the colors of the images F4 and F5, for example, the average saturation. May be determined to be different from the color of the video. Further, the luminance and color of the frame object H3 may be determined based on the luminance and color of the images F4 and F5.

Furthermore, although the example in which the color and brightness of the frame object H3 are determined based on the brightness of the entire video has been described, the color and brightness of the frame object H3 may be determined based on the average brightness of only the foreground video portion F11. good. That is, the color and brightness of the frame object H3 may be determined so that the brightness of the foreground video portion F11 and the brightness of the frame object H3 are different. In this case, since the difference between the frame object H3 and the foreground video portion F11 is significant, the depth of the foreground video portion F11 can be effectively enhanced.
Similarly, the color and brightness of the frame object H3 may be determined based on the average brightness of only the background video portion F12. That is, the color and luminance of the frame object H3 may be determined so that the luminance of the background video portion F12 and the luminance of the frame object H3 are different. In this case, since the difference between the frame object H3 and the background video portion F12 is conspicuous, the depth of the background video portion F12 can be effectively enhanced.
Further, the average luminance of the foreground video portion F11 and the background video portion F12 is calculated separately, and the luminance and color of the frame object H3 are determined so that the calculated average luminance and the luminance of the frame object H3 are different. You may comprise as follows. In this case, since the differences among the frame object H3, the foreground video portion F11, and the background video portion F12 are conspicuous, the depths of the foreground video portion F11 and the background video portion F12 can be effectively enhanced.

  The frame object generation unit 15 generates a frame object H3 having the shape determined by the projection conversion unit 15d and the color determined by the color determination unit 15e, and outputs the generated frame object H3 to the video composition unit 16. .

<Video composition part>
The video composition unit 16 acquires the foreground video part F11 and the background video part F12 output from the video division unit 13 and the frame object H3 output from the frame object generation unit 15. However, when the frame object H3 is not output from the frame object generation unit 15, for example, when the video F2 is processed, only the foreground video portion F21 and the background video portion F22 are acquired.
When the frame object H3 is not output from the frame object generation unit 15, the background video part F22 and the foreground video part F21 are synthesized in this order, that is, a video similar to the input video F2 is generated and output to the display unit 2. .
When the frame object H3 is output from the frame object generation unit 15, the following processing is executed.

  FIG. 14 is an explanatory diagram conceptually showing the processing content of the video composition unit 16. The video composition unit 16 receives the foreground video portion F11 and the background video portion F12 output from the video division unit 13 and the frame object H3 output from the frame object generation unit 15. Then, as shown in FIGS. 14A and 14B, the video compositing unit 16 superimposes the background so that the frame object H3 is superimposed on the background video portion F12 and further the foreground video portion F11 is superimposed on the frame object H3. The video part F12, the frame object H3, and the foreground video part F11 are combined. If the shape and dimensions of the video and the frame object H3 do not match, as shown in FIG. 14B, an area outside the frame object H3 is generated, but the video composition unit 16 is outside the frame object H3. In order not to display the protruding background video portion F12, predetermined complementary videos I1 and I2 are synthesized in the area. The foreground video portion F11 that protrudes outside the frame object H3 is displayed as it is. That is, the foreground image portion F11 is displayed so as to be superimposed on the complementary images I1 and I2. The complementary images I1 and I2 are arbitrary images such as a monochrome image and a wall texture, for example. When the background video portion F12 that protrudes outside the frame object H3 is displayed as it is, there is a possibility that the viewer may erroneously recognize the depth of the background video portion F12. By covering the protruding video portion, it is possible to prevent the depth from being misidentified and to effectively enhance the depth of the video.

  In addition, when the video around the display device can be acquired, the video may be displayed as a complementary video.

  The video composition unit 16 outputs a composite video obtained by combining the background video part F12, the frame object H3, and the foreground video part F11 to the external display unit 2.

  The video processing procedure for one frame of video F1 and F2 has been described above. However, when processing a plurality of frames of video constituting a moving image, the same video processing may be executed for each video.

  FIG. 15 is an explanatory diagram conceptually showing a moving image obtained by selectively assigning the frame object H3 to a plurality of videos constituting the moving image. For example, when the moving image Q1 is given as shown in FIG. 15, the above-described processing is performed on the video of each frame constituting the moving image Q1. The moving image Q1 is composed of a plurality of frame groups Q01 to Q07. In particular, for the images constituting the frame groups Q01, Q04, Q06, the foreground image portion is not in contact with the end of the image, and for the images constituting the frame groups Q02, Q03, Q05, Q07, the foreground image portion is the image. Touching the edge. That is, the moving image Q1 has the frame group Q01, Q04, Q06, which is determined to have the effect of emphasizing popping out by adding the frame object H3, and the effect of emphasizing popping out even when the frame object H3 is added. It consists of frame groups Q02, Q03, Q05, and Q07 that cannot be obtained. In this case, when the above-described processing is executed for each of the frame groups Q01 to Q07, a moving image Q2 in which the frame object H3 is selectively superimposed only on the video that forms the frame groups Q01, Q04, and Q06 is obtained.

The display unit 2 is, for example, a liquid crystal display panel, a plasma display, or an organic EL (Electro-Luminescence) display. The display unit 2 receives the composite video output from the video processing device 1 and displays the composite video.
Although the display unit 2 is exemplified as the output destination of the composite video, a printer, a transmission device, and other various output devices may be employed as long as the composite video can be output.

  FIG. 16 is a flowchart showing the flow of a video processing method performed by the video processing apparatus 1. When an instruction to start the processing operation is given, each component starts operation, and the video acquisition unit 11 acquires videos input to the video processing device 1, for example, videos F1 and F2, and acquires the acquired videos F1 and F2. Is output to the video dividing unit 13 (step S11). Next, the depth information acquisition unit 12 acquires the depth information input to the video processing device 1 and outputs the acquired depth information to the video division unit 13 (step S12).

  Then, the video dividing unit 13 inputs the videos F1 and F2 and the depth information, and determines the arrangement position of the frame object H3 based on the videos F1 and F2 and the depth information (Step S13). Next, the video dividing unit 13 divides the videos F1 and F2 into the foreground video parts F11 and F21 and the background video parts F12 and F22 based on the depth information and the positions of the frame objects H3. The divided foreground video portions F11 and F12 and background video portions F12 and F22 are output to the frame object generation unit 15 and the video composition unit 16 (step S14).

  Next, the pop-out information acquisition unit 14 acquires the pop-out information input to the video processing device 1, and outputs the acquired pop-out information to the frame object generation unit 15 (step S15).

  Then, the frame object addition determination unit 15a of the frame object generation unit 15 specifies the video portions corresponding to the end portions of the input videos F1 and F2 that are the foreground video portions F11 and F21 (step S16). For example, the frame object addition determination unit 15a specifies pixels constituting the foreground video portions F11 and F21 on any of the upper side, the lower side, the left side, and the right side in the videos F1 and F2, and calculates the number thereof.

  Next, the frame object provision determination unit 15a determines whether or not the frame object H3 should be imparted based on the number of pixels calculated in step S16 (step S17). That is, the frame object addition determination unit 15a determines whether or not the effect of emphasizing popping out can be obtained by adding the frame object H3 to the videos F1 and F2. Specifically, the frame object addition determination unit 15a determines that the frame object H3 should be added when the foreground video portions F11 and F21 do not include the end video portions of the input images F1 and F2. To do. Further, the frame object addition determination unit 15a is a case where the foreground video portions F11 and F21 include the video portions at the ends of the input videos F1 and F2, and is included in the foreground video portions F11 and F21. If the number of pixels in the video portion at the end is less than a predetermined threshold, it may be determined that the frame object H3 should be added.

  When it is determined that the frame object should not be added (step S17: NO), the frame object generation unit 15 does not generate the frame object H3, and the video composition unit 16 performs, for example, the foreground video portion F21 and the background video portion. F22 is synthesized in this order, and the video obtained by synthesis, that is, the acquired initial video F2 is output to the display unit 2 (step S21), and the process is completed.

  When it is determined that a frame object should be added (step S17: YES), the frame object generation unit 15 generates a frame object H3 and outputs the generated frame object H3 to the video composition unit 16 (step S18). .

  FIG. 17 is a flowchart showing an operation flow of the frame object generation unit 15. The frame object generation unit 15 reads the original three-dimensional frame object H1 from the storage unit 15b (step S31). Then, the rotation processing unit 15c of the frame object generation unit 15 executes a process of rotating the original 3D frame object H1 according to the pop-out information (step S32), and the projective conversion unit 15d performs the 3D after the rotation process. The shape of the frame object H3 is determined by projective transformation of the frame objects H11 and H21 (step S33).

  Then, the color determination unit 15e determines the luminance and color of the frame object H3 based on the luminance and color of the video (step S34), and finishes the process related to the generation of the frame object H3.

  Following the processing in step S18, the video composition unit 16 inputs the foreground video portion F11, the background video portion F12, and the frame object H3, and superimposes the background video portion F12, the frame object H3, and the foreground video portion F11 in this order. The complementary images I1 and I2 are synthesized, and the synthesized video obtained by the synthesis is output to the display unit 2 (step S19).

Then, the display unit 2 inputs the synthesized video output from the video synthesizing unit 16, displays the synthesized video (Step S20), and ends the process.
The video processing procedure for one frame of video has been described above. However, when processing a plurality of frames of video constituting a moving image, the same video processing may be executed for each video.
Note that if the arrangement position, shape, and color of the frame object H3 change suddenly with respect to a plurality of frames of video, the viewer may feel uncomfortable, so the arrangement position determined for each video adjacent in time series, A low-pass filter that keeps the amount of change in the generated shape and color constant may be provided.

  In the video processing device 1 and the video processing method configured as described above, it is possible to improve the sense of depth of video only by image processing without using a dedicated video display device and special glasses.

  Note that the video processing apparatus 1 and the video processing method according to the present embodiment include a liquid crystal television including a display unit 2, a television such as an organic EL television and a plasma television, a still image camera including the display unit 2, a video camera, and a mobile phone. , PDAs (Personal Digital Assistants) and other portable devices, personal computers, information displays, video output BD (Blu-ray Disc) recorders, DVD (Digital Versatile Disc) recorders, HDD (Hard Disk Drive) recorders, etc. The present invention can be applied to a recorder, a digital photo frame, and other various furniture and home appliances provided with a display.

(Modification 1)
Since the video processing apparatus according to the modification 1 is configured to selectively output only the video F1 to which the frame object H3 is attached, it differs from the above-described embodiment. The differences will be described.

  FIG. 18 is a block diagram illustrating a configuration example of the video processing apparatus 101 according to the first modification. The video processing apparatus 101 according to the modification 1 includes an output video editing unit 117 that acquires the video output from the video synthesis unit 16, edits the acquired video, and outputs the edited video.

  The output video editing unit 117 is configured to switch whether or not the acquired video is output depending on whether or not the video is provided with the frame object H3. That is, the output video editing unit 117 determines whether or not the acquired video is a video to which the frame object H3 is added. If the acquired video is the video to which the frame object H3 is added, the output video editing unit 117 sends the video to the display unit 2. When the video is output and the frame object H3 is not attached, the video is not output to the display unit 2.

  FIG. 19 is an explanatory diagram conceptually showing a moving image obtained by selectively arranging videos provided with the frame object H3. As shown in FIG. 19, a moving image Q2 is obtained by selectively assigning a frame object H3 to a plurality of videos constituting the moving image Q1. Further, the output video editing unit 117 generates a moving image Q3 including the frame groups Q01, Q04, and Q06 to which the frame object H3 is assigned. As a result, it is possible to output a moving image Q3 composed only of a video to which the frame object H3 is added, that is, only a video having a pop-out feeling enhancement effect.

(Modification 2)
FIG. 20 is a block diagram illustrating a configuration example of the video processing apparatus 201 according to the second modification. In this embodiment, the depth information is configured to be acquired separately from the video. However, the video processing apparatus 201 according to the modification 2 obtains the depth information from the video acquired by the video acquisition unit 211 by various operations. Is configured to get. Specifically, since the configurations of the video acquisition unit 211 and the depth information acquisition unit 212 are different, the differences will be mainly described below.

  The video acquisition unit 211 acquires a video to be processed to improve the stereoscopic effect or the depth feeling, outputs the acquired video to the video dividing unit 13, and outputs the acquired video to the depth information acquisition unit 212.

  The depth information acquisition unit 212 receives the video output from the video acquisition unit 211, calculates depth information based on the input video, and outputs the calculated depth information to the video division unit 13.

  As a method for calculating depth information, for example, a method disclosed in Japanese Patent Laid-Open No. 9-161074 may be used.

  Further, when the video is encoded by some method, depth information may be created from the encoded information. For example, in MPEG-4 (Moving Picture Expert Group phase 4), which is one of the standard moving image standards created by the Moving Picture Experts Group (MPEG), encoding can be performed in units of objects such as backgrounds and persons. If the video is encoded using the function and the background and the person are encoded separately, using this information, for example, the background depth is assigned to “2” and the depth of the person is assigned to “1”. Create depth information. However, the smaller the number, the closer to the front.

  In the second modification, the video can be divided into the foreground video portion F11 and the background video portion F12 without inserting depth information to the video processing device 201, and the frame object H3 can be inserted. Can be emphasized.

(Modification 3)
FIG. 21 is a block diagram illustrating a configuration example of the video processing device 301 according to the third modification. In the present embodiment, the pop-up information is obtained separately from the video. However, the video processing device 301 according to the modification 3 obtains the pop-out information by various operations from the video acquired by the video acquisition unit 311. Is configured to get. Specifically, since the configurations of the video acquisition unit 311 and the pop-out information acquisition unit 314 are different, the difference will be mainly described below.

  The video acquisition unit 311 acquires a video to be processed for improving a stereoscopic effect or a depth feeling, in particular, a moving image encoded in units of objects such as a background and a person, and the acquired video is supplied to the video dividing unit 13. Output to the pop-out information acquisition unit 314.

The pop-out information acquisition unit 314 calculates a change in the moving direction and size of the object in the video constituting the continuous frame. Then, the pop-out information acquisition unit 314 calculates an X-axis vector component of the pop-out information based on the amount of movement of the object in the horizontal direction. In the three-dimensional space shown in FIG. 12, when the object is moving in the positive X-axis direction, the X-axis vector component of the pop-out information is set to a positive value, and the value is set to be larger as the movement amount of the object is larger. Conversely, when the object is moving in the negative X-axis direction, the X-axis vector component of the pop-out information is set to a negative value, and the absolute value of the value is set larger as the amount of movement of the object increases.
Similarly, the pop-out information acquisition unit 314 calculates the Y-axis vector component of the pop-out information based on the amount of vertical movement of the object.
Further, when the pop-out information acquisition unit 314 changes in the direction in which the size of the object increases, the Z-axis vector component of the pop-out information is set to a positive value, and the larger the change amount of the object size, the larger the value Set a larger value. Conversely, if the object size is changing in a decreasing direction, the X-axis vector component of the pop-out information is set to a negative value, and the absolute value of the value is set to increase as the amount of change in the object size increases. To do.

  In the third modification, the video can be divided into the foreground video portion F11 and the background video portion F12 without inserting the jump information to the video processing device 301, and the frame object H3 can be inserted. Can be emphasized.

  Note that the modification 2 and the modification 3 may be combined, and the depth information and the pop-out information may be calculated from the video input to the video processing device 301. In this case, the depth of the video can be emphasized without giving both the depth information and the pop-out information to the video processing device 301.

(Modification 4)
In the embodiment, the frame-type frame object H3 is illustrated as the depth-enhanced video that emphasizes the depth of the video. However, the video processing device 1 according to the modification 4 displays the curtain object H401 instead of the frame object H3. It is configured as follows. Specifically, the video processing device 1 according to the modification 4 includes a curtain object generation unit (not shown) instead of the frame object generation unit 15.

  FIG. 22 is a schematic diagram illustrating a curtain object H401 that is an example of a depth-enhanced video. The curtain object generation unit stores curtain-shaped curtain objects H401 located on both sides in the horizontal direction of the video, and outputs the curtain objects H401 to the video composition unit 16. The shape and color of the curtain object H401 are constant regardless of the content of the video. Needless to say, the curtain object generation unit inputs the foreground video portion F11 and the background video portion F12, and changes the color and luminance of the curtain object H401 based on the luminance of the foreground video portion F11 and the background video portion F12. You may comprise. In addition, the original three-dimensional curtain object having a three-dimensional shape is stored, the pop-out information is input, and the original three-dimensional curtain object is rotated and projectively converted using the pop-out information, whereby the two-dimensional shape curtain object H401 is obtained. May be configured to generate.

  As an example of the depth-enhanced video, the frame shape is illustrated in the embodiment, and the curtain shape is illustrated in the fourth modification. However, the shape of the depth-enhanced video is not limited to this as long as the depth of the video can be emphasized. For example, a bracket-enhanced depth-enhanced video may be employed. It should be noted that the depth-enhanced video is preferably configured to be located on the end side of the video so that the main part of the background video is not hidden.

(Modification 5)
In the embodiment, when the video pop-out information has only the Z-axis component as shown in FIG. 12B, the shape of the frame object H503 is not particularly deformed, so that the pop-out in the Z-axis direction cannot be emphasized. . When the pop-out information includes only the Z-axis component, the video processing device 1 according to the modified example 5 changes the shape of the frame object H503 so as to protrude in the Z-axis direction, thereby moving the frame object H503 toward the viewer side. It is constructed so that the popping out of the can be emphasized. Since only the processing content of the frame object generation unit 15 is different from the embodiment, the above differences will be mainly described below.

  FIG. 23 is an explanatory diagram conceptually illustrating a method for determining the shape of the frame object H503 according to the fifth modification. When the frame object generation unit 15 includes only the Z-axis component of the pop-out information, or when the Z-axis component is larger than the X-axis component and the Y-axis component by a predetermined value or more, particularly when the Z-axis component is positive, FIG. As shown in FIG. 4, the horizontal center portion of the original three-dimensional frame object H501 is bent so as to protrude in the positive direction along the X axis with a substantially central portion in the horizontal direction as a mountain, and further, the horizontal frame portion (long side portion of the frame) of the original three-dimensional frame object H501 Is deformed into a three-dimensional shape that is spread in the vertical direction. Then, the frame object generation unit 15 calculates a two-dimensional shape obtained by projecting and transforming the deformed three-dimensional frame object H501 onto the XY plane, and determines the calculated two-dimensional shape as the shape of the frame object H503.

  On the other hand, when the Z-axis component is negative, the frame object generation unit 15 bends the original three-dimensional frame object H501 so that it protrudes in the negative direction of the X-axis with the substantially horizontal central portion of the original three-dimensional frame object H501 as a valley. The horizontal frame portion (long side portion of the frame) of the object H501 is deformed into a three-dimensional shape that is narrowed by pushing in the vertical direction. Then, the frame object generation unit 15 calculates a two-dimensional shape obtained by projecting the deformed three-dimensional frame object H501 onto the XY plane, and determines the calculated two-dimensional shape as the shape of the frame object.

  The processing content of the video composition unit 16 is the same as that of the present embodiment. The video composition unit 16 synthesizes the background video part F12 so that the frame object H503, the complementary videos I501, I502, I503, and I504, and the foreground video part F11 are sequentially superimposed, and the synthesized video part obtained by the synthesis is externally generated. Output to.

  In the video processing device 1 and the video processing method according to the modified example 5, there are two images in which the object moves in the Z-axis direction, that is, the object approaches the near side, or the object that pops out toward the near side. Even in a video in which the pop-out directions of the left and right are different, for example, a video in which a person located in the center spreads both hands toward the left and right ends of the screen, the pop-out feeling can be emphasized.

(Modification 6)
FIG. 24 is a block diagram illustrating a video processing apparatus according to the sixth modification. The video processing apparatus according to the modified example 6 is realized by causing the computer 3 to execute the computer program 4a according to the present invention.

  The computer 3 includes a CPU (Central Processing Unit) 31 that controls the entire apparatus. The CPU 31 includes a ROM (Read Only Memory) 32, a RAM (Random Access Memory) 33 that stores temporary information generated in accordance with the calculation, and a recording medium that records the computer program 4a according to the embodiment of the present invention. 4, for example, an external storage device 34 that reads a computer program 4a from a CD-ROM, an internal storage device 35 such as a hard disk that records the computer program 4a read by the external storage device 34, an input unit 36, and an output unit 37 It is connected. The CPU 31 implements the video processing method according to the present invention by reading the computer program 4a from the internal storage device 35 into the RAM 33 and executing various arithmetic processes. The processing procedure of the CPU 31 is as shown in FIGS. 16 and 17, and the processing procedure of steps S11 to 21 and steps S31 to 34 is executed. Since the processing procedure is the same as the processing contents of each component constituting the video processing device 1 according to the present embodiment and the modification 5, detailed description thereof is omitted.

  In the computer 3 and the computer program 4a according to the modified example 6, the computer 3 can function as the video processing apparatus according to the present embodiment, and the video processing method according to the present embodiment can be performed. The same effects as in the embodiment and the second to fourth modifications are obtained.

  Needless to say, the computer program 4a according to the sixth modification is not limited to the one recorded in the recording medium 4, and is downloaded, stored, and executed via a wired or wireless communication network. May be.

  In addition, it should be considered that the embodiment disclosed this time is illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the meanings described above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Display part 3 Computer 4 Recording medium 4a Computer program 11 Image | video acquisition part 12 Depth information acquisition part 13 Image | video division | segmentation part 14 Pop-out information acquisition part 15 Frame object generation part 16 Image | video synthetic | combination part 15a Frame object provision determination part 15b Memory | storage 15c Rotation processing unit 15d Projection conversion unit 15e Color determination unit 31 CPU
32 ROM
33 RAM
34 External storage device 36 Input unit 37 Output unit 117 Output video editing unit F11 Foreground video part F12 Background video part H1 Original 3D frame object H3 Frame object

Claims (11)

An image processing apparatus that performs processing for enhancing a sense of depth of an input image,
Depth information acquisition means for acquiring depth information indicating the distance in the depth direction of each of a plurality of video parts constituting the video;
Video dividing means for dividing the video into a plurality of video parts having different distances in the depth direction based on the depth information acquired by the depth information acquiring means and the video;
A depth-enhanced video for emphasizing the depth of the video is superimposed on the one video portion divided by the video dividing means, and the depth-enhanced video has a distance in the depth direction relative to the one video portion. Video synthesizing means for synthesizing each video part and depth-enhanced video so that other short video parts are superimposed;
Determination means for determining whether or not a depth-enhanced video should be superimposed based on another video portion divided by the video dividing means,
The video composition means
A video processing apparatus characterized in that when the determination means determines that a depth-enhanced video should be superimposed, a depth-enhanced video is superimposed. Means for identifying a video part corresponding to the video at the end of the input video, which is another video part divided by the video splitting means;
The determination means includes
The video processing according to claim 1, wherein when the video portion at the end of the input video is not included in the other video portions, it is determined that the depth-enhanced video should be superimposed. apparatus. Means for identifying a video part corresponding to the video at the end of the input video, which is another video part divided by the video splitting means;
The determination means includes
If the video portion at the end of the input video is included in another video portion, should the depth-enhanced video be superimposed according to the size of the area of the video portion at the end included in the other video portion? The video processing apparatus according to claim 1, wherein it is determined whether or not. Based on the luminance or color of the input video, comprising generation means for generating a depth-enhanced video having a luminance or color different from that of the video
The video composition means
The video processing apparatus according to any one of claims 1 to 3, wherein the depth-emphasized video generated by the generating unit is synthesized. The generating means includes
Based on the luminance or color of one video portion and / or another video portion divided by the video dividing means, a depth-enhanced video having a luminance or color different from that of the video portion is generated. The video processing apparatus according to claim 4. It is configured to input multiple videos in chronological order,
The apparatus according to claim 1, further comprising means for selecting a video obtained by synthesizing the depth-enhanced video from a video obtained by synthesizing the depth-enhanced video and a video obtained by synthesizing the depth-enhanced video. Item 6. The video processing device according to any one of Items 5 to 5. It is configured to input multiple videos in chronological order,
Movement direction information acquisition means for acquiring movement direction information indicating the movement direction of the video portion between each video input in chronological order;
Generating means for generating a depth-enhanced video having a shape corresponding to the moving direction information acquired by the moving direction information acquiring means,
The video composition means
The video processing apparatus according to any one of claims 1 to 3, wherein the depth-emphasized video generated by the generation unit is synthesized. It is configured to input multiple videos in chronological order,
A moving direction information acquisition means for acquiring moving direction information indicating the moving direction of the video portion between the videos input in time series order,
The generating means includes
6. The video processing apparatus according to claim 4, wherein a depth-enhanced video having a shape corresponding to the movement direction information acquired by the movement direction information acquisition unit is generated. A storage means for storing a predetermined three-dimensional image;
The generating means includes
The three-dimensional image stored in the storage means and the three-dimensional image so that the movement direction indicated by the movement direction information of one video portion acquired by the movement direction information acquisition means has a predetermined positional relationship. A rotation processing means for rotating
The depth-enhanced video having a two-dimensional shape obtained by projecting the three-dimensional image rotated by the rotation processing unit onto a predetermined two-dimensional plane is generated. The video processing apparatus described in 1. An image processing method for performing a process of enhancing the sense of depth of an input image,
Obtaining depth information indicating a distance in a depth direction of each of a plurality of video portions constituting the video;
Based on the acquired depth information and the video, the video is divided into a plurality of video parts having different distances in the depth direction,
Determining whether or not to superimpose a depth-enhanced video for emphasizing the depth of the video on the video part based on the other divided video parts;
When it is determined that the depth-enhanced video is to be superimposed, a depth-enhanced video for emphasizing the depth of the video is superimposed on the divided one video portion, and further, the depth-enhanced video is compared with the one video portion. A video processing method comprising: synthesizing each video portion and depth-enhanced video so that another video portion having a short distance in the depth direction is superimposed. A computer program for causing a computer to execute processing for enhancing a sense of depth of an image,
In the computer,
Dividing the video into a plurality of video parts having different distances in the depth direction based on the depth information indicating the distance in the depth direction of each of the video parts constituting the video and the video; and
Determining whether or not a depth-enhanced video for emphasizing the depth of the video should be superimposed on the video portion based on the other divided video parts;
When it is determined that the depth-enhanced video is to be superimposed, a depth-enhanced video for emphasizing the depth of the video is superimposed on the divided one video portion, and further, the depth-enhanced video is more than the one video portion. And a step of synthesizing each video portion and the depth-enhanced video so that another video portion having a short distance in the depth direction is superimposed.