JP4815004B2 - Multi-view image encoding device - Google Patents

Description

  The present invention relates to multi-viewpoint image processing technology, and more particularly to reduction of processing load in viewpoint image encoding processing.

  By using images corresponding to a plurality of viewpoints, it is possible to express a video with a higher sense of presence that cannot be obtained by a conventional single viewpoint image alone. As a typical use example of a multi-viewpoint image, there are a stereoscopic image display technique and an arbitrary viewpoint image display technique. In the stereoscopic image display technique, the displayed image itself is a planar image, that is, two-dimensional information, and the images 501 and 502 having parallax with respect to the left and right eyes of the observer are observed as shown in FIG. By giving the image, the image 503 perceived in the brain gives a pseudo three-dimensional feeling similar to that when observing an actual three-dimensional object / three-dimensional space.

  In addition, the arbitrary viewpoint image display technique uses image data 601v to 603v photographed from a plurality of viewpoints and distance information 601d to 603d between the camera and the subject, as shown in FIG. By generating the images 604v, 605v, etc., it is possible to observe an image from an observer's favorite position, that is, an arbitrary viewpoint.

  As described above, the arbitrary viewpoint image generation / display technique enables observation of an image from an observer's favorite position including a viewpoint that has not been shot. In order to generate an arbitrary viewpoint image, position information (depth information) of each subject in the video is required. For example, when the depth information is not used, the area A that can be seen from one viewpoint is hidden behind the object B when viewed from another viewpoint, but in order to reproduce such a relationship in all arbitrary viewpoints, It is necessary to prepare an infinite number of viewpoint images. However, if there is depth information, an image of an arbitrary viewpoint can be reproduced with a small number of viewpoint images.

For example, the non-patent literature description discloses a method for generating a higher-quality arbitrary viewpoint image based on the basic method of 3D Warping, which is the basis of the arbitrary viewpoint image generation technique. This method uses a depth image related to a viewpoint image. From the virtual viewpoint that is the viewpoint to be observed by using the input two viewpoint images and the corresponding depth images in the following procedure. Generate an image. That is,
(1) A virtual camera is installed and a depth image is projected onto a virtual viewpoint.
(2) Smooth the projected depth image.
(3) Map the pixel value of the actual image to the smoothed depth image.
(4) The remaining pixels are restored using surrounding pixels.
In this way, by using the two viewpoint images and the depth image thereof, it is possible to generate an image from an arbitrary viewpoint in the vicinity of these viewpoints.

  Such a technique for generating an arbitrary viewpoint image can be applied to the above-described stereoscopic image display for improving and improving the sense of reality. For example, it is assumed that there are stereoscopic display image data 701v and 702v obtained by photographing subjects 704 and 705 with two cameras 701 and 702 as shown in FIG. However, since the camera interval 706 is too far from the distance between the left and right eyes of the person (which is said to be around 65 mm), the image becomes unnatural when viewed as a stereoscopic image, or perceived as a stereoscopic image at all. It becomes an image that can not be.

  In such a case, by applying the above-described arbitrary viewpoint image generation technique and generating the image 703v at the virtual viewpoint position 703 corresponding to the distance 707 between the left and right eyes of the person from the camera position 701, the images 701v and 703v are obtained. It becomes possible to observe as a proper stereoscopic image.

  On the other hand, if the distance between the two cameras 701 and 702 is too narrow, the captured images 701v and 702v are stereoscopic images with almost no stereoscopic effect. By setting a virtual viewpoint corresponding to the distance between the left and right eyes from the viewpoint and generating an image at the virtual viewpoint, it is possible to observe a stereoscopic image with a sufficient stereoscopic effect. Furthermore, by applying the above principle, it is possible to observe a stereoscopic image from an arbitrary viewpoint, or to adjust the stereoscopic effect when observing a stereoscopic image from an arbitrary viewpoint.

  As described above, the expression function of the image display system can be improved by using a plurality of viewpoint images and corresponding depth images. However, on the other hand, since depth image data is required, there is a problem that the amount of code at the time of recording / transmission increases accordingly. In order to solve this problem, various devices have been made in the multi-view image encoding / decoding device.

  For example, Patent Document 1 discloses a technique for controlling a motion vector search range according to perspective information that is a distance from an image viewpoint when multi-view image encoding is performed. The perspective information of the image is acquired from the right eye image and the left eye image. Using this information, the motion vector search range is narrowed in the region far from the viewpoint, and the motion vector search range is widened in the region near the viewpoint, thereby degrading the image quality of the region near the viewpoint in the desired data amount. Thus, it is possible to perform encoding without causing the error to occur.

JP 2001-285895 A

Mori, et al .: Free viewpoint image generation by 3D warping using depth images, IEICE General Conference, Information and Systems Proceedings 2, D-11-7, 2008

However, in the case of following the method of Patent Document 1, when the ratio of the area of the region close to the viewpoint in the entire image is larger than the area of the region far from the viewpoint, the ratio of the region having a large motion vector search range increases. The amount of motion vector search for the entire screen increases. Due to this increase in processing load, there is a possibility that a problem such as inadequate encoding processing may occur when real-time processing is required (for example, television relaying). In particular, when the number of pixels of an image and the input frame rate increase, the processing load increases more significantly, which may hinder real-time processing. Furthermore, when the number of viewpoints of viewpoint images used for stereoscopic video increases, the load increases.
An object of the present invention is to reduce the processing load in the encoding process of a multi-viewpoint image.

In order to solve the above problems, a multi-view image encoding apparatus according to the present invention includes the following units.
(1) Depending on the depth information and the viewpoint image, an encoding mode in which motion compensation is performed by motion vector search between two image frames or an encoding mode in which motion compensation is not performed by motion vector search is performed. A coding mode selection unit that selects and outputs coding mode selection information indicating the selected coding mode; a viewpoint image coding unit that codes the viewpoint image according to the coding mode selection information; and the depth Depth information encoding means for encoding information , wherein the encoding mode selection means determines the depth according to the number of pixels of the viewpoint image, or the number of pixels and the frame rate, or the number of pixels, the frame rate, and the number of viewpoints. Determining a threshold value for information, and depending on a comparison result between the threshold value and the depth information for each small region in the viewpoint image, Select the encoding mode for each area .

(2) The coding mode selection means outputs the coding mode selection information for selecting a coding mode in which motion compensation is not performed by the motion vector search for a small region closer to the viewpoint than the threshold .
(3) The encoding mode selection means selects the encoding mode selection information for selecting an encoding mode for performing motion compensation by searching for a motion vector between the two image frames for a small region farther from the viewpoint than the threshold. Output .
(4) The depth information encoding means encodes depth information according to the encoding mode selection information .

  By providing the above configuration, the multi-view image encoding device of the present invention has the following effects. That is, when compressing and encoding a multi-viewpoint image, the processing load of the encoding process can be reduced by selecting an encoding mode according to the depth information. In this case, the selection of the encoding mode is easy because it is only a numerical value comparison process.

It is a block diagram which shows the internal structure of the multiview image coding apparatus which is 1st embodiment of this invention. It is a figure which shows the image which binarized the viewpoint image, the corresponding depth image, and the depth image by the encoding mode. It is a figure which shows the processing flow in an encoding mode selection part. It is a block diagram which shows the internal structure of the multiview image coding apparatus which is 2nd embodiment of this invention. It is a conceptual explanatory drawing of the stereoscopic image display which is background art. It is a conceptual explanatory view of arbitrary viewpoint image generation which is background art. It is a figure explaining the correction process of the stereoscopic image display using arbitrary viewpoint image generation techniques. It is a figure which shows the processing flow in an encoding mode selection part.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. A multi-view image encoding apparatus according to the first embodiment of the present invention will be described. FIG. 1 shows an internal block diagram of the multi-view image encoding apparatus of the present embodiment. The multi-viewpoint image encoding apparatus 100 in FIG. 1 is an apparatus for reducing the amount of information by performing encoding processing using a plurality of viewpoint images and corresponding depth information as input. Hereinafter, the operation of each functional block in the figure will be described.

  The encoding mode selection unit 101 outputs encoding mode selection information based on the depth information and the information on the number of pixels of the viewpoint image. Here, the encoding mode is an encoding method performed for each small region obtained by dividing an image frame into small regions when performing moving image encoding, and includes intra-frame (intra) prediction and inter-frame (inter) prediction. There are modes such as. In inter prediction, motion compensation is performed by motion vector search between two image frames.

  The viewpoint image encoding unit 102 performs compression encoding of the viewpoint image. At that time, ON / OFF selection of motion vector search is performed based on the encoding mode selection information from the encoding mode selection unit 101. The depth information encoding unit 103 performs compression encoding similarly to the viewpoint image encoding unit 102. The multiplexing unit 104 multiplexes the encoded viewpoint image and depth information, and outputs them as encoded data to the outside of the apparatus.

  Here, the processing of the encoding mode selection unit 101 will be described in detail. FIG. 3 shows a flowchart of the process of the encoding mode selection unit 101. First, the encoding mode selection unit 101 divides depth information as shown in FIG. 2B into small regions bx (x = 1,..., N) (step S1), and for each small region bx. Zx (x = 1,..., N) is calculated as an average value of the depth values (step S2).

  For the depth information, for example, a numerical value corresponding to the distance from the viewpoint of the subject corresponding to each pixel is given to the viewpoint image as shown in FIG. 2A, and is expressed as a luminance image as shown in FIG. Is done. At this time, it is assumed that the depth value Zx is larger as it is closer to the viewpoint, and is smaller as it is farther from the viewpoint, and the luminance image in FIG.

In general, the depth information indicates how far the object on the viewpoint image is from the camera position (actually converted to a larger value as it gets closer), and the depth value for each pixel by each camera, A maximum depth value (closest) and a minimum depth value (farthest) are defined.
As a method for generating depth information, for example, there are a method using a distance measuring device using infrared rays and a method using software. In the method using infrared rays, the distance is measured using the time until the emitted light beam returns. In addition, as a method using software, there is a method of calculating a distance from a pixel shift when matching each pixel of the left and right viewpoint images. The present invention does not limit the depth information generation method, and a depth information generation method based on a conventionally known method can be appropriately applied.

  On the other hand, for the viewpoint image, the pixel number p is acquired from the viewpoint image, and the threshold value θz is determined according to the pixel number p (step S3). The threshold value θz is determined using a function f (p) that decreases as the number of pixels p increases. Finally, ON / OFF of motion vector search is determined using the depth value Zx and the threshold value θz (step S4). Specifically, when Zx is equal to or greater than θz, the motion vector search in the small region bx is turned off (0), and when Zx is smaller than θz, the motion vector search is turned on (1). As a result, a binary image as shown in FIG. 2C is output as encoding mode selection information.

  The viewpoint image encoding unit 102 controls the encoding mode using the viewpoint image and the encoding mode selection information. Specifically, when the coding mode selection information of the small area bx is 1, a motion vector search is performed, and when the coding mode selection information is 0, no motion vector search is performed, and other coding processes are performed. The processing load is reduced by performing encoding using (for example, intra prediction).

  In the processing of the encoding mode selection unit 101, the method of comparing the depth average value Zx for each small region in the depth information with the threshold value has been described above. Zx is the maximum value or the minimum value of the depth value for each small region. It is also good. When the maximum value is used, the processing load can be further reduced because the number of small areas determined to be motion vector search OFF is increased compared to the case where the average value is used. Since the number of small areas determined to be search ON increases, it is useful in cases where emphasis is placed on improving coding efficiency rather than reducing processing load.

  Next, a multi-view image encoding apparatus that is a second embodiment of the present invention will be described. FIG. 4 shows an internal block diagram of the multi-view image encoding apparatus of the present embodiment. Since the processing in the encoding mode selection unit 101, the viewpoint image encoding unit 102, and the multiplexing unit 104 is the same as that in the first embodiment, description thereof is omitted. In the present embodiment, the depth information encoding unit 103 also performs encoding according to the encoding mode selection information. That is, regarding the depth information, it is possible to reduce the processing load by controlling not to perform the motion vector search for the small area close to the viewpoint.

  Note that the processing of the encoding mode selection unit may be configured as in the flowchart shown in FIG. That is, when the encoding mode is selected, not only the depth information and the number of pixels of the viewpoint image but also the encoding frame rate is considered. Specifically, the threshold value θz is determined according to two values of the number of pixels p and the frame rate r (step S3 ′). The other processes (steps S1, S2, and S4) are the same as those in the first embodiment, and a description thereof will be omitted.

  In addition, when selecting an encoding mode, the number of viewpoint images to be encoded may be further considered in addition to the above. As an example in which the number of viewpoint images is different, for example, there is a technique of stereoscopic television using three or more viewpoint images. This is mainly a technology for autostereoscopic television. However, in autostereoscopic television, if there are only two viewpoints, the area where the image appears stereoscopic is very narrow. Since the image does not look stereoscopic, a method of expanding a stereoscopically viewable region using images from a plurality of viewpoints is used. Although this viewpoint image can be generated by a viewpoint image synthesis technique, a method for transmitting a plurality of images taken by an actual camera is considered necessary when a higher-definition image is required. As another example, there is a correspondence to an arbitrary viewpoint image. Arbitrary viewpoint image technology is a technology that can synthesize images of arbitrary viewpoints using a group of images taken from various positions and angles, and this naturally requires a considerable number of viewpoints. Conceivable.

When such a multi-viewpoint image is applied, if the number of viewpoints to be encoded increases, the load of the encoding process increases accordingly. Therefore, the process of the encoding mode selection unit is configured as described below. As a result, an increase in processing load can be suppressed.
For example, when the number of viewpoints v exceeds a predetermined threshold value, the threshold value θz for the depth is reset again. Alternatively, the threshold θz is determined by comprehensively considering the number of pixels, the frame rate, and the number of viewpoints. In addition, ON / OFF of the encoding mode selection process itself may be determined based on the number of viewpoints v. That is, the encoding selection mode selection unit can determine a threshold for depth information based on the number of pixels of the viewpoint image, or the number of pixels and the frame rate, or the number of pixels, the frame rate, and the number of viewpoints. Here, in any case, the processing load in encoding can be further reduced.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and the design and the like within the scope of the present invention are also within the scope of the claims. include.

DESCRIPTION OF SYMBOLS 100 ... Multiview image coding apparatus, 101 ... Coding mode selection part, 102 ... Viewpoint image coding part, 103 ... Depth information coding part, 104 ... Multiplexing part.

Claims (4)

A multi-viewpoint image encoding device that encodes a plurality of viewpoint images and corresponding depth information,
Based on the depth information and the viewpoint image , a coding mode is selected which is either a coding mode in which motion compensation is performed by motion vector search between two image frames or a coding mode in which motion compensation is not performed by motion vector search. Encoding mode selection means for outputting encoding mode selection information indicating the selected encoding mode;
Viewpoint image encoding means for encoding the viewpoint image according to the encoding mode selection information ;
Depth information encoding means for encoding the depth information ;
The encoding mode selection unit determines a threshold for the depth information based on the number of pixels of the viewpoint image, or the number of pixels and the frame rate, or the number of pixels, the frame rate, and the number of viewpoints. A multi- view image encoding apparatus , wherein an encoding mode is selected for each small region in accordance with a comparison result with the depth information for each small region . The encoding mode selection means outputs the encoding mode selection information for selecting an encoding mode in which motion compensation is not performed by the motion vector search for a small region closer to the viewpoint than the threshold value, The multi-view image encoding apparatus according to claim 1 . The encoding mode selection means outputs the encoding mode selection information for selecting an encoding mode for performing motion compensation by searching for a motion vector between the two image frames for a small region farther from the viewpoint than the threshold. The multi-viewpoint image encoding apparatus according to claim 1 , wherein: The multi-view image encoding apparatus according to any one of claims 1 to 3 , wherein the depth information encoding unit encodes depth information according to the encoding mode selection information.