JP6028965B2 - Depth map resampling apparatus and method, program, and depth map resampling system - Google Patents

Description

  The present invention relates to a depth map resampling apparatus and method, a program, and a depth map resampling system used for generating a multi-view video.

  Currently, twin-lens stereoscopic television and twin-lens stereoscopic movies are becoming widespread. However, since only a part of the stereoscopic viewing factors is realized, there is no motion parallax and it is unnatural, or there is eye fatigue caused by wearing glasses. Therefore, there is a demand for practical use of a more natural autostereoscopic image that does not require special glasses for observing the stereoscopic image.

  An autostereoscopic image is composed of a number of viewpoint images. For this reason, when an autostereoscopic image is to be displayed, it is necessary to store and transmit a large number of viewpoint images, which requires a large amount of data and is difficult to put into practical use. Therefore, conventionally, a method of restoring a stereoscopic video using a depth map that is depth information of a viewpoint video is known (see, for example, Patent Document 1).

  Here, an example of the depth map is shown in FIG. FIG. 8A is a depth map obtained when, for example, a floor is photographed, and the bottom of the rectangle indicates a near side of the subject and is whitish and bright. The upper side of the rectangle indicates the back of the subject and is dark and dark. In this example, the floor is one subject, and the depth changes continuously and is smooth. Further, when the subject is, for example, a staircase, a depth map in which the depth changes in a staircase shape as shown in FIG. 8B is obtained. Although not shown, when there are a plurality of subjects having different depths, a near subject is displayed as whitish and a far subject is displayed as black.

  In a conventional method using a depth map, a viewpoint video obtained by thinning out the number of viewpoints from a multi-view video is accumulated and transmitted by adding the depth map to the multi-view video. A multi-view video in which the number of viewpoints is thinned in this way is also referred to as a small number of viewpoint videos. Then, by projecting the small number of viewpoint videos using the depth map at the transmission destination, the stereoscopic video can be restored by synthesizing the thinned viewpoint videos at the transmission source.

  FIG. 9A is a schematic block diagram of the image encoding device described in Patent Document 1. The image encoding device shown in FIG. 9 includes an image signal encoding unit 107 that encodes a multi-view video (image signal), and a depth signal encoding unit 108 that encodes a depth map (depth signal) of the multi-view video. And. The image encoding apparatus also includes an encoding management unit 101 that encodes encoding management information indicating information related to encoding, and parameter information encoding that encodes the encoded information and camera parameter information. Part 110. The image encoding apparatus includes a unitization unit 109, and generates an encoded bit string from outputs of the image signal encoding unit 107, the depth signal encoding unit 108, and the parameter information encoding unit 110. This encoded bit string is stored and transmitted, and is decoded at the transmission destination.

  The image signal encoding unit 107 shown in FIG. 9A encodes an image from a viewpoint to be used as a reference among the viewpoint videos (image signals), and encodes the other images between a plurality of images. Interframe predictive coding. In addition, the depth map (depth signal) of one or more viewpoints is similarly subjected to intra-frame prediction encoding or inter-frame prediction encoding by the depth signal encoding unit 108. For this reason, the depth signal encoding unit 108 also includes an intra-frame prediction encoding 108a and an inter-frame prediction encoding 108b, as shown in FIG. 9B. The depth signal encoding unit 108 compresses all pixels by, for example, MPEG. At this time, the number of pixels of the depth map is not changed by encoding. In short, all encoded viewpoint videos and depth maps have the same number of pixels as the original.

JP 2010-157821 A JP 2001-86345 A

  For example, as shown in FIG. 8A, the depth map has a property of smoothly continuing on the same subject regardless of the subject texture and depending only on the distance from the camera to the subject. . That is, the depth value changes smoothly in one subject (for example, the floor) (inside the subject). Therefore, the resolution of the depth map does not have to be the same as the resolution of the viewpoint video in order to reduce the amount of data when storing and transmitting autostereoscopic video composed of a large number of viewpoint videos. You can lower the resolution. However, in the encoding method described in Patent Document 1, all encoded viewpoint videos and depth maps have the same number of pixels as the original. Therefore, encoding and transmitting the depth map with the original number of pixels has a problem that encoding efficiency is poor.

  Therefore, it is conceivable to resample the depth map to a small number of pixels to reduce the amount of data to be stored and transmitted. A conventional image processing apparatus that performs general re-sampling is described in Patent Document 2, for example. FIG. 10A is a block diagram schematically showing the configuration of the image re-sampling unit 220 in the image processing apparatus 200, and FIG. 10B is a flowchart showing the processing flow. In the image re-sampling unit 220, the LPF 223 receives the original image, passes through the low-pass filter, and removes a high-frequency component that causes aliasing distortion (step S201). The image from which the high frequency component has been removed is input to the sub-sampling unit 224. Then, the sub-sampling unit 224 thins out pixels and reduces the image size (step S202). Thereby, the generated reduced image is output. Thus, in the case of resampling, it has been common to perform resampling by applying a low-pass filter to pixels in order to prevent aliasing distortion.

  However, if the depth map to be added to the multi-view video is resampled, the low-pass filter (LPF 223) generates an intermediate depth value that was not found in the original depth map, and the quality of the depth map deteriorates. In addition, there is a problem that distortion occurs in a multi-view video synthesized using this.

  Therefore, in the present invention, it is an object to provide a depth map resampling apparatus and method, a program, and a depth map resampling system that can reduce deterioration in the quality of a depth map, which is depth information of a multi-view video. And

  In order to solve the above-mentioned problem, a depth map resampling apparatus according to claim 1 of the present invention thins out pixels from a first depth map that is depth information of a multi-view video obtained by photographing a subject from multiple viewpoints. Re-sampling of a depth map is performed by inputting a depth map obtained by thinning out pixels through a predetermined accumulation and transmission path and restoring the depth value of the thinned pixels by interpolation or copying. An apparatus comprising a depth value determining unit, a depth interpolating unit, a depth copying unit, and a median filter.

According to such a configuration, the resampling device is configured such that the depth value difference between two adjacent pixels in the second depth map, which is a depth map thinned out by the depth value determination unit, is a predetermined value. It is determined whether it is smaller than the first threshold. Here, the first threshold value is determined in advance so as to represent the depth value inside the subject if the difference in depth value is smaller than the first threshold value, and to represent the depth value of the boundary of the subject otherwise. If the difference between the depth values of two adjacent pixels in the second depth map is smaller than the first threshold value, the resampling device uses the depth interpolation unit to determine the depth values of the two adjacent pixels as a predetermined value. The value obtained through the first low-pass filter is set as a depth value obtained by interpolating the pixels thinned out between the two adjacent pixels. Thereby, the depth value inside the subject can be smoothly continued. If the difference between the depth values of two adjacent pixels in the second depth map is greater than or equal to the first threshold value, the resampling device uses the depth copying unit to determine which of the two adjacent pixels is larger than the other. A value obtained by copying the depth value is set as the depth value of the pixels thinned out between the two adjacent pixels. Thus, by interpolating the depth value thinned out at the boundary of the subject with the depth value of the foreground having a large value, it is possible to restore a multi-view video with less sense of discomfort. Then, the depth value interpolated by the depth interpolation means, and the depth value is copied by the depth copying means, the depth value of the second depth map, the third depth map combined, resampling apparatus , A median filter applies a window of a predetermined size that identifies a pixel to be processed and its surrounding pixels to each pixel, and replaces the depth value of each pixel with the center value of all the depth values in the window. The Thereby, the depth value roughly restored at the boundary of the subject can be accurately restored.

  The depth map re-sampling device according to claim 2 of the present invention is the re-sampling device according to claim 1, wherein an edge for determining the presence / absence of an edge of a subject is provided before the processing by the depth value determination means. It is further provided with determination means and low-pass filter means for generating a new depth value when there is no edge.

  According to such a configuration, the resampling device is configured such that the difference between the depth values of two neighboring pixels separated by a predetermined number of pixels in the second depth map input by the edge determination unit is a predetermined value. It is determined whether or not the threshold value is equal to or greater than a second threshold value. If the threshold value is equal to or greater than the second threshold value, it is determined that an edge of the subject exists between the two neighboring pixels, and a difference in depth value is smaller than the second threshold value. If it is determined that the edge does not exist. Among these, when there is no edge, that is, when the difference between the depth values of the two neighboring pixels is smaller than the second threshold value, the re-sampling device uses the low-pass filter means in the second depth map. A value obtained by passing a depth value of a pixel having a predetermined number of neighboring pixels through a predetermined second low-pass filter is obtained as a central pixel among the pixels having the predetermined number of neighboring pixels. A new depth value for. Therefore, even if encoding distortion is added during transmission and accumulation of the depth map with pixels thinned out, it is possible to remove the influence of the encoding distortion and restore the depth map in which the depth value continuously changes. it can. On the other hand, if the depth value difference is equal to or greater than the second threshold value, the resampling device determines that the boundary of the subject exists between two neighboring pixels, and performs the process using the second low-pass filter described above. Absent. Therefore, when the resampling device removes the influence of coding distortion and restores the depth map in which the depth value changes continuously, the depth value near the boundary of the subject originally provided in the original depth map Can maintain the original value.

  In order to solve the above problem, a depth map re-sampling system according to claim 3 of the present invention provides a first depth map which is depth information of a multi-view video obtained by photographing a subject from multiple viewpoints. A first re-sampling device that inputs and creates a depth map with thinned pixels; and a thinned-out pixel input from the first re-sampling device via a predetermined accumulation and transmission path A depth map resampling system comprising: a second resampling device that restores the depth values of the thinned pixels from the second depth map by interpolation or copying, wherein the first resampling system includes: And the second resampling device includes a depth value determining unit, a depth value determining unit, and a depth value determining unit. Interpolating means, depth copying means, and media Characterized in that it comprises a Ann filter, the.

According to this configuration, in the depth map resampling system, the first resampling device creates a depth map in which pixels are thinned out by a process of thinning out pixels without passing through a low-pass filter. Therefore, the depth map thinned out by this pixel does not include a depth value that was not originally present in the first depth map. Therefore, it is possible to prevent the occurrence of problems such as a part of the foreground and background subjects overlapping each other and the foreground and background pixels being repeatedly arranged. In the depth map re-sampling system, the second re-sampling device is configured such that the second re-sampling device includes two adjacent ones in the second depth map that is the depth map thinned out by the depth value determination unit. It is determined whether or not the difference in pixel depth value is smaller than a predetermined first threshold value. When the difference between the depth values of two adjacent pixels in the second depth map is smaller than the first threshold, the second resampling device uses the depth interpolation unit to determine the depth value of the two adjacent pixels. Is a depth value obtained by interpolating the pixels thinned out between the two adjacent pixels. When the difference between the depth values of two adjacent pixels in the second depth map is equal to or greater than the first threshold value, the second resampling device uses the depth copying unit to determine the two adjacent pixels from others. A value obtained by copying the larger depth value is set as the depth value of the pixels thinned out between the two adjacent pixels. Then, the depth value interpolated by the depth interpolation means, and the depth value is copied by the depth copying means, the depth value of the second depth map, the third depth map combined, second resampling The converting device applies a window of a predetermined size that identifies a pixel to be processed and a surrounding pixel group to each pixel by a median filter, and sets the depth value of each pixel to the center of all the depth values in the window. Replace with value.

  In order to solve the above problem, the resampling method according to claim 4 of the present invention performs pixel thinning from a first depth map that is depth information of a multi-view video obtained by photographing a subject from multiple viewpoints. Depth map re-sampling device for inputting depth map thinned out pixels via a predetermined accumulation and transmission path created by performing processing and restoring depth values of the thinned pixels by interpolation or copying A re-sampling method according to claim 1, wherein the re-sampling device has a predetermined size that specifies at least a predetermined first threshold value, a predetermined first low-pass filter, a pixel to be processed, and a surrounding pixel group. Storage means for storing the window, and processing means, wherein the processing means includes a depth value determination processing step, a depth interpolation processing step, a depth copying processing step, and a median filter processing step. Characterized in that row.

According to such a procedure, the processing means of the resampling apparatus has the depth of two pixels adjacent to each other in the second depth map that is the depth map thinned out in the depth value determination processing step. It is determined whether or not the value difference is smaller than the first threshold value. In the depth interpolation processing step, the processing unit of the resampling apparatus may determine that the two adjacent pixels when the difference between the depth values of the two adjacent pixels in the second depth map is smaller than the first threshold value. A value obtained by passing the depth value of the first low-pass filter through the first low-pass filter is a depth value obtained by interpolating the pixels thinned out between the two adjacent pixels. Then, the processing means of the resampling apparatus, in the depth copying processing step, if the difference between the depth values of two adjacent pixels in the second depth map is not less than the first threshold value, the two adjacent pixels A value obtained by copying the depth value larger than the other for is set as the depth value of the pixel thinned out between the two adjacent pixels. Then, the processing means of the resampling apparatus at median filter processing step, the depth value interpolated by the depth interpolation processing step, a depth value that is copied by the depth copy processing step, the second depth map In the third depth map in which the depth values of the pixels are combined, the window of the predetermined size is applied to each pixel, and the depth value of each pixel is replaced with the center value of all the depth values in the window.

  The depth map re-sampling method according to claim 5 of the present invention is the re-sampling method according to claim 4, wherein the storage means of the re-sampling device has a predetermined second threshold value and a predetermined first threshold value. Two low-pass filters, and the processing means includes two neighboring pixels separated by a predetermined number of pixels in the input second depth map before the depth value determination processing step. It is determined whether or not the difference in depth value is equal to or greater than the second threshold value. If the depth value is equal to or greater than the second threshold value, an edge determination processing step is performed to determine that the edge of the subject is present. When it is determined that there is no edge of the subject because the difference in depth value is smaller than the second threshold value, the depth value of the pixel having the predetermined number of pixels in the vicinity is set to a predetermined second low frequency range. Obtained through a pass filter A low-pass filter processing step that sets a value to be a new depth value for a central pixel of the neighboring pixels having a predetermined number of pixels, and in the edge determination processing step, When it is determined that there is an edge, the depth value determination processing step is executed by skipping the low-pass filter processing step.

  According to such a procedure, when the processing unit of the resampling apparatus determines that the difference in the depth value is smaller than the second threshold value in the edge determination processing step, the processing unit in the low-pass filter processing step A depth value of a predetermined number of pixels is passed through a predetermined second low-pass filter to generate a new depth value. Therefore, even if encoding distortion is added during transmission and accumulation of the depth map with pixels thinned out, it is possible to remove the influence of the encoding distortion and restore the depth map in which the depth value continuously changes. it can. Further, when it is determined in the edge determination processing step that the depth value difference is equal to or larger than the second threshold value, the low-pass filter processing step is skipped, so the vicinity of the boundary of the subject originally provided in the original depth map As for the depth value of, the original value can be maintained.

  A depth map resampling program according to claim 6 of the present invention is created by a process of performing pixel thinning from a first depth map which is depth information of a multi-view video obtained by photographing a subject from multiple viewpoints. In order to input a depth map with thinned pixels through a predetermined accumulation and transmission path, and to restore the depth value of the thinned pixels by interpolation or copying, a computer is provided with a depth value judging means, a depth This is a program for functioning as interpolation means, depth copying means, and median filter.

According to such a configuration, the re-sampling program is configured such that the depth value difference between two adjacent pixels in the second depth map, which is the depth map thinned out by the depth value determination unit, is a predetermined value. It is determined whether it is smaller than the first threshold. When the difference between the depth values of two adjacent pixels in the second depth map is smaller than the first threshold, the resampling program uses the depth interpolation means to calculate the depth values of the two adjacent pixels to a predetermined value. The value obtained through the first low-pass filter is set as a depth value obtained by interpolating the pixels thinned out between the two adjacent pixels. If the difference between the depth values of two adjacent pixels in the second depth map is equal to or greater than the first threshold, the resampling program uses the depth copying unit to determine which of the two adjacent pixels is larger than the other. A value obtained by copying the depth value is set as the depth value of the pixels thinned out between the two adjacent pixels. Then, the depth value interpolated by the depth interpolation means, and the depth value is copied by the depth copying means, the depth value of the second depth map, the third depth map combined, resampling program , A median filter applies a window of a predetermined size that identifies a pixel to be processed and its surrounding pixels to each pixel, and replaces the depth value of each pixel with the center value of all the depth values in the window. The

  According to the present invention, a depth map similar to the boundary position of the subject in the original depth map before thinning out pixels can be restored from the depth map thinned out of pixels. Therefore, it is possible to reduce the deterioration of the quality of the depth map when re-sampling the depth map in which the amount of data stored and transmitted together with the multi-viewpoint video is reduced. Therefore, it is possible to reduce distortion in the multi-view video synthesized using the restored depth map.

It is a block diagram which shows the structure of a resampling system provided with the resampling apparatus of the depth map which concerns on 1st Embodiment of this invention. It is a block diagram which shows the whole structure of the transmission / reception system of the stereo image using the resampling system of the depth map which concerns on embodiment of this invention. It is explanatory drawing (the 1) of the process which decompress | restores the depth map of the original pixel number. It is explanatory drawing (the 2) of the process which decompress | restores the depth map of the original pixel number. It is explanatory drawing which shows typically the state transition of the depth map by the process of the resampling apparatus which concerns on embodiment of this invention. It is a flowchart which shows the whole flow of the resampling process by the resampling system which concerns on embodiment of this invention. It is a block diagram which shows the structure of the resampling system provided with the resampling apparatus of the depth map which concerns on 2nd Embodiment of this invention. FIG. 8 is an explanatory diagram of coding distortion in the resampling system shown in FIG. 7, (a) to (c) are coding distortions in the depth map and state transitions due to the removal thereof, and (d) is a coding distortion removing filter. An example of a tap coefficient is shown. It is a block diagram which shows typically the structure of the encoding apparatus of the conventional multiview video. It is a figure which shows typically the structure and operation | movement of the image processing apparatus which compression-encodes the conventional image data.

  A depth map resampling apparatus and method, a program, and a depth map resampling system according to the present invention will be described in detail with reference to the drawings.

(First embodiment)
[Outline of resampling system for depth map]
The depth map re-sampling system according to the first embodiment is a system for re-sampling a depth map that is depth information of a multi-view video obtained by photographing a subject from multiple viewpoints. The depth map resampling system 1 shown in FIG. 1 includes a first resampling device 10 and a second resampling device 20 as depth map resampling devices. The first resampler 10 and the second resampler 20 are connected by a predetermined accumulation / transmission path 30. In the first embodiment, it is assumed that the encoding distortion in the storage / transmission path 30 is sufficiently small and can be ignored. That is, the storage / transmission path 30 can be equivalently regarded as one electric wire. A description of the entire stereoscopic video transmission / reception system using the resampling system 1 will be given later. In FIG. 1 and FIG. 2, transmission of the depth map is indicated by a thick solid arrow.

  The first resampling device 10 and the second resampling device 20 include an arithmetic device such as a CPU, a storage device such as a memory and a hard disk, and an interface device that transmits and receives various information that is input or output. I have. Hereinafter, the depth map input to the first resampling apparatus 10 is formally called a first depth map, and the depth map input to the second resampling apparatus 20 is formally. It is also called a second depth map.

  The first resampling device 10 inputs a first depth map and creates a depth map with pixels thinned out. The first resampling apparatus 10 creates a depth map in which pixels are thinned out by a process of thinning out pixels from the first depth map without passing through a low-pass filter.

  The second resampling device (depth map resampling device) 20 receives a second depth map in which pixels are thinned out and input from the first resampling device 10 via the storage / transmission path 30. The depth values of the thinned pixels are restored by interpolation or copying. The second resampling device 20 outputs the restored depth map. The detailed internal configuration will be described later.

[Stereoscopic video transmission and reception system]
Here, FIG. 2 shows an overall configuration of a stereoscopic video transmission / reception system using the depth map resampling system according to the embodiment of the present invention. The stereoscopic video transmission / reception system 2 includes a depth map re-sampling system 1 (first re-sampling device 10 and second re-sampling device 20), a storage / transmission path 30, and a multi-view video synthesizing device 40. And.

  The accumulation / transmission path 30 includes an encoding unit 31, a decoding unit 32, an encoding unit 33, and a decoding unit 34. The encoding unit 31 encodes the depth map that has been thinned out by the first resampling apparatus 10. The encoding method of the image by the encoding unit 31 is, for example, MPEG-4 AVC (H.264). Since MPEG-4 AVC is a well-known moving image compression method, the description thereof is omitted.

  The decoding unit 32 decodes the encoded, stored and transmitted depth map. The decoding unit 32 decodes the encoded depth map by a decoding method corresponding to the encoding method by the encoding unit 31. The decoded depth map is input to the second resampling device 20. The depth map input to the second resampling device 20 is a second depth map in which pixels are thinned out. In the first embodiment, since the encoding distortion in the storage / transmission path 30 is ignored, the second depth map input to the second resampler 20 is used as the first resampler. 10 is regarded as the same as the depth map in which pixels are thinned out.

The encoding unit 33 encodes a small number of viewpoint videos (multi-viewpoint videos with a reduced number of viewpoints) attached to the original depth map (first depth map). The encoding of the image by the encoding unit 33 is the same as that of the encoding unit 31.
The decoding unit 34 decodes a small number of viewpoint videos that have been encoded, stored, and transmitted. The decoding of the video by the decoding unit 34 is the same as that of the decoding unit 32. A small number of decoded viewpoint videos are input to the multi-view video synthesis apparatus 40.

  The multi-view video synthesizing device 40 synthesizes a multi-view video from a small number of input viewpoint videos using the restored depth map that is an output from the second resampling device 20. The multi-view video synthesized here is displayed as a stereoscopic video on a stereoscopic display (not shown).

[Configuration of Second Resampler]
Here, the details of the configuration of the second resampling apparatus 20 will be described.
The second resampling device 20 is realized by controlling the hardware resources described above by the cooperation of the hardware device and software, and as a processing means as shown in FIG. , A depth value determiner 21, a depth interpolator 22, a depth copier 23, and a median filter 24. In addition, the second resampling device 20 stores at least a predetermined first threshold value, a predetermined first low-pass filter, a pixel to be processed, and a surrounding pixel group in a storage unit (not shown). The size of the window is memorized.

  The depth value determiner (depth value determination means) 21 determines whether the difference between the depth values of two adjacent pixels in the second depth map, which is a depth map thinned out from the input pixels, is smaller than a predetermined first threshold value. It is to determine whether or not. Here, the depth value of a pixel has a larger value as the color is white at a pixel near the subject, and a smaller value as the color is darker at a pixel at the back side of the subject. For example, when the depth value is represented by 8 bits, the value is “0” to “255”. Further, the first threshold value is determined in advance in order to define whether or not the two adjacent pixels in the thinned second depth map are present in the same subject. That is, when the depth value difference is smaller than the first threshold value, the two pixels are in the same subject, and when the depth value difference is greater than or equal to the first threshold value, there is a different subject boundary between the two pixels. Shall. The depth value determiner 21 passes the process to the depth interpolator 22 when it is determined that the depth value difference is smaller than the first threshold value, and if the depth value difference is greater than or equal to the first threshold value, the depth value determiner 21 performs the process. It is passed to the copying machine 23.

  The depth interpolator (depth interpolation means) 22 passes the depth values of the two adjacent pixels through a low-pass filter when the difference between the depth values of the two adjacent pixels in the second depth map is smaller than the first threshold value. The value obtained in this way is a depth value obtained by interpolating the pixels thinned out between two adjacent pixels. The low-pass filter (first low-pass filter) in the depth interpolator 22 is, for example, a 2-tap low-pass filter with tap coefficients of (1/2, 1/2). When the difference is smaller than the first threshold, that is, within the same subject, the depth interpolator 22 interpolates the value obtained by applying the low-pass filter as the depth value of the thinned pixel, and calculates the depth value. The depth map of the original number of pixels is restored by making it continue smoothly. A processing example will be described later. The depth interpolator 22 interpolates the depth value and then passes the processing to the median filter 24.

  The depth copying device (depth copying means) 23, when the difference between the depth values of two adjacent pixels in the second depth map is equal to or larger than the first threshold value, copies the larger depth value for the two adjacent pixels. The value obtained in this way is used as the depth value of the pixels thinned out between two adjacent pixels. When the difference is greater than or equal to the first threshold value, that is, at the boundary between different subjects, the depth copying unit 23 copies the larger depth value of the adjacent pixels as the depth value of the thinned pixel, Restore the depth map of the original number of pixels. A processing example will be described later. The depth copier 23 copies the depth value and then passes the processing to the median filter 24.

The median filter 24 applies a median selection filter to the restored depth map of the original number of pixels. The median filter 24 is a depth map (hereinafter, a third depth map) obtained by combining the depth value interpolated by the depth interpolator 22, the depth value copied by the depth copy unit 23, and the depth value of the second depth map. In other words, a window having a predetermined size for specifying a pixel to be processed and a surrounding pixel group is applied to each pixel, and the depth value of each pixel is replaced with the center value of all the depth values in the window. The median filter 24 applies a median value selection filter to the third depth map, thereby smoothing the boundary portion of the subject without generating a depth value that was not in the original depth map (first depth map). it can. A processing example will be described later.

[Example of processing to restore the depth map of the original number of pixels]
3 and 4 are explanatory diagrams of processing for restoring the depth map of the original number of pixels by the depth interpolator 22 and the depth copier 23 of the second resampling apparatus 20. The graph shown in FIG. 3A shows a part of the depth value of a pixel in a normal stereoscopic video depth map (for example, see FIG. 8A) cut out in the horizontal direction (X-axis direction). Yes. The horizontal axis of the graph shown in FIG. 3A indicates the position of the pixel in the X-axis direction, and the vertical axis indicates the depth value of the pixel.

In FIG. 3A, the depth value indicates the original depth value, and here, the position of the pixel and the set of the original depth value are represented by (X ₁₁ , a), (X ₁₂ , b), (X ₁₃ , c), (X ₁₄ , d), (X ₁₅ , e), (X ₁₆ , f). Actually, the depth values of the pixels are similarly arranged on the left side of the position X ₁₁ and the right side of the position X ₁₆ . The depth values a, b, and c are the depth values of the foreground subject and have a large value, and the depth values d, e, and f are the depth values of the background subject and have a small depth value. . Therefore, there is a large gap between the depth value c and the depth value d.

In the depth map resampling system 1, in the first resampling device 10, the original depth value shown in FIG. 3A is directly thinned out without passing through a low-pass filter. The depth value indicated by the solid line in FIG. Note that in the example shown in FIG. 3B, the thinning of pixels at the odd pixel positions (X ₁₁ , X ₁₃ , X ₁₅ ) is indicated by broken lines. Thereby, (X ₁₂ , b), (X ₁₄ , d), (X ₁₆ , f) are stored and transmitted as the thinned depth map.

  The method of comparison with the present invention (method of passing through a conventional low-pass filter) has the following problems. As shown in the graph of FIG. 3A, a portion where the depth value changes rapidly due to a large gap between the depth value c and the depth value d contains many high frequency components. If this pixel is thinned out as it is, and then the thinned pixel is interpolated with surrounding pixels, a frequency component different from the original is generated, and so-called aliasing distortion occurs. In order to prevent this aliasing, in the comparison method, as is done in normal video, first, on the code side, each pixel is passed through a low-pass filter to remove high-frequency components, and then the pixel Perform thinning. Thereafter, on the decoding side, the remaining pixels are passed through a low-pass filter to interpolate the thinned pixels. (For example, see Patent Document 2)

FIG. 3C shows the depth value after the original depth value of FIG. 3A is passed through the low-pass filter on the code side by the comparison method. Here, as the low-pass filter, for example, a 3-tap filter having tap coefficients of (1/4, 2/4, 1/4) is used. As a result, at the pixel position X ₁₃ , the original depth value b of the foreground subject and the original depth value d of the background subject are mixed, so that the value α is different from the original depth value c. Similarly, the original depth value c of the foreground subject and the original depth value e of the background subject are mixed at the pixel position X ₁₄ , so that the value β is different from the original depth value d. Thereafter, in FIG. 3C, for example, pixel thinning is performed at odd pixel positions (X ₁₁ , X ₁₃ , X ₁₅ ), and as a thinned depth map, (X ₁₂ , b), (X ₁₄ , β), (X ₁₆ , f) are stored and transmitted. After that, when the decoding side passes the accumulated and transmitted depth values through, for example, a 2-tap low-pass filter with tap coefficients of (1/2, 1/2), and interpolates the thinned depth values, as shown in FIG. 3 (c), in for example, the pixel positions X _13, is restored value close to the value alpha, for example, the pixel positions X _14, a value close to the value β is restored.

In the depth map restored by the comparison method, the intermediate level depth values α and β that were not found in the original depth map are thus restored. When the depth map restored by the comparison method is used to synthesize a viewpoint video at a position different from the original viewpoint position, the following problem occurs. That is, in the view image synthesized in different positions, since the depth value of the depth map represents the moving distance of the lateral direction of each pixel, the pixel positions X ₁₃ is other pixel (position X of the same subject (foreground subject) ₁₁ pixels, the pixel position X ₁₂₎ and for depth values are different, the moving distance in the lateral direction is different from the other pixels of the same subject. Similarly, the pixel at the position X ₁₄ is different in depth value from the other pixels (the pixel at the position X _{15 and} the pixel at the position X ₁₆ ) of the same subject (background subject). The moving distance in the horizontal direction is different. As a result, there are problems that some of the foreground and background subjects overlap each other, and the foreground and background pixels are repeatedly arranged in a ringing pattern.

  On the other hand, the depth map re-sampling system 1 of the present invention uses the first re-sampling device 10 shown in FIG. 1 to convert the original depth value shown in FIG. The pixel value is directly thinned out without passing through to obtain a depth value indicated by a solid line in FIG. These depth values include only the original depth value. Therefore, it is possible to solve the problem caused by the comparison method.

Next, in the depth map resampling system 1 of the present invention, a procedure in which the second resampling device 20 restores the thinned pixels on the decoding side will be described. In the second re-sampling device 20, first, the depth value determiner 21 examines the difference between the depth values of the thinned adjacent pixels, which is indicated by the solid line in FIG. Among these, for example, the difference between the depth values of the pixel at the position X _{14 and} the pixel at the position X ₁₆ corresponding to the portion where the original same depth value is continuous like d, e, f in the background subject. Is smaller than a first threshold value (for example, 5) (adjacent pixels are of the same subject). In this case, at the depth interpolator 22, these adjacent pixels, the value obtained for instance through a 2-tap low-pass filter tap coefficients (1 / 2,1 / 2), decimated positions X ₁₅ The depth value of the pixel of. In this example, as a result, the original depth value e is restored to the pixel positions X _15. In addition, for the pixel at position X ₁₁ where the same depth value is continuous like a and b in the foreground subject, the depth value of the pixel at position X ₁₀ (not shown) is also a 2-tap low range. The original depth value a is restored by passing through a pass filter.

On the other hand, in the depth value determiner 21, for example, the difference between the depth values for the pixel at the position X _{12 and} the pixel at the position X ₁₄ indicated by the solid line in FIG. 3B is equal to or greater than the first threshold (for example, 5). It is determined that there is a pixel (adjacent pixels are determined to be from different subjects). In this case, the depth copier 23 sets the depth value of the larger adjacent pixel as the depth value of the thinned pixel. In this example, since the depth value of the pixel at the position X ₁₂ is larger, as a result, the original depth value c is restored for the pixel at the position X ₁₃ . Therefore, the depth map shown in FIG. 4A is restored from the depth map (second map) thinned out in FIG. The restored depth map is the same as the original depth map (first map) in FIG.

In the example of FIG. 3 (a), thinning at large but the gap has to be one, the same pixel position in the case where the position of the gap is deviated by one of the depth values between the positions X ₁₃ and position X ₁₄ The final result is slightly different. Since the result at this time is equivalent to the result obtained when pixel decimation is performed at even pixel positions (X ₁₂ , X ₁₄ , X ₁₆ ) with the same gap position, an example in that case will be described. In the example shown in FIG. 4B, the first re-sampling apparatus 10 has an even pixel position (X ₁₂ , X ₁₄ , X ₁₆ ) from the original depth map (first map) in FIG. The dotted line indicates that pixel thinning has been performed. Therefore, (X ₁₁ , a), (X ₁₃ , c), (X ₁₅ , e) are stored and transmitted as the thinned depth map. In this case, for example, the depth value d at the pixel position X ₁₄ of the background-side object are thinned out.

Then, the determination of the second resampling apparatus 20, relative to the depth value of the pixel positions X ₁₄ of the background-side object is given depth value c of the foreground side object (see FIG. 4 (c)). In this case, on the restored depth map, the depth values (a, b, c) of the pixel positions (X ₁₁ , X ₁₂ , X ₁₃ ) of the foreground subject are all restored, and further, the foreground subject is Only one pixel is fat. As a result, in the multi-view video synthesized using the restored depth map, the foreground subject maintains its shape without distortion, and an incorrect pixel exists for one pixel at the boundary. Usually, the pixel at the subject boundary is often a mixture of the depth values of the pixels of both subjects, and the wrong pixel on the background side of the boundary is not conspicuous due to the masking effect of the boundary of different subjects (textures) .

  In addition, as a human psychology, a person usually tends to concentrate and see more closely before a stereoscopic image. In other words, people are more concerned about distortion in the foreground of a stereoscopic image (foreground subject), so the distortion in the foreground is more noticeable. On the other hand, the rear side of the stereoscopic image (background subject, wall, etc.) tends not to be concerned even if the depth value is slightly shifted.

[Example of processing by median filter]
FIG. 5 is an explanatory diagram of processing by the median filter 24 of the second resampling device 20 and schematically shows state transition of the depth map. FIG. 5A is a two-dimensional representation of a part of a normal stereoscopic video depth map (see, for example, FIG. 8). A square portion surrounded by a dotted line represents one pixel, and has a staircase shape. The thick solid line represents the boundary of the subject and has different depth values on the upper side and the lower side of the boundary. Here, it is assumed that all pixels above the boundary have the depth value a of the background subject, and all pixels below the boundary have the depth value b (b> a) of the foreground subject.

  In FIG. 5A, for example, the position of the pixel in the first row of the depth map is indicated by coordinates (11, 11) to (11, 16). The depth values of the pixels in the first row are all a. In addition, for example, the position of the pixel in the sixth row of the depth map is indicated by coordinates (16, 11) to (16, 16). The depth values of the pixels in the sixth row are all b. Although only 6 × 6 36 pixels are shown, actually, the pixel is located on the left side / upper side of the pixel at the coordinate (11, 11) and the right side / lower side of the pixel at the coordinate (16, 16) position. Similarly, pixels and depth values are arranged.

  In the depth map resampling system 1, in the first resampling apparatus 10, the pixels in the original depth map shown in FIG. One pixel is thinned out. Accordingly, the 6 × 6 36 pixel portion shown in FIG. 5A becomes a 3 × 3 9 pixel portion shown in FIG. 5B. For example, on the first line of the thinned depth map, the pixels at the original coordinates (11, 11), (11, 13), (11, 15) are arranged. In the second row, pixels at the original coordinates (13, 11), (13, 13), (13, 15) are arranged, and in the third row, the original coordinates (15, 11) are arranged. ), (15, 13), and the pixels at the positions (15, 15) are arranged. That is, in the depth map shown in FIG. 5A, only the depth value of the upper left pixel in the portion that was originally 2 × 2 4 pixels is the depth value remaining in the depth map shown in FIG. is there. This thinned depth map is stored and transmitted.

  In the depth map resampling system 1, the accumulated and transmitted depth map is input to the second resampler 20. The second re-sampling device 20 first calculates, for example, a difference between adjacent depth values in the depth map (second depth map thinned out in FIG. 5B) input by the depth value determiner 21. Investigate. When the difference is smaller than the first threshold value (for example, 5), the depth value of the adjacent pixel is thinned with a value obtained by passing a 2-tap low-pass filter with a tap coefficient (1/2, 1/2), for example. When the pixel is interpolated and the difference is equal to or greater than the first threshold (for example, 5), the larger depth value of the adjacent pixels is copied as the thinned depth value. Next, for example, a difference between depth values adjacent to the left and right is examined, and the same determination process is performed. As a result, the depth map (third depth map) of FIG. 5C is obtained. For example, the difference between the depth values adjacent to the left and right may be checked, and then the difference between the depth values adjacent to the top and bottom may be checked.

  Specifically, when adjacent to each other, for example, the depth value a of the pixel at the position of the original coordinate (13, 11) and the depth value of the pixel at the position of the original coordinate (15, 11) From the difference from b, b, which is the larger depth value, is assigned to the pixel at the position indicated by reference numeral 501 in FIG. When adjacent to the left and right, for example, the depth value a of the pixel at the position of the original coordinate (11, 13), the depth value a of the pixel at the position of the original coordinate (11, 15), From the difference, the pixel at the position indicated by the reference numeral 502 in FIG. 5C is assigned an averaged depth value a. Similarly, the pixel at the position indicated by reference numeral 503 in FIG. 5C is assigned b, which is the greater depth value for the left and right pixels. For the pixel at the position indicated by reference numeral 504 in FIG. 5C, the pixels adjacent to the left and right are obtained first, and then b, which is the larger depth value for the left and right pixels, is obtained. Is granted. Pixels at other positions are restored in the same manner.

  When the depth map (third depth map) in FIG. 5C is compared with the original depth map (FIG. 5A), the resolution of the oblique subject boundary line is reduced and has a large step shape. That is, the three-step staircase indicating the boundary of the depth value in the original depth map is a one-step staircase in the third depth map, and the height of one step is doubled.

  In the present embodiment, the median filter 24 has, for example, a (3 × 3) pixel window, and the center depth value when the nine depth values in the window are arranged in order from the smallest. We decided to replace it with the (fifth) depth value. Specifically, the window of (3 × 3) pixels has nine pixels including a pixel at a position indicated by reference numeral 504 in FIG. 5C as a central target pixel and eight surrounding pixel groups. When enclosing a pixel group, the depth value of these nine pixels is 4 for the larger depth value b and 5 for the smaller depth value a, so the selected median is a.

  Further, the window of (3 × 3) pixels is aligned with the surrounding eight pixel groups with the pixel at the position of the original coordinates (13, 13) in FIG. In the case of enclosing nine pixel groups, the depth value of these nine pixels is 5 for the larger depth value b and 4 for the smaller depth value a, so the median value selected is b. .

  As a result, as shown in FIG. 5 (d), the thick dashed boundary in the final depth map changes to a solid line. In FIG. 5C, for example, the pixel at the position indicated by reference numeral 504 has a depth value changed to a as indicated by reference numeral 504a in FIG. Further, the depth value of the pixel located at the original coordinate (13, 13) in FIG. 5C is changed to b as indicated by reference numeral 505 in FIG. 5D. That is, as shown in FIG. 5C, the depth map that has become a large step and doubles the height of one step is changed to a height of one step on the boundary of the depth map by the processing of the median filter 24. Returning to the original, the boundary line is smoothed finely, and is close to the shape of the original boundary line in FIG. Therefore, the distortion of the depth map restored by resampling is reduced by the processing of the median filter 24.

  The boundary position of the subject indicated by the solid line in FIG. 5D is a portion where the foreground subject is thicker by one pixel than the original boundary position shown in FIG. 506). However, since the depth values of the visually important foreground subject are all restored, the foreground subject in the multi-view video synthesized by the depth map is not distorted. Further, the pixels of the multi-view video synthesized with the wrong depth value of the background subject are not conspicuous for the reason described above, and the quality of the synthesized video is improved.

[Resampling method of depth map]
A depth map resampling method according to the first embodiment of the present invention will be described with reference to FIG. 6 (refer to FIG. 1 as appropriate). FIG. 6 is a flowchart showing an overall flow of the resampling process by the resampling system according to the first embodiment of the present invention.

As shown in FIG. 6, the resampling method is roughly divided into two steps. One is processing by the first resampling device 10 (step S1: first resampling processing), and the other is processing by the second resampling device 20 (step S2: second). Re-sampling process).
Step S2 includes steps S23 (depth value determination processing), step S24 (depth interpolation processing), step S25 (depth copying processing), and step S26 (median filter processing) as internal processing. In addition, the process of S21 and S22 which is an internal process of step S2 is demonstrated in 2nd Embodiment.

  In step S1 (first resampling process), the first resampling device 10 inputs a depth map (first depth map), thins out the pixels without passing through the low-pass filter, and accumulates them. Send to transmission line 30 Then, the depth map stored and transmitted in step S11 is input to the second resampling device 20. The second resampling device 20 proceeds to step S23.

  In step S23 (depth value determination processing), the second resampling apparatus 20 uses the depth value determiner 21 to determine whether the difference between the depth values of adjacent pixels in the input depth map (second depth map) is a threshold ( It is determined whether it is smaller than the first threshold). As a result of the determination by the depth value determiner 21, if the difference between the depth values of adjacent pixels is smaller than the threshold value (first threshold value), the second resampling device 20 proceeds to the depth interpolation process S24. On the other hand, in other cases, the second resampling apparatus 20 proceeds to the depth copying process S25.

  In step S24 (depth interpolation processing), the second resampler 20 thins out values obtained by passing adjacent pixels of the second depth map through the low-pass filter by the depth interpolator 22. The depth value of the pixel. Following step S24, the second resampling device 20 proceeds to step S26.

  In step S25 (depth copying process), the second re-sampling device 20 uses the depth copying unit 23 to set the depth value of the larger pixel of the adjacent pixels of the second depth map to the depth value of the pixel that has been thinned out. . Following step S25, the second resampling apparatus 20 proceeds to step S26.

  In step S26 (median filter processing), the second resampling device 20 uses the median filter 24 to set the depth value of each pixel of the third depth map to the center of all the depth values in the predetermined surrounding window. Replace with value and output.

  The present invention is a program that causes a general computer to function as the depth value determiner 21, the depth interpolator 22, the depth copier 23, and the median filter 24 without constructing dedicated hardware. By using the (depth map resampling program), the second resampling device 20 shown in FIG. 1 can be realized. This program (depth map resampling program) can be distributed via a communication line, or can be written on a recording medium such as a CD-ROM for distribution.

  According to the depth map resampling apparatus and method, program, and depth map resampling system according to the first embodiment, the amount of data is reduced while suppressing distortion of the depth map, and the depth map and a small number of viewpoint videos 3D video storage and transmission efficiency can be improved. In addition, by using the restored depth map, it is possible to reduce distortion in a three-dimensional video composed of a multi-view video synthesized from a small number of viewpoint videos, and to provide high efficiency and high quality. As a result, a 3D image storage / transmission device or service using a depth map re-sampling system can easily store and transmit a depth map and provide high-quality 3D images.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to the said embodiment. For example, in the above embodiment, the encoding distortion in the storage / transmission path 30 is negligible, but this can be taken into account. In the storage / transmission path 30 shown in FIG. 2, in order to increase the encoding efficiency, the encoding unit 31 shown in FIG. It is common. Therefore, an embodiment assuming a case where the encoding distortion and the decoding element 32 shown in FIG. 2 are subjected to encoding distortion will be described as a second embodiment.

(Second Embodiment)
[Outline of resampling system for depth map]
FIG. 7 is a block diagram showing a configuration of a resampling system including a depth map resampling apparatus according to the second embodiment of the present invention. The depth map resampling system 1A shown in FIG. 7 includes a first resampling device 10 and a second resampling device 20A. The storage / transmission path connecting the first resampling apparatus 10 and the second resampling apparatus 20A is denoted as the storage / transmission path 30A and distinguished from the first embodiment. In the second embodiment, encoding distortion is equivalently added to the thinned depth map in the storage / transmission path 30A.

The second re-sampling apparatus 20A includes an edge determination unit 25 that determines the presence or absence of an edge of the subject, and a low-pass that generates a new depth value when there is no edge before the processing by the depth value determination unit 21. And a filter 26. In the second resampling apparatus 20A, the same components as those of the second resampling apparatus 20 according to the first embodiment are denoted by the same reference numerals, and description thereof is omitted.
The second resampling apparatus 20A stores the first threshold value, the first low-pass filter, and the window having the predetermined size in a storage unit (not shown), and further, a predetermined second threshold value. And a predetermined second low-pass filter.

  The edge discriminator (edge discriminating means) 25 includes two neighboring pixels separated by a predetermined number of pixels in the depth map (thinned second depth map) input to the second resampling apparatus 20A. It is determined whether or not the difference in pixel depth value is equal to or greater than a predetermined second threshold value. Here, the second threshold value is determined in advance in order to define whether or not the edge of the subject exists between two neighboring pixels in the second depth map. That is, when the depth value difference is equal to or larger than the second threshold, the subject edge exists between the two pixels, and when the depth value difference is smaller than the second threshold value, the subject edge is between the two pixels. Shall not exist. The magnitude relationship between the second threshold and the first threshold is not particularly limited. In the following description, the first threshold value is 5 and the second threshold value is 10 as an example.

The edge determination unit 25 determines whether or not there is an edge of the subject in order to determine whether or not to apply an encoding distortion removal filter by the low-pass filter 26 in the previous stage of the processing by the depth value determination unit 21. is there. Therefore, the difference between the depth values of two neighboring pixels separated by a predetermined number of pixels may be any of the following (A1), (A2), or (A3), for example.
(A1) Neighboring pixels (A2) When N is a natural number of 3 or more, the difference between the minimum depth value and the maximum depth value among N pixels of the N tap filter (A3) is a natural number of 3 or more Difference in depth value of pixels at both ends among N pixels of N tap filter

  Hereinafter, as an example, the description will be made assuming that the depth value is the difference between the depth value of the leftmost pixel and the depth value of the rightmost pixel among the 7 pixels of the 7-tap filter.

  When the edge determination unit 25 determines that the difference in depth value is equal to or greater than the second threshold (when it is determined that an edge of the subject exists), the edge determination unit 25 passes the process to the depth value determination unit 21, and the difference in depth value is the second. When it is determined that it is smaller than the threshold value (when it is determined that there is no subject edge), the process is passed to the low-pass filter 26.

  The low-pass filter (low-pass filter means) 26, when the difference between the depth values of the two neighboring pixels is smaller than the second threshold in the second depth map, the depth of the pixels having the predetermined number of neighboring pixels. A value obtained by passing the value through a predetermined low-pass filter (second low-pass filter) is set as a new depth value for a central pixel among neighboring pixels having a predetermined number of pixels. The low-pass filter (low-pass filter means) 26 generates a new depth value, and then passes the processing to the depth value determiner 21. The low-pass filter (low-pass filter means) 26 does not perform processing when the difference between the depth values of two neighboring pixels is greater than or equal to the second threshold in the second depth map.

  The low-pass filter 26 is, for example, a filter with tap number k = 7 having tap coefficient values as shown in FIG.

  This filter multiplies the depth values of the first pixel and the seventh pixel by a factor of 1/64, and the depth values of the second pixel and the sixth pixel, respectively, by a factor of 6 for seven consecutive pixels in the depth map. / 64, the depth value of the third pixel and the fifth pixel is multiplied by the coefficient 15/64, the depth value of the fourth pixel is multiplied by the coefficient 20/64, and all are added to This is the pixel depth value.

[Example of coding distortion removal processing]
Here, an example of coding distortion removal processing will be described with reference to FIG. 8 (see FIGS. 2 and 7 as appropriate). FIGS. 8A to 8C show an example of state transition caused by coding distortion of the depth map and its removal. FIG. 8A shows an example of the depth map thinned out by the first re-sampling apparatus 10 and an original depth map without coding distortion. In this example, for example, the floor is one subject, and the depth value smoothly changes from bottom to top.

  Here, in the storage / transmission path 30A, the encoding unit 31 shown in FIG. 2 encodes a plurality of pixels such as 2 × 2 pixels and 8 × 8 pixels in units of blocks in the MPEG-4 AVC method, for example. It is assumed that the data is compressed into one average value and is upsampled by the decoding unit 32. When encoding distortion is added to the depth map in the accumulation / transmission path 30A in this way, as shown in FIG. 8 (b), the depth map in which the depth value slightly changes in a step shape is the second depth. The map is given to the second resampling apparatus 20A. The difference between the original depth value and the stepwise depth value is the coding distortion.

  If a 3D image is composed of multi-view images synthesized using a depth value (depth map in FIG. 8B) that has received such coding distortion, the depth position of the 3D image is not correct. The problem of appearing distorted. Although the distortion of the depth value is slight, the feeling of depth, such as the floor that should be smoothly continuous from the front to the back, is felt as if it changes in a staircase shape, and the sense of incongruity increases.

  On the other hand, since the second resampling apparatus 20A includes the low-pass filter 26, the encoding distortion can be removed. By applying the filter shown in FIG. 8D to a depth map subjected to block distortion (see FIG. 8B), a smooth depth map from which block distortion has been removed is obtained.

  Here, it is assumed that the second resampling apparatus 20A does not include the edge determination unit 25. In this case, it is conceivable that the low-pass filter 26 causes the depth value changing portion originally present in the subject boundary portion to generate new distortion. However, since the second resampling apparatus 20A includes the edge determination unit 25, such a situation can be prevented in advance. That is, the edge determination unit 25 detects the amount of change in the depth value of neighboring pixels, and determines that the portion where the depth value changes larger than the second threshold (for example, 10) is the boundary portion of the original subject. The low-pass filter 26 can be bypassed.

  In the present embodiment, in order to prevent the pixels at the boundary portion of the subject from being affected by the low-pass filter 26, the depth between the first pixel position and the seventh pixel position is determined as the difference between the neighboring depth values. The presence or absence of an edge is determined based on the difference in values. In this way, the depth value can be held at the boundary portion of the subject, and the depth value other than the boundary portion of the subject can be changed to a smooth depth value. Therefore, as shown in FIG. 8C, an original thinned depth map from which only coding distortion has been removed is obtained.

[Resampling method of depth map]
A depth map resampling method according to the second embodiment of the present invention will be described with reference to FIG. 6 (refer to FIG. 7 as appropriate). The depth map resampling method according to the second embodiment includes steps S21 (edge determination processing) and step S22 (low-pass filter) before step S23 (depth value determination processing) in the internal processing of step S2. Is different from the first embodiment. The same processes as those in the first embodiment are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The depth map (first depth map) created in step S1 (first resampling process) is stored and transmitted in step S12, and is input to the second resampling device 20A. The second resampling apparatus 20A proceeds to step S21.

  In step S21 (edge determination processing), the second resampling apparatus 20A determines that the difference in the depth value of neighboring pixels is a threshold value (second threshold value) in the second depth map thinned out by the edge determination unit 25. It is determined whether or not this is the case. When the difference between the depth values of the neighboring pixels is smaller than the second threshold value, the second resampling apparatus 20A proceeds to step S22, and in other cases, skips step S22 and proceeds to step S23.

  In step S22 (low-pass filter processing), the second re-sampling apparatus 20A uses the low-pass filter 26 to pass the second depth map through the low-pass filter to obtain a new depth value. And Subsequently, the second resampler 20A proceeds to step S23.

  According to the second embodiment, the same effects as those of the first embodiment can be obtained, and even if encoding distortion is added during transmission and accumulation of the depth map with pixels thinned out, the influence of the encoding distortion is removed. Thus, the depth map in which the depth value continuously changes can be restored. At that time, the original value of the depth value near the boundary of the subject originally provided in the original depth map can be maintained.

  The depth map resampling apparatus according to the present invention can be used for stereoscopic television broadcasting, stereoscopic video recorders, stereoscopic movies, educational equipment using stereoscopic video, exhibition equipment, Internet services, and the like. Furthermore, the depth map resampling device according to the present invention can be used for a free viewpoint television or a free viewpoint movie in which the viewer can freely change the viewpoint position.

1,1A Depth Map Re-sampling System 2 Stereo Image Transmission / Reception System 10 First Re-Sampling Device (Depth Map Re-sampling Device)
20, 20A Second resampling device (depth map resampling device)
21 Depth value determination device (depth value determination means)
22 Depth interpolator (depth interpolation means)
23 Depth copier (depth copying means)
24 median filter 25 edge determination device (edge determination means)
26 Low-pass filter (low-pass filter means)
30, 30A Accumulation / transmission path 31, 33 Encoding unit 32, 34 Decoding unit 40 Multi-viewpoint video synthesis device

Claims (6)

A depth map created by a process of performing pixel decimation from a first depth map, which is depth information of a multi-view video obtained by photographing a subject from multiple viewpoints, and thinned out through a predetermined accumulation and transmission path. A depth map resampling device that inputs and restores the depth values of the thinned pixels by interpolation or copying,
A depth value determination means for determining whether or not a difference between depth values of two adjacent pixels in the second depth map, which is a depth map thinned out by the input pixels, is smaller than a predetermined first threshold;
When the difference between the depth values of two adjacent pixels in the second depth map is smaller than the first threshold, a value obtained by passing the depth values of the two adjacent pixels through a predetermined first low-pass filter is obtained. A depth interpolation means for making a depth value interpolated for pixels thinned between the two adjacent pixels;
When the difference between the depth values of two adjacent pixels in the second depth map is equal to or greater than the first threshold value, a value obtained by copying the larger depth value for the two adjacent pixels is set to the adjacent value. Depth copying means for making the depth value of the pixels thinned between the two pixels
A depth value interpolated by the depth interpolation means, and the depth value is copied by the depth copying means, the depth value of the second depth map, the third depth map combined, pixel and its surroundings to be processed A median filter that applies a window of a predetermined size that identifies a pixel group of each pixel to each pixel, and replaces the depth value of each pixel with the center value of all the depth values in the window. Resampler for depth map. Before the processing by the depth value determination means, further comprises an edge determination means for determining the presence or absence of an edge of the subject, and a low-pass filter means for generating a new depth value when there is no edge,
The edge determination means determines whether or not the difference between the depth values of two neighboring pixels separated by a predetermined number of pixels in the input second depth map is equal to or greater than a predetermined second threshold; If it is equal to or greater than the second threshold, it is determined that an edge of the subject exists between the two neighboring pixels,
In the second depth map, when the difference between the depth values of the two neighboring pixels is smaller than the second threshold value, the low-pass filter means calculates the depth value of the pixel having the predetermined number of pixels in the neighborhood. 2. The value obtained by passing through a predetermined second low-pass filter is set as a new depth value for a central pixel among the neighboring pixels having a predetermined number of pixels. Desampling device for depth map as described in. A first re-sampling device that inputs a first depth map, which is depth information of a multi-view video obtained by photographing a subject from multiple viewpoints, and creates a depth map with thinned pixels; and From the second depth map obtained by thinning out the pixels, which is input from the resampling device via a predetermined accumulation and transmission path, a second re-sampling process that restores the depth values of the thinned pixels by interpolation or copying. A depth map resampling system comprising a sampling device,
The first resampling device includes:
Creating a depth map thinned out of the pixels by a process of thinning out pixels without passing through a low-pass filter from the first depth map;
The second resampling device is:
A depth value determination means for determining whether or not a difference between depth values of two adjacent pixels in the second depth map, which is a depth map thinned out by the input pixels, is smaller than a predetermined first threshold;
When the difference between the depth values of two adjacent pixels in the second depth map is smaller than the first threshold, a value obtained by passing the depth values of the two adjacent pixels through a predetermined first low-pass filter is obtained. A depth interpolation means for making a depth value interpolated for pixels thinned between the two adjacent pixels;
When the difference between the depth values of two adjacent pixels in the second depth map is equal to or greater than the first threshold value, a value obtained by copying the larger depth value for the two adjacent pixels is set to the adjacent value. Depth copying means for making the depth value of the pixels thinned between the two pixels
A depth value interpolated by the depth interpolation means, and the depth value is copied by the depth copying means, the depth value of the second depth map, the third depth map combined, pixel and its surroundings to be processed A median filter that applies a window of a predetermined size that identifies a pixel group of each pixel to each pixel, and replaces the depth value of each pixel with the center value of all the depth values in the window. Depth map resampling system. A depth map created by a process of performing pixel decimation from a first depth map, which is depth information of a multi-view video obtained by photographing a subject from multiple viewpoints, and thinned out through a predetermined accumulation and transmission path. Re-sampling method by a re-sampling device of a depth map that inputs and restores the depth values of the thinned pixels by interpolation or copying,
The re-sampling device stores at least a predetermined first threshold, a predetermined first low-pass filter, and a window having a predetermined size for specifying a pixel to be processed and a surrounding pixel group. And processing means,
The processing means includes
A depth value determination processing step for determining whether or not a difference between depth values of two adjacent pixels in the second depth map thinned out by the input is smaller than the first threshold;
When the difference between the depth values of two adjacent pixels in the second depth map is smaller than the first threshold value, a value obtained by passing the depth values of the two adjacent pixels through the first low-pass filter, Depth interpolation processing step to be a depth value interpolated for pixels thinned between the two adjacent pixels;
When the difference between the depth values of two adjacent pixels in the second depth map is equal to or greater than the first threshold value, a value obtained by copying the larger depth value for the two adjacent pixels is set to the adjacent value. A depth copy processing step for setting a pixel depth value between two pixels to be
A depth value interpolated by the depth interpolation processing step, a depth value that is copied by the depth copying process step, the depth value of the second depth map, the third depth map combined, the predetermined size And a median filter processing step of applying a window of each pixel to each pixel and replacing the depth value of each pixel with the center value of all the depth values in the window. . The storage means of the resampler further stores a predetermined second threshold value and a predetermined second low-pass filter,
The processing means includes
Before the depth value determination processing step,
In the input second depth map, it is determined whether or not the difference between the depth values of two neighboring pixels separated by a predetermined number of pixels is equal to or greater than the second threshold, and is greater than or equal to the second threshold. , Performing an edge determination processing step for determining that an edge of the subject exists between the two neighboring pixels,
In the edge determination processing step, when it is determined that the edge of the subject does not exist because the difference in depth value is smaller than the second threshold value, a depth value of a predetermined number of pixels in the neighborhood is determined in advance. And further executing a low-pass filter processing step in which a value obtained by passing through the second low-pass filter is a new depth value for a central pixel of the neighboring pixels having the predetermined number of pixels. And
5. The depth value determination processing step is executed according to claim 4, wherein when the edge determination processing step determines that an edge of the subject exists, the depth value determination processing step is executed by skipping the low-pass filter processing step. How to resample the depth map. A depth map created by a process of performing pixel decimation from a first depth map, which is depth information of a multi-view video obtained by photographing a subject from multiple viewpoints, and thinned out through a predetermined accumulation and transmission path. In order to restore the depth value of the thinned pixel by interpolation or copying,
A depth value determining means for determining whether or not a difference between depth values of two adjacent pixels in the second depth map thinned out from the input pixels is smaller than a predetermined first threshold;
When the difference between the depth values of two adjacent pixels in the second depth map is smaller than the first threshold, a value obtained by passing the depth values of the two adjacent pixels through a predetermined first low-pass filter is obtained. , A depth interpolation means for making a depth value interpolated for the pixels thinned out between the two adjacent pixels,
When the difference between the depth values of two adjacent pixels in the second depth map is equal to or greater than the first threshold value, a value obtained by copying the larger depth value for the two adjacent pixels is set to the adjacent value. Depth copying means for making a depth value of a pixel thinned between two pixels
A depth value interpolated by the depth interpolation means, and the depth value is copied by the depth copying means, the depth value of the second depth map, the third depth map combined, pixel and its surroundings to be processed A depth map for functioning as a median filter that applies a window of a predetermined size that identifies a pixel group of each pixel to each pixel and replaces the depth value of each pixel with the center value of all the depth values in the window. Resampling program.