JP2022522504A - Image depth map processing - Google Patents

Abstract

Methods of processing depth maps include receiving images and corresponding depth maps (301). The depth value of the first depth map of the corresponding depth map is updated based on the depth value of at least the second depth map of the corresponding depth map (303). Updates are based on weighted joins of candidate depth values determined from other maps. The weight for the candidate depth value from the second depth map projects the pixel value in the first image corresponding to the updated depth and the position of the updated depth value using the candidate depth value onto the third image. It is determined based on the similarity with the pixel value in the third image at the position determined by. In this way, a more consistent depth map can be generated.

Description

The present invention relates to, but is not limited to, processing depth maps for images, and in particular, processing depth maps that support view compositing for virtual reality applications.

In recent years, the variety and scope of image and video applications has increased significantly, and new services and methods of utilizing and consuming video are continually being developed and introduced.

For example, one growing service is to provide image sequences in a way that allows the observer to actively interact with the system to change rendering parameters. A very attractive feature in many applications is to change the effective viewing position and orientation of the observer, for example, allowing the observer to move around and "look around" within the presented scene. Ability.

Such features can, in particular, allow a virtual reality experience to be provided to the user. This allows the user, for example, to move around in the virtual environment (relatively) freely and dynamically change their position and where they are looking. Typically, such virtual reality applications are based on a 3D model of the scene, which is dynamically evaluated to provide a particular requested view. This approach is well known from gaming applications such as in the category of first-person shooters for computers and consoles.

Further, especially in a virtual reality application, it is desirable that the presented image is a three-dimensional image. In fact, in order to optimize the observer's immersive feeling, it is preferable for the user to experience the presented scene as a three-dimensional scene. In fact, the virtual reality experience should preferably allow the user to select their location, camera perspective, and moment of time with respect to the virtual world.

Many virtual reality applications are based on a given model of the scene, typically an artificial model of the virtual world. In many cases, it is desirable that virtual reality experiences be provided based on real-world captures.

Many systems, especially those based on real-world scenes, provide an image representation of the scene, which includes the image and depth for one or more capture points / viewpoints in the scene. The image + depth representation provides a very efficient characterization of the real world scene in particular, and the characterization is not only relatively easy to generate by capturing the real world scene, but also other than what was captured. It is also very suitable for renderers that synthesize viewpoint views. For example, the renderer can be configured to dynamically generate a view that matches the current local observer pose. For example, the observer pose can be dynamically determined and the view can be dynamically generated based on the image and, for example, the provided depth map to match this observer pose.

In many practical systems, a calibrated multi-view camera rig can be used to enable playback for users who have different perspectives on the captured scene. The application involves selecting individual perspectives during a sports match, ie playing a 3D scene captured on an enhanced or virtual reality headset.

"You Yang ET AL., Cross-View Multi-Lateral Filter for Compressed MultiView Depth Video", IEEE TRANSACTIONS ON IMAGE PROCESSING., Vol. 28, no. 1, 1 January 2019 (2019-01-01), pages 302- 315, XP055614403, US ISSN: 1057-7149, DOI: 10.1109 / TIP.2018.2867740 "is a view to improve the quality of compressed depth maps / videos within the framework of asymmetric multiview video with depth compression. It discloses an inter-multilateral filtering scheme, which enhances the distorted depth map through non-local candidates selected from the current and adjacent viewpoints in different time slots. , These candidates are clustered into macro superpixels that show the physical and semantic interrelationships between views, spatial and temporal preah.

"WOLFF KATJA ET AL., Point Cloud Noise and Outlier Removal for ImageBased 3D Reconstruction, 2016 FOURTH INTERNATIONAL CONFERENCE ON 3D VISION (3DV), IEEE, 25 October 2016 (2016-10-25), pages 118-127, XP033027617, DOI : 10.1109 / 3DV.2016.20 "discloses an algorithm that uses the input image and the corresponding depth map to remove pixels that are geometrically or photographicly inconsistent with the colored surface implied by the input. This allows standard surface reconstruction methods to perform less smoothing and thus achieve higher quality.

Depth maps are often provided to predict views from these other perspectives to provide a smooth transition between discretely captured viewpoints and some extrapolations beyond the captured viewpoints. Used for compositing.

Depth maps are typically generated using (multi-view) stereo matching between captured cameras, or more directly by using depth sensors (structured light or time-based). Generated. However, such depth maps obtained from depth sensors or parallax estimation processes. In essence, it has errors and inaccuracies that can lead to errors in the combined view. This reduces the observer's experience.

Therefore, an improved approach for generating and processing depth maps would be advantageous. In particular, improved behavior, increased flexibility, improved virtual reality experience, reduced complexity, easier implementation, improved depth maps, increased composite image quality, improved rendering, Systems and / or approaches that enable an improved user experience and / or improved performance and / or operation are advantageous.

Therefore, the present invention preferably attempts to reduce, reduce or eliminate one or more of the above drawbacks alone or in any combination.

According to one aspect of the invention, a method of processing a depth map is provided, the step of receiving multiple images representing scenes from different view poses and the corresponding depth map, and the corresponding depth map. A step of updating the depth value of our first depth map based on at least the depth value of the second depth map of the corresponding depth maps, wherein the first depth map is for the first image. The second depth map has a step, which is for a second image, and the updating step is the first depth pixel of the first depth map at the first depth map position in the first depth map. The first candidate depth value is at least one of the second depth pixels of the second depth map at the second depth map position in the second depth map. 2 A step that determines the first depth value of the first depth pixel by a weighted combination of a step determined according to the depth value and a plurality of depth candidate values for the first depth map position. The weighted coupling comprises a step comprising the first candidate depth value weighted by the first weight, and the step of determining the first depth value is the first image with respect to the first depth map position. The step of determining the position of the first image in the image and the step of determining the position of the third image in the third image of the plurality of images, wherein the third image position is the first candidate depth value. A step, an image pixel value in the third image with respect to the third image position, and a first image in the first image with respect to the first image position, corresponding to the projection of the first image position onto the third image. It has a step of determining a first match error index indicating a difference from an image pixel value, and a step of determining the first weight according to the first match error index.

This approach can, in many embodiments, provide an improved depth map, in particular a set of depth maps with increased consistency. This approach can enable improved view consistency when images are combined based on the image and the updated depth map.

We find that discrepancies between depth maps are often more perceptible than consistent errors or noise between depth maps, and certain approaches provide a more consistent and updated depth map. I got the insight that I can. This method can be used as a depth refinement algorithm to improve the quality of the depth map for a set of multi-view images of the scene.

This approach can be facilitated in many embodiments and can be implemented with relatively low complexity and resource requirements.

Positions in the image can directly correspond to positions in the corresponding depth map and vice versa. There may be a one-to-one correspondence between the position in the image and the position in the corresponding depth map. In many embodiments, the pixel positions can be the same in the image and the corresponding depth map can contain one pixel for each pixel in the image.

In some embodiments, the weights may be binary (eg, 1 or 0) and the weighted joins may be selective.

It will be appreciated that the term projection can often refer to the projection of 3D spatial coordinates in a scene onto an image or 2D image coordinates (u, v) in a depth map. However, the projection is a mapping between the dimensional image coordinates (u, v) of a scene point from one image or depth map to another, that is, from a set of image coordinates (u ₁ , v ₁ ) in one pose to another pose. It may also refer to a mapping of image coordinates (u ₂ , v ₂ ) to a set. Such projections between image coordinates for images corresponding to different view poses / viewpoints are typically performed with the corresponding spatial scene points in mind, especially the depth of the scene points.

In some embodiments, the step of determining the first depth value is a step of projecting the first depth map position onto a third depth map position in the third depth map of the corresponding depth maps. The third depth map is for the third image, and the projection is performed based on the first candidate depth value, the step and the image pixels in the third image with respect to the third depth map position. A step of determining a first match error index that indicates the difference between the value and the image pixel value in the first image with respect to the first depth map position, and the first weight according to the first match error index. Has a step to determine.

According to the optional features of the present invention, the step of determining the first candidate depth value is based on at least one of the second value and the first original depth value of the first depth map. It has a step of determining a second depth map position with respect to the first depth map position by projection between the first view pose of one image and the second view pose of the second image.

This can provide particularly advantageous performance in many embodiments, and in particular can enable improved depth maps with improved consistency in many scenarios.

The projection may be from the second depth map position to the first depth map position, and thus from the second view pose to the first view pose, based on the first original depth value.

The projection may be from the second depth map position to the first depth map position, and thus from the second view pose to the first view pose, based on the second depth value.

The original depth value may be the unupdated depth value of the first depth map.

The original depth value may be the depth value of the first depth map received by the receiver.

In some embodiments, the step of determining the first depth value is the step of projecting the first depth map position onto the third depth map position in the third depth map of the corresponding depth maps. , The third depth map is for a third image, and the projection is based on the first candidate depth value, with a step and an image pixel value in the third image with respect to the third depth map position. , A step of determining a first match error index indicating the difference between the first depth map position and the image pixel value in the first image, and determining the first weight according to the first match error index. And have steps to do.

According to the optional features of the present invention, the weighted coupling includes candidate depth values determined from the area of the second depth map determined according to the first depth map position.

This can provide increased depth map consistency in many embodiments. The first candidate depth value can be derived from one or more depth values in this area.

According to the optional features of the present invention, the area of the second depth map is determined as the area around the second depth map position, and the second depth map position is the first depth map position in the first depth map. Determined as the depth map position in the same second depth map.

This allows for efficient determination of appropriate depth values to consider due to low complexity and low resources.

According to the optional features of the present invention, the area of the second depth map is determined by projection from the first depth map position based on the original depth value in the first depth map at the first depth map position. Determined as the area around a position in the depth map.

This can provide increased depth map consistency in many embodiments. The original depth value may be the depth value of the first depth map received by the receiver.

According to the optional features of the present invention, the method further comprises an image pixel value in the second image for the second depth map position and an image pixel value in the first image for the first depth map position. It has a step of determining a second match error index indicating the difference between the two, and the determination of the first weight is further made according to the second match error index.

This can provide an improved depth map in many embodiments.

According to any feature of the invention, the method further comprises image pixel values in other images for the depth map position corresponding to the first depth map position and in the first image for the first depth map position. It has a step of determining an additional match error index indicating the difference between the image pixel value and the first weight, and the determination of the first weight is further performed according to the additional match error index.

This can provide an improved depth map in many embodiments.

According to the optional features of the present invention, the weighted coupling comprises the depth value of the first depth map in the area around the first depth map position.

This can provide an improved depth map in many embodiments.

According to the features of the options of the present invention, the first weight depends on the confidence value of the first candidate depth value.

This can provide an improved depth map in many scenarios.

According to the optional feature of the present invention, only the depth value of the first depth map whose confidence value is less than the threshold value is updated.

This can provide an improved depth map in many scenarios, and in particular can reduce the risk of accurate depth values being updated by less accurate depth values.

According to any feature of the invention, the method further provides a set of depth values for the second depth map to include in the weighted coupling, a confidence value for which the depth value of the set of depth values is greater than or equal to a threshold. Have the steps to choose according to the requirement that you must have.

This can provide an improved depth map in many scenarios.

According to any feature of the invention, the method further comprises projecting a given depth map position for a given depth value in a given depth map onto the corresponding positions in a plurality of corresponding depth maps. , A step of determining a variability scale for a set of depth values having the given depth value and the depth value of the corresponding position in the plurality of corresponding depth maps, and the given variability scale according to the variation scale. It has a step of determining a confidence value for a depth map position.

This can provide a particularly favorable determination of confidence values that can result in an improved depth map.

According to the optional features of the present invention, the method further addresses a given depth map position for a given depth value in a given depth map in another depth map based on the given depth value. The step to project to the position and the corresponding position in the other depth map to the test position in the given depth map based on the depth value at the corresponding position in the other depth map. It has a step of projecting and a step of determining a confidence value for the given depth map position according to the distance between the given depth map position and the test position.

This can provide a particularly favorable determination of confidence values that can result in an improved depth map.

According to one aspect of the invention, a device for processing a depth map is provided, the device comprising a receiver for receiving a plurality of images representing scenes from different view poses and a corresponding depth map. An updater for performing a step of updating the depth value of the first depth map of the corresponding depth map based on at least the depth value of the second depth map of the corresponding depth map. The first depth map is for the first image, the second depth map is for the second image, has an updater, and the updating step is the first depth map in the first depth map. It is a step of determining the first candidate depth value for the first depth pixel of the first depth map at the position, and the first candidate depth value is the second at the second depth map position in the second depth map. A weighted combination of a step and a plurality of candidate depth values for the first depth map position, which is determined according to at least one second depth value of the second depth pixel of the depth map, of the first depth pixel. A step of determining a first depth value, wherein the weighted combination has a step including the first candidate depth value weighted by the first weight, and the step of determining the first depth value is The first step is to determine the position of the first image in the first image with respect to the position of the first depth map, and the step is to determine the position of the third image in the third image among the plurality of images. The three image positions include a step corresponding to the projection of the first image position onto the third image based on the first candidate depth value, and an image pixel value in the third image with respect to the third image position. A step of determining a first match error index indicating a difference between the first image position and an image pixel value in the first image, and a step of determining the first weight according to the first match error index. And have.

These and other aspects, features and advantages of the present invention will be apparent from and described with reference to the embodiments described below.

Embodiments of the present invention will be described with reference to the drawings as merely examples.
The figure which shows the example of the configuration for providing a virtual reality experience. The figure which shows an example of the element of the apparatus for processing a depth map according to some embodiments of this invention. The figure which shows the example of the element of the method of processing a depth map by some embodiments of this invention. The figure which shows an example of the camera composition for capturing a scene. The figure which shows the example of the element of the method of updating the depth map by some embodiments of this invention. The figure which shows the example of the element of the method of determining the weight by some embodiments of this invention. The figure which shows the example of the processing of a depth map and an image by some embodiments of this invention.

Although the following description focuses on embodiments of the invention applicable to virtual reality experiences, the invention is not limited to this application and many other systems and applications, such as specific applications including view compositing. Please understand that it can be applied to the system.

Virtual experiences that allow users to move around in the virtual world are becoming more and more popular, and services are being developed to meet such demands. However, providing efficient virtual reality services is very difficult, especially if the experience is based on a capture of the real world environment rather than a completely virtually virtualized artificial world.

In many virtual reality applications, the observer pose input is determined to reflect the pose of the virtual observer in the scene. The virtual reality device / system / application then generates one or more images corresponding to the view and viewport of the scene for the observer corresponding to the observer pose.

Typically, a virtual reality application produces 3D output in the form of separate view images for the left and right eyes. These can then be presented to the user, typically by appropriate means such as the individual left-eye and right-eye displays of the VR headset. In other embodiments, the image may be presented, for example, on an automatic stereoscopic display (in this case, a larger number of view images may be generated for the observer pose), or, in fact, some. In embodiments, only a single 2D image may be generated (eg, using a conventional 2D display).

Observer pose input may be determined differently in different applications. In many embodiments, the physical movement of the user can be tracked directly. For example, a camera that surveys the user area can detect and track the user's head (or eyes). In many embodiments, the user can wear a VR headset that can be tracked by external and / or internal means. For example, the headset can include a headset, and thus an accelerometer and gyroscope that provide information about head movement and rotation. In some examples, the VR headset can transmit a signal or can be equipped with a (eg, visual) identifier that allows an external sensor to determine the movement of the VR headset.

In some systems, observer poses may be provided by manual means, for example, by the user manually controlling a joystick or similar manual input. For example, the user manually moves the virtual observer in the scene by controlling the first analog joystick with one hand and the virtual observer manually moves the second analog joystick with the other hand. You can manually control the direction you are in.

Some applications can use a combination of a manual approach and an automated approach to generate an input observer pose. For example, the headset can track the orientation of the head and the movement / position of the observer in the scene can be controlled by the user using the joystick.

Image generation is based on the proper representation of the virtual world / environment / scene. Some applications can provide a complete 3D model of the scene, and the view of the scene from a particular observer pose can be determined by evaluating this model.

In many practical systems, a scene can be represented by an image representation that includes image data. The image data may typically include one or more images associated with one or more capture poses or anchor poses, specifically images about one or more viewports. Often, each viewport corresponds to a particular pose. Image representations that include one or more images can be used, and each image represents a view in a given viewport for a given view pose. Such view poses or positions to which image data is provided are often referred to as anchor poses or positions, or capture poses or positions (image data typically has a position and orientation corresponding to the capture pose). (Because it can correspond to an image captured or will be captured by a camera placed in the scene).

The image is typically associated with depth information, specifically a depth image or map is typically provided. Such a depth map can provide a depth value for each pixel in the corresponding image, which indicates the distance from the camera / anchor / capture position to the object / scene point depicted by the pixel. .. Therefore, the pixel value may be considered to represent a ray from an object / point in the scene to the camera's capture device, and the depth value for the pixel can reflect the length of this ray.

In many embodiments, the resolution of the image and the corresponding depth map can be the same and thus can include individual depth values for each pixel in the image, i.e. the depth map is for each pixel in the image. It may contain one depth value. In other embodiments, the resolutions may vary, for example, the depth map may have a lower resolution so that one depth value can be applied to multiple image pixels. The following description focuses on embodiments where the resolution of the image and the corresponding depth map are the same and therefore there is a separate depth map pixel (depth map pixel) for each image pixel (image pixel). Guess.

The depth value can be any value that indicates the depth to the pixel, and thus it can be any value that indicates the distance from the camera position to the object in the scene depicted by a given pixel. The depth value may be, for example, a parallax value, a z coordinate, a distance measure, or the like.

Many typical VR applications can proceed to provide a viewport corresponding to the viewport for the current observer pose based on such an image + depth representation, and the image is a viewer. Dynamically updated to reflect changes in poses and (in some cases) generated based on image data representing virtual scenes / environments / worlds. Applications can do this by performing view composition and view shift algorithms, as is known to those of skill in the art.

In this field, the terms placement and pose are used as general terms for position and / or orientation / orientation. For example, a combination of object, camera, head or view position and orientation / orientation may be referred to as a pose or alignment. Therefore, the placement or pose index can contain 6 values / components / degrees of freedom, where each value / component typically describes the individual characteristics of the position / location or orientation / orientation of the corresponding object. do. Of course, in many situations, for example, if one or more components are considered fixed or irrelevant (eg, if all objects are at the same height and are considered to have a horizontal orientation, then the four components are considered. A complete representation of the pose of an object can be provided), placement or pose may be considered or represented with less component. In the following, the term pose is used to refer to a position and / or orientation that can be represented by one to six values (corresponding to the maximum degree of freedom possible).

Many VR applications are based on maximum degrees of freedom, that is, poses with three degrees of freedom each for position and orientation, resulting in a total of six degrees of freedom. Thus, a pose can be represented by a set or vector of six values representing six degrees of freedom, and thus the pose vector can give a three-dimensional position and / or a three-dimensional directional representation. However, it will be appreciated that in other embodiments, the pose may be represented by a smaller value.

The pose can be at least one of orientation and position. The pose value can indicate at least one of the orientation value and the position value.

A system or entity based on providing the observer with maximum degrees of freedom is commonly referred to as having 6 degrees of freedom (6DoF). Many systems and entities provide only directions or positions, which are typically known to have 3 degrees of freedom (3DoF).

Depending on the system, the VR application may be provided locally to the observer, for example, by a stand-alone device that does not use any remote VR data or processing and has no access. For example, a device such as a game console comprises a storage device for storing scene data, an input for receiving / generating an observer pose, and a processor for generating a corresponding image from the scene data. Can be done.

In other systems, VR applications can be implemented and run remotely from the observer. For example, a user-local device can detect / receive motion / pause data sent to a remote device that processes the data to generate an observer pose. The remote device can then generate an appropriate view image for the observer pose based on the scene data that describes the scene. The view images are then sent to a device local to the observer to whom they are presented. For example, a remote device can directly generate a video stream (typically a stereo / 3D video stream) presented directly by the local device. Therefore, in such an example, the local device may not perform any VR processing except to send the travel data and present the received video data.

In many systems, functionality can be distributed across local and remote devices. For example, the local device can process the received input and sensor data to generate an observer pose that is continuously transmitted to the remote VR device. The remote VR device can then generate the corresponding view images and send them to the local device for presentation. In other systems, the remote VR device does not have to generate the view image directly, but the relevant scene data may be selected and sent to the local device, and the local device presents the view image. May be generated. For example, a remote VR device can identify the closest capture point, extract the corresponding scene data (eg, spherical image and depth data from the capture point) and send it to the local device. The local device can then process the received scene data to generate an image for a particular current view pose. A view pose typically corresponds to a head pose, and a reference to a view pose can be considered equivalent to a reference to a head pose.

For many applications, especially broadcast services, the source may send scene data in the form of an image (including video) representation of the scene that does not depend on the observer pose. For example, an image representation for a single view sphere for a single capture position can be sent to multiple clients. Individual clients can then locally compose the view image that corresponds to the current observer pose.

Applications of particular interest are presented to support a limited amount of movement and follow small movements and rotations corresponding to a virtually static observer who makes only small movements and rotations of the head. The view to be updated is updated. For example, a sitting observer can turn his head and move it slightly, and the views / images presented are adapted to follow these pose changes. Such an approach can provide a very immersive, eg video experience. For example, an observer watching a sporting event may feel that he or she is in a particular spot in the arena.

Applications with such limited degrees of freedom have the advantage of providing an improved experience without the need for accurate representation of scenes from many different locations, thereby significantly reducing capture requirements. .. Similarly, the amount of data that needs to be provided to the renderer can be significantly reduced. In fact, in many scenarios, only images for a single point of view and typically depth data need to be provided to a local renderer that can now generate the desired view.

This approach is particularly suitable for applications where data needs to be communicated from source to destination over a bandwidth-limited communication channel, such as broadcast or client-server applications.

FIG. 1 shows such an example of a VR system in which the remote VR client device 101 cooperates with the VR server 103 via a network 105 such as the Internet. The server 103 may be configured to support potentially a large number of client devices 101 at the same time.

The VR server 103 supports a broadcast experience, for example, by transmitting an image signal having an image representation in the form of image data that can be used by a client device to locally synthesize a view image corresponding to the appropriate pose. obtain.

FIG. 2 shows exemplary elements of an exemplary implementation of a device for processing a depth map. Specifically, this device may be mounted on the VR server 103, and will be described with reference to this. FIG. 3 shows a flow chart for how to process the depth map performed by the device of FIG.

The device / VR server 103 includes a receiver 201 that performs step 301 in which a plurality of images representing scenes from different view poses and corresponding depth maps are received.

The image contains light intensity information, and the pixel value of the image reflects the light intensity value. In some examples, the pixel value may be a single value, such as the brightness of a grayscale image, but in many embodiments, the pixel value is, for example, a color channel value for a color image ( Sub) may be a set or vector of values (eg RGB or Yuv values may be provided).

The depth map of the image can contain the depth values of the same viewport. For example, for each pixel of an image for a given view / capture / anchor pose, the corresponding depth map contains pixels with depth values. Thus, the same location in the image and its corresponding depth map provides the light intensity and depth of the rays corresponding to the pixels, respectively. In some embodiments, the depth map may have a lower resolution, for example, one depth map pixel may correspond to multiple image pixels. However, in such cases, there may still be a direct paired position correspondence between the position in the depth map and the position in the depth map (including the subpixel position).

For brevity and complexity, the discussion below focuses on an example where only three images and the corresponding depth map are provided. Further, it is assumed that these images are provided by a linear arrangement of cameras that capture scenes from three different viewpoint positions and have the same orientation as shown in FIG.

It will be appreciated that in many embodiments, a significant number of images are often received and the scene is often captured from a significant number of capture poses.

The receiver is simply fed to a depth map updater called updater 203 for brevity below. Updater 203 performs step 303, where one or more (and typically all) of the received depth maps are updated. The update involves updating the depth value of the first received depth map based on at least the depth value of the second received depth map. Therefore, updates between depth maps and between view poses are performed to generate an improved depth map.

In this example, the updater 203 is coupled to an image signal generator 205 that performs step 305, which step generates an image signal containing the image received with the updated depth map. The image signal can then be transmitted, for example, to the VR client device 101, where it can be used as the basis for synthesizing the view image for the current observer pose.

In this example, the depth map update is performed in the VR server 103, and the updated depth map is delivered to the VR client device 101. However, in other embodiments, the depth map update may be performed, for example, in the VR client device 101. For example, the receiver 201 may be part of the VR client device 101 and receives an image and a corresponding depth map from the VR server 103. The received depth map may then be updated by the updater 203, and instead of the image signal generator 205, the device is configured to generate a new view based on the image and the updated depth map. It may be equipped with a renderer or a view image synthesizer.

In yet another embodiment, all processing can be performed on a single device. For example, the same device can receive the captured information directly and generate an initial depth map, for example by parallax estimation. The resulting depth map can be updated and the device synthesizer can dynamically generate new views.

Therefore, the location of the described features and the specific use of the updated depth map will depend on the preferences and requirements of the individual embodiments.

Therefore, the depth map update is based on one or more of the other depth maps that represent the depth for different images from different spatial locations. For depth maps, this approach is important not only for the absolute accuracy or reliability of individual depth values or depth maps, but also for the consistency between different depth maps, which is important for the resulting perceptual quality. Use the perception that there is.

In fact, heuristic insights are that when errors or inaccuracies are inconsistent across depth maps, that is, when they change across the source view, they are virtual scenes when the viewer repositions them. Is perceived to be particularly harmful because it is perceived as vibrating.

Such view consistency is not always fully adhered to during the depth map estimation process. For example, this is the case when a separate depth sensor is used to obtain a depth map for each view. In that case, the depth data is captured completely independently. In other extreme cases where all views are used to estimate depth (eg using a plane sweep algorithm), the result depends on the particular multiview parallax algorithm used and its parameter settings. , Still may be inconsistent. The specific approach described below, in many scenarios, mitigates such problems, improves consistency between depth maps, and thus updates depth maps to improve the perceived image quality. Can be done. This approach can improve the quality of the depth map for a set of multi-view images of the scene.

Figure 5 shows a flow chart for updates performed on a pixel in a depth map. This approach may be repeated for some or all of the depth map pixels to generate an updated first depth map. This process can then be repeated for other depth maps.

The update of the pixels in the depth map, which will be referred to below as the first depth map, which will be referred to below as the first depth pixel, will start in step 501 and the first candidate depth value will be determined for the first depth pixel. The position of the first depth pixel in the first depth map is called the first depth map position. Corresponding terms are used in other figures, changing only the numeric labels.

The first candidate depth value is determined according to at least one second depth value, which is the depth value of the second depth pixel at the second depth map position in the second depth map. Therefore, the first candidate depth value is determined from one or more depth values in another depth map. The first candidate depth value can be an estimate of the correct depth value for the first depth pixel, in particular, based on the information contained in the second depth map.

In step 503 following step 501, the updated first depth value is determined for the first depth pixel by a weighted combination of a plurality of candidate depth values for the first depth map position. The first candidate depth value determined in step 503 is included in the weighted join.

Therefore, in step 501, one of a plurality of candidate depth values for subsequent joins is determined. In most embodiments, the plurality of candidate depth values describe the first candidate depth value in step 501 with respect to the other depth values in the second depth map and / or the depth values in the other depth map. It can be determined by repeating the process.

In many embodiments, one or more candidate depth values can be determined in other ways or from other sources. In many embodiments, one or more of the candidate depth values can be depth values from the first depth map, such as depth values in the vicinity of the first depth pixel. In many embodiments, the original first depth value, i.e., the depth value for the first depth pixel in the first depth map received by receiver 201, may be included as one of the candidate depth values.

Therefore, the updater 205 can perform a weighted combination of candidate depth values including at least one candidate depth value determined as described above. The number, characteristics, origin, etc. of any other candidate depth value will depend on the preferences and requirements of the individual embodiments, as well as the desired actual depth update operation.

For example, in some embodiments, the weighted join can include only the first candidate depth value and the original depth value determined in step 501. In such cases, only a single weight for the first candidate depth value may be determined, for example, and the weight for the original depth value may be constant.

As another example, in some embodiments, the weighted join is a value determined from another depth map and / or position, the original depth value, the depth value in the neighborhood in the first depth map, or actually. Can be a combination of a large number of candidate depth values, including those based on depth values in alternative depth maps, such as depth maps that use different depth estimation algorithms. In such a more complex embodiment, the weights may be determined, for example, for each candidate depth value.

Use any suitable form of weighted coupling, including non-linear coupling or selective coupling (one candidate depth value is weighted 1 and all other candidate depth values are weighted 0). It will be understood that it can be done. However, in many embodiments, linear combinations, specifically weighted averages, can be used.

は、ステップ501について説明されたように少なくとも1つが生成される候補奥行き値ｚ_i

のセットの加重平均であってもよい。この場合、重み付け結合は、以下のように与えられるフィルタ関数に対応し得る。

ここで、

はビューｋのピクセル位置（u, v）における更新された奥行き値であり、z_iはi番目の入力候補奥行き値であり、w_iは i番目の入力候補奥行き値の重みである。 Therefore, as a specific example, the updated depth value for the image coordinates (u, v) in the depth map / view k.

Is a candidate depth value z _i for which at least one is generated as described for step 501.

It may be a weighted average of a set of. In this case, the weighted join may correspond to the filter function given as follows.

here,

Is the updated depth value at the pixel position (u, v) of the view k, z _i is the i-th input candidate depth value, and w _i is the weight of the i-th input candidate depth value.

This method uses a specific approach for determining the weight of the first candidate depth value, i.e. the first weight. This approach is illustrated with reference to the flowchart of FIG. 6 and the image and depth map examples of FIG.

Figure 7 shows an example where three images and three corresponding depth maps are provided / considered. The first image 701 is provided with the first depth map 703. Similarly, the second image 705 is provided with the second depth map 707 and the third image 709 is provided with the third depth map 711. The following description focuses on determining the first weight for the first depth value of the first depth map 703, based on the depth values from the second depth map 707, and also considers the third image 709.

Therefore, the determination of the first weight (for the first candidate depth value) is based on one or more second depth values for the second depth pixel at the second depth map position in the second depth map 707. Determined for pixel / first depth map position. Specifically, the first candidate depth value can be determined as the second depth value at the corresponding position in the second depth map 707, as indicated by arrow 713 in FIG.

The determination of the first weight begins in step 601 where the renewer determines the first image position in the first image 701 corresponding to the first depth map position, as indicated by arrow 715. Typically, this may simply be the same position and image coordinates. The pixels in the first image 701 corresponding to this first image position are called the first image pixels.

Then, the updater 203 proceeds to step 603 to determine the third image position in the third image 709 of the plurality of images, where the third image position is based on the first candidate depth value. Corresponds to the projection of one image position onto the third image. The third image position can be determined by direct projection from the image coordinates of the first image 701, as indicated by arrow 717.

Therefore, the updater 203 proceeds to project the first image position to the third image position in the third image 709. The projection is based on the first candidate depth value. Therefore, the projection of the first image position onto the third image 709 is based on a depth value that can be considered an estimate of the first depth value determined based on the second depth map 707.

In some embodiments, the determination of the third image position can be based on the projection of the depth map position. For example, updater 203 can proceed to project the first depth map position (the position of the first depth pixel) to the third depth map position in the third depth map 711, as indicated by arrow 719. .. The projection is based on the first candidate depth value. Therefore, the projection of the first depth map position onto the third depth map 711 is based on a depth value that can be considered an estimate of the first depth value determined based on the second depth map 707.

The third image position can then be determined as the image position in the third image 709 corresponding to the third depth map position, as indicated by arrow 721.

It will be understood that the two approaches are equivalent.

Projection from one depth map / image to a different depth map / image is of the one depth map / depth map in the image / different depth map representing the same scene point as the image position / depth map in the image / image position. It may be a decision. Since depth maps / images represent different view / capture poses, the parallax effect results in a shift in image position with respect to a given point in the scene. This shift depends on changes in view pose and the depth of points in the scene. Projection from one image / depth map to another image / depth map can therefore also be referred to as an image / depth map position shift or determination.

As an example, the image coordinates (u, v) _l of a view (l) and its depth value z _l (u, v) are projected onto the corresponding image coordinates (u, v) _k of an adjacent view (k). This can be done, for example, in the case of a fluoroscopic camera, in the following steps:
1. 1. Image coordinates (u, v) _l are unprojected using z_ _l in 3D space (x, y, z) _l using camera-specific parameters (focal length, principal point) of camera (l). To.
2. 2. Unprojected points in the camera (l) coordinate system (x, y, z) _l are the coordinates of the camera (k) using their relative external parameters (camera rotation matrix R and translation vector t). Converted to the system (x, y, z) _k .
3. 3. Finally, the point (x, y, z) _k is projected onto the image plane of the camera (k) (using the camera eigenvalues of k) to obtain the image coordinates (u, v) _k .

Similar mechanisms can be used for other camera projection types, such as enterprise resource planning (ERP).

In the approach described, the projection based on the first candidate depth value is to the scene point of the first depth map / image position with the depth of the first candidate depth value (and between the first view pose and the third view pose). It may be considered to correspond to the determination of the third depth map / image position (for changes in the view pose of).

Different depths result in different shifts, in this case the image and depth between the first view pose for the first depth map 703 and the first image 701 and the third view pose for the third depth map 711 and the third image 709. The map position shift is based on at least one depth value in the second depth map 707.

In step 603, the updater 203 actually has the same scene point as the first image pixel in the first image 701 if the first candidate depth value is actually the correct value for the first depth value and the first image pixel. The position in the third depth map 711 and the third image 709 that will reflect the above is determined accordingly. Any deviation of the first candidate depth value from the correct value can result in the wrong position determined in the third image 709. Note that the scene point refers to a scene point on a ray associated with a pixel, but does not necessarily have to be the frontmost scene point for both view poses. For example, if the scene points seen from the first view pose are obscured by (more) foreground objects than from the second view pose, the depth map and image depth values may represent different scene points. Yes, and therefore can potentially have very different values.

Step 603 is followed by step 605 to generate a first matching error index based on the contents of the first image 701 and the third image 709 at the first and third image positions, respectively. Specifically, the image pixel value in the third image at the third image position is searched. In some embodiments, the image pixel value can be determined as the image pixel value in the third image 709 where the third depth map position in the third depth map 711 provides the depth value. That is, in many embodiments where the same resolution is used for the third depth map 711 and the third image 709, the direct determination of the position in the third image 709 corresponding to the first depth map position (arrow 719) It will be understood that it is equivalent to determining the position in the third depth map 711 and extracting the corresponding image pixel.

Similarly, the updater 203 proceeds to extract the pixel values in the first image 701 at the first image position. It then proceeds to determine a first match error index that indicates the difference between these two image pixel values. It will be appreciated that any suitable difference scale can be used, for example, a simple absolute difference, for example a root-squared difference applied to the pixel value components of multiple color channels.

Therefore, the updater 203 determines 605 as the first match error index indicating the difference between the image pixel value in the third image with respect to the third image position and the image pixel value in the first image with respect to the first image position. ..

Next, the updater 203 proceeds to step 607, and the first weight is determined according to the first match error index. It will be appreciated that the particular approach for determining the first weight from the first match error index may depend on the individual embodiment. In many embodiments, complex considerations can be used, including, for example, other match error indicators, further examples will be provided later.

As an example of low complexity, in some embodiments the first weight may be determined as a monotonic reduction function of the first match error index, and in many embodiments it is determined without considering other parameters. May be good.

For example, in an example where a weighted join contains only the first candidate depth value and the original depth value of the first depth pixel, the join applies a fixed weight to the original depth value, the lower the first match error index. An increasing first weight can be applied (usually with additional weight normalization).

The first match error index can be thought of as reflecting how well the first and third images match when representing a given scene point. If there is no shielding difference between the first and third images and the first candidate depth value is the correct value, then the image pixel values should be the same and the first match error index is zero. There should be. If the first candidate depth value deviates from the correct value, the image pixels in the third image may not directly correspond to the same scene point, and therefore the first match error index may increase. If there is a change in shielding, the error is likely to be very high. Therefore, the first match error index can provide a good indicator of how accurate and appropriate the first candidate depth value is for the first depth pixel.

In different embodiments, different approaches can be used to determine the first candidate depth value from one or more of the depth values in the second depth map. Similarly, different approaches can be used to determine which candidate values are generated for the weighted join. Specifically, a plurality of candidate values may be generated from the depth values of the second depth map, and the weights may be calculated individually for each of them according to the approach described with respect to FIG.

In many embodiments, the determination of which second depth value to use to derive the first candidate depth value is with the first depth map so that the corresponding position in the two depth maps is determined. Depends on the projection to and from the second depth map. Specifically, in many embodiments, the first candidate depth value may be determined as the second depth value at the second depth map position that is considered to correspond to the first depth map position, i.e., the second depth. The value is chosen as a depth value that is considered to represent the same scene point.

The determination of the corresponding first depth map position and second depth map position can be based on the projection from the first depth map to the second depth map, i.e., based on the original first depth value or the first. It can be based on the projection from the 2 depth map to the 1st depth map, that is, on the 2nd depth value. In some embodiments, projections in both directions may be performed and, for example, an average of these may be used.

Therefore, determining the first candidate depth value is based on at least one of the second value and the first original depth value of the first depth map, the first view pose and the first of the first image. The projection between the two image's second view poses may include determining the second depth map position relative to the first depth map position.

For example, for a given first pixel in the first depth map, the updater 203 extracts the depth value and uses it to determine the corresponding first depth map position in the second depth map. It can be projected to the corresponding second depth map position. Next, the second depth value at this position can be extracted and used as the first candidate depth value.

As another example, for a given second pixel in the second depth map, the updater 203 extracts the depth value and uses it to determine the corresponding second depth map position in the first depth. It can be projected onto the corresponding first depth map position in the map. The second depth value can then be extracted and used as the first candidate depth value for the first depth pixel at the first depth map position.

In such an embodiment, the depth value in the second depth map is directly used as the first candidate depth value. However, the two depth map pixels represent the distance to the same scene point (without occlusion), but the distance from different viewpoints, so the depth values may be different. In many practical embodiments, this difference in distance from the camera / view pose to the same scene point at different positions is not significant and can be ignored. Therefore, in many embodiments, it can be assumed that the cameras are perfectly aligned, look in the same direction, and have the same position. In that case, if the object is flat and parallel to the image sensor, the depth can actually be the same in the two corresponding depth maps. Deviations from this scenario are often small enough to be ignored.

However, in some embodiments, the determination of the first candidate depth value from the second depth value can include a projection that modifies the depth value. This can be based on more detailed geometric calculations, including taking into account the projective geometry of both views.

In some embodiments, two or more second depth values can be used to generate the first candidate depth value. For example, spatial interpolation may be performed between different depth values to compensate for projections that are not aligned with the center of the pixel.

As another example, in some embodiments, the first candidate depth value may be determined as a result of spatial filtering applied to the second depth map by the kernel centered on the second depth map position.

The following description focuses on embodiments in which each candidate depth value depends only on a single second depth value and is still equal to the second depth value.

In many embodiments, the weighted coupling can further include a plurality of candidate depth values determined from different second depth values.

Specifically, in many embodiments, the weighted join can include candidate depth values from a region of the second depth map. This area may typically be determined based on the first depth map position. Specifically, the second depth map position may be determined by projection (either or both directions) as described above, and the area may be a region around this second depth map position (eg, a predetermined contour). May have).

This approach may provide a set of candidate depth values for the first depth pixel in the first depth map accordingly. For each of the candidate depth values, the updater 203 can perform the method of FIG. 6 to determine the weight of the weighted join.

A special advantage of this approach is that the choice of a second depth value for the candidate depth value is not overly important, as subsequent weighting properly weights good and bad candidates. Therefore, in many embodiments, a relatively less complex approach can be used to select candidates.

In many embodiments, the area is simply as a predetermined area around a position in the second depth map determined by projection from the first depth map to the second depth map based on the original first depth value, for example. It may be decided. In fact, in many embodiments, the projection can even be replaced by simply selecting the area as the area around the depth map position in the second depth map that is the same as the depth map position in the first depth map. Therefore, the approach is simply to select a candidate set of depth values by selecting the second depth value within the area around the first pixel position in the first depth map and the same position in the second depth map. Can be done.

Such an approach can actually provide efficient operation while reducing resource usage. This approach may be particularly suitable when the area size is relatively large compared to the position / parallax shifts that occur between depth maps.

As mentioned above, many different approaches can be used to determine the weight of individual candidate depth values in a weighted join.

In many embodiments, the first weight may be further determined in response to additional match error indicators determined for images other than the third image. In many embodiments, the approach described can be used to generate a match error index for all images except the first image. Then, for example, a combined match error index can be generated as the average of these, from which the first weight can be determined.

ここで、e_kl(z_i)は、候補z_iを所与としたビューkとビューlとの間のマッチ誤差である。マッチ誤差は、例えば、単一ピクセルに対する色差に依存し得るか、あるいは、ピクセル位置（u,v）の周りの空間平均として計算され得る。ビュー（l≠k）にわたる最小のマッチ誤差を計算する代わりに、平均値や中央値などを使用してもよい。多くの実施形態では、この評価関数は、好ましくは、遮蔽によって引き起こされるマッチ誤差異常値に対してロバストである。 Specifically, the first weight may depend on a match error metric, which is a function of a separate match error from the filtered view to all other views (l ≠ k). An example of a metric for determining weights for candidate z _i is:

Here, e _kl (z _i ) is the match error between the view k and the view l given the candidate z _i . The match error can depend, for example, on the color difference for a single pixel, or it can be calculated as a spatial average around the pixel position (u, v). Instead of calculating the minimum match error over the view (l ≠ k), you may use the mean, median, and so on. In many embodiments, this merit function is preferably robust to match error outliers caused by shielding.

In many embodiments, the second match error index can be determined for the second image, i.e., for the view in which the first candidate depth value was generated. The determination of this second match error index can use the same approach described for the first match error index, where the second match error index is the image pixel value in the second image with respect to the second depth map position. Can be generated to show the difference between and the image pixel value in the first image with respect to the first depth map position.

The first weight can then be determined according to both the first match error index and the second match error index (and perhaps other match error indicators or parameters).

In some embodiments, this weighting determination can take into account, for example, not only the average match error index, but also the relative differences between the match indicators. For example, if the first match error index is relatively small, but the second match error index is relatively large, this is due to the obstruction that occurs in the second image with respect to the first image (but not in the third image). there's a possibility that. Therefore, the first weight may be reduced or set to zero.

Other examples of weight consideration can use statistical measures such as, for example, a median match error or another displacement value. The same rationale as above applies here. For example, if you have a linear camera array of nine cameras, all looking in the same direction, four anchors on the left or four anchors on the right around the object edge with respect to the center camera. Will always see the unobstructed area. In this case, the good total weight for a candidate can be a function of only the four lowest of all eight match errors.

In many embodiments, the weighted join can include other depth values of the first depth map itself. Specifically, the weighted join may include a set of depth pixels in the first depth map around the first depth position. For example, a given spatial kernel can be applied to the first depth map for lowpass filtering of the first depth map. The weighting of spatially lowpass filtered first depth map values and candidate depth values from other views is variable, for example, by applying a fixed weight to the lowpass filtered depth values and relative to the first candidate depth value. May be fitted by applying the first weight of.

In many embodiments, the determination of the weight, specifically the first weight, also depends on the confidence value of the depth value.

Depth estimates and measurements are inherently noisy and can have a variety of errors and variations. In addition to depth estimation, many depth estimation and measurement algorithms can also generate confidence values that indicate how reliable the depth estimation provided is. For example, parallax estimation can be based on detecting matching areas in different images, and confidence values can be generated to reflect how similar the matching areas are.

Confidence values can be used in different ways. For example, in many embodiments, the first weight for the first candidate depth value is the confidence value for the first candidate depth value, in particular the confidence value for the second depth value used to generate the first candidate depth value. Can depend on. The first weight may be a monotonically increasing function of confidence values for the second depth value, so the first weight is the underlying depth value (s) used to generate the first candidate depth value. ) Can be increased as the reliability increases. Therefore, the weighted coupling can be biased towards a depth value that is considered reliable and accurate.

In embodiments, the confidence value for the depth map can be used to select which depth value / pixel is updated and for which depth pixel the depth value remains unchanged. Specifically, the updater 203 may be configured to select only the depth values / pixels of the first depth map whose confidence value is less than the threshold value as to be updated.

Therefore, instead of updating all the pixels in the first depth map, the updater 203 specifically identifies the depth values that are considered unreliable and updates only those values. This can prevent, for example, a very accurate and reliable depth estimate from being replaced by a more uncertain value generated from the depth value from another viewpoint in many embodiments. Depth map can be improved.

In some embodiments, the set of depth values in the second depth map contained in the weighted join may depend on the confidence value of the depth values by contributing to different candidate depth values or the same candidate depth values. Specifically, only depth values with confidence values above a given threshold can be included, and all other depth values can be excluded from processing.

For example, updater 203 can generate a modified second depth map by first scanning the second depth map and removing all depth values whose confidence values are below the threshold. You can then use the modified second depth map to perform the above processing, and if no such second depth value is present in the second depth map, everything that requires a second depth value. Operation is bypassed. For example, if no such value exists, no candidate depth value for the second depth value is generated.

In some embodiments, the updater 203 may be configured to generate confidence values for depth values.

In some embodiments, confidence values for a given depth value in a given depth map are determined in response to variations in depth values in other depth maps for corresponding positions in these depth maps. Can be done.

The updater 203 first projects a depth map position for a given depth value for which a confidence value has been determined to a corresponding position in a plurality of other depth maps, typically all of them. be able to.

Specifically, for a given depth value at image coordinates (u, v) _k in the depth map k, another set L of depth maps (typically for adjacent views) is determined. For each of these depth maps (l ∈ L), the corresponding image coordinates (u, v) _l (l ∈ L) are calculated by reprojection.

The updater 203 can then take into account the depth values in these other depth maps at these corresponding locations. We can proceed to determine the variability scale of these depth values at the corresponding positions. Any suitable measure of variability (eg, variance measure) can be used.

The updater 203 can then proceed from this variability scale to determine a confidence value for a given depth map position, specifically an increase in the degree of variability can indicate a decrease in confidence value. .. Therefore, the confidence value may be a monotonically decreasing function of the variability scale.

Specifically, given a depth value z _k and a set of corresponding adjacent depth values z _l (l ∈ L) at (u, v) _l , the trust is based on the consistency of these depth values. You can calculate the metric. For example, the variance of these depth values can be used as a confidence metric. In this case, a small variance means a high degree of reliability.

It is desirable to make this decision more robust to outliers that may result from the corresponding image coordinates (u, v) _k being occluded by objects or camera boundaries in the scene. Often. One concrete way to do this is to select two adjacent views l ₀ and l ₁ on either side of the camera view (k) and use the minimum depth difference:

In some embodiments, the confidence value for a given depth value in a given depth map projects the corresponding given depth position onto another depth map, and then the two depth values in the two depth maps. Can be determined by assessing the error resulting from projecting it back using.

Thus, the updater 203 can initially project a given depth map position onto another depth map based on a given depth value. The depth value at this projected position is then retrieved and the position in the other depth map is reprojected to the original depth map based on the other depth value. This results in a test position that is exactly the same as the original depth map position if the two depth values for the projection exactly match (eg, taking into account the characteristics and geometry of the camera and capture). However, noise and error can make a difference between the two positions.

Therefore, the updater 203 can proceed to determine the confidence value of a given depth map position depending on the distance between the given depth map position and the test position. The smaller the distance, the higher the confidence value, so the confidence value can be determined as a monotonous decrease function of the distance. In many embodiments, multiple other depth maps, and thus distances, can be taken into account.

ここで、f((u,v)_l; z_l)は、奥行き値z_lを用いた隣接するビューlからビューkへ逆投影された画像座標を示す。ノルム

は、L1またはL2、あるいは任意の他のノルムであってもよい。信頼値は、この値の単調減少関数として決定されることができる。 Therefore, in some embodiments, the confidence value can be determined based on the geometrical consistency of the motion vector. Let d _kl be a two-dimensional motion vector that moves the pixel (u, v) _k given the depth z _k to the adjacent view l. Each corresponding pixel position (u, v) _l in the adjacent view l has its own depth z _l , resulting in the vector d _lk returning to the view k. Ideally, if the error is zero, all of these vectors map back exactly to the original point (u, v) _k . But in general, this is not the case, and certainly not the case for unreliable areas. Therefore, a good measure of the lack of reliability is the mean error at the back projection position. This error metric can be formulated as follows:

Here, f ((u, v) _l ; z _l ) indicates the image coordinates back-projected from the adjacent view l to the view k using the depth value z _l . Norm

May be L1 or L2, or any other norm. The confidence value can be determined as a monotonically decreasing function of this value.

It will be appreciated that the term "candidate" does not imply any limitation on the depth value and the term "candidate depth value" can refer to any depth value contained in the weighted join.

It will be appreciated that the above description for clarity has described embodiments of the invention with reference to different functional circuits, units and processors. However, it will be clear that any suitable distribution of functions between different functional circuits, units or processors can be used without departing from the present invention. For example, a function that has been shown to be performed by a separate processor or controller may be performed by the same processor or controller. Therefore, references to specific functional units or circuits should be viewed only as references to appropriate means for providing the described functionality, rather than indicating a strict logical or physical structure or organization. be.

The present invention can be implemented in any suitable form, including hardware, software, firmware or any combination thereof. The invention may optionally be implemented at least partially as computer software running on one or more data processors and / or digital signal processors. The elements and components of embodiments of the invention may be physically, functionally and logically implemented in any suitable manner. In fact, a function may be implemented in a single unit, in multiple units, or as part of another functional unit. Accordingly, the invention may be implemented in a single unit or may be physically and functionally distributed among different units, circuits and processors.

The present invention has been described in connection with some embodiments, but is not intended to be limited to the particular embodiments described herein. Rather, the scope of the invention is limited only by the appended claims. Further, while certain features may appear to be described in relation to a particular embodiment, one of ordinary skill in the art will recognize that various features of the described embodiments can be combined according to the present invention. Let's do it. In the claims, the term "comprising" does not preclude the existence of other elements or steps.

Further, although listed individually, multiple means, elements, circuits or method steps may be implemented, for example, by a single circuit, unit or processor. Further, individual features may be included in different claims, which may be advantageously combined in some cases, and inclusion in different claims is not feasible and / Or it does not mean that it is not advantageous. Also, including a feature in one category of claims does not imply a limitation to this category, but rather that the feature is equally applicable to other claims categories as needed. Is shown. Furthermore, the order of the features in the claims does not mean the particular order in which the features must operate, and in particular the order of the individual steps in the claims of the method is such that the steps are performed in this order. It does not mean that it must be done. Rather, the steps can be performed in any suitable order. Moreover, references to the singular do not exclude multiple. Therefore, the terms "a", "an", "first", "second", etc. do not exclude multiple terms, and the terms "first", "second", "third", etc. are labels. As used as, it does not imply any limitation other than to explicitly identify the corresponding feature and should not be construed as limiting the scope of the claim in any way. The reference numerals in the claims are provided merely as clear examples and should not be construed as limiting the scope of the claims in any way.

Claims (15)

How to handle a depth map,
Steps to receive multiple images representing scenes from different view poses and corresponding depth maps, and
A step of updating the depth value of the first depth map of the corresponding depth map based on the depth value of at least the second depth map of the corresponding depth map, wherein the first depth map is the first. The step and the step, which is for one image and the second depth map is for the second image,
Have,
The step to update is
It is a step of determining the first candidate depth value for the first depth pixel of the first depth map at the first depth map position in the first depth map, and the first candidate depth value is the second depth map. A step and a step determined according to at least one second depth value of the second depth pixel of the second depth map at the second depth map position in.
A step of determining the first depth value of the first depth pixel by weighted coupling of a plurality of depth candidate values for the first depth map position, wherein the weighted coupling is weighted by the first weight. Steps and steps, including the first candidate depth value,
Have,
The step of determining the first depth value is
A step of determining the first image position in the first image with respect to the first depth map position, and
It is a step of determining the third image position in the third image among the plurality of images, and the third image position is to the third image of the first image position based on the first candidate depth value. And the steps that correspond to the projection of
A step of determining a first match error index indicating the difference between the image pixel value in the third image with respect to the third image position and the image pixel value in the first image with respect to the first image position.
The step of determining the first weight according to the first match error index, and
The method. The step of determining the first candidate depth value is the first view pose of the first image and the first view pose based on at least one of the second value and the first original depth value of the first depth map. 2. The method of claim 1, comprising determining the second depth map position relative to the first depth map position by projection between the second view pose of the image. The method of claim 1 or 2, wherein the weighted coupling comprises a candidate depth value determined from a region of the second depth map determined according to the first depth map position. The area of the second depth map is determined as an area around the second depth map position, and the second depth map position is equal to the first depth map position in the first depth map. The method of claim 3, wherein the depth map position in the map is determined. In the second depth map, the region of the second depth map is determined by projection from the first depth map position based on the original depth value in the first depth map at the first depth map position. The method of claim 3, wherein the area around the position is determined. A step of determining a second match error index that indicates the difference between the image pixel value in the second image for the second depth map position and the image pixel value in the first image for the first depth map position. The method according to any one of claims 1 to 5, wherein the step of determining the first weight is performed according to the second match error index. An additional match error indicating the difference between an image pixel value in another image for the depth map position corresponding to the first depth map position and the image pixel value in the first image for the first depth map position. The method according to any one of claims 1 to 6, further comprising a step of determining an index, wherein the step of determining the first weight is performed in accordance with the additional match error index. The method of any one of claims 1 to 7, wherein the weighted coupling comprises a depth value of the first depth map in a region around the first depth map position. The method according to any one of claims 1 to 8, wherein the first weight depends on the confidence value of the first candidate depth value. The method according to any one of claims 1 to 9, wherein only the depth value of the first depth map whose reliability value is below the threshold value is updated. Further steps to select the set of depth values in the second depth map for inclusion in the weighted coupling, according to the requirement that the depth values in the set of depth values must have a confidence value greater than or equal to the threshold. The method according to any one of claims 1 to 10. A step of projecting a given depth map position for a given depth value in a given depth map onto the corresponding positions in the corresponding depth map.
A step of determining a variability scale for a set of depth values having said given depth value and the depth value at said corresponding position in the plurality of said corresponding depth maps.
The step of determining the confidence value for the given depth map position according to the variation scale,
The method according to any one of claims 1 to 11, further comprising. A step of projecting a given depth map position for a given depth value in a given depth map onto a corresponding position in another depth map, the projection being based on said given depth value. , Steps and
A step of projecting the corresponding position in the other depth map to a test position in the given depth map, wherein the projection is a depth value at the corresponding position in the other depth map. Based on the steps and
A step of determining a confidence value for a given depth map position according to the distance between the given depth map position and the test position.
The method according to any one of claims 1 to 12, further comprising. A device for processing depth maps,
A receiver for receiving multiple images representing scenes from different view poses and a corresponding depth map,
An updater for performing a step of updating the depth value of the first depth map of the corresponding depth map based on at least the depth value of the second depth map of the corresponding depth map. The updater and the updater, the first depth map is for the first image and the second depth map is for the second image.
Have,
The updating step is a step of determining the first candidate depth value for the first depth pixel of the first depth map at the first depth map position in the first depth map, and the first candidate depth value is , A step, which is determined according to at least one second depth value of the second depth pixel of the second depth map at the second depth map position in the second depth map.
A step of determining the first depth value of the first depth pixel by weighted coupling of a plurality of candidate depth values for the first depth map position, wherein the weighted coupling is weighted by the first weight. Steps and steps, including the first candidate depth value,
Have,
The step of determining the first depth value is
A step of determining the first image position in the first image with respect to the first depth map position, and
It is a step of determining the third image position in the third image among the plurality of images, and the third image position is to the third image of the first image position based on the first candidate depth value. And the steps that correspond to the projection of
A step of determining a first match error index indicating the difference between the image pixel value in the third image with respect to the third image position and the image pixel value in the first image with respect to the first image position.
The step of determining the first weight according to the first match error index, and
A device with. A computer program that is executed by a computer and causes the computer to perform the method according to any one of claims 1 to 13.

Images