RU2817803C2 - Image signal representing scene - Google Patents

Abstract

FIELD: imaging devices.

SUBSTANCE: invention relates to an apparatus and methods for generating an image signal. Method comprises receiving a plurality of source images representing a scene from different viewing positions, wherein the viewing posture comprises a viewing position and a viewing direction; formation of multiple combined images from source images, wherein each combined image is obtained from a set of at least two source images of a plurality of source images, each corresponding pixel of the merged image represents a scene property at the point of intersection of the corresponding beam with the scene, wherein the beam has an onset position and a direction which together define a beam position, and wherein each combined image includes at least two different pixels representing beam poses for different corresponding beam pose start positions; determination of prediction quality measures for elements of a plurality of source images, wherein the prediction quality measure for the element of the first source image indicates the difference between the pixel values in the element and the predicted pixel values in the element, and the predicted pixel values are pixel values obtained by predicting pixel values in an element based on a plurality of combined images; determining segments of source images containing elements for which the prediction quality measure indicates a difference above a threshold value; and generating an image signal containing image data, representing combined images, and image data representing segments of original images.

EFFECT: high rate of data transmission owing to formation of merged images.

16 cl, 13 dwg

Description

TECHNICAL FIELD

The present invention relates to an image signal representing a scene, and particularly, but not exclusively, to generating an image signal representing a scene and rendering images from that image signal as part of a virtual reality application.

BACKGROUND OF THE ART

With the continuous development and introduction of new services and ways of using and consuming video, recent years have seen a significant increase in the variety and range of image and video applications.

For example, one increasingly popular service is to provide image sequences so that the viewer can actively and dynamically interact with the system to change rendering parameters.

A very attractive feature of many applications is the ability to change the viewer's effective viewing position and viewing direction so that, for example, the viewer can move and "look around" the scene being presented.

Such a feature may, in particular, provide the user with a virtual reality experience.

Thanks to this, the user can, for example, move (relatively) freely in the virtual environment and dynamically change his position and the direction in which he is looking.

Typically, such virtual reality applications are based on a 3D model of the scene, with the model dynamically evaluated to provide the specific view requested.

This approach is well known, for example, from gaming applications for computers and consoles, for example, from the category of first-person shooting games.

It is also desirable, particularly for virtual reality applications, that the image being presented is a three-dimensional image.

Indeed, to optimize viewer immersion, it is generally preferable for the user to experience the presented scene as a 3D scene.

After all, the experience of virtual reality should preferably allow the user to choose his own position, camera viewpoint and point in time relative to the virtual world.

Typically, virtual reality applications are inherently limited in that they are based on a predefined model of the scene and usually an artificial model of the virtual world.

It is often desirable that a virtual reality experience can be provided by capturing the real world.

However, in many cases this approach is limited or typically requires building a virtual model of the real world from the captured real world data.

In this case, the feeling of virtual reality is formed by evaluating this model.

However, current approaches are generally suboptimal and often tend to place high demands on computing and communication resources and/or provide a suboptimal user experience, for example due to reduced quality or limited freedom.

In many systems, for example, particularly those based on a real world scene, an image representation of the scene is provided, where the image representation includes images and depth for one or more capture points/viewpoints in the scene.

The image-plus-depth view provides very efficient characterization, especially of real-world scenes, and this characterization is not only quite easily formed by capturing a real-world scene, but is also very suitable for a renderer synthesizing views for viewpoints other than those used in the capture.

For example, the renderer may be configured to dynamically generate views that correspond to the viewer's current local pose.

For example, the viewer's pose can be determined dynamically, and views can be dynamically generated according to that viewer's pose based on provided images and, for example, depth maps.

However, such image representations typically result in very high data rates for a given image quality.

To ensure good scene capture and in particular to solve the problem of occlusion phenomenon, it is desirable to capture the scene from capture positions close to each other and covering a large range of positions.

Accordingly, a relatively large number of images are required.

In addition, the capture viewports of the cameras often overlap and, therefore, the image set usually contains a large amount of redundant information.

Typically, these problems are independent of the specific gripper configuration and, in particular, whether linear or, for example, circular gripper configurations are used.

Thus, while many of the conventional presentations and image formats may provide good performance in many applications and services, they tend to be suboptimal in at least some circumstances.

Therefore, an improved approach to processing and generating an image signal containing a scene image representation would be useful.

In particular, a system and/or approach that would improve performance, increase flexibility, improve the virtual reality experience, reduce data rates, increase efficiency, facilitate distribution, reduce complexity, facilitate implementation, reduce storage requirements, improve image quality would be beneficial , improve rendering, improve user experience, improve the trade-off between image quality and bit rate, and/or improve performance and/or operation.

DISCLOSURE OF THE INVENTION

Accordingly, the present invention is directed to preferably attenuating, mitigating or eliminating one or more of the above-mentioned disadvantages, individually or in any combination.

According to an aspect of the present invention, there is provided an apparatus for generating an image signal, comprising: a receiver for receiving a plurality of source images representing a scene from various viewing poses; a fused image generator for generating a plurality of fused images from the source images, wherein each fused image is obtained from a set of at least two source images of the plurality of source images, each pixel of the fused image represents a scene for a ray pose, and the ray poses for each fused image include at least two different positions, wherein the beam pose for the pixel represents the pose for the beam in the viewing direction for the pixel and from the viewing position for the pixel; evaluation means for determining prediction quality measures for elements of the plurality of source images, wherein the prediction quality measure for an element of the first source image indicates a difference between pixel values in the first source image for pixels in the element and predicted pixel values for pixels in the element, and the predicted pixel values are values pixels obtained by predicting pixels in an element from a plurality of merged images; a qualifier for identifying segments of source images containing elements for which the prediction quality measure indicates a difference above a threshold value; and an image signal generator for generating an image signal containing image data representing the merged images and image data representing segments of the original images.

The invention can provide improved scene representation and can provide improved image quality of rendered images depending on the data rate of the image signal in many image variations and scenarios.

In many embodiments, a more efficient scene presentation can be provided, for example, allowing a given quality to be achieved at a reduced data rate.

This approach may provide a more flexible and efficient approach to rendering scene images and may allow better adaptation to, for example, scene properties.

This approach can, in many embodiments, utilize a scene image representation suitable for a flexible, efficient, and high-performance virtual reality (VR) application.

In many implementations, it can enable or provide a virtual reality application with a substantially improved trade-off between image quality and data rate.

In many implementations, this may improve the perceived image quality and/or reduce the data transfer rate.

This approach may be suitable, for example, for broadcast video services that support movement and head rotation adaptation at the receiving end.

The source images may be, in particular, light-intensive images with associated depth information, such as depth maps.

This approach may, in particular, allow fusion images to be optimized for foreground and background information respectively, with segments providing additional data where particularly relevant.

The image signal generator may be configured to use more efficient coding of concatenated images than segments.

However, segments may typically represent a relatively small fraction of the fused image data.

In accordance with an optional feature of the present invention, the fused image generator is configured to generate at least a first fused image of a plurality of fused images by synthesizing the appearance of pixels of the first fused image from the plurality of source images, wherein each pixel of the first fused image represents a scene for a ray pose, and the ray pose for the first image include at least two different positions.

This may provide particularly effective performance in many implementations and may, for example, allow the generation of fused images for viewing poses that can (usually in combination) provide a particularly effective representation of the scene.

In accordance with an optional feature of the present invention, the dot product between the vertical vector and the pixel cross product vectors is non-negative for at least 90% of the pixels of the first merged image, wherein the pixel cross product vector for the pixel is the cross product between the ray direction for the pixel and the vector from the center point for different viewing poses to the beam position per pixel.

This can provide particularly efficient and advantageous generation of fused images in many embodiments.

This may in particular provide a low complexity approach to defining a fused image that provides efficient representation of background data at the expense of the tendency to provide a side-biased view.

In accordance with an optional feature of the present invention, the fused image generator is configured to generate a second fused image of the plurality of fused images by synthesizing the pixel appearance of the second fused image from the plurality of source images, wherein each pixel of the second fused image represents a scene for a ray pose and a ray pose for the first image include at least two different provisions; and wherein the dot product between the vertical vector and the pixel cross product vectors is non-positive for at least 90% of the pixels of the second merged image.

This can provide particularly efficient and advantageous generation of fused images in many embodiments.

This may in particular provide a low complexity approach to defining a fused image that provides efficient representation of background data at the cost of tending to provide views offset to other views laterally.

In accordance with an optional feature of the present invention, the ray poses of the first fused image are selected to be close to the boundary of a region containing different viewing poses of the plurality of source images.

This may work effectively in many embodiments and may, for example, provide improved background information via the image signal, thereby facilitating and/or improving view synthesis from the image signal.

In accordance with an optional feature of the present invention, each of the ray poses of the first merged image is determined to be less than a first distance from the boundary of a region containing different viewing poses of a plurality of source images, the first distance being no more than 50% of the maximum internal distance between points on the boundary.

This may work effectively in many embodiments and may, for example, provide improved background information via the image signal, thereby facilitating and/or improving view synthesis from the image signal.

In some embodiments, the first distance is no more than 25% or 10% of the maximum internal distance.

In some embodiments, at least one viewing pose of the fused image is defined at a distance less than a first distance from the boundary of a region containing different viewing poses of a plurality of source images, wherein the first distance is no more than 20%, 10%, or even 5% of the maximum distance between the two viewing poses from various viewing positions.

In some embodiments, at least one viewing pose of the fused images is determined to be at least a minimum distance from the center point of the various viewing poses, wherein the minimum distance is at least 50%, 75%, or even 90% of the distance from the center point to boundaries of a region containing various viewing poses of the plurality of source images along a line through a center point and at least one viewing pose.

In accordance with an optional feature of the present invention, the fused image generator is configured, for each pixel of a first fused image of the plurality of fused images: to determine a corresponding pixel in each of the source images of the view for which the corresponding pixel is present, wherein the corresponding pixel is a pixel that represents the same the ray direction is the same as the pixel of the first combined image; selecting a pixel value for a pixel of the first composite image as the pixel value of a corresponding pixel in the original image of the view for which the corresponding pixel represents a ray having a greatest distance from a center point for various viewing poses, wherein the greatest distance is determined in a first direction along a first axis perpendicular to beam direction for the corresponding pixel.

This can provide particularly efficient and advantageous generation of fused images in many embodiments.

This may in particular provide a low complexity approach to defining a fused image that provides efficient representation of background data at the expense of the tendency to provide a side-biased view.

In accordance with an optional feature of the present invention, determining the corresponding pixels includes resampling each source image into an image representation representing at least a portion of the surface of a viewing sphere surrounding the viewing poses, and defining the corresponding pixels as pixels having the same position in the image representation.

This can provide particularly efficient and accurate detection of the corresponding pixels.

The viewing sphere surface may, for example, be represented by an equirectangular representation or a cube map representation.

Each ray of the viewing sphere may have a ray direction, and resampling the original image may involve setting the pixel value of the viewing sphere to the pixel value of the original image with the same ray direction.

In accordance with an optional feature of the present invention, the fused image generator is configured to, for each pixel of the second fused image: select a pixel value for a pixel in the second fused image as the pixel value of a corresponding pixel in the original image of the view for which the corresponding pixel represents a ray having the greatest distance from the center point in the opposite direction to the first direction.

This can provide particularly efficient and advantageous generation of fused images in many embodiments.

This may in particular provide a low complexity approach to defining a fused image that provides efficient representation of background data at the expense of the tendency to provide a side-biased view.

In addition, the second fused image may complement the first fused image by providing side views from an opposite direction, thereby combining with the first fused image to provide a particularly effective representation of the scene and, in particular, background information.

In accordance with an optional feature of the present invention, the fused image generator is configured, for each pixel of the third fused image: to select a pixel value for a pixel in the third fused image as the pixel value of a corresponding pixel in the original image of the view for which the corresponding pixel represents a ray having the shortest distance from the center point.

This can provide particularly efficient and advantageous generation of fused images in many embodiments.

The third fused image may complement the first (and second) fused image(s) by providing a more frontal view of the scene, which may provide an improved view of foreground objects in the scene.

In accordance with an optional feature of the present invention, the fused image generator is configured, for each pixel of the fourth fused image: to select a pixel value for a pixel in the fourth fused image as the pixel value of a corresponding pixel in the original image of the view for which the corresponding pixel represents a ray having the greatest distance from the center point in a second direction along a second axis perpendicular to the ray direction for the corresponding pixel, the first axis and the second axis having different directions.

This can provide particularly efficient and advantageous generation of fused images in many embodiments and can provide improved scene representation.

In accordance with an optional feature of the present invention, the fused image generator is configured to generate provenance data for the first fused image, wherein the provenance data indicates which of the fused images is the source for each pixel of the first fused image; and the image signal generator is configured to include origin data in the image signal.

This may provide particularly efficient operation in many implementations.

According to an optional feature of the present invention, the image signal generator is configured to include source viewing pose data in the image signal, wherein the source viewing pose data indicates different viewing poses for the source images.

This may provide particularly efficient operation in many implementations.

According to an aspect of the present invention, there is provided an apparatus for receiving an image signal, comprising: a receiver for receiving an image signal, wherein the image signal comprises: a plurality of merged images, each merged image representing image data obtained from a set of at least two original images of the plurality source images representing the scene from different viewing poses, each pixel of the fused image represents the scene for a ray pose, and the ray poses for each fused image include at least two different positions, wherein the ray pose for a pixel represents the pose for a ray in the viewing direction for pixel and from the viewing position for the pixel; image data for a set of segments of a plurality of source images, wherein the segment for the first source image contains at least one pixel of the first source image for which a prediction quality measure for predicting the segment from the plurality of merged images is below a threshold value; and a processor for processing the image signal.

In accordance with an aspect of the present invention, there is provided a method for generating an image signal, including: receiving a plurality of source images representing a scene from various viewing poses; generating a plurality of fused images from the source images, wherein each fused image is derived from a set of at least two source images of the plurality of source images, each pixel of the fused image represents a scene for a ray pose, and the ray poses for each fused image include at least two different positions, wherein a beam pose for a pixel represents a beam pose in a viewing direction for a pixel and from a viewing position for a pixel; determining prediction quality measures for elements of the plurality of source images, wherein the prediction quality measure for an element of the first source image indicates the difference between pixel values in the first source image for pixels in the element and predicted pixel values for pixels in the element, and the predicted pixel values are the pixel values obtained by predicting pixels in an element from the plurality of merged images; identifying segments of source images containing elements for which the prediction quality measure indicates a difference above a threshold value; and generating an image signal containing image data representing the merged images and image data representing segments of the original images.

According to an aspect of the present invention, there is provided a method for receiving an image signal, comprising: receiving an image signal, wherein the image signal comprises: a plurality of merged images, each merged image representing image data obtained from a set of at least two source images of the plurality of source images, representing a scene from different viewing poses, each pixel of the fused image represents a scene for a ray pose, and the ray poses for each fused image include at least two different positions, wherein a ray pose for a pixel represents a pose for a ray in the viewing direction of the pixel and from view position for pixel; image data for a set of segments of a plurality of source images, wherein the segment for the first source image contains at least one pixel of the first source image for which a prediction quality measure for predicting the segment from the plurality of merged images is below a threshold value; and image signal processing.

According to an aspect of the present invention, there is provided an image signal comprising: a plurality of fused images, each fused image representing image data obtained from a set of at least two source images of a plurality of source images representing a scene from different viewing poses, each pixel of the fused image representing a scene for a ray pose, and the ray poses for each fused image include at least two different positions, wherein the ray pose for a pixel represents a pose for the ray in a viewing direction for the pixel and from a viewing position for the pixel; image data for a set of segments of the plurality of source images, wherein the segment for the first source image contains at least one pixel of the first source image for which a measure of prediction quality for predicting the segment from the plurality of merged images is below a threshold value.

These and other aspects, features and/or advantages of the present invention will become apparent from the embodiments described later in this document and will be explained with reference to the embodiment(s).

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will be described only by way of example with reference to the drawings, in which:

In FIG. 1 shows an example of a layout for providing virtual reality perception;

In FIG. 2 shows an example layout for a scene gripper;

In FIG. 3 shows an example layout for a scene gripper;

In FIG. 4 shows an example of elements of a device in accordance with some embodiments of the present invention;

In FIG. 5 shows an example of elements of a device in accordance with some embodiments of the present invention;

In FIG. 6 illustrates an example of pixel selection in accordance with some embodiments of the present invention; And

In FIG. 7 illustrates an example of pixel selection in accordance with some embodiments of the present invention;

In FIG. 8 illustrates an example of beam pose arrangement elements for a fused image generated in accordance with some embodiments of the present invention;

In FIG. 9 illustrates an example of beam pose arrangement elements for a fusion image generated in accordance with some embodiments of the present invention;

In FIG. 10 illustrates an example of beam pose arrangement elements for a fusion image generated in accordance with some embodiments of the present invention;

In FIG. 11 illustrates an example of beam pose arrangement elements for a fused image generated in accordance with some embodiments of the present invention;

In FIG. 12 illustrates an example of beam pose arrangement elements for a fused image generated in accordance with some embodiments of the present invention; And

In FIG. 13 illustrates an example of beam pose arrangement elements for a fused image generated in accordance with some embodiments of the present invention.

IMPLEMENTATION OF THE INVENTION

Virtual perceptions, which allow the user to navigate in a virtual world, are becoming increasingly popular, and services are being developed to meet this demand.

However, providing effective virtual reality services is very challenging, particularly if the perception is to be based on capturing the real world environment rather than a fully virtualized artificial world.

Many virtual reality applications define viewer pose input data that reflects the pose of the virtual viewer in the scene.

The virtual reality device/system/application then generates one or more images corresponding to views and viewports of the scene for the viewer corresponding to the viewer's pose.

Typically, a virtual reality application produces 3D output in the form of separate images of the view for the left and right eyes.

They can then be presented to the user through suitable means, typically such as separate left- and right-eye displays of a virtual reality (VR) headset.

In other embodiments, the image may, for example, be presented on an autostereoscopic display (in which case more views of the viewer's pose may be generated) or, in fact, in some embodiments, only a single two-dimensional image may be generated (for example, using conventional two-dimensional display).

The viewer pose input can be determined in different ways in different applications.

In many embodiments, the user's physical movement can be tracked directly.

For example, a camera capturing a user's area can detect and track the user's head (or even eyes).

In many embodiments, the user may wear a virtual reality headset that can be tracked externally and/or internally.

For example, the headset may contain accelerometers and gyroscopes that provide information about the movement and rotation of the headset and therefore the head.

In some examples, the virtual reality headset may transmit signals or contain (eg, visual) identifiers that allow an external sensor to detect movement of the virtual reality headset.

In some systems, the viewer's pose may be provided by manual means, for example, by the user manually operating a joystick or similar manual input means.

For example, a user can manually move a virtual spectator around a stage by operating a first analog stick with one hand, and manually control the direction in which the virtual spectator is looking by manually moving a second analog stick with the other hand.

Some applications may use a combination of manual and automated approaches to generate the viewer's input pose.

For example, the headset can track head orientation, and the viewer's movement/position in the scene can be controlled by the user using a joystick.

The generation of images is based on a suitable representation of the virtual world/environment/scene.

In some applications, a complete 3D model for a scene may be provided, and views of the scene from a particular viewer pose can be determined by evaluating this model.

In many practical systems, a scene can be represented by an image representation containing image data.

The image data may typically comprise images associated with one or more gripping poses or anchoring poses, and in particular may include images for one or more viewports, each viewport corresponding to a particular pose.

An image representation may be used comprising one or more images, where each image represents a view of a given viewport for a given viewing pose.

Such poses or viewing positions for which image data are provided are often called anchor poses or positions or capture poses or positions (since the image data may typically correspond to images that are, or will be, captured by cameras positioned in the scene and orientation corresponding to the gripping posture).

Many typical virtual reality applications may, based on such an image representation, proceed to provide view images corresponding to viewports for the scene for the viewer's current pose, wherein the images are dynamically updated to reflect changes in the viewer's pose, and wherein the images are generated based on image data representing (possibly ) virtual stage/environment/world.

The application may do this by executing synthesis and view shifting algorithms as known to one of ordinary skill in the art.

In the art, the terms "position" and "posture" are used as a general term for position and/or direction/orientation.

The combination of position and direction/orientation, such as an object, camera, head, or view, can be called pose or orientation.

Thus, a location or pose specification may include six values/components/DOF, with each value/component/DOF describing a different property of the position/location or orientation/direction of the corresponding object.

Of course, in many situations, location or pose can be taken into account or represented using fewer components, for example, if one or more components are considered fixed or irrelevant (for example, if all objects are considered to be located at the same height and have a horizontal orientation, then four components can provide a complete representation of the object's pose).

In the following, the term “pose” is used to refer to position and/or orientation, which can be represented by one to six values (corresponding to the maximum possible degrees of freedom).

Many virtual reality applications are based on a pose that has maximum degrees of freedom, i.e. three degrees of freedom for each position and orientation, resulting in a total of six degrees of freedom.

Thus, a pose may be represented by a set or vector of six values representing six degrees of freedom and, therefore, the pose vector may provide an indication of a three-dimensional position and/or a three-dimensional direction.

However, it is understood that in other embodiments the pose may be represented by fewer values.

The pose may be at least one of orientation and position.

The pose value may indicate at least one of an orientation value and a position value.

A system or object based on providing maximum degrees of freedom to the viewer is usually referred to as having 6 degrees of freedom (6DoF).

Many systems and objects provide only orientation or position and are commonly referred to as having 3 degrees of freedom (3DoF).

In some systems, the virtual reality application may be provided to the viewer locally, for example, by a standalone device that does not use, or even have any access to, any remote virtual reality data or processing.

For example, a device such as a game console may include storage for storing scene data, an input for receiving/generating a viewer's pose, and a processor for generating corresponding images from the scene data.

In other systems, the virtual reality application may be implemented and executed remotely from the viewer.

For example, a device local to the user may detect/receive motion/pose data transmitted to a remote device, which processes the data to generate the viewer's pose.

The remote device can then generate suitable view images for the viewer's pose based on the scene data describing the scene.

The view images are then transferred to a device local to the viewer where they are presented.

For example, a remote device can directly generate a video stream (typically a stereo/3D video stream) that is directly presented by the local device.

Therefore, in such an example, the local device may not perform any virtual reality processing except for transmitting motion data and presenting received video data.

In many systems, functionality can be distributed between a local device and a remote device.

For example, a local device may process received inputs and sensor data to generate viewer poses that are continuously transmitted to a remote virtual reality device.

The remote VR device can then generate appropriate images of the view and transmit them to the local device for presentation.

In other systems, the remote virtual reality device may not directly generate images of the view, but may select relevant scene data and transmit it to the local device, which can then generate images of the view it represents.

For example, a remote VR device can identify a nearby capture point, extract relevant scene data (e.g., spherical image and depth data from the capture point), and transmit it to the local device.

The local device can then process the received scene data to generate images for the specific current viewing pose.

Generally, viewing pose will correspond to head pose, and references to viewing pose can generally be treated equally as corresponding references to head pose.

In many applications, especially for broadcast services, a source may convey scene data as an image (including video) representation of the scene that is independent of the viewer's pose.

For example, an image representation for one field of view for one grip position may be shared with multiple clients.

Individual clients can then locally synthesize view images corresponding to the viewer's current pose.

An application of particular interest is one that supports movement of a limited amount so that the presented views are updated with small movements and rotations corresponding to an essentially static viewer making only small head movements and rotations.

For example, a seated viewer may turn their head and move it slightly, with the views/images presented adapting to these posture changes.

This approach can provide a highly immersive experience, such as video.

For example, a spectator watching a sporting event may feel as if he is present at a specific location in the arena.

The advantage of such constrained freedom applications is to provide improved perception without necessarily accurately representing the scene from many different positions, thereby significantly reducing capture requirements.

Likewise, the amount of data that needs to be provided to the renderer can be significantly reduced.

Indeed, in many scenarios, it is necessary to provide only the image and, usually, depth data for a single viewpoint, and the local renderer can form the required views from this.

This approach may be particularly suitable for applications in which data must be transferred from a source to a destination over a limited bandwidth communication channel, such as, for example, a broadcast application or a client-server application.

In FIG. 1 shows such an example of a virtual reality system in which a remote virtual reality client device 101 communicates with a virtual reality server 103, for example, through a network 105 such as the Internet.

The server 103 may be configured to support a potentially large number of client devices 101 simultaneously.

The virtual reality server 103 may, for example, support broadcast perception by transmitting an image signal containing an image representation in the form of image data that can be used by client devices to locally synthesize view images corresponding to appropriate poses.

Therefore, in many applications, such as the one shown in FIG. 1, it may be desirable to capture a scene and form an effective image representation that can be efficiently included in the image signal.

The image signal can then be transmitted to various devices, which can locally synthesize views for viewing poses other than grasping poses.

To this end, the image representation may typically include depth information and, for example, images with associated depth may be provided.

For example, depth maps can be obtained using stereo capture in combination with disparity estimation or using range sensors, and these depth maps can be provided along with light-intensity images.

However, a particular problem with such approaches is that changing the viewing pose can change the occlusion characteristics, resulting in background segments that are not visible in a given captured image but become visible for a different viewing pose.

To solve this problem, a relatively large number of cameras are often used to capture a scene.

In FIG. Figure 2 shows an example of capture with a circular 8-view multi-camera setup.

In this example, the cameras are facing outwards.

As you can see, different cameras, and therefore different captures/source images, may have views of different parts of the scene.

For example, background area 1 is only visible from camera 2.

However, as can be seen, most of the scene is visible from many cameras, and hence a significant amount of redundant information is generated.

In FIG. Figure 3 shows an example of a linear set of cameras.

Again, cameras provide information about different parts of the scene, for example c1 is the only camera capturing area 2, c3 is the only camera capturing area 4, and c4 is the only camera capturing area 3.

At the same time, some parts of the scene are captured by more than one camera.

For example, all cameras capture the front of foreground objects fg1 and fg2, with some cameras capturing better than others.

In FIG. Figure 3 shows example A for four cameras and example B for two cameras.

As can be seen, the four-camera setup provides better scene capture (background areas 4 bg), but of course also generates more data containing more redundant data.

The disadvantage of multi-view capture relative to a single central view is obviously the increased amount of image data.

Another disadvantage is the huge number of pixels generated, i.e. the bit rate in pixels that must be processed and that the decoder must produce.

It also requires increased complexity and resource use to synthesize species during reproduction.

A specific approach that uses a more efficient and less redundant image representation of the captured views will be described next.

It aims to preserve some spatial and temporal coherence in image data to improve the efficiency of video encoders.

It reduces bit rates, pixel rates, and view synthesis complexity on the playback side.

This representation contains a plurality of merged images, each formed from two or more original images (which in particular may be captured 3D images, e.g. represented as an image plus a depth map), typically only considering a portion of each original Images.

Merged images can serve as the basis for view synthesis and provide significant information about the scene.

The merged images can be generated with a bias towards more external views of the scene and, in particular, towards the boundaries of the capture area.

In some embodiments, one or more central merged images may also be provided.

In many embodiments, each of the merged images represents views from different viewing positions, i.e. each image may contain at least pixels that correspond to different viewing/grabbing/anchoring poses.

In particular, each pixel of the fused image may represent a ray pose corresponding to an origin/position and direction/orientation for a ray from that origin/position directed in that direction/orientation and ending at a point/object in the scene represented by the pixel value for that pixel.

At least two pixels of the merged image may have different ray origins/positions.

For example, in some embodiments, the pixels of the merged image may be divided into N groups, with all pixels in the group having the same origin/position, but different from the other groups.

N can be two or more.

In some embodiments, N may be equal to the maximum number of horizontal pixels in a row (and/or the number of columns in a merged image), and even in some embodiments, N may be equal to the number of pixels, i.e. all pixels can have a unique ray origin/pose.

Thus, a ray pose for a pixel may represent an origin/position and/or an orientation/position for a ray between the origin/position and a point in the scene represented by the pixel.

The origin/position may be, in particular, the viewing position for the pixel, and the orientation/direction may be the viewing direction for the pixel.

It can effectively represent a ray of light that will be captured at a ray position from the ray direction for a pixel, and therefore reflects a ray of light that is represented by a pixel value.

Thus, each pixel can represent a scene as viewed from the viewing position in the viewing direction.

The viewing position and viewing direction respectively determine the beam.

Each pixel may have an associated line of sight from the viewing position for the pixel and in the viewing direction for the pixel.

Each pixel represents a scene for a ray pose, which is the ray pose from the view point/position for the pixel in the view direction.

In particular, a pixel may represent a scene point (a point in the scene) at which the line of sight intersects an object in the scene (including the background).

A pixel can represent rays of light from a point in the scene to the viewing position and in the viewing direction.

The view ray may be a ray from the view position in a direction intersecting a point in the scene.

In addition, the fused images are augmented with segments or fragments of captured species that were identified as not being well predicted from the fused images.

In this way, a number, typically a relatively large number, of typically small segments are determined and included in specially presented individual portions of the captured images that can provide information about elements of the scene not well represented by the combined images.

The advantage of this representation is that different encodings can be provided for different parts of the image data to be transmitted.

For example, efficient and complex coding and compression can be applied to spliced images since they typically constitute the largest portion of the image signal, whereas segments can often benefit from less efficient coding.

Additionally, the fused images can be shaped to be well suited for efficient encoding, for example by being shaped to resemble conventional images and thereby allow efficient image encoding approaches.

In contrast, the properties of segments can vary much more depending on the specific characteristics of the images, and are therefore much more difficult to encode as efficiently.

However, this is not a problem since segments typically provide much less image data.

In FIG. 4 shows an example of an apparatus for generating an image signal that includes representing a plurality of source images of a scene from different source viewing poses (anchor poses) as described above.

This device will also be referred to as the image signal transmitter 400.

The image signal transmitter 400 may, for example, be contained in the virtual reality server 103 shown in FIG. 1

In FIG. 5 shows an example of an apparatus for rendering view images based on a received image signal, which includes representing a plurality of images of a scene.

The device may, in particular, receive an image data signal generated by the device shown in FIG. 4, and process it to render images for specific viewing poses.

The device shown in FIG. 5 will also be referred to as the image signal receiver 500.

The image signal receiver 500 may, for example, be included in the client device 101 shown in FIG. 1

The image signal transmitter 400 includes an image source receiver 401, which is configured to receive a plurality of scene source images.

The source images can represent views of the scene from different viewing poses.

The source images can usually be captured images, i.e. captured by cameras of a multi-camera installation.

The source images may, for example, contain images from a number of equally spaced capture cameras or from a circle of cameras.

In many implementations, the source images may be 3D images containing 2D images with associated depth information.

The 2D images may be, in particular, view images for view windows of the scene from the corresponding grip pose, and the 2D image may be accompanied by an image or depth map containing depth values for each pixel of the 2D image.

A 2D image can be a texture map.

The 2D image may be a light-intensive image.

The depth values may be, for example, disparity values or distance values, such as those indicated by a z coordinate.

In some implementations, the source image may be a 3D image in the form of a texture map with an associated 3D mesh.

In some implementations, such texture map and mesh representations may be converted by the image source receiver to image plus depth representations before further processing by the image signal transmitter 400.

The image source receiver 401 accordingly receives a plurality of source images that characterize and represent a scene from various source viewing poses.

Such a set of source images will allow view images to be generated for other poses using algorithms such as view shift, as known to one skilled in the art.

Accordingly, the image signal transmitter 400 is configured to generate an image signal that contains image data for the source images, and transmits this data to the remote device for local rendering.

However, direct transmission of all source images would require an unfeasibly high data rate and would contain a large amount of redundant information.

The image signal transmitter 400 is configured to reduce the data rate by using an image representation as described previously.

In particular, the input source receiver 401 is connected to a fusion image generator 403, which is configured to generate a plurality of fusion images.

Merged images contain information obtained from multiple original images.

The exact approach to obtaining fused images may differ between different implementations, and specific examples will be described in more detail later.

In some embodiments, a fused image may be generated by selecting pixels from different source images.

In other embodiments, alternatively or additionally, the fused images may form one or more fused images by synthesizing the view from the original images.

However, while each fused image contains contributions from at least two, and often more, source images, only a portion of the individual source images are typically considered for each fused image.

Thus, for each source image used to form a given composite image, there are some pixels that are excluded/discarded.

Therefore, the pixel values generated for a particular merged image are independent of the values of those pixels.

The merged images may be formed such that each image does not simply represent one viewing/capturing/anchoring position, but rather represents two or more viewing/capturing/anchoring positions.

In particular, the origin/position of the ray for at least some pixels in one fused image will be different, and therefore one fused image can represent a view of the scene from different directions.

Accordingly, the fused image generator 403 may be configured to generate a plurality of fused images from the original images, each fused image being derived from a set of at least two original images, and wherein the production of the first fused image typically includes only a portion of each these at least two source images.

In addition, each pixel of a given fused image represents a scene for a ray pose, and the ray poses for each fused image may include at least two different poses.

The fused image generator 403 is connected to an estimator 405, which is supplied with the fused images and the original images.

The estimator 405 is configured to determine measures of prediction quality for elements of the source images.

The element may be an individual pixel, and the estimator 405 may be configured to determine a measure of prediction quality for each pixel of each source image.

In other implementations, elements may contain multiple pixels, and each element may be a group of pixels.

For example, a measure of prediction quality may be defined for blocks, such as 4 × 4 or 16 × 16 pixel blocks.

This may reduce the granularity of the segments or fragments being defined, but can significantly reduce processing complexity and resource usage.

The prediction quality measure for a given element is configured to indicate the difference between the pixel values in the first source image for the pixels in the element and the predicted pixel values for the pixels in the element.

Thus, an element may be formed by one or more pixels, and a measure of prediction quality for an element may indicate the difference between the pixel values for those pixels in the original source image and the pixel values for the pixels that result from the prediction from the fused images.

It is clear that different implementations may use different approaches to determining measures of forecast quality.

In particular, in many embodiments, the estimator 405 may proceed to actually perform a prediction of each source image from the fused images.

It can then, for each individual image and each individual pixel, determine the difference between the original pixel value and the predicted pixel value.

It will be understood that any suitable difference measure may be used, such as, for example, the absolute difference, the difference of the square roots of sums of squares applied to the components of a pixel value, such as a plurality of color channels, etc.

Therefore, such prediction may simulate view prediction/synthesis that may be performed by image signal receiver 500 to generate views for viewing poses of source images.

Thus, prediction quality measures reflect how well a fusion image receiver can generate initial source images from only the fusion images.

The predicted image for the original image from the fused images may be an image for the viewing pose of the original image generated by synthesizing the view from the fused images.

View synthesis typically involves a shift in viewing pose and, typically, a shift in viewing position.

The view synthesis may be a view shift image synthesis.

Predicting the first image from the second image may be, in particular, synthesizing the view of the image at the viewing pose of the first image based on the second image (and its viewing pose).

Thus, the prediction operation for predicting the first image from the second image may be to shift the viewing pose of the second image from the viewing pose associated with the viewing pose of the first image.

It is clear that different implementations may use different methods and algorithms for view synthesis and prediction.

In many implementations, a view synthesis/prediction algorithm may be used that takes as input a synthesis view pose for which a synthesized image is to be generated, and a plurality of input images, each associated with a different view pose.

The view synthesis algorithm can then generate a synthesized image for a given viewing pose based on the input images, which may typically contain both a texture map and a depth map.

A number of such algorithms are known, and any algorithm can be used without prejudice to the present invention.

As an example of such an approach, intermediate synthesized/predicted images may first be generated for each input image.

This can be achieved, for example, by first generating a mesh for the input image based on the image's depth map.

Then, based on geometric calculations, the mesh can be deformed/shifted from the viewing pose of the input image to the viewing pose of the synthesis.

The vertices of the resulting mesh can then be projected onto an intermediate synthesized/predicted image, and a texture map can be overlaid on this image.

For example, such a process could be implemented using vertex processing and fragment shaders known, for example, from standard graphics pipelines.

Thus, for each of the input images, an intermediate synthesized/predicted image (hereinafter referred to simply as an intermediate predicted image) for the synthesis viewing pose can be generated.

The intermediate predicted images can then be combined with each other, for example, by weighted combining/summation or by selective combining.

For example, in some embodiments, each pixel of the synthesized/predicted image for a synthesis view pose may be formed by selecting the pixel from the intermediate predicted image that is furthest forward, or the pixel may be formed by weighted summing the corresponding pixel value for all intermediate predicted images, wherein the weight for a given intermediate predicted image depends on the depth defined for the pixel.

This combining operation is also called the mixing operation.

In some embodiments, measurements of forecast quality may be made without performing a full forecast, but rather, an indirect measure of forecast quality may be used.

For example, a measure of prediction quality can be determined indirectly by estimating the process parameter involved in the species shift.

For example, the amount of geometric distortion (stretch) resulting in a primitive (usually a triangle) when performing a viewing pose shift.

The greater the geometric distortion, the lower the measure of prediction quality for any pixel represented by that primitive.

Thus, the estimator 405 may determine prediction quality measures for elements of a plurality of source images, wherein the prediction quality measure for an element of the first source image indicates the difference between predicted pixel values in an element predicted from the plurality of merged images and pixel values in an element in the first source image. .

The estimator 405 is coupled to a determiner 407, which is configured to determine segments of source images containing elements for which a prediction quality measure indicates a difference above a threshold value/prediction quality measure indicating a prediction quality below a threshold value.

The segments may correspond to individual elements that are determined by the estimator 405 and for which the prediction quality measure is below a quality threshold.

However, in many implementations, the determinant 407 may be configured to form segments by grouping such elements, and indeed the grouping may also include some elements for which the prediction quality measure is above a threshold.

For example, in some embodiments, the qualifier 407 may be configured to form segments by grouping all adjacent elements that have a predictive quality measure below a quality threshold (hereinafter referred to as low predictive quality measures and poor quality elements, respectively).

In other implementations, the determiner 407 may, for example, be configured to select segments of a given size and shape for the images so that they include as many low quality elements as possible.

The determiner 407 accordingly generates a set of segments that include low quality features, and therefore, that cannot be accurately predicted from the fused images.

Typically, these segments will correspond to a small fraction of the original images and therefore a relatively small amount of data and image pixels.

The determiner 407 and the merged image generator 403 are connected to an image signal generator 409 that receives the merged images and segments.

The image signal generator 409 is configured to generate an image signal that contains image data representing merged images and image data representing segments.

The image signal generator 409 may, in particular, encode the merged images and segments, and may, in particular, do so differently and use different encoding algorithms and standards for the merged images and for the segments.

Typically, the fused images are encoded using efficient image encoding algorithms and standards, or high-efficiency video encoding algorithms and standards if the images are frames of a video signal.

Segment encoding may generally be less efficient.

For example, segments may be combined into segmented images, where each image typically may contain segments from a plurality of source images.

Such concatenated segmented images can then be encoded using a standard image or video encoding algorithm.

However, due to the mixed and partial nature of such concatenated segmented images, coding is typically less efficient than for conventional full images.

As another example, due to the sparse nature of segments, they cannot be stored in full frames/images.

In some implementations, the segments may, for example, be represented as meshes in 3D space using a Virtual Reality Modeling Language (VRML).

Typically, image data for segments may be accompanied by metadata indicating the origin of the segments, such as, for example, original image coordinates and camera/original image origin.

In one example, the image signal is transmitted to an image signal receiver 500 that is part of the virtual reality client device 101.

The image signal receiver 500 includes an image signal receiver 501 that receives the image signal from the image signal transmitter 400.

The image signal receiver 501 is configured to decode the image signal to reconstruct the combined images and segments.

The image signal receiver 501 is connected to an image processor 503, which is configured to process the image signal and, in particular, the combined images and segments.

In many implementations, image processor 503 may be configured to synthesize view images for different viewing poses based on the fused images and segments.

In some embodiments, image processor 503 may first begin synthesizing the source images.

The portions of the synthesized original images for which a segment is included in the image signal can then be replaced by the image data provided by the segments.

The resulting source images can then be used for conventional image synthesis.

In other implementations, the merged images and segments can be used directly without first reconstructing the original images.

It is understood that the image signal transmitter 400 and the image signal receiver 500 contain the required functionality for exchanging the image signal, including functionality for encoding, modulation, transmission, reception, etc. image signal.

It is understood that such functionality will depend on the preferences and requirements of the individual implementation, and that such technologies are known to one skilled in the art and therefore, for the sake of clarity and brevity, will not be discussed further herein.

Different implementations may use different approaches to generating fused images.

In some implementations, the fused image generator 403 may be configured to generate fused images by selecting pixels from the original images.

For example, for each pixel in the fused image, the fused image generator 403 may select one pixel in one of the original images.

The image and/or depth map contains pixels having values that can be considered to represent a corresponding image property (light intensity/intensities or depth) of a scene along a ray having a ray direction (orientation) from the origin (position) of the ray.

The ray onset is typically the viewing pose for the image, but may in some representation vary depending on the pixels (such as, for example, omnidirectional stereo, in which the image itself can be viewed as having a viewing pose corresponding to the center of the circle of the omnidirectional video, but each pixel has a separate viewing pose corresponding to its position on the circumference of the omnidirectional video).

The direction of the ray can typically vary across pixels, particularly for images in which all pixels have the same ray origin (i.e. there is one common viewing pose).

The origin and/or direction of the beam is also often referred to as the beam pose or beam projection pose.

Thus, each pixel is associated with a position representing the origin of a ray/straight line.

Each pixel is also associated with a direction, which is the direction of the ray/straight line from the origin.

Accordingly, each pixel is associated with a ray/straight line that is defined by a position/origin and a direction from that position/origin.

The value of a pixel is specified by the corresponding property for the scene at the first intersection of the ray for the pixel and the scene object (including the background).

Thus, a pixel value represents a property of the scene along a ray/straight line starting at the ray origin position and having a direction associated with the pixel.

A pixel value represents a property of the scene along a ray having the pixel's ray pose.

Thus, the fusion image generator 403 can, for a given first pixel in the generated fusion image, determine the corresponding pixels in the original images as pixels that represent the same ray direction.

The corresponding pixels may suitably be a pixel that represents the same ray direction, but which has different positions, since the source images may correspond to different positions.

Thus, in principle, the fused image generator 403 can, for a given pixel in the fused image, determine the ray direction and then determine all pixels in the original images that have the same (within a given similarity requirement) ray directions and consider them to be corresponding pixels.

Therefore, the corresponding pixels will generally have the same ray direction but different ray positions/origins.

Views from different images of the original viewing poses can, for example, be resampled so that the corresponding image coordinates have corresponding ray directions.

For example, when original views are presented in partial equirectangular projection format, they are resampled into a full 360°/180° version.

For example, you can define a viewing sphere that surrounds the entire original viewing configuration.

This field of view can be divided into pixels, with each pixel having a ray direction.

For a given source image, each pixel can be resampled into a viewport representation by setting the viewport pixel value for a given ray direction to the pixel value for a pixel in the source view that has the same ray direction.

Resampling the original images into a full viewport surface representation will typically result in N partially filled images, since individual images typically have limited viewports, where N is the number of original images.

However, viewports tend to overlap, and accordingly a set of viewport surface representations typically provide multiple pixel values for any given direction.

The fused image generator 403 can now proceed to generate at least one, but typically multiple fused images by selecting between corresponding pixels.

In particular, a first fused image may be generated to cover a portion of the scene.

For example, a fused image of a given size may be generated to cover a specific region of pixels in the field of view representations and thereby describe that region of the scene.

In some embodiments, each of the fused images may span the entire scene and include the entire surface of the viewing sphere.

For each pixel in the first fused image, the fused image generator 403 can now consider the corresponding pixels in the viewport views and proceed to select pixels.

In particular, the fused image generator 403 may generate a first fused image by selecting a pixel value for the fused image as a pixel value for a corresponding pixel in the original image of the view for which the corresponding pixel represents a ray having the greatest distance from a center point in a first direction along a first axis, perpendicular to the ray direction for the corresponding pixel

The distance from the center point to the ray direction can be defined as the distance between the rays of the center point and the corresponding pixel for a pixel in the merged image.

The selection can be explained using FIG. 6, which is based on an example of a circular configuration of the initial viewing pose having a center point C.

This example looks at identifying a pixel in a fused image that has ray direction rc.

This direction is captured by cameras/source views 1–4, and thus there are four corresponding pixels.

Each of these four corresponding pixels represents different poses and accordingly represents rays starting at different positions, as shown.

Accordingly, between these rays and the ray rc of the combined image there are offset distances p1-p4 corresponding to the distance between the center point C and the rays if they are extended back (to the intersection with axis 601).

In FIG. 6 also shows the direction/axis 601 perpendicular to the ray rc.

The fusion image generator 403 can now, for the first fusion image, select the corresponding pixel for which the ray distance in that direction is greatest.

So in this case, the pixel value of the merged image will be chosen as the pixel value for camera/view 1 since p1 is the largest distance in that direction.

Typically, the fused image generator 403 may then proceed to determine a second fused image by performing the same operation, but selecting the corresponding pixels that have the greatest distance in the opposite direction (we can consider that generating the first and second fused images can be accomplished by selecting the largest positive and negative distances, respectively, with respect to the first direction, if the distance is measured as positive when it is in the same direction as the axis, and negative when it is in the other direction).

Thus, in this case, the fused image generator 403 will select the pixel value of the fused image as the pixel value for camera/view 4 since p4 is the largest distance in that direction.

In many implementations, the fusion image generator 403 may then proceed to generate a third fusion image by performing the same operation but selecting the corresponding pixels that have the smallest distance in any direction (smallest absolute distance).

Thus, in this case, the fused image generator 403 will select the pixel value of the fused image as the pixel value for camera/view 3 since p3 is the smallest distance.

Thus, the fused image generator 403 can thereby generate three fused images for the same portion of the scene (and possibly the entire scene).

One of the images will correspond to a selection of pixels that provide most of the side view of the scene in one direction, one that represents most of the side view of the scene from the opposite direction, and one that represents most of the central view of the scene.

This can be illustrated in FIG. 7, which shows viewing directions selected from each view/camera, respectively, for the central fused image and the two side fused images.

In this way, the resulting images provide a very efficient representation of the scene, with one fused image typically providing a representation for foreground objects and the other two fused to provide background-focused data.

In some embodiments, the fusion image generator 403 may be configured to further generate one or more fusion images by selecting corresponding pixels according to an axis direction that is perpendicular to the beam direction, but different from the previously used axis direction.

This approach may be suitable for non-planar configurations of the initial viewing pose (e.g., 3D configurations).

For example, for a spherical configuration of the initial viewing pose, more than two planes may be considered.

For example, planes at 0, 60, and 120 degrees may be considered, or two orthogonal planes (eg, left-to-right and top-to-bottom planes) may be considered.

In some implementations, the fused images may be generated by synthesizing/predicting the view from the original images.

The image generator 103 may, in particular, generate fused images representing views of the scene from different viewing positions and, in particular, from different viewing positions different from the viewing positions of the original images.

In addition, unlike conventional image synthesis, the fused image is not generated to represent a view of a scene from a single viewing/capture position, but can represent the scene from different viewing positions even within the same fused image.

Thus, a fused image can be formed by generating pixel values for pixels in the fused image by synthesizing/predicting the view from the original images, but with the pixel values representing different viewing positions.

Specifically, for a given pixel in a fused image, view synthesis/prediction can be performed to determine the pixel value corresponding to a particular ray pose for that pixel.

This can be repeated for all pixels in the merged image, but at least some of the pixels have different ray poses.

For example, a single fused image may provide a 360° view of the scene corresponding to, for example, the surface of a viewing sphere that surrounds the entire configuration of the original viewing pose.

However, these views of different parts of the scene may be presented from different positions within the same merged image.

In FIG. 8 shows an example in which the fused image contains pixels representing two different ray positions (and therefore viewing positions of the pixel), namely, a first ray origin 801 that is used for pixels representing one hemisphere, and a second ray origin 803 representing another hemisphere.

For each of these beam positions/starts, pixels with different beam directions are provided, as shown by the arrows.

In this particular example, the initial viewing pose configuration contains eight initial views (1–8) in a circular arrangement.

Each camera's view provides only part of the view, say a 90° view, but with overlap between views.

For a given pixel in the fused image, there may be an associated ray pose, and the pixel value for that pose can be determined by synthesizing/predicting the view from these source views.

In principle, each pixel of a fused image can be synthesized individually, but many implementations perform fused synthesis across multiple pixels.

For example, for the first position 801, one 180° image may be synthesized from the original images (e.g., using positions 2, 1, 8, 7, 6, 5, 4), and for the second position 803, one 180° image may be synthesized ° from the original images (for example, using positions 6, 5, 4, 3, 2, 1, 8).

Then, by combining these images, a merged image can be formed.

If individually synthesized images overlap, fusion or blending can be used to form a combined image.

In an alternative implementation, overlapping portions of the merged images may be suppressed, for example, by assigning a reserved color or depth value.

This increases the efficiency of video encoding.

In many embodiments, one or more fused images may be formed to represent the scene from a viewpoint that provides a more lateral view of the scene.

For example, in FIG. 8, the center of the viewing circle corresponds to the center point of the original viewing poses and the center of the beam origin positions for the fused image.

However, the beam directions for a given beam origin 801, 803 do not point in a predominantly radial direction, but rather provide a sideways view of the scene.

Specifically, in this example, both the first beam start 801 and the second beam start 803 provide views in the left direction, i.e. the beam directions for both beams point to the left when looking at the beam origins 801, 803 from the center point.

The image generator 103 may proceed to generate a second fused image that represents a different view of the scene, and in particular, it is advantageous that it may generate a second view of the scene that is complementary to the first view but facing in the opposite direction.

For example, image generator 103 may generate a second merged image that uses the same beam origins but the beam directions point in the opposite direction.

For example, the image generator 103 may generate a second fused image corresponding to the configuration shown in FIG. 9.

These two images can provide a very effective and complementary representation of the scene, and can generally provide an improved representation of the background portions of the scene.

In many embodiments, the fused image may also include one or more images that are shaped to provide a more frontal view, such as the configuration shown in FIG. 10.

Such an example may, in many implementations, provide an improved view of the front of foreground objects.

It will be understood that different beam start configurations may be used in different embodiments, and that, in particular, more beam starts may be used.

For example, in FIG. 11 and 12 show examples of two complementary configurations for generating side-facing fusion images, with the origins of the rays distributed along a curve (specifically a circle), in this case surrounding the initial viewing configuration (often such a curve will be chosen strictly according to the configuration of the initial viewing poses) .

The figures only show the origins and poses for a portion of the circle/curve, and it is understood that many embodiments will generate a full spherical or 360° view.

Indeed, Fig. 7 can be taken as an illustration of yet another exemplary configuration in which three merged images are formed based on eight beam positions on a circle around a center point.

For the first merged image, radial directions of the circle are selected, for the second image, ray directions are selected at 90° to the right of the radius, and for the third image, ray directions are selected at 90° to the left of the radius.

This combination of fused images can provide a very effective fused view of the scene.

Thus, in some embodiments, image generator 103 may be configured to generate pixel values for fused images for specific ray poses by synthesizing the view from the original images.

Beam poses can be chosen differently for different fused images.

Specifically, in many embodiments, the beam poses for one image may be selected to provide a side view of the scene from the origin of the beam, and the beam poses of another image may be selected to provide a complementary side view.

In particular, the ray poses for the first fused image may be such that the dot product between the vertical vector and the pixel cross product vectors is non-negative for at least 90% (sometimes 95% or even all) of the pixels in the first fused image.

The pixel-to-pixel cross product vector is defined as the cross product between the ray direction for a pixel and the vector from the center point for different initial viewing poses to the ray position for a pixel.

The center point for the initial viewing poses can be formed as the middle or median position for the initial viewing poses.

For example, one could average over the coordinate (e.g., x, y, z) of each pose, and the resulting average could be the center point.

It should be noted that the center point for the configuration is not (necessarily) located at the center of the smallest circle/sphere containing the original viewing poses.

Thus, the vector from the center point to the origin of the ray for a given pixel is a vector in scene space that defines the distance and direction from the center point to the viewing position for that pixel.

The direction of a ray can be represented by (any) vector that has the same direction, i.e. this could be a vector from the origin of the ray to a point in the scene represented by the pixel (and therefore could also be a vector in scene space).

The cross product between such two vectors will be perpendicular to both of them.

In the case of a horizontal plane (in the scene coordinate system), the direction of the ray to the left (as viewed from the center point) will result in a cross product vector that has an upward component, i.e. has a positive z component, where z indicates the height.

The cross product vector will be upward for any left-handed view regardless of the origin, for example it will be upward for all pixels/beam poses in FIG. 8.

Conversely, for a right-handed view, the cross product vector will be downward for all ray poses, such as the result for all pixels/beam poses in FIG. 9 would be a negative z-coordinate.

The dot product between a vertical vector in scene space and all vectors that have a positive z-coordinate will be the same, specifically it will be positive for an up-pointing vertical vector and negative for a downward-pointing vertical vector.

Conversely, for a negative z-coordinate, the dot product will be negative for an upward-pointing vertical vector and positive for a downward-pointing vertical vector.

Accordingly, the dot product will have the same sign for all right-handed beam poses and the opposite sign for all left-handed beam poses.

In some scenarios, the result may be a null vector or dot product (for example, for polar points on the viewing circle), and for such ray poses the sign will be no different from either left-handed views or right-handed views.

It is clear that the above considerations are also valid, with appropriate amendments, in relation to a three-dimensional representation, for example, when the origins of the ray are located on a sphere.

Thus, in some embodiments, at least 90%, and in some embodiments, at least 95%, or even all, of the pixels in the merged image produce a dot product that does not differ in sign, i.e. at least that many pixels will have a sideways view in the same direction.

In some implementations, the merged images may be configured to have guard bands, or, for example, some particular edge pixels may have particular circumstances for which the dot product could potentially not satisfy this requirement.

However, for the vast majority of pixels this requirement is satisfied and the pixels provide appropriate side views.

Additionally, in many embodiments, at least two merged images satisfy these requirements, but the signs of the dot product are opposite.

Thus, for one fused image, at least 90% of the pixels may represent a right-handed view, and for another fused image, at least 90% of the pixels may represent a left-handed view.

Merged images can be generated for poses that provide a particularly effective view of the scene.

The inventors realized that in many scenarios it may be particularly advantageous to generate fused images for viewing poses that result in a more lateral view of the main portion of the scene, and also that for a given configuration of source views it may be advantageous to generate at least some views that are close to the extreme positions of the configuration, rather than close to the center of the configuration.

Thus, in many embodiments, at least one, and typically at least two, fused images are generated for the beam poses that are close to the boundary of the region corresponding to the configuration of the original viewing poses.

In particular, this region may be a region of space (a group or set of points in space) that is bounded by the largest polygon that can be formed using at least some of the view positions as vertices for the straight lines of the polygon.

A polygon may be a planar figure that is limited by a finite chain of line segments that loop together to form a closed chain or path, and it may include a one-dimensional configuration such as in FIG. 2A (also known as a degenerate polygon).

In the case of a three-dimensional configuration, the region may correspond to the largest possible polyhedron formed by at least some of the initial viewing positions.

Thus, the region may be the largest polygon or polyhedron that can be formed using at least some of the original view positions as vertices for the lines of the polygon or polyhedron.

Alternatively, the region containing the various viewing poses of a plurality of source images may be the smallest line, circle, or sphere containing all viewing positions.

In particular, the region may be the smallest sphere that contains all of the original view positions.

Thus, in many embodiments, the beam pose of at least one of the fused images is selected close to the boundary of the region containing the original viewing pose configuration.

In many embodiments, at least one beam position of the fused images is determined to be less than a first distance from the boundary of the region, wherein the first distance is no more than 50%, or in many cases 25% or 10%, of the maximum (internal) distance between points on border of the region.

Thus, the minimum distance from the viewing pose position to the boundary can be no more than 50%, 25%, or 10% of the maximum distance to the boundary.

This can be illustrated using FIG. 13, which shows the original viewpoints indicated by the black dots.

In FIG. 13 also shows the area corresponding to the smallest sphere that includes these viewpoints.

In this example, the viewing configuration is a planar two-dimensional configuration, and viewing a sphere is reduced to viewing a circle 1301.

In FIG. 13 also shows the ray pose 1303 for the merged image, which is close to the sphere/circle/region boundary.

In particular, the minimum distance d _min to the border/edge of the region is much smaller (about 10%) than the maximum distance d _max to the border/edge of the region.

In some embodiments, the beam poses of the fused image may be determined at a distance less than a first distance from the region boundary, with the first distance being no more than 20%, or even 10% or 5%, of the maximum distance between the two original viewing poses.

In this example, where the region is defined as the smallest sphere/circle containing all the original viewing poses, the maximum distance between two viewing poses is equal to the diameters of the sphere/circle, and hence the viewing pose of the merged image can be chosen such that the minimum distance d _min satisfies this requirement.

In some embodiments, the beam poses of the fused image may be determined to be at least a minimum distance from the center point of the various viewing poses, where the minimum distance is at least 50%, or as often as 75% or 90%, of the distance from the center point to the boundary along a line passing through the center point and pose of the beam.

In some embodiments, the two viewing poses for the fused images are selected such that the distance between them is at least 80%, and sometimes even 90% or 95%, of the maximum distance between two intersection points of a line passing through the two viewing poses.

For example, if a line is drawn through two poses, the distance between the two poses is at least 80%, 90%, or 95% of the distance between the points at which the line intersects the circle.

In some embodiments, the maximum distance between two ray poses of the first fused image is at least 80% of the maximum distance between boundary points of a region containing different viewing poses of the plurality of source images.

The inventors have realized that the approach of generating fused images for positions close to the border/edge of the region containing the original viewing poses can be particularly advantageous since it typically provides more information about background objects in the scene.

Most background data is usually captured by cameras or in areas of the image that are generally more to the side compared to the central viewpoint.

This can be advantageously combined with a more centrally fused image since it typically provides improved image information for foreground objects.

In many implementations, the image signal generator 409 may be configured to optionally include metadata for the generated image data.

In particular, the fusion image generator 403 may generate provenance data for the fused images, wherein the provenance data indicates which of the original images is the source for individual pixels in the fused images.

The image signal generator 409 may then include this data in a generated image signal.

In many embodiments, the image signal generator 409 may include source viewing pose data indicating viewing poses for the source images.

In particular, this data may include data defining the position and direction of each source image/view.

The image signal may accordingly comprise metadata which indicates, possibly on a per-pixel basis, the position and direction for which the pixel values are provided, i.e. indicating the pose of the beam.

Accordingly, the image signal receiver 500 may be configured to process this data to perform, for example, view synthesis.

For example, for each pixel of three views formed by selecting the corresponding pixels, the metadata may include an indication of the source view identifier.

This can result in three cue maps - one for the central view and two for the side views.

The labels can then be associated with specific viewing pose data containing, for example, camera optics and multi-camera setup geometry.

It will be understood that in the foregoing description, embodiments of the present invention are set forth for clarity with reference to different functional circuits, blocks, and processors.

However, it is understood that any suitable distribution of functionality between different functional circuits, blocks or processors can be used without prejudice to the present invention.

For example, functionality shown to be implemented by separate processors or controllers may be implemented by the same processor or controllers.

Therefore, references to specific functional blocks or diagrams should be considered only as references to suitable means to provide the functionality described, and not as an indication of a strict logical or physical structure or organization.

The present invention may be implemented in any suitable form, including hardware, software, firmware, or any combination thereof.

Optionally, the present invention may be implemented, at least in part, in the form of computer software running on one or more processors and/or digital signal processors.

The elements and components of an embodiment of the present invention may be physically, functionally, and logically implemented in any suitable manner.

In reality, functionality may be implemented in a single block, in multiple blocks, or as part of other functional blocks.

Therefore, the present invention may be implemented in a single block or may be physically or functionally distributed among different blocks, circuits and processors.

Although the present invention has been described in connection with certain embodiments, this should not be construed as limiting it to the particular form set forth herein.

On the contrary, the scope of the present invention is limited only by the appended claims.

In addition, although it may appear that a feature is described in connection with specific embodiments, one skilled in the art will understand that various features of the described embodiments can be combined in accordance with the present invention.

In the claims, the term “comprising/comprising” does not exclude the presence of other elements or steps.

In addition, although multiple means, elements, circuits, or method steps are listed separately, they may be implemented, for example, by a single circuit, block, or processor.

Further, although individual features may be included in different claims, they may be effectively combined, and inclusion in different claims does not mean that the combination of features is infeasible and/or disadvantageous.

In addition, the inclusion of a feature in one category of claims does not imply limitation to that category, but, on the contrary, indicates that the feature can equally be applied to other categories of claims when appropriate.

In addition, the order of features in a claim does not imply the specific order in which those features must be processed, and, in particular, the order of individual steps in a method claim does not mean that the steps must be performed in that order.

Instead, the steps may be performed in any suitable order.

Moreover, references in the singular do not exclude the plural.

Therefore, references using singular means, singular adjectives “first”, “second”, etc. do not exclude the multitude.

Reference numerals in the claims are provided solely by way of clarifying example and should not be construed as limiting the scope of the claims in any way.

Claims (39)

1. A device for generating an image signal, containing: a receiver (401) for receiving a plurality of source images representing a scene from different viewing poses, the viewing pose comprising a viewing position and a viewing direction; a fused image generator (403) for generating a plurality of fused images from the source images, each fused image being obtained from a set of at least two source images of the plurality of source images, each corresponding pixel of the fused image representing a property of the scene at the point of intersection of the corresponding ray with the scene, wherein a ray has an origin position and a direction that together define a ray pose, and wherein each merged image includes at least two pixels representing ray poses for various corresponding ray pose origin positions; evaluation means (405) for determining prediction quality measures for elements of the plurality of source images, wherein the prediction quality measure for an element of the first source image indicates the difference between pixel values in the element and predicted pixel values in the element, and the predicted pixel values are pixel values obtained as a result predicting pixel values in an element based on the plurality of merged images; a qualifier (407) for determining segments of source images containing elements for which the prediction quality measure indicates a difference above a threshold value; And an image signal generator (409) for generating an image signal containing image data representing the merged images and image data representing segments of the original images. 2. The apparatus of claim 1, wherein the fused image generator (403) is configured to generate at least a first fused image of the plurality of fused images by synthesizing the appearance of pixels of the first fused image from the plurality of source images, wherein each pixel of the first fused image represents a scene for beam poses, and the beam poses for the first image include beam poses for at least two different beam pose start positions. 3. The apparatus of claim 2, wherein the dot product between the vertical vector and the pixel cross product vectors is non-negative for at least 90% of the pixels of the first merged image, wherein the pixel cross product vector for the pixel is the cross product between the ray pose direction for the pixel and vector from the center point for different viewing poses to the position of the beginning of the ray pose for the pixel. 4. The apparatus of claim 3, wherein the fused image generator (403) is configured to generate a second fused image of the plurality of fused images by synthesizing the appearance of pixels of the second fused image from the plurality of original images, wherein each pixel of the second fused image represents a scene for a ray pose, and the beam poses for the second image include beam poses for at least two different beam pose start positions; And wherein the dot product between the vertical vector and the pixel cross product vectors is non-positive for at least 90% of the pixels of the second fused image. 5. The apparatus of claim 2, wherein the ray poses of the first merged image are selected to be close to the boundary of a region containing different viewing poses of the plurality of source images. 6. The device according to claim 2 or 3, wherein each of the ray poses of the first combined image is defined at a distance less than the first distance from the boundary of the region containing various viewing poses of the plurality of source images, and the first distance is no more than 50% of the maximum internal distance between points on the border. 7. The apparatus of any preceding claim, wherein the fused image generator (403) is configured to, for each pixel of the first fused image, of a plurality of fused images: determining a corresponding pixel in each of the source images of the view for which the corresponding pixel is present, the corresponding pixel being a pixel that represents the same ray direction as a pixel of the first merged image; selecting a pixel value for a pixel of the first merged image as the pixel value of a corresponding pixel in the original image of the view for which the corresponding pixel represents a ray having a greatest distance from a center point for various viewing poses, which greatest distance is determined in a first direction along a first axis perpendicular to ray direction for the corresponding pixel. 8. The apparatus of claim 7, wherein determining the corresponding pixels includes resampling each source image into an image representation representing at least a portion of the surface of a viewing sphere surrounding the viewing poses, and determining the corresponding pixels as pixels having the same position in the representation Images. 9. The device according to claim 7 or 8, in which the fused image generator (403) is configured to, for each pixel of the second fused image: selecting a pixel value for a pixel in the second merged image as a pixel value of a corresponding pixel in the original image of the view for which the corresponding pixel represents a ray having the greatest distance from the center point in a direction opposite to the first direction. 10. Device according to any one of paragraphs. 7-9, in which the fusion image generator (403) is configured to: for each pixel of the third merged image: selecting a pixel value for a pixel in the third merged image as a pixel value of a corresponding pixel in the original image of the view for which the corresponding pixel represents a ray having the shortest distance from the center point. 11. Device according to any one of paragraphs. 7-10, in which the fusion image generator (403) is configured to: for each pixel in the fourth merged image: selecting a pixel value for a pixel in the fourth merged image as the pixel value of a corresponding pixel in the original image of the view for which the corresponding pixel represents a ray having the greatest distance from a center point in a second direction along a second axis perpendicular to the direction of the ray for the corresponding pixel, wherein the first axis and the second axis have different directions. 12. Device according to any one of paragraphs. 7-11, wherein the fused image generator (403) is configured to generate provenance data for the first fused image, wherein the provenance data indicates which of the fused images is the source for each pixel of the first fused image; and the image signal generator (409) is configured to include origin data in the image signal. 13. The apparatus of any preceding claim, wherein the image signal generator (403) is configured to include source viewing pose data in the image signal, wherein the source viewing pose data indicates different viewing poses for the source images. 14. A device for receiving an image signal, comprising: a receiver (501) for receiving an image signal containing: a plurality of fused images, each fused image representing image data derived from a set of at least two source images of a plurality of source images representing a scene from different viewing poses, wherein the viewing pose comprises a viewing position and a viewing direction, each corresponding pixel of the fused image representing a property scenes at the point of intersection of the corresponding ray with the scene, wherein the ray has an origin position and a direction that together define a ray pose, and wherein each merged image includes at least two different pixels representing ray poses for different corresponding ray pose origin positions; image data for a set of segments of a plurality of source images, wherein the segment for the first source image contains at least one pixel of the first source image for which a measure of prediction quality related to predicting pixel values in the segment based on the plurality of merged images is below a threshold value; And a processor (503) for processing the data signal. 15. A method for generating an image signal, including: receiving a plurality of source images representing the scene from different viewing poses, the viewing pose comprising a viewing position and a viewing direction; generating a plurality of fused images from the source images, each fused image being derived from a set of at least two source images of the plurality of source images, each corresponding pixel of the fused image representing a property of the scene at the point of intersection of the corresponding ray with the scene, the ray having an origin position and a direction, which together define a beam pose, and wherein each combined image includes at least two different pixels representing beam poses for different corresponding beam pose origin positions; determining prediction quality measures for elements of the plurality of source images, wherein the prediction quality measure for an element of the first source image indicates the difference between pixel values in the element and predicted pixel values in the element, and the predicted pixel values are pixel values obtained by predicting pixel values in the element on based on multiple merged images; identifying segments of source images containing elements for which the prediction quality measure indicates a difference above a threshold value; And generating an image signal containing image data representing the merged images and image data representing segments of the original images. 16. A method for processing an image signal, including: receiving an image signal containing: a plurality of fused images, each fused image representing image data derived from a set of at least two source images of a plurality of source images representing a scene from different viewing poses, wherein the viewing pose comprises a viewing position and a viewing direction, each corresponding pixel of the fused image representing a property scenes at the point of intersection of the corresponding ray with the scene, wherein the ray has an origin position and a direction that together define a ray pose, and wherein each merged image includes at least two pixels representing ray poses for various corresponding ray pose origin positions; image data for a set of segments of a plurality of source images, wherein the segment for the first source image contains at least one pixel of the first source image for which a measure of prediction quality related to predicting pixel values in the segment based on the plurality of merged images is below a threshold value; and image signal processing.