KR101969943B1 - Method and apparatus for reducing spherical image bandwidth to a user headset - Google Patents

Abstract

The method includes determining at least one preferred view viewpoint associated with a three-dimensional (3D) image; Encoding a first portion of the 3D image corresponding to the at least one preferred view perspective with a first quality; And encoding a second portion of the 3D image with a second quality, wherein the first quality is higher quality than the second quality.

Description

Method and apparatus for reducing spherical image bandwidth to a user headset

This application claims the benefit of “Methods and Apparatus for Reducing Spherical Image Bandwidth to a User Headset” of US Patent Application No. 62 / 167,121, filed May 27, 2015, which is incorporated herein by reference in its entirety. It is integrated.

Embodiments relate to spherical video streaming.

Spherical video (or other three-dimensional video) streaming can consume a significant amount of system resources. For example, an encoded spherical image may include a large number of transmission bits that may consume a significant amount of bandwidth as well as processing and memory associated with the encoder and decoder.

Example embodiments disclose a system and method for optimizing video streaming, 3D video streaming and / or spherical video streaming.

In a general aspect, a method includes determining at least one preferred view viewpoint associated with a three dimensional (3D) image, encoding a first portion of the 3D image corresponding to the at least one preferred view viewpoint with a first quality, and Encoding a second portion of the 3D image with a second quality, wherein the first quality is higher quality than the second quality.

In another general aspect, a server and / or a streaming server includes a controller that determines at least one preferred view viewpoint associated with a three-dimensional (3D) image, and a first quality of the 3D image corresponding to the at least one preferred view viewpoint with a first quality. And an encoder for encoding the portion and encoding the second portion of the 3D image at a second quality, wherein the first quality is higher quality compared to the second quality.

In another general aspect, a method includes receiving a request for a streaming video, the request including an indication of a user view viewpoint associated with the three-dimensional (3D) video; Determining whether a user view viewpoint is stored in the view viewpoint data storage; Incrementing a counter associated with the user view viewpoint if it is determined that the user view viewpoint is stored in the view viewpoint data storage; And if it is determined that the user view viewpoint is not stored in the view viewpoint data storage, adding the user view viewpoint to the view viewpoint data storage and setting a counter related to the user view viewpoint to one.

Implementations can include one or more of the following features: For example, the method (or implementation on a server) may comprise storing a first portion of a 3D image in a data storage device; Storing a second portion of the 3D image in a data storage device; Receiving a request for streaming video; Streaming the first portion of the 3D image and the second portion of the 3D image from the data storage device as a streaming streaming image. The method (or implementation at a server) comprises receiving a request for streaming video, the request comprising an indication of a user view point of view; Selecting a 3D image corresponding to the user view viewpoint as the encoded first portion of the 3D image; And streaming the selected first portion of the 3D image and the second portion of the 3D image as a streaming image.

The method (or implementation at a server) receives a request for streaming video, the request including an indication of a user view viewpoint associated with the 3D video; Determining whether a user view viewpoint is stored in the view viewpoint data storage; If it is determined that the user view viewpoint is stored in the view viewpoint data storage, incrementing a counter associated with the user view viewpoint; And if it is determined that the user view viewpoint is not stored in the view viewpoint data storage, adding the user view viewpoint to the view viewpoint data storage and setting a counter related to the user view viewpoint to one. The method (or implementation at a server) wherein the encoding of the second portion of the 3D video comprises using at least one first quality of service QoS parameter in a first pass encoding operation. Encoding the first portion may further comprise using at least one second quality of service QoS parameter in a second pass encoding operation.

For example, determining the at least one preferred view viewpoint associated with the 3D image is based on at least one of a historical viewed reference point and a historical viewed view viewpoint. The at least one preferred view viewpoint associated with the 3D image is based on at least one of a direction of the viewer of the 3D image, a position of the viewer of the 3D image, a point of the viewer of the 3D image, and an imaging point of the focus 3D image viewer. Determining at least one preferred view viewpoint associated with the 3D image is based on a default view viewpoint, wherein the default view viewpoint is a characteristic of a user of the display device, a user of the associated display device, a director's cut, and a 3D image. It depends on the property. For example, the method (or implementation on a server) may further include repeatedly encoding at least one portion of the second portion of the first quality 3D image and streaming at least one portion of the second portion. May contain

Exemplary embodiments disclosed herein may disclose a system and method for optimizing video streaming, 3D video streaming, and / or spherical video streaming.

Exemplary embodiments will be more fully understood from the following detailed description and the accompanying drawings, in which like elements are denoted by like reference numerals and are given only in an illustrative manner and thus do not limit the exemplary embodiment.
1A shows a two-dimensional (2D) representation of a sphere in accordance with at least one exemplary embodiment.
1B shows an unpackaged cylindrical representation of a 2D representation of a sphere as a 2D rectangular representation.
2-5 illustrate a method of encoding a streaming spherical image in accordance with at least one exemplary embodiment.
6A illustrates an image encoder system according to at least one example embodiment.
6B illustrates an image decoder system according to at least one example embodiment.
7A shows a flow diagram for a video encoder system according to at least one example embodiment.
7B is a flowchart of an image decoder system according to at least one exemplary embodiment.
8 illustrates a system according to at least one example embodiment.
9 is a schematic block diagram of a computer device and a mobile computer device that may be used to implement the techniques described herein.
These drawings are intended to explain the general characteristics of the methods, structures and / or materials used in certain exemplary embodiments and to supplement the written description provided below. However, these drawings may not accurately reflect and do not extend the exact structural or performance characteristics of any given embodiment and should not be construed as defining or limiting the range of values or characteristics encompassed by the exemplary embodiments. . For example, the relative thickness and positioning of the structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numerals in the various figures is intended to indicate the presence of similar or identical elements or features.

The example embodiments may include various modifications and alternative forms, but the embodiments are shown by way of example in the drawings and will be described in detail herein. However, although there is no intention to limit the example embodiments to the specific forms disclosed, on the contrary, the example embodiments should be understood to include all modifications, equivalents, and alternatives falling within the scope of the claims. Like reference numerals denote like elements throughout the drawings.

Exemplary embodiments include spherical images based on priorities (eg, director's cut, historical viewing, etc.) for viewing video (streaming), streaming 3D video, and viewing a portion of the spherical video (by the viewer of the video). A system and method configured to optimize the streaming of (and / or other three-dimensional images), for example, a director's cut may be a view viewpoint selected by the director or the image producer, wherein the director's cut is the director or producer of the image. Can be based on the camera (of multiple cameras) view you have selected or viewed.

The spherical image, the spherical image and / or the frame of the spherical image may have a perspective. For example, the spherical image may be a globe image. The internal viewpoint may be a view from the center of the earth looking outward. Or the internal view may be a view of the universe on Earth. The outer view can be a view of the earth from space. As another example, the viewpoint may be based on a portion of the image that is visible. That is, the viewable perspective may be a viewpoint that the viewer (wearer) can see. The view point of view may be part of the spherical image in front of the viewer. For example, when viewed from an internal point of view, the viewer may lie on the ground (eg, Earth) to look at the universe. The viewer can see the moon, sun or a specific star in the image. However, the ground on which the viewer lays is included in the spherical image but outside the current viewing point. In this example, the viewer can turn her head and the ground will be included at the point of view of the surroundings. The viewer could flip and the ground would be in view, but the moon, sun or stars would not.

The view point from the external point of view may be a portion of the spherical image that is not blocked (eg, by another portion of the image) and / or a portion of the spherical image that is not invisibly bent. Other portions of the spherical image may result in a visible point of view from an external point of view by movement (eg, rotation) of the spherical image and / or movement of the spherical image. Thus, the view point of view is a portion of the spherical image that is within the visible range of the spherical image viewer.

Spherical images are images that do not change with time. For example, a spherical image from an internal viewpoint with respect to the earth can display the moon and stars in one location. Spherical images (or series of images), on the other hand, can change over time. For example, spherical images from internal viewpoints associated with the earth can represent plane stripes across moving moons and stars (eg, due to earth rotation) and / or images (eg, the sky).

1A is a two-dimensional (2D) representation of a sphere. As shown in FIG. 1A, the sphere 100 (eg, as a frame of a spherical image or spherical image) represents the directions of the internal viewpoints 105, 110, the external viewpoint 115, and the visible viewpoints 120, 125, 130. . The view point 120 may be part of the sphere 100 as can be seen from the inner view point 105. The view point 125 may be part of the sphere 100 as seen from an external view point.

1B shows an unpacked cylindrical representation 150 that is a 2D representation of sphere 100 as a 2D rectangular representation. The isometric projection of the image shown by the unpacked cylindrical representation 150 may appear as an image that is stretched as it runs vertically (up and down as shown in FIG. 1B) from the midline between points A and B. The 2D rectangular representation may be decomposed as a C × R matrix of N × N blocks. For example, as shown in FIG. 1B, the unpacked cylindrical representation 150 is a 30 × 16 matrix of N × N blocks. However, other CxR dimensions are within the scope of this disclosure. The blocks may be blocks (or pixel blocks) such as 2x2, 2x4, 4x4, 4x8, 8x8, 8x16, 16x16, and the like.

Spherical images are continuous images in all directions. Thus, if the spherical image is decomposed into a plurality of blocks, the plurality of blocks will continue over the spherical image. That is, there are no edges or boundaries as in 2D images. In example implementations, adjacent terminal blocks can be adjacent to the boundary of the 2D representation. In addition, the adjacent end block may be a block contiguous to the block at the boundary of the 2D representation. For example, the adjacent end blocks are associated with two or more boundaries of the two-dimensional representation. In other words, since the spherical image is a continuous image in all directions, the adjacent ends may be associated with the upper boundary (eg, of the row of blocks) and the lower boundary in the image or frame and / or the left boundary (in the image or frame). For example, in a block row) and to the right boundary.

For example, if isometric is used, adjacent end blocks may be blocks at the other end of the column or row. For example, as shown in FIG. 1B, blocks 160 and 170 may be respective adjacent end blocks (relative to columns) relative to one another. In addition, blocks 180 and 185 may be respective adjacent end blocks (based on columns) relative to one another. In addition, blocks 165 and 175 may be respective adjacent end blocks (relative to rows) with respect to each other. The view point 192 may include (and / or overlap) at least one block. Blocks may be encoded as an area of an image, an area of a frame, a portion or subset of an image or frame, a group of blocks, or the like. Hereinafter, this block group may be referred to as a tile or a tile group. For example, tiles 190 and 195 are shown in groups of four blocks in FIG. 1B. Tile 195 is shown to be within view point 192.

In exemplary embodiments, in addition to streaming a frame of encoded spherical image, a tile (or tile group) selected based on at least one reference point that the viewer (at least at one historically watched reference point or view viewpoints) often viewed The view viewpoint can be encoded, for example, at higher quality (eg, high resolution and / or low distortion) and can be streamed with (or as part of) an encoded frame of the spherical image. Thus, during playback, the viewer can see the decoded (higher quality) tiles while the full spherical image is playing, and the full spherical image is also available if the viewer's view point changes to a frequently viewed view point. The viewer can also change the view position or switch to a different view point. If another view point is included in at least one reference point that the viewer frequently sees, the reproduced image is of higher quality (e.g., not one of the at least one reference point that the viewer often sees). For example, higher resolution). One advantage of encoding and streaming only selected portions or subsets of images or frames at higher quality is that the selected images or frames of the spherical image can be decoded and played back at higher quality without necessarily encoding, streaming or decoding. In addition, by decoding the entire spherical image with higher quality, it is possible to increase the efficiency of bandwidth usage and processing and memory resources associated with the encoder and decoder.

In a head mounted display (HMD), the viewer experiences visual virtual reality through the use of a left (eg, left eye) display and a right (eg, right eye) display that projects a recognized three-dimensional (3D) image. . According to example implementations, spherical (eg, 3D) images or images are stored on a server. The video or image may be encoded and streamed from the server to the HMD. The spherical image or image may be encoded as a left image and a right image packaged with metadata for the left image and the right image (eg, in a data packet). The left image and the right image are then decoded and displayed by the left (eg left eye) display and right (eg right eye) display.

The system (s) and method (s) described herein are applicable to both left and right images and refer to images, frames, portions of images, portions of frames, tiles, and / or the like throughout the specification. It may include. That is, the encoded data that is passed from the server (eg streaming server) to the user device (eg HMD) and then decoded for display may be a left image and / or right image associated with the 3D image or image. have.

2-5 are flowcharts of a method according to an exemplary embodiment. The steps described in connection with FIGS. 2-5 may include, but are not limited to, memory (eg, at least one associated with an apparatus (eg, as shown in FIGS. 6A, 6B, 7A, 7B, and 8C, described below). May be executed due to the execution of software code stored in the memory 610 of the memory 610, and may be executed by at least one processor (eg, at least one processor 605) associated with the apparatus. However, alternative embodiments such as systems implemented as special purpose processors may be considered. The steps described below are described as being executed by a processor, but the steps are not necessarily executed by the same processor. In other words, at least one processor may execute the steps described below with respect to FIGS.

2 illustrates a method of storing a historical view viewpoint, where the historical refers to a view viewpoint that users have previously requested. For example, FIG. 2 may illustrate building a database of view points of view generally viewed in spherical video streams. As shown in FIG. 2, an indication of the view viewpoint is received in step S205. For example, a tile may be required by an apparatus that includes a decoder. The tile request may include information based on a viewpoint or view viewpoint related to the direction, position, point, or focus of the viewer on the spherical image. The viewpoint or view viewpoint may be a user viewpoint viewpoint or a view viewpoint of the HMD user. For example, the view viewpoint (eg, user view viewpoint) may be a latitude and longitude position on the spherical image (eg, as an internal viewpoint or an external viewpoint). The view, perspective or view perspective may be determined from the side of the cube based on the spherical image. The display of the view viewpoint may also include spherical image information. In an example implementation, the indication of the view viewpoint can include information regarding a frame (eg, a frame sequence) associated with the view viewpoint. For example, the view (eg, latitude and longitude location or side) may be transmitted from the user device with the HMD to the streaming server using, for example, Hypertext Transfer Protocol (HTTP).

In step S210, it is determined whether a view viewpoint (eg, a user view viewpoint) is stored in the view viewpoint data store. For example, data storage (eg, view viewpoint data storage 815) may be queried or filtered based on information associated with the view viewpoint or the user view viewpoint. For example, the data storage device may be queried or filtered based on the latitude and longitude position on the spherical image of the view viewpoint as well as the time stamp in the spherical image where the view viewpoint is viewed. The time stamp may be a time and / or a range of time associated with playing a spherical image. The query or filter may be used to determine spatial proximity (e.g., how close the current view point is to a given storage view point in time) and / or temporal proximity (e.g., how close the given storage time stamp is to the current time stamp). Can be based If the query or filter returns a result, the view point is stored in the data store. Otherwise, the view point is not stored in the data store. If the view viewpoint is stored in the view viewpoint data storage, processing continues from step S215 to step S220. Otherwise, processing continues at step S225.

In step S220, a counter or ranking (or rank value) associated with the received view viewpoint is incremented. For example, the data storage device may include a data table including a history view viewpoint (eg, the data storage device may be a database including a plurality of data tables). The data table can be accessed (eg, each uniquely) for the view point of view. The data table may include identification information of the view viewpoint, information related to the view viewpoint, and a counter indicating the number of times the view viewpoint is requested. The counter may be incremented each time a view point is requested. Data stored in the data table may be anonymously processed. That is, data may be stored such that there is no reference (or identification) to a user, device, session, or the like. Therefore, data stored in the data table cannot be distinguished based on the user or viewer of the image. In an example implementation, data stored in a data table may be classified based on the user without identifying the user. For example, the data may include a user and / or other age, age range, gender, type or role (eg musician or crowd).

In step S225, the view viewpoint is added to the view viewpoint data storage. For example, identification of the view viewpoint, information related to the view viewpoint, and a counter (or rank value) set to 1 may be stored in a data table including the view viewpoint history.

In an example embodiment, tiles associated with at least one preferred view viewpoint may be encoded with higher QoS. The QoS may be an implementation of the quality discussed above (eg, an encoder variable input defining the quality). For example, an encoder (eg, image encoder 625) may individually encode tiles associated with the 3D image. Tiles associated with at least one preferred view viewpoint may be encoded with a higher QoS than tiles associated with the rest of the 3D image. In an example implementation, the 3D image may be encoded using a first QoS parameter (s) (eg, in the first pass) or at least one first QoS parameter used in the first encoding pass. In addition, the tile associated with the at least one preferred view viewpoint that can be encoded is encoded using the second QoS parameter (s) (eg, in the second pass) or at least one second QoS parameter used in the second encoding pass. Can be. In this example implementation, the second QoS is higher QoS than the first QoS. In another example implementation, the 3D image may be encoded as a plurality of tiles representing the 3D image. Tiles associated with the at least one preferred view viewpoint may be encoded using the second QoS parameter (s). The remaining tiles may be encoded using the first QoS parameter (s).

In another alternative implementation (and / or further implementation), the encoder projects a tile associated with the at least one preferred view viewpoint using a projection technique or algorithm other than that used to generate the 2D representation of the rest of the 3D image frame. can do. Some projections may introduce distortion in certain areas of the frame. Thus, projecting tiles differently from spherical frames can improve the quality of the final image and / or use pixels more efficiently (e.g., reduce computer computation or place less strain on the user's eyes). ). In one example implementation, the spherical image can be rotated before projecting the tiles in order to direct the tiles to the least distorted position based on the projection algorithm. In another example implementation, a tile may use (and / or modify) a projection algorithm based on the tile's position. For example, projecting a spherical image frame into a 2D representation may use isometric projection, while projecting a spherical image frame into a representation comprising a portion to be selected as a tile may use cubic projection.

3 illustrates a method of streaming 3D video. 3 describes a scenario in which streaming 3D video is encoded on demand during a live streaming event. As shown in FIG. 3, in step S305, a request for streaming a 3D image is received. For example, a streamable 3D image, part of 3D image, or part of a tile may be requested by a device that includes a decoder (eg, through user interaction with a media application). The request may include information based on a viewpoint or view viewpoint associated with the viewer's direction, position, point or focal point on the spherical image. Information based on the viewpoint or view viewpoint may be based on the current direction or the default (eg, initialization) direction. The default direction may be, for example, a detector cut for the 3D image.

In step S310, at least one preferred view viewpoint is determined. For example, data storage (eg, view viewpoint data storage 815) may be queried or filtered based on information associated with the view viewpoint. The data storage device may be queried or filtered based on the latitude and longitude positions on the spherical image of the view point of view. In an example implementation at least one preferred view viewpoint may be based on the view viewpoint history. Therefore, the data storage device may include a data table including view viewpoint details. The preference may be represented by the number of requested view viewpoints. Thus, the query or filter may include filtering the results below a threshold counter value. That is, the parameter set for the query of the data table including the history view viewpoint may include a value for the counter or rank that the query result should be equal to or greater than the threshold of the counter. The query result of the data table including the history view viewpoint may be set as at least one preferred view viewpoint.

In addition, the default preferred view viewpoint (or view viewpoints) may be associated with the 3D image. The default preference view viewpoint may be a director cut, a point of interest (eg, a horizontal line, a moving object, a priority object), and / or the like. For example, the purpose of the game may be to destroy an object (eg, a building or a vehicle). This object may be labled as a priority object. The view viewpoint including the priority object may be displayed as the preferred view viewpoint. The default preference view viewpoint may additionally be included in the history view viewpoint or an alternative to the history view viewpoint. The default direction may be, for example, an initial set of preferred view viewpoints (eg, for lack of historical data when the image is first uploaded) based on an automated computer vision algorithm. The vision algorithm may determine a preferred view viewpoint portion of the image with motion or complex details or a nearby object in stereo to infer what interests and / or features were present in the preferred view of the other historical image.

Other factors may be used when determining at least one preferred view viewpoint. For example, the at least one preferred view viewpoint may be a historical view viewpoint within (eg, close to) a range of the current view viewpoint. For example, the at least one preference view viewpoint may be a history view viewpoint that is within the range (eg, proximate) of the history view viewpoint of the current user to which the current user belongs or the group (type or category) to which the current user belongs. That is, the at least one preferred view viewpoint may include view viewpoints (or tiles) that are close in distance and / or close in time to the stored historical view viewpoint. The default preferred view viewpoint (s) may be stored in the data storage 815 including the historical view viewpoint or in a separate (eg, additional) data storage not shown.

In step S315, the 3D image is encoded with at least one encoding parameter based on at least one preferred view viewpoint. For example, the 3D image (or part thereof) may be encoded such that portions including at least one preferred view viewpoint are encoded differently from the rest of the 3D image. As such, the portions including at least one preferred view viewpoint may be encoded with higher QoS than the rest of the 3D image. As a result, when rendered on the HMD, the portions including at least one preferred view viewpoint may have a higher resolution than the rest of the 3D image.

In step S320, the encoded 3D video is streamed. For example, tiles can be included in a packet for transmission. The packet may include a compressed video bit 10A. The packet may comprise a spherical image frame and an encoded 2D representation of an encoded tile (or a plurality of tiles). The packet may include a header for transmission. The header may include, among other things, information indicating the mode or technique used in intra-frame coding by the encoder. The header may include information indicating parameters used when converting a frame of a spherical image frame into a 2D rectangular representation. The header may include information representing the encoded 2D rectangular representation and parameters used to achieve QoS of the encoded tile. As described above, the QoS of tiles associated with at least one preferred view viewpoint may be different (eg, higher) than tiles not associated with at least one preferred view viewpoint.

3D video streaming can be implemented using priority steps. For example, low (or minimum standard) QoS encoded video data may be streamed in the first priority step. This allows the user of the HMD to start a virtual reality experience. Higher QoS video can then be streamed to the HMD to replace the previously streamed low (or minimum standard) QoS encoded video data (eg, data stored in buffer 830). For example, in the second step, higher quality image or image data may be streamed based on the current view viewpoint. In subsequent steps, higher QoS image or image data may be streamed based on one or more preferred view viewpoints. This may continue until the HMD buffer contains only substantially high QoS video or image data. In addition, this staged streaming can continue to cycle with progressively higher QoS video or image data. That is, after the first iteration, the HMD includes video or image data encoded in the first QoS, after the second iteration, the HMD includes video or image data encoded in the second QoS, and after the third iteration, the HMD is encoded in the third QoS. Included image or image data. In an exemplary implementation, the second QoS is higher than the first QoS and the third QoS is higher than the second QoS.

The encoder 625 may operate off-line as part of the setup procedure to make the spherical image available for streaming. Each of the plurality of tiles may be stored in the view frame storage device 795. Each of the plurality of tiles is indexed such that each of the plurality of tiles can be stored with a reference (eg, time dependent) to a frame (eg, time dependent) and a field of view (eg, view dependent). Thus, each of the plurality of tiles depends on time and view, view or view viewpoint and can be called up based on time and view dependencies.

As such, in an example implementation, the encoder 625 may be configured to execute a loop in which a frame is selected such that a portion of the frame is selected as a tile based on the view point of view. The tile is then encoded and stored. The loop continues to cycle through the plurality of view viewpoints. When a desired number of view points, for example, every 5 degrees vertically and every 5 degrees horizontally of the spherical image, are stored as tiles, a new frame is selected, and the process requires that the mode frames of the spherical image select the desired number of tiles. Repeat until you have it. In an example implementation, a tile associated with at least one preferred view viewpoint may be encoded with a higher QoS than a tile that is not a tile associated with at least one preferred view viewpoint. However, this is only one example implementation of encoding and storing tiles.

4 shows a method of storing an encoded 3D image. 4 illustrates a scenario in which streaming 3D video is pre-encoded and stored for later streaming. As shown in FIG. 4, at step S405, at least one preferred view viewpoint for the 3D image is determined. For example, data storage (eg, view viewpoint data storage 815) may be queried or filtered based on information associated with the view viewpoint. The data storage device may be queried or filtered based on the latitude and longitude locations on the spherical image at the view point of view. In at least one example implementation, the at least one preferred view viewpoint may be based on a history view viewpoint. As such, the data table may include historical view viewpoints. The preference may indicate how often the view point was requested. Thus, the query or filter may include filtering the results below a threshold counter value. That is, the parameters set for the query of the data table including the history view viewpoint may include a counter value for which the query result should be greater than or equal to the threshold of the counter. The query result of the data table including the history view viewpoint may be set as at least one preferred view viewpoint.

Also, the default preferred view viewpoint (or view viewpoint) may be associated with the 3D image. Default preference view viewpoints may be director cuts, points of interest (eg, horizontal lines, moving objects, priority objects), and / or the like.

For example, the purpose of the game may be to destroy an object (eg, a building or a vehicle). This object may be labled as a priority object. The view viewpoint including the priority object may be displayed as the preferred view viewpoint. The default preference view viewpoint may additionally be included in the history view viewpoint or an alternative to the history view viewpoint. Other factors may be used when determining at least one preferred view viewpoint. For example, the at least one preferred view viewpoint can be a historical view viewpoint that is within (eg, proximate) the range of the current view viewpoint. For example, the at least one preferred view viewpoint may be a history view viewpoint that is within (eg, close to) a history view viewpoint of the current user or a group (type or category) to which the current user belongs. The default preference view viewpoint (s) may be stored in a data table that includes a historical view viewpoint or in a separate (eg, additional) data table.

In step S410, the 3D image is encoded with at least one encoding parameter based on at least one preferred view viewpoint. For example, a frame of the 3D image may be selected and a portion of the frame may be selected as a tile based on the view viewpoint. The tile is then encoded. In one embodiment, tiles associated with at least one preferred view viewpoint may be encoded with higher QoS. Tiles associated with at least one preferred view viewpoint may be encoded with higher QoS than tiles associated with the remaining 3D image.

In an alternative implementation (and / or further implementation), the encoder may project a tile associated with at least one preferred view point of view using a projection technique or algorithm different from that used when generating the 2D representation of the remainder of the 3D image frame. Can be. Some projections may cause distortion in certain areas of the frame. Thus, projecting tiles differently from spherical frames can improve the quality of the final image and / or use pixels more efficiently. In one example implementation, the spherical image can be rotated before projecting the tile to direct the tile to the least distorted position based on the projection algorithm. In another example implementation, a tile may use (and / or modify) a projection algorithm based on the position of the tile. For example, projecting a spherical image frame into a 2D representation may use isometric projection, while projecting a spherical image frame into a representation comprising a portion to be selected as a tile may use cubic projection.

In step S415, the encoded 3D image is stored. For example, each of the plurality of tiles may be stored in the view frame storage 795. Each of the plurality of tiles associated with the 3D image may be indexed such that each of the plurality of tiles is stored as a reference to a frame (eg, time dependent) and a view (eg, time dependent). Thus, each of the plurality of tiles depends on time and view, view or view viewpoint and can be called up based on time and view dependencies.

In example implementations, the 3D image (eg, related tiles) can be encoded and stored with various encoding parameters. Thus, the 3D image can be stored in different encoding states. The states may vary depending on QoS. For example, the 3D video may be stored as a plurality of tiles, each encoded with the same QoS. For example, the 3D image may be stored as a plurality of tiles each encoded with different QoS. For example, the 3D image may be stored as a plurality of tiles encoded with QoS based on the encoded at least one preferred view viewpoint.

5 illustrates a method of determining a preferred view viewpoint for a 3D image. The preferred view viewpoint of the 3D image may be added to the preferred view viewpoint based on the history view of the 3D image. As shown in FIG. 6, at step S505, at least one default view viewpoint is determined. For example, the default preferred view viewpoint (s) may be stored in a data table included in data storage (eg, view viewpoint data storage 815). The data storage device may be queried or filtered based on the default display for the 3D image. If the query or filter returns a result, the 3D image has an associated default view viewpoint (s). Otherwise, the 3D image does not have an associated default view viewpoint. Default preference view viewpoints may be director cuts, points of interest (eg, horizontal lines, moving objects, priority objects), and / or the like. For example, the purpose of the game may be to destroy an object (eg, a building or a vehicle). This object can be recognized as a priority object. The view viewpoint including the priority object may be displayed as the preferred view viewpoint.

In step S510, at least one view viewpoint is determined based on user characteristics / preferences / categories. For example, a user of an HMD may have characteristics based on previous use of the HMD. The characteristic may be based on statistical viewing preferences (eg, preferences for viewing objects up close as opposed to distant objects). For example, a user of the HMD can store user preferences associated with the HMD. The preference may be general (eg, attracted to movement) or image specific (eg, preferring to focus on a guitar list for music performance). For example, an HMD user can belong to a group or category (eg, males between 15 and 22 years old). For example, the user characteristic / preference / category may be stored in a data table included in the data storage device (eg, view viewpoint data storage device 815). The data storage device may be queried or filtered based on the default display for the 3D image. If the query or filter returns a result, the 3D image has at least one associated preferred view viewpoint (s) based on relevant characteristics / preferences / categories for the user; otherwise, the 3D image does not have an associated view viewpoint based on the user. Do not.

In step S515, at least one view viewpoint based on the region of interest is determined. For example, the region of interest may be a current view viewpoint. For example, the at least one preferred view viewpoint may be a historical view viewpoint within a range (eg, proximate) of the current view viewpoint. For example, the at least one preferred view viewpoint may be a history view viewpoint that is within (eg, close to) a history view viewpoint of the current user or a group (type or category) to which the current user belongs.

In step S520, at least one view viewpoint based on at least one system characteristic is determined. For example, the HMD may have a function to enhance the user experience. One feature may be enhanced audio. Thus, in a virtual reality environment, a user may be attracted to certain sounds (eg, a game user may be attracted to an explosion). The preferred view viewpoint can be based on the view viewpoint containing these audible queues. In step S525, at least one preferred view viewpoint for the 3D image based on each of the above-described view viewpoint determinations and / or combination / sub combinations thereof is determined. For example, the at least one preferred view viewpoint can be generated by merging or combining the results of the query described above.

In the example of FIG. 6A, image encoder system 600 may be or include at least one computing device and may virtually represent any computing device configured to perform the methods described herein. As such, image encoder system 600 may include various components that may be used to implement the techniques described herein, or different or future versions thereof. By way of example, image encoder system 600 is shown to include at least one processor 605 as well as at least one memory 610 (eg, non-transitory computer readable storage medium).

6A illustrates an image encoder system according to at least one example embodiment. As shown in FIG. 6A, at least one processor 605, at least one memory 610, a controller 620, and an image encoder 625 are included. The at least one processor 605, at least one memory 610, the controller 620, and the image encoder 625 are communicatively coupled via the bus 615.

At least one processor 605 may be used to execute instructions stored on at least one memory 610 to implement various features and functions described herein, or additional or alternative features and functions. The at least one processor 605 and at least one memory 610 may be used for a variety of other purposes. In particular, at least one memory 610 may represent an example of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein.

At least one memory 610 may be configured to store data and / or information associated with the image encoder system 600. For example, at least one memory 610 may be configured to store a codec associated with encoding spherical images. For example, the at least one memory may be configured to store code associated with selecting a portion of a frame of the spherical image as a tile to be encoded separately from the encoding of the spherical image. At least one memory 610 may be a shared resource. As described in more detail below, the tile may be a plurality of pixels selected based on the viewer's viewing point of view during playback of a spherical viewer (eg, an HMD). The plurality of pixels may be a block, a plurality of blocks, or a macro block that may include a portion of a spherical image visible to the user. For example, video encoder system 600 may be an element of a larger system (eg, server, personal computer, mobile device, etc.). Thus, at least one memory 610 may be configured to store data and / or information related to other elements (eg, image / video serving, web browsing or wired / wireless communication) within a larger system.

The controller 620 may be configured to generate various control signals and to transmit control signals to various blocks of the image encoder system 600. Controller 620 may be configured to generate control signals to implement the techniques described below. The controller 620 may be configured to control the image encoder 625 to encode an image, an image sequence, an image frame, an image sequence, etc. according to an example embodiment. For example, the controller 620 may generate control signals corresponding to parameters for encoding the spherical image. More details related to the functions and operation of the image encoder 625 and the controller 620 will be described below with reference to at least FIGS. 7A, 4A, 5A and 6-9.

The video encoder 625 may be configured to receive the video stream input 5 and output the compressed (eg, encoded) video bits 10. The video encoder 625 may convert the video stream input 5 into individual video frames. The video stream input 5 may also be an image, so that the compressed (eg, encoded) video bit 10 may also be a compressed image bit. The image encoder 625 may further convert each individual image frame (or image) into a matrix of blocks (hereinafter referred to as blocks). For example, an image frame (or image) may be converted into a 16x16, 16x8, 8x8, 8x4, 4x4, 4x2, 2x2 or similar matrix of blocks each having multiple pixels. Although these example matrices are listed, the example embodiments are not so limited.

The compressed video bit 10 may represent the output of the video encoder system 600. For example, the compressed image bit 10 may represent an encoded image frame (or encoded image). For example, the compressed image bit 10 may be ready for transmission to a receiving device (not shown). For example, the image bits may be sent to a system transceiver (not shown) for transmission to the receiving device.

At least one processor 605 may be configured to execute computer instructions associated with controller 620 and / or image encoder 625. At least one processor 605 may be a shared resource. For example, image encoder system 600 may be an element of a larger system (eg, a mobile device). Thus, at least one processor 605 may be configured to execute computer instructions associated with other elements (eg, image / video serving, web browsing or wired or wireless communication) within a large system.

In the example of FIG. 6B, image decoder system 650 may be at least one computing device, and may virtually represent any computing device configured to perform the methods described herein. As such, image decoder system 650 may include various components that may be used to implement the techniques described herein, or different versions or future versions thereof. By way of example, image decoder system 650 is shown to include at least one processor 655 as well as at least one memory 660 (eg, a computer readable storage medium).

Thus, at least one processor 655 may execute instructions stored on at least one memory 660 to implement the various features and functions described herein, or additional or alternative features and functions. At least one processor 655 and at least one memory 660 may be used for a variety of other purposes. In particular, at least one memory 660 may represent an example of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein. According to example embodiments, video encoder system 600 and video decoder system 650 may be included in the same large system (eg, personal computer, mobile device, etc.). According to an example embodiment, the video decoder system 650 may be configured to implement the inverse or reverse technique described with respect to the video encoder system 600.

At least one memory 660 may be configured to store data and / or information associated with the image decoder system 650. For example, at least one memory 610 may be configured to store a codec associated with decoding the encoded spherical image data. For example, the at least one memory may be configured to store code associated with decoding the encoded tile and the separately encoded spherical image frame and code for replacing the pixels of the decoded spherical image frame with the decoded tile. At least one memory 660 may be a shared resource. For example, video decoder system 650 may be an element of a larger system (eg, a personal computer, mobile device, etc.). Thus, at least one memory 660 may be configured to store data and / or information related to other elements (eg, web browsing or wireless communication) within a larger system.

The controller 670 may be configured to generate various control signals and to pass the control signals to various blocks of the image decoder system 650. The controller 670 may be configured to generate a control signal to implement the image decoding technique described below. The controller 670 may be configured to control the image decoder 675 to decode the image frame according to an example embodiment. The controller 670 may be configured to generate a control signal corresponding to the decoded image. Further details related to the function and operation of the image decoder 675 and controller 670 will be described below.

The video decoder 675 may be configured to receive the compressed (eg, encoded) video bit 10 and output the video stream 5. The image decoder 675 may convert the individual image frames of the compressed image bits 10 into the image stream 5. The compressed (eg, encoded) image bit 10 may also be a compressed image bit, so that the image stream 5 may also be an image.

At least one processor 655 may be configured to execute computer instructions associated with controller 670 and / or image decoder 675. At least one processor 655 may be a shared resource. For example, video decoder system 650 may be an element of a larger system (eg, a personal computer, mobile device, etc.). Thus, at least one processor 655 may be configured to execute computer instructions associated with other elements (eg, web browsing or wireless communication) within a larger system.

7A and 7B show flowcharts for the video encoder 625 shown in FIG. 6A and the video decoder shown in FIG. 6B, respectively, according to one exemplary embodiment. The image encoder 625 (disclosed above) is a spherical-2D representation block 705, a prediction block 710, a transform block 715, a quantization block 720, an entropy encoding block 725, an inverse quantization block 730. 740, inverse transform block 735, reconstruction block 740, loop filter block 745, tile representation block 790, and view frame storage 795. Other structural variations of the video encoder 625 may be used to encode the input video stream 5. As shown in FIG. 7A, the dotted line represents the reconstruction path between some blocks and the solid line represents the forward path between some blocks. Each of the aforementioned blocks (e.g., as shown in Figure 6A) may be executed as software code stored in a memory (e.g., at least one memory 610) associated with the image encoder system; It may be executed by at least one associated processor (eg, at least one processor 605). However, alternative embodiments may be contemplated, such as an image encoder implemented as a special purpose processor. For example, each of the aforementioned blocks (alone and / or in combination) may be an application specific integrated circuit or ASIC. For example, the ASIC may be configured as a transform block 715 and / or a quantization block 720.

Spherical-2D representation block 705 may be configured to map a spherical frame or image to a 2D representation of a spherical frame or image. For example, the spheres can be projected onto surfaces of different shapes (eg, squares, rectangles, cylinders and / or cubes). The projection can be, for example, conformal or semi anisotropic.

Prediction block 710 may be configured to use image frame consistency (eg, pixels that have not changed compared to previously encoded pixels). Predictions can include two types. For example, the prediction may include intra-frame prediction and inter-frame prediction. Intra frame prediction relates to predicting pixel values in a block of a picture neighboring the same picture and relative to previously coded reference samples. In intra-frame prediction, samples are reconstructed within the same frame to reduce residual errors that are coded by transforms (e.g., entropy encoding block 725) and entropy coding (e.g., entropy encoding block 725). It is predicted from pixels. Inter-frame prediction relates to predicting pixel values in blocks of an image relative to data of previously coded images.

The transform block 715 can be configured to transform pixel values from the spatial domain into transform coefficients of the transform domain. The transform coefficients may correspond to a two-dimensional matrix of coefficients that are generally the same size as the original block. That is, there may be as many transform coefficients as the pixels in the original block. However, due to the transformation, some of the transformation coefficients may have a value equal to zero.

The transform block 715 transforms the residual (from the prediction block 710) into, for example, transform coefficients in the frequency domain. Generally, transformations include Karhunen-Loeve Transform (KLT), Discrete Cosine Transform (DCT), Singular Value Decomposition Transform (SVD), and Asymmetric Discrete Sine Transform (ADST).

Quantization block 720 may be configured to reduce the data in each transform coefficient. Quantization may include mapping values in a relatively large range to values in a relatively small range to reduce the amount of data needed to represent the quantized transform coefficients. Quantization block 720 may transform the transform coefficients into discrete quantum values called quantized transform coefficients or quantization levels. For example, quantization block 720 may be configured to add zero to data associated with the transform coefficients. For example, the encoding standard can define 128 quantization levels in a scalar quantization process.

The quantized transform coefficients are then entropy encoded by entropy encoding block 725. The entropy-encoded coefficients are output along with the information needed to decode the block, such as the prediction type used, motion vector, and quantizer values. The compressed image bit 10 may be formatted using various schemes such as run-length encoding (RLE) and zero-run coding.

The reconstruction path shown in FIG. 17A exists to ensure both image encoder 625 and image decoder 675 using the same reference frame to decode compressed image bit 10 (or compressed image bit). . The reconstruction path includes inverse quantizing the quantized transform coefficients in inverse quantization block 730 and inverse transforming the inverse quantized transform coefficients in inverse transform block 735 to produce a differential residual (differential residual). To perform functions similar to those occurring during the decoding process described in more detail below. In reconstruction block 740, the predictive block predicted in the prediction block 710 may be added to the differential residual to generate the reconstructed block. Loop filter 745 may then be applied to the reconstructed block to reduce distortion, such as blocking artifacts.

Tile representation block 790 may be configured to convert an image and / or frame into a plurality of tiles. A tile can be a group of pixels. The tile may be a number of pixels selected based on the view or view point of view. The plurality of pixels may be blocks, plurality of blocks, or macro blocks that may include portions of a spherical image that are visible to (or expected to see) the user. Like tiles, parts of a spherical image can have length and width. Portions of the spherical image can be two-dimensional or substantially two-dimensional. The tiles can have varying sizes (eg, how many parts of the sphere the tiles cover). For example, the size of the tiles may be encoded and streamed based on, for example, how wide the viewer's field of view is, how close to other tiles, and / or how fast the user is turning their head. For example, if the viewer continues to look around, larger, lower quality tiles may be selected. However, if the viewer is focusing on one point of view, a smaller, more detailed tile can be selected.

In one implementation, tile representation block 790 initiates an instruction to cause spherical-2D representation block 705 to generate a tile. In another implementation, tile representation block 790 generates a tile. In either implementation, each tile is encoded separately. In another implementation, tile representation block 790 initiates a command to view frame storage 795 such that view frame storage 795 stores the encoded image and / or video frame as a tile. The tile representation block 790 may initiate a command to the view frame storage 795 to cause the view frame storage 795 to store a tile having information or metadata about the tile. For example, information or metadata about a tile may include an indication of tile location within an image or frame, information related to tile encoding (e.g., resolution, bandwidth and / or 3D-2D projection algorithm), one or more regions of interest and / or Or the like.

According to an example implementation, encoder 625 may encode a portion of a frame, frame, and / or tile multiple times each with a different quality (or quality of service). As such, the aforementioned information or metadata about a tile may include an indication of QoS in which a frame, frame and / or a portion of the tile is encoded.

QoS may be based on a compression algorithm, resolution, transmission rate, and / or encoding scheme. Thus, encoder 625 may use a different compression algorithm and / or encoding scheme for each frame, frame and / or portion of a tile. For example, an encoded tile may have a higher QoS than a frame encoded (associated with the tile) by the encoder 625. As described above, encoder 625 may be configured to encode a 2D representation of a spherical image frame. Thus, a tile (as a view point containing part of a spherical image frame) may be encoded with a higher QoS than the 2D representation of the spherical image frame. The QoS can affect the resolution of the frame as it is decoded. Thus, a tile (as a visible point of view that includes a portion of a spherical image frame) may be encoded such that the tile has a higher resolution of the frame when it is decoded compared to the decoded 2D representation of the spherical image frame. Tile representation block 790 may indicate the QoS to which the tile should be encoded. The tile representation block 790 may select QoS based on whether the frame, frames and / or portions of the tiles are seed regions and / or similar and interested regions that are within the region of interest. The region of interest and the seed region are described in detail below.

The image encoder 625 described above with respect to FIG. 7A includes the block shown. However, example implementations are not so limited. Additional blocks may be added based on the different image encoding schemes and / or techniques used. In addition, each block shown in video encoder 625 described above with respect to FIG. 7A may be an optional block based on the different video encoding and / or technology used.

7B is a schematic block diagram of a decoder 675 configured to decode the compressed image bit 10 (or the compressed image bit). Similar to the reconstruction path of the encoder 625 described previously, the decoder block 675 includes an entropy decoding block 750, an inverse quantization block 755, an inverse transform block 760, a reconstruction block 765, a loop filter block. 770, prediction block 775, deblocking filter block 780, and 2D representation-spherical block 785.

Data elements in the compressed image bit 10 (e.g., using Context Adaptive Binary Arithmetic Decoding) are decoded by entropy decoding block 750 to produce a set of quantized transform coefficients. Can be. Inverse quantization block 755 inverse quantizes the quantized transform coefficients, and inverse transform block 760 inverse transforms (using ADST) the inverse quantized transform coefficients to the same as produced by the reconstruction step of encoder 625. Create a differential residual.

Using the header information decoded from the compressed video bits 10, the decoder 675 may use the prediction block 775 to generate the same prediction block as generated at the encoder 675. The predictive block may be added to the differential residual to produce a reconstruction block by reconstruction block 7750. Loop filter block 770 may be applied to the reconstructed block to reduce blocking artifacts. 780 may be applied to the reconstructed block to reduce blocking distortion, and the result is output as image stream 5.

The 2D representation for spherical block 785 may be configured to map the 2D representation of the spherical frame or image to the spherical frame or image. For example, mapping a 2D representation of a spherical frame or image to a spherical frame or image may be the inverse of the 3D-2D mapping performed by encoder 625.

The image decoder 675 described above with respect to FIG. 7B includes the block shown. However, exemplary embodiments are not limited thereto. Additional blocks may be added based on the different video encoding schemes and / or techniques used. In addition, each of the blocks shown in video decoder 675 described above with respect to FIG. 7B may be an optional block based on different video encoding configurations and / or techniques used.

Encoder 625 and decoder 675 may be configured to encode spherical images and / or images and decode spherical images and / or images, respectively. A spherical image is an image including a plurality of pixels organized into a spherical surface. In other words, the spherical image is a continuous image in all directions. Thus, the viewer of the spherical image can reposition or reorient in any direction (e.g., up, down, left, right, or any combination thereof) and (eg, change her head or eyes). By moving) one part of the image can be viewed continuously.

In an example implementation, the parameters used at and / or by the encoder 625 may be used by other elements of the encoder 405. For example, a motion vector used to encode a 2D representation (eg, as used in prediction) can be used to encode a tile. Further, the prediction block 710, the transform block 715, the quantization block 720, the entropy encoding block 725, the inverse quantization block 730, the inverse transform block 735, the reconstruction block 740, and the loop filter block ( The parameters used and / or determined at 745 may be shared between encoder 625 and encoder 405.

A spherical image frame or portion of an image can be treated as an image. Thus, the portion of the spherical image frame can be transformed (or decomposed) into a CxR matrix of blocks (hereinafter referred to as blocks). For example, the portion of the spherical image frame may be converted into a CxR matrix of 16x16, 16x8, 8x8, 8x4, 4x4, 4x2, 2x2 or similar matrix of blocks each having multiple pixels.

8 illustrates a system 800 in accordance with at least one example embodiment. As shown in FIG. 8, the system 700 includes a controller 620, a controller 670, an image encoder 625, a view frame store 795, and a direction sensor (s) 835. The controller 620 further includes a view position control module 805, a tile control module 810, and a view viewpoint data storage 815. The controller 670 further includes a view position determination module 820, a tile request module 825, and a buffer 830.

According to an example implementation, the direction sensor 835 detects the direction (or change of direction) of the viewer's eye (or head), and the viewing position determination module 820 based on the detected direction, view or point of view Determine the viewpoint, and the tile request module 825 delivers the view, viewpoint or view viewpoint as part of a request for a tile or a plurality of tiles (in addition to the spherical image). According to another example implementation, the orientation sensor 835 detects a direction (or change in orientation) based on the HMD or the image panning direction rendered on the display. For example, a user of the HMD can change the depth of focus. This means that HMD users can change their focus to something close to a distant object (or vice versa), with or without a change in orientation. For example, a user may use a mouse, trackpad, or gesture (eg, a touch sensitive display) to select, move, drag, and expand a portion of a spherical image or image rendered on the display.

The request for the tile may be delivered together with the request for the frame of the spherical image. The request for the tile may be delivered together with the request for the frame of the spherical image. For example, a request for a tile may correspond to a changed view, viewpoint or view viewpoint when it is necessary to replace a previously requested and / or queried tile.

The view position control module 805 receives and processes a request for a tile. For example, the view position control module 805 may determine the position and frame of one or more tiles or a plurality of tiles in a frame within the frame based on the view. The view position control module 805 may then instruct the tile control module 810 to select a tile or a plurality of tiles. Selecting a tile or a plurality of tiles may include passing parameters to image encoder 625. The parameter may be used by the image encoder 625 during encoding of spherical images and / or tiles. Alternatively, selecting a tile or a plurality of tiles may include selecting a tile or a plurality of tiles from view frame storage 795.

Accordingly, the tile control module 810 may be configured to select a tile (or a plurality of tiles) based on the view or the viewpoint or the viewpoint of the user viewing the spherical image. The tile may be a plurality of pixels selected based on the view. The plurality of pixels may be a block, a plurality of blocks, or a macro block that may include a portion of a spherical image visible to the user. Portions of the spherical image can have a length and a width. Portions of the spherical image can be two-dimensional or substantially two-dimensional. The tiles may have varying sizes (eg how much of the sphere the tiles occupy). For example, the size of the tiles may be encoded and streamed based on, for example, how wide the viewer's field of view is and / or how fast the user is turning their head. For example, if the viewer keeps looking around, larger and lower quality tiles may be selected. However, if the viewer is focusing on one point of view, a smaller, more detailed tile may be selected.

Accordingly, the direction sensor 835 may be configured to detect the direction (or change of direction) of the eye (or head) of the viewer (viewer). For example, the orientation sensor 835 may include an accelerometer for detecting movement and a gyroscope for detecting a direction. Alternatively, or in addition, the orientation sensor 835 may include a camera or infrared sensor focused on the viewer's eyes or head to determine the orientation of the viewer's eyes or head. Alternatively, or in addition, the orientation sensor 835 may determine a portion of the spherical image or image rendered on the display to detect the orientation of the spherical image or image. The direction sensor 825 may be configured to communicate the direction so that the view positioning module 820 changes the direction information.

The view position determining module 820 may be configured to determine a view or view point of view (eg, a portion of the spherical image the viewer is currently viewing) in relation to the spherical image. The view, viewpoint or view viewpoint may be determined as a position, a point or a focus on a spherical image. For example, the view may be a latitude and longitude location in the spherical image. The view, viewpoint or view viewpoint may be determined to the side of the cube based on the spherical image. The view (eg, latitude and longitude location or side) may be passed to the view position control module 805 using, for example, the Hypertext Transfer Protocol (HTTP).

The view position control module 805 may be configured to determine a view position (eg, a frame and a position within a frame) of a tile or a plurality of tiles in the spherical image. For example, the view position control module 805 may select a rectangle centered on the view position, point or focus (eg, latitude and longitude position or side). The tile control module 810 can be configured to select a rectangle as a tile or a plurality of tiles. The tile control module 810 instructs the image encoder 625 to encode the selected tile or a plurality of tiles (eg, via parameters or configuration settings) and / or the tile control module 810 stores the view frame. It may be configured to select a tile from unit 795 or a plurality of tiles.

As can be seen, the systems 600 and 650 shown in FIGS. 6A and 6B and the system 800 disclosed in FIG. 8 are generic computer devices 900 and / or general mobile computer devices described below in connection with FIG. It may be implemented as an element and / or extension of the switch 950. Alternatively or additionally, the systems 600 and 650 shown in FIGS. 6A and 6B and / or the system 800 disclosed in FIG. 8 may be a generic computer device 900 and / or a generic mobile computer device 950. ) May be implemented in a system separate from general computer device 900 and / or general mobile computer device 950 having some or all of the features described below.

9 is a schematic block diagram of a computer device and a mobile computer device that may be used to implement the techniques described herein. 9 is an example of a generic computer device 900 and a generic mobile computer device 950, which may be used in conjunction with the techniques described herein. Computing device 900 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, PDAs, servers, blade servers, mainframes, and other suitable computers. Computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices.

The computing device 900 includes a processor 902, a memory 904, a storage device 906, a high speed interface 908 and a low speed bus 914 and storage devices connected to the memory 904 and the fast expansion port 910. A low speed interface 912 coupled to 906. Each component 902, 904, 906, 908, 910 and 912 is interconnected using a variety of buses and may be mounted on a common motherboard or otherwise appropriately. The processor 902 includes instructions stored in memory 904 or storage 906 to display graphical information for a GUI on an external input / output device, such as a display 916 connected to a high speed interface 908. Process the execution command in the computing device 900. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, with multiple memories and types of memory. In addition, multiple computing devices 900 may be connected with each device (eg, a server bank, blade server group, or multiprocessor system) that provides a partition of the required operations.

Memory 904 stores information within computing device 900. In one implementation, the memory 904 is a volatile memory unit. In another implementation, the memory 904 is a nonvolatile memory unit (s). Memory 904 may also be another form of computer readable media, such as a magnetic or optical disk.

The storage device 906 can provide a mass storage capacity for the computing device 900. In one implementation, storage device 906 is an array of devices including a floppy disk device, hard disk device, optical disk device or tape device, flash memory or other similar solid state memory device, or a device in a storage area network or other configuration. Or may be a computer readable medium. The computer program product can be reliably embodied in an information medium. The computer program product may also include instructions that, when executed, perform one or more methods as described above. The information medium is a computer or machine readable medium, such as memory 904, storage 906, or memory on processor 902.

The high speed controller 908 manages bandwidth-intensive operations for the computing device 900, and the low speed controller 912 manages low bandwidth-intensive operations. This function assignment is merely exemplary. In one implementation, high speed controller 908 is a high speed expansion port 910 that can accommodate memory 904, display 916 (eg, via a graphics processor or accelerator), and various expansion cards (not shown). ) Is combined. In this implementation, the low speed controller 912 is coupled to the storage device 906 and the low speed expansion port 914. The low speed expansion ports, which may include various communication ports (eg, USB, Bluetooth, Ethernet, Wireless Ethernet), include one or more input / output devices such as keyboards, pointing devices, scanners, or networking devices such as switches or routers, For example, it can be coupled via a network adapter.

Computing device 900 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 920 or may be implemented multiple times in a group of such servers. It may also be implemented as part of the lab server system 924. It may also be implemented in a personal computer, such as laptop computer 922. Alternatively, components from computing device 900 may be combined with other components within a mobile device. Each of these devices may include one or more of computing devices 900 and 950, and the entire system communicates with each other. It may be composed of a plurality of computing devices (900, 950).

The computing device 950 includes a processor 952, a memory 964, an input / output device such as a display 954, a communication interface 966, and a transceiver 968, among other components. The device 950 may include a storage device such as a micro drive or other device to provide additional storage capacity. Each of the components 950, 952, 964, 954, 966 and 968 are interconnected using a variety of buses, and some components may be mounted on a common motherboard or in other suitable manners.

The processor 952 can execute instructions within the computing device. The processor may be implemented as a chipset of a chip comprising individual and multiple analog and digital processors, including instructions stored in memory 964. The processor may provide coordination of other components of the device 950 such as, for example, control of the user interface, applications executed by the device 950 and wireless communication by the device 950.

The processor 952 may communicate with a user through the display interface 954 coupled via the control interface 958 and the display 956. The display 954 may include, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 956 may include suitable circuitry for driving the display 954 to provide graphics and other information to a user. The control interface 958 can convert commands for receiving commands from a user and submitting to the processor 952. In addition, the external interface 962 can communicate with the processor 952 to enable near field communication of the device 950 with other devices. External interface 962 may be provided, for example, in wired communication in some implementations or for wireless communication in other implementations, and multiple interfaces may also be used. Memory 964 stores information within the computing device.

Memory 964 may be implemented as one or more of computer-readable media or media, volatile memory units or units, and non-volatile memory units or units. Expansion memory 974 may also be provided and connected to device 950 via expansion interface 972, which may include, for example, a Single In Line Memory Module (SIMM) card interface. In particular, the expansion memory 974 may include instructions for performing or supplementing the above-described process, and may also include security information. Thus, for example, expansion memory 974 may be provided as a security module for device 950 and may be programmed with instructions to allow secure use of device 950. In addition, the security application may perform tasks such as placing identification information on the SIMM card through the SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and / or NVRAM memory, as discussed below. In one implementation, a computer program product is specifically embodied in an information medium. The computer program product includes instructions which, when executed, perform one or more methods, such as those described above. The information medium is a computer- or machine-readable medium such as, for example, memory 964, expansion memory 974, or memory on processor 952, which can be received via transceiver 968 or external interface 962. .

Device 950 may communicate wirelessly via communication interface 966, which may include digital signal processing circuitry if desired. The communication interface 966 may provide communication under various modes or protocols, particularly GSM voice calling, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS. Such communication may occur via, for example, a radio frequency transceiver 968. In addition, near field communication may occur, such as using Bluetooth, WiFi, or other transceivers (not shown). In addition, the Global Positioning System (GPS) receiver module 970 may provide additional navigation and location related wireless data to the device 950 that may be suitably used by an application running on the device 950. The audio codec 960 may receive voice information from the user and convert it into usable digital information. Audio codec 960 may likewise generate an audible sound for the user, such as through a speaker in handset of device 950. Such sounds may include sounds from voice telephone calls, may include recorded sounds (eg, voice messages, music files) and may also include sounds generated by an application running on device 950. It may be.

Computing device 950 may be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a cellular telephone 980. It may also be implemented as part of a smartphone 982, personal digital assistant, or other similar mobile device.

Some of the example embodiments described above are disclosed as processes or methods described as flow diagrams. Although the flowchart describes the operations as sequential processes, many of the operations can be performed in parallel, in parallel or concurrently. In addition, the order of the operations may be rearranged. The process may end when the operation is completed but may also have additional steps not included in the figures. The process may correspond to a method, a function, a procedure, a subroutine, a subprogram, or the like.

Some of the foregoing methods disclosed by the flowcharts may be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing a required task may be stored on a machine or computer readable medium, such as a storage medium. The processor (s) may perform the necessary tasks.

The specific structural and functional details disclosed herein are merely representative examples for the purpose of describing exemplary embodiments. However, example embodiments may be embodied in many alternative forms and should not be construed as limited to the embodiments set forth herein.

Although the terms first, second, etc. may be used herein to describe various components, it will be understood that these components should not be limited by these terms. This term is only used to distinguish one element from another. For example, the first component may be referred to as the second component, and similarly, the second component may be referred to as the first component without departing from the scope of the exemplary embodiments. As used herein, the term includes any and all combinations of one or more of the items listed in relation thereto.

When a component is referred to as being connected or connected to another component, it may be appreciated that the component may be directly connected or connected to another component or that an intervening component may exist. In contrast, an intervening component does not exist when the component is referred to as being directly connected or directly connected to another component. Other words used to describe the relationships between components should be interpreted in a similar manner (eg, between vs. directly between, adjacent vs. direct, etc.).

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms a, an, and the are intended to include the plural forms unless the context clearly indicates otherwise. The terminology used herein includes, but is not limited to, specifying and including and / or including the specified features, integers, steps, operations, components and / or presence of components, one or more other features, integers, It may include steps, actions, elements and / or components.

In addition, it should be appreciated that in some alternative implementations, the functions / acts mentioned may deviate from the order recorded in the figures. For example, two figures shown in succession may actually be executed simultaneously or in reverse order depending on the function / operation involved.

Portions of the foregoing exemplary embodiments and corresponding detailed descriptions are provided in connection with algorithms and symbolic representations of operations on data bits in software, or computer memory. These descriptions and representations are the ones by which those of ordinary skill in the art effectively convey the substance of their work to others skilled in the art. The term algorithm, as used herein, implies that there may be a consistent sequence of steps leading to a desired result as is generally used. This step is a step that requires physical manipulation of the physical quantity. Generally, though not necessarily, these quantities take the form of optical, electrical or magnetic signals capable of being stored, transmitted, combined, compared and otherwise manipulated. It is proven to be convenient to refer to this signal as bits, values, elements, symbols, letters, terms, numbers, etc., mainly for common uses.

In the above exemplary embodiment, the reference to the operations and the symbolic representation of the operations (eg in the form of a flowchart), which may be implemented as a program module or a functional process, perform a particular task or implement a particular abstract data type. Existing structural elements include routines, programs, objects, components, data structures, etc. that can be described and / or implemented using existing hardware. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application integrated circuits, field programmable gate arrays (FPGAs) computers, and the like.

However, all of these terms and similar terms are related to the appropriate physical quantities and are merely convenient indications applied to these quantities. Except as otherwise specifically described or otherwise apparent in the discussion, such terms as processing or computing or calculating or determining a representation refer to data expressed in physical or electronic quantities in a register or memory of a computer system or in a memory or register of a computer system. Or operations and processes of a computer system or similar electronic computing device that manipulates and transforms into other information storage devices, transmission devices, or other data similarly represented by physical quantities within a display.

It should also be noted that software implemented aspects of the exemplary embodiments are typically encoded on some form of non-transitory program storage medium or implemented on some type of transmission medium. The program storage medium may be magnetic (eg, floppy disk or hard drive) or optical (eg, compact disc read-only memory, or CD ROM), and may be read-only or random access. Similarly, the transmission medium may be a twisted wire pair, coaxial cable, optical fiber, or other suitable transmission medium known in the art. Example embodiments are not limited by any given aspect of the implementation.

Finally, while the appended claims describe particular combinations of features described herein, the scope of the present disclosure is not limited to the specific combinations that are subsequently claimed, but instead include features or any combination of features. Note that it expands. The embodiments herein are disclosed whether or not this particular combination is specifically listed in the appended claims.

Claims (20)

Storing the view viewpoint as one of a plurality of historical view perspectives for the 3D image based on the historically viewed view viewpoint associated with the three-dimensional (3D) image;
Determining whether a rank value associated with one of the plurality of history view views is greater than a threshold and satisfies a condition for the remainder of the plurality of history view views;
In response to determining that the rank value is greater than a threshold and satisfies the condition, selecting one of the plurality of historical view viewpoints as at least one preferred view viewpoint associated with a 3D image;
Mapping a frame of the 3D image to a two-dimensional (2D) representation;
Encoding a first portion of a 2D representation of the frame corresponding to the at least one preferred view viewpoint with a first quality; And
Encoding a second portion of the 2D representation of the frame with a second quality, wherein the first quality is higher quality compared to the second quality. The method of claim 1,
Storing the encoded first portion of the 2D representation of the frame in a data storage device;
Storing a second portion of the 2D representation of the frame in a data storage device in association with the encoded first portion of the 2D representation of the frame as an encoded profile of the 3D image;
Receiving a request for a frame of a streaming video; And
In response to receiving a request for a frame of the streaming video, streaming at least a portion of the encoded frame of the 3D video from a data storage device as a frame of the streaming video. . The method of claim 1,
Receiving a request for a frame of a streaming video, the request comprising an indication of a user view point of view;
Selecting the encoded first portion of the 2D representation of a frame of 3D image corresponding to a user view viewpoint; And
In response to receiving a request for a frame of the streaming image, select the encoded first portion of the 2D representation of the frame of the 3D image and the encoded second portion of the 2D representation of the frame of the 3D image. And streaming as a frame of the streaming video. The method of claim 1,
Receiving a request for a frame of a streaming video, the request comprising an indication of a user view viewpoint associated with the 3D video;
Determining whether a user view viewpoint is stored in the view viewpoint data storage;
In response to determining that the user view viewpoint is stored in the view viewpoint data storage, incrementing a counter associated with the user view viewpoint; And
In response to determining that the user view viewpoint is stored in the view viewpoint data storage, adding the user view viewpoint to the view viewpoint data storage and setting a counter associated with the user view viewpoint to one. . The method of claim 1,
Encoding the second portion of the 2D representation of the frame of the 3D image comprises using at least one first QoS parameter in a first pass encoding operation,
Encoding the first portion of the 2D representation of a frame of a 3D image comprises using at least one QoS parameter in a second pass encoding operation. The method of claim 1,
And determining at least one preferred view viewpoint associated with the 3D image is based on at least one of a history viewed reference point and a history viewed view viewpoint. The method of claim 1,
The at least one preferred view viewpoint associated with the 3D image is based on at least one of a viewer direction of the 3D image, a viewer position of the 3D image, a viewer point of the 3D image, and a viewer imaging point of the 3D image. The method of claim 1,
Determining at least one preferred view viewpoint associated with the 3D image is based on a default view viewpoint,
The default view point is
User characteristics of the display device,
The characteristics of the group related to the user of the display,
Director's cut, and
And at least one of the characteristics of the 3D image. The method of claim 1,
Iteratively encoding at least one portion of the second portion of the 2D representation of the frame of the 3D image with a first quality; And
Streaming said at least one portion of said second portion of said 2D representation of a frame of 3D video. A memory configured to store a plurality of historical view viewpoints for the 3D image based on historically viewed view viewpoints associated with three-dimensional (3D) images, wherein the plurality of historical view viewpoints are associated with a plurality of users of the 3D images Become;
As a controller, the controller,
Determine whether a rank value associated with one of the plurality of history view views is greater than a threshold and satisfies a condition for the remainder of the plurality of history view views; And
In response to determining that the rank value is greater than a threshold and satisfies the condition, select one of the plurality of historical view viewpoints as at least one preferred view viewpoint associated with a 3D image;
An encoder, wherein the encoder,
Encode a first portion of a 2D representation of the 3D image frame corresponding to the at least one preferred view viewpoint with a first quality; And
And encoding a second portion of the 2D representation of the frame with a second quality, wherein the first quality is higher quality compared to the second quality. The method of claim 10, wherein the controller,
Store the encoded first portion of the 2D representation of the frame in a data storage device,
Storing a second portion of the 2D representation of the frame in relation to the encoded first portion of the 2D representation of the frame as an encoded profile of a 3D image in a data storage device,
By receiving a request for a frame of a streaming video,
And in response to receiving the request for the frame of the streaming video, stream at least a portion of the encoded frame of the 3D video as a frame of the streaming video from a data storage device. The method of claim 10, wherein the controller,
Receiving a request for a frame of the streaming video, the request including an indication of a user view point of view,
Select the encoded first portion of the 2D representation of the frame of the 3D image corresponding to a user view point of view,
In response to receiving a request for a frame of the streaming image, select the encoded first portion of the 2D representation of the frame of the 3D image and the encoded second portion of the 2D representation of the frame of the 3D image. And further configured to stream as a frame of the streaming video. The method of claim 10, wherein the controller,
Receiving a request for a frame of a streaming video including an indication of a user's view point in relation to the 3D image,
Determine if a user view point is stored in the view point data storage,
If it is determined that the user view viewpoint is stored in the view viewpoint data storage, increment the counter associated with the user view viewpoint,
And if it is determined that the user view viewpoint is not stored in the view viewpoint data storage, the streaming server is further configured to add the user view viewpoint to the view viewpoint data storage and set a counter associated with the user view viewpoint to one. The method of claim 10,
Encoding the second portion of the 2D representation of a frame of 3D image comprises using at least one first QoS parameter in a first pass encoding operation,
Encoding the first portion of the 2D representation of a frame of 3D video comprises using at least one second QoS parameter in a second pass encoding operation. delete The method of claim 10,
The at least one preferred view viewpoint associated with the 3D image is based on at least one of a viewer direction of the 3D image, a viewer position of the 3D image, a viewer point of the 3D image, and a viewer imaging point of the 3D image. The method of claim 10,
Determining at least one preferred view viewpoint associated with the 3D image is based on a default view viewpoint,
The default view point is
Characteristics of display device users,
Characteristics of groups related to display device users,
Director's cut, and
Streaming server, characterized in that based on at least one of the characteristics of the 3D image. The method of claim 10, wherein the controller,
Iteratively encoding at least one portion of the second portion of the 2D representation of the frame of the 3D image with a first quality,
And the at least one portion of the second portion of the 2D representation of a frame of 3D video. Receiving a request for a frame of a streaming image, the request includes an indication of a user view viewpoint associated with a 33D (3D) image, wherein the user view viewpoint is a plurality of historical views of the 33D (3D) image One of the historical view perspectives, representing a focal point of the user on a portion of the 33D (3D) image;
Determining whether the user view viewpoint is stored in a view viewpoint data storage;
In response to determining that the user view viewpoint is stored in the view viewpoint data storage:
Increase a rank value associated with the user view point of view,
Determine whether the rank value is greater than a threshold and satisfies a condition for the remainder of the plurality of historical view viewpoints, and
In response to determining that the rank value is greater than the threshold and satisfies the condition, selecting the user view viewpoint as at least one preferred view viewpoint associated with the three-dimensional (3D) image; And
In response to determining that the user view viewpoint is not stored in the view viewpoint data storage, adding the user viewpoint to the view viewpoint data storage and setting a rank value associated with the user view viewpoint to one. How to. The method of claim 19,
Mapping a frame of the 3D image to a two-dimensional (2D) representation;
Encoding a first portion of a 2D representation of the frame corresponding to the at least one preferred view viewpoint with a first quality; And
Encoding a second portion of the 2D representation of the frame with a second quality, wherein the first quality is higher quality than the second quality.

Images