JP6672327B2 - Method and apparatus for reducing spherical video bandwidth to a user headset - Google Patents

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application filed May 27, 2015, entitled "Method and Apparatus to Reduce Spherical Video Bandwidth to User Headset". Claim the benefit of the entitled U.S. Application Serial No. 62 / 167,121. That application is incorporated herein by reference in its entirety.

FIELD Embodiments relate to streaming spherical video.

Background Streaming spherical video (or other three-dimensional video) can consume a significant amount of system resources. For example, an encoded spherical video may include a large number of bits for transmission, which may consume a significant amount of bandwidth and processing and memory associated with the encoder and decoder.

Overview Exemplary embodiments describe systems and methods that optimize streaming video, streaming 3D video, and / or streaming spherical video.

  In one general aspect, a method includes determining at least one preferred view perspective associated with a three-dimensional (3D) video and identifying a first portion of the 3D video corresponding to the at least one preferred view perspective. Encoding at a first quality and encoding a second portion of the 3D video at a second quality, wherein the first quality is higher quality than the second quality.

  In another general aspect, a server and / or a streaming server includes a controller configured to determine at least one preferred view perspective associated with three-dimensional (3D) video, and an encoder, wherein the encoder comprises: The first portion of the 3D video corresponding to the at least one preferred view perspective is configured with a first quality and the second portion of the 3D video is configured with a second quality. The quality is higher quality as compared to the second quality.

  In yet another general aspect, a method includes receiving a request for streaming video, wherein the request includes displaying a user view perspective associated with the three-dimensional (3D) video, and the method further comprises: Determining whether the perspective is stored in the view perspective data store; and, if determining that the user view perspective is stored in the view perspective data store, incrementing a ranking value associated with the user view perspective. If it is determined that the user view perspective is not stored in the view perspective data store, the user view perspective is added to the view perspective data store and the user view is stored. And a step of setting a ranking value associated with the perspective to 1.

  Implementations may include one or more of the following features. For example, the method (or server implementation) includes storing a first portion of the 3D video in a data store, storing a second portion of the 3D video in a data store, The method may further include receiving the request and streaming the first portion of the 3D video and the second portion of the 3D video from the data store as streaming video. The method (or server implementation) further comprises receiving a request for streaming video, the request including displaying a user view perspective, the method further comprising: displaying the 3D video corresponding to the user view perspective; Selecting may be as a coded first portion of the 3D video, and streaming the selected first portion of the 3D video and a second portion of the 3D video as streaming video.

  The method (or implementation on a server) further comprises receiving a request for streaming video, wherein the request includes displaying a user view perspective associated with the 3D video, wherein the method further comprises: Determining whether the view is stored in the view perspective data store; and, if determining that the user view perspective is stored in the view perspective data store, incrementing a counter associated with the user view perspective; and If the user view perspective is not stored in the view perspective data store, the user view perspective is added to the view perspective data store, and the user view The counter associated with Pekutibu may include a step of setting to 1. The method (or server implementation) includes the step of encoding the second portion of the 3D video using at least one first QoS (Quality of Service) parameter in a first pass encoding operation. And encoding the first portion of the 3D video may include using at least one second Quality of Service (QoS) parameter in a second pass encoding operation. .

  For example, determining at least one preferred view perspective associated with the 3D video is based on at least one of a historically viewed reference point and a previously viewed view perspective. . The at least one preferred view perspective associated with the 3D video is a 3D video viewer orientation, a 3D video viewer location, a 3D video viewer point, and a 3D video viewer focus. Based on at least one. Determining at least one preferred view perspective associated with the 3D video is based on a default view perspective, wherein the default view perspective includes characteristics of a user of the display device, characteristics of a group associated with the user of the display device, Director's cut and / or 3D video characteristics. For example, the method (or server implementation) includes repeatedly encoding at least a portion of the second portion of the 3D video at a first quality and at least a portion of the second portion of the 3D video. Streaming.

Example embodiments will be more fully understood from the detailed description provided herein below and the accompanying drawings. In the figures, the same elements are denoted by the same reference numerals, and they are given by way of example only and thus are not limitations of the exemplary embodiments.
FIG. 4 illustrates a two-dimensional (2D) representation of a sphere according to at least one example embodiment. FIG. 4 is a diagram illustrating an expanded cylindrical representation of a 2D representation of a sphere as a 2D rectangular representation. FIG. 3 illustrates a method for encoding streaming spherical video, according to at least one example embodiment; FIG. 3 illustrates a method for encoding streaming spherical video, according to at least one example embodiment; FIG. 3 illustrates a method for encoding streaming spherical video, according to at least one example embodiment; FIG. 3 illustrates a method for encoding streaming spherical video, according to at least one example embodiment; FIG. 2 illustrates a video encoder system according to at least one example embodiment. FIG. 4 illustrates a video decoder system according to at least one example embodiment. FIG. 4 illustrates a flow diagram for a video encoder system according to at least one example embodiment; FIG. 4 illustrates a flow diagram for a video decoder system according to at least one example embodiment; FIG. 2 illustrates a system according to at least one example embodiment. FIG. 2 is a schematic block diagram of a computing device and a mobile computing device that may be used to implement the techniques described herein.

  It is noted that these figures are intended to illustrate general features of the methods, structures and / or materials utilized in certain exemplary embodiments, and are intended to supplement the description provided below. Have been. However, these drawings are not to scale and may not accurately reflect the structural or performance characteristics of any given embodiment itself, and may not define or define the characteristics encompassed by the exemplary embodiment. It should not be construed as limiting. For example, the positioning of structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in the various drawings is intended to indicate the presence of similar or identical elements or features.

DETAILED DESCRIPTION OF EMBODIMENTS While exemplary embodiments may include various modifications and alternatives, the embodiments are shown by way of example in the drawings and will now be described in detail. However, there is no intention to limit the exemplary embodiments to the particular forms disclosed, but rather the exemplary embodiments are intended to cover all modifications, equivalents, and alternatives falling within the scope of the appended claims. Should be understood. Like numbers refer to like elements throughout the description of the figures.

  An exemplary embodiment may include streaming video, 3D video streaming, spherical video (and / or other three-dimensional video) streaming, as well as preferentially viewed portions of spherical video (by video viewers). For example, systems and methods configured to optimize based on director's cuts, historical viewings, etc. are described. For example, a director's cut may be a view perspective as selected by the video director (director) or producer. The director's cut may be based on the camera's (multi-camera) view selected or seen when the video director or producer recorded the video.

  The spherical video, the frames of the spherical video, and / or the spherical image may have a perspective. For example, the spherical image may be an image of the earth. The interior perspective may be a view looking outward from the center of the earth. Alternatively, the internal perspective may be looking at space on Earth. The external perspective may be a view looking down from space towards the earth. As another example, the perspective may be based on a portion of the image that is visible. In other words, the visible perspective may be what the viewer can see. The visible perspective may be part of a spherical image in front of the viewer. For example, when viewed from an interior perspective, a viewer may be lying on the ground (eg, the earth) and looking at the universe. Viewers may see the moon, sun, or certain stars in the image. However, although the ground on which the viewer is lying is included in the spherical image, the ground is outside the current visible perspective. In this example, when the viewer turns his or her head, the ground will be included in the surrounding visible perspective. If the viewer is prone, the ground will be in the visible perspective, but the moon, sun, or stars will not be in the visible perspective.

  The visible perspective from the external perspective may be a portion of the spherical image that is not obstructed (eg, by another portion of the image) and / or a portion of the spherical image that is not curved until invisible. By moving (eg, rotating) the spherical image and / or by moving the spherical image, another portion of the spherical image may be brought from the external perspective into the visible perspective. Thus, a visible perspective is a portion of a spherical image that is within the visible range of a viewer of the spherical image.

  The spherical image is an image that does not change with time. For example, a spherical image from an interior perspective, such as about the earth, may show the moon and stars in one location. On the other hand, spherical videos (or sequences of images) may change over time. For example, a spherical video from an internal perspective, such as about the earth, may show moving moons and stars (eg, due to the Earth's rotation) and / or contrails across the image (eg, sky).

  FIG. 1A is a two-dimensional (2D) representation of a sphere. As shown in FIG. 1A, a sphere 100 (eg, as a spherical image or frame of a spherical video) shows the orientation of an inner perspective 105, 110, an outer perspective 115, and a visible perspective 120, 125, 130. The visible perspective 120 may be part of a spherical image as seen from the interior perspective 110. The visible perspective 120 may be a part of the sphere 100 as seen from the interior perspective 105. The visible perspective 125 may be a part of the sphere 100 as seen from the external perspective 115.

  FIG. 1B is a diagram illustrating an expanded cylindrical representation 150 of the 2D representation of the sphere 100 as a 2D rectangular representation. The equirectangular projection of the image, shown as an expanded cylinder representation 150, appears as an expanded image as the image advances vertically (up and down as shown in FIG. 1B) from the center line between points A and B. obtain. The 2D rectangular representation can be decomposed as a CxR matrix of NxN blocks. For example, as shown in FIG. 1B, the illustrated expanded cylindrical representation 150 is a 30 × 16 matrix of N × N blocks. However, other C × R dimensions are within the scope of this disclosure. The blocks may be blocks (or blocks of pixels) such as 2 × 2, 2 × 4, 4 × 4, 4 × 8, 8 × 8, 8 × 16, 16 × 16, and the like.

  A spherical image is an image that is continuous in all directions. Therefore, if the spherical image is decomposed into a plurality of blocks, the plurality of blocks will be close to each other throughout the spherical image. In other words, there are no edges or boundaries as in 2D images. In an exemplary implementation, adjacent end blocks may be adjacent to a boundary of the 2D representation. In addition, the adjacent end block may be a block adjacent to a block on the boundary of the 2D representation. For example, adjacent end blocks are associated with two or more boundaries of a two-dimensional representation. In other words, because the spherical image is an image that is continuous in all directions, adjacent edges may be associated with upper and lower boundaries (eg, of a row of blocks) in the image or frame, and / or It may be associated with left and right boundaries (eg, of a row of blocks) in an image or frame.

  For example, if equirectangular projection is used, the adjacent end block may be the block at the other end of the column or row. For example, as shown in FIG. 1B, blocks 160 and 170 may be their respective (per column) adjacent end blocks. Further, the blocks 180 and 185 may be adjacent end blocks (for each column). Further, the blocks 165 and 175 may be respective adjacent (row-by-row) end blocks. View perspective 192 may include at least one block (and / or may overlap with at least one block). A block may be encoded as a region of an image, a region of a frame, a part or subset of an image or a frame, a group of blocks, and the like. Hereinafter, this group of blocks may be referred to as a tile or a group of tiles. For example, in FIG. 1B, tiles 190 and 195 are illustrated as a group of four blocks. Tiles 195 are shown as being in view perspective 192.

  In an exemplary embodiment, in addition to streaming the frames of the encoded spherical video, as a selected tile (or group of tiles) based on at least one reference point frequently viewed by a viewer. The view perspective (eg, at least one reference point or view perspective seen so far) is encoded, for example, with higher quality (eg, higher resolution and / or less distortion) and the encoded spherical video It can be streamed with (or as part of) the frame. Thus, during playback, the viewer can see the decoded tiles (with higher quality) while the entire spherical video is being played, and the viewer's view perspective is reduced to the views frequently viewed by the viewer. The entire spherical video is still available, even if turned into a perspective. The viewer can also change the viewing position or switch to another view perspective. If the other view perspective is included in at least one reference point frequently viewed by the viewer, the played video is (e.g., one of the at least one reference point frequently viewed by the viewer). It may be of higher quality (eg, higher resolution) compared to some other (not one) view perspective. One advantage of encoding and streaming only selected portions or subsets of images or frames with higher quality is that the entire spherical video does not necessarily have to be encoded, streamed, and decoded with higher quality. Has the advantage that the selected image or frame of the spherical video can be decoded and played back with higher quality, so that the bandwidth usage and the processing and memory resources associated with the encoder and decoder To increase efficiency.

  In a head-mounted display (HMD), the viewer can visually sense through the use of left (eg, left-eye) and right (eg, right-eye) displays that project a perceived three-dimensional (3D) video or image. Experience virtual reality. According to an exemplary embodiment, a spherical (eg, 3D) video or image is stored on a server. Video or images may be encoded and streamed from the server to the HMD. A spherical video or image may be encoded as a left image and a right image. The left and right images are packaged (eg, in data packets) with metadata about the left and right images. The left and right images are then decoded and displayed by a left (eg, left eye) display and a right (eg, right eye) display.

  The systems and methods described herein are applicable to both left and right images, and throughout this disclosure, depending on the use case, images, frames, parts of images, parts of frames, tiles, etc. It is called. In other words, the encoded data communicated from the server (eg, a streaming server) to a user device (eg, an HMD) and then decoded for display is a left and / or right image associated with the 3D video or image. possible.

  2 to 5 are flowcharts of a method according to an exemplary embodiment. The steps described with respect to FIGS. 2-5 may include a memory associated with the device (eg, as shown in FIGS. 6A, 6B, 7A, 7B, and 8 (described below) (eg, at least It may be performed by execution of software code stored in one memory 610) and executed by at least one processor (eg, at least one processor 605) associated with the device. However, alternative embodiments are conceivable, such as a system embodied as a special purpose processor. Although the steps described below are described as being performed by a processor, these steps are not necessarily performed by the same processor. In other words, at least one processor may perform the steps described below with respect to FIGS.

  FIG. 2 illustrates a method for storing a previous view perspective, where "historical" refers to a view perspective previously requested by a user. For example, FIG. 2 may show the construction of a database of view perspectives commonly found in spherical video streams. As shown in FIG. 2, in step S205, a display of a view perspective is received. For example, a tile may be requested by a device that includes a decoder. The tile request may include information based on a perspective or view perspective regarding the viewer's orientation, position, point, or focus on the spherical video. The perspective or view perspective may be a user view perspective, ie, a view perspective of a user of the HMD. For example, a view perspective (eg, a user view perspective) may be a latitude and longitude position on a spherical video (eg, as an internal or external perspective). A view, perspective, or view perspective may be determined as a cube edge based on the spherical video. The display of the view perspective may also include spherical video information. In an exemplary implementation, the display of the view perspective may include information (eg, a frame sequence) about a frame associated with the view perspective. For example, views (eg, latitude and longitude locations, or sides) communicate from a user device (including a controller associated with an HMD) to a streaming server using, for example, the Hypertext Transfer Protocol (HTTP). Can be done.

  In step S210, it is determined whether a view perspective (eg, a user view perspective) is stored in a view perspective data store. For example, a data store (eg, view perspective data store 815) may be queried or filtered based on information associated with a view perspective or a user view perspective. For example, the data store may be queried or filtered based on the latitude and longitude location on the spherical video of the view perspective and the timestamp in the spherical video where the view perspective was viewed. The time stamp may be a time and / or time range associated with playing the spherical video. The query or filter may determine spatial proximity (eg, how close the current view perspective is to a given stored view perspective) and / or temporal proximity (eg, the current timestamp may be stored at a given stored timestamp). Based on how close the timestamp is). If the query or filter returns results, the view perspective is stored in the data store. If no results are returned, the view perspective has not been stored in the data store. If the view perspective is stored in the view perspective data store, in step S215, the process continues to step S220. If not, the process continues to step S225.

  In step S220, a counter or ranking (or ranking value) associated with the received view perspective is incremented. For example, the data store may include a data table that includes a previous view perspective (eg, the data store may be a database that includes multiple data tables). The data table may be a keyed (eg, unique to each) view perspective. The data table may include identification information of the view perspective, information associated with the view perspective, and a counter indicating how many times the view perspective has been requested. The counter may be incremented each time a view perspective is requested. The data stored in the data table may be anonymized. In other words, the data may be stored such that the user, device, session, etc. are not mentioned (or there is no identifying information of the user, device, session, etc.). As such, the data stored in the data tables cannot be distinguished based on the video user or viewer. In an exemplary implementation, data stored in the data tables may be categorized based on a user without identifying the user. For example, the data may include the user's age, age group, gender, type or role (eg, musician or audience), and the like.

  In step S225, the view perspective is added to the view perspective data store. For example, the identification information of the view perspective, the information associated with the view perspective, and the counter (or ranking value) set to 1 may be stored in a data table including the past view perspectives.

  In an exemplary embodiment, tiles associated with at least one preferred view perspective may be encoded with higher QoS. QoS can be an implementation of the above-mentioned quality (eg, an encoder variable input that defines the quality). For example, an encoder (eg, video encoder 625) can individually encode tiles associated with 3D video. Tiles associated with at least one preferred view perspective may be encoded with a higher QoS than tiles associated with the rest of the 3D video. In an exemplary implementation, the 3D video uses a first QoS parameter (eg, in a first pass) or at least one first QoS parameter used in a first encoding pass. Can be encoded. In addition, the tile associated with the at least one preferred view perspective may include a second QoS parameter (eg, in a second pass) or at least one second QoS parameter used in a second encoding pass. May be encoded. In this example implementation, the second QoS is a higher QoS than the first QoS. In another example implementation, the 3D video may be encoded as multiple tiles representing the 3D video. Tiles associated with at least one preferred view perspective may be encoded using the second QoS parameter. The remaining tiles may be encoded using the first QoS parameter.

  In alternative implementations (and / or additional implementations), an encoder is used to generate a tile associated with at least one preferred view perspective to generate a remaining 2D representation of the 3D video frame. The projection can be performed using a different projection technique or algorithm. Some projections may have distortion in certain areas of the frame. Thus, projecting tiles differently than spherical frames can improve the quality of the final image and / or use pixels more efficiently (e.g., use less computational or user The burden on the eyes is reduced). In one example implementation, the spherical image can be rotated before projecting the tiles to direct the tiles to locations where distortion is minimal based on the projection algorithm. In another example implementation, the tile may use (and / or modify) a projection algorithm based on the location of the tile. For example, projecting a spherical video frame into a 2D representation can use an equirectangular projection, while projecting a spherical video frame onto a representation that includes a portion to be selected as a tile can use a cubic projection.

  FIG. 3 shows a method for streaming 3D video. FIG. 3 illustrates a scenario where streaming 3D video is encoded on demand during a live streaming event or the like. As shown in FIG. 3, in step S305, a request to stream 3D video is received. For example, 3D video, 3D video portions, or tiles available for streaming may be requested by a device including a decoder (eg, via user interaction with a media application). The request may include information based on a perspective or view perspective regarding the orientation, position, point, or focus of the viewer on the spherical video. Information based on a perspective or view perspective may be based on a current orientation or a default (eg, initialized) orientation. The default orientation may be, for example, director's cut for 3D video.

  At step S310, at least one preferred view perspective is determined. For example, a data store (eg, view perspective data store 815) may be queried or filtered based on information associated with the view perspective. The data store may be queried or filtered based on the latitude and longitude position on the spherical video in the view perspective. In an exemplary implementation, at least one preferred view perspective may be based on a previous view perspective. As such, the data store may include a data table that includes the previous view perspective. Preferences may be displayed depending on how many times the view perspective has been requested. Thus, a query or filter may include removing results that are less than a threshold counter value. In other words, the parameters set for the data table query including the previous view perspective may include values for counters or rankings. Here, the result of the query must be above the threshold for the counter. The results of a query of the data table containing the previous view perspective may be set as at least one preferred view perspective.

  In addition, a default preferred view perspective (or multiple such view perspectives) may be associated with the 3D video. Default preferred view perspectives may be director's cuts, points of interest (eg, horizon, moving objects, priority objects), and so on. For example, the purpose of some games may be to destroy objects (eg, buildings or vehicles). This object may be labeled as a priority object. The view perspective that includes the priority object may be displayed as a preferred view perspective. A default preferred view perspective may be included in addition to, or instead of, a previous view perspective. The default orientation can be, for example, the first set of preferred view perspectives based on, for example, an automatic computer vision algorithm (eg, because there is no previous data if the video was first uploaded). The vision algorithm can be used for video preferred, with motion or complex details, or nearby stereo objects to infer what might be interesting, and / or other features that were present in the preferred view of the video. The view perspective portion may be determined.

  Other factors can be used in determining at least one preferred view perspective. For example, the at least one preferred view perspective may be a previous view perspective that is within range of the current view perspective (eg, close to the current view perspective). For example, at least one preferred view perspective is one that is within (eg, has approached) the previous view perspective of the current user or of the group (type or category) to which the current user belongs. View perspective. In other words, the at least one preferred view perspective may include a view perspective (or tile) that is close in distance and / or close in time to a stored previous view perspective. The default preferred view perspective may be stored in the data store 815 containing the previous view perspective, or in a separate (eg, additional) data store not shown.

  In step S315, the 3D video is encoded using at least one encoding parameter based on at least one preferred view perspective. For example, the 3D video (or a portion thereof) may be encoded such that a portion including at least one preferred view perspective is encoded differently than the rest of the 3D video. As such, portions that include at least one preferred view perspective may be encoded with higher QoS than the rest of the 3D video. As a result, when rendered on the HMD, the portion containing at least one preferred view perspective may have a higher resolution than the rest of the 3D video.

  In step S320, the encoded 3D video is streamed. For example, a tile may be included in a packet for transmission. The packet may include the compressed video bits 10A. The packet may include an encoded 2D representation of the spherical video frame and the encoded tile (or tiles). The packet may include a transmission header. The header may include information indicating, among other things, the mode or scheme usage in intra-frame encoding by the encoder. The header may include information indicating parameters used to convert the frames of the spherical video frame into a 2D rectangular representation. The header may include information indicating an encoded 2D rectangular representation and parameters used to obtain the QoS of the encoded tile. As described above, the QoS of tiles associated with at least one preferred view perspective may be different (eg, higher) than tiles not associated with at least one preferred view perspective.

  Streaming 3D video can be achieved through the use of a priority stage. For example, in a first priority stage, video data encoded at a low (or lowest reference) QoS may be streamed. This allows the user of the HMD to start a virtual reality experience. The higher QoS video may then be streamed to the HMD to replace previously streamed video data encoded with a lower (or lowest reference) QoS (eg, data stored in buffer 830). . As an example, in a second stage, higher quality video or image data may be streamed based on the current view perspective. In the next stage, higher QoS video or image data may be streamed based on one or more preferred view perspectives. This may continue until the HMD buffer contains substantially only high QoS video or image data. In addition, this gradual streaming may loop with video or image data with progressively higher QoS. In other words, after the first iteration, the HMD includes video or image data encoded with a first QoS, and after the second iteration, the HMD includes video or image encoded with a second QoS. It includes image data, and after a third iteration, the HMD contains video or image data encoded with a third QoS. In an exemplary implementation, the second QoS is higher than the first QoS, and the third QoS is higher than the second QoS.

  The encoder 625 may operate offline as part of a set-up procedure to make the spherical video available for streaming. Each of the plurality of tiles may be stored in view frame storage 795. Each of the plurality of tiles may be indexed such that each of the plurality of tiles may be stored with reference to a frame (eg, time-dependent) and with reference to a view (eg, view-dependent). Thus, each of the multiple tiles is dependent on time and view, perspective, or view perspective, and may be recalled based on time and view dependencies.

  Thus, in an exemplary implementation, encoder 625 may be configured to perform a loop in which a frame is selected and a portion of the frame is selected as a tile based on a view perspective. The tiles are then encoded and stored. The loop continues to cycle through multiple view perspectives. If the desired number of view perspectives are saved as tiles, e.g., 5 degrees centered on the vertical line and 5 degrees centered on the horizontal line of the spherical image, a new frame is selected and the process proceeds with all frames of the spherical video. Until it has the desired number of tiles reserved for them. In an exemplary embodiment, tiles associated with at least one preferred view perspective may be encoded with a higher QoS than tiles that are not tiles associated with at least one preferred view perspective. This is just one example implementation for encoding and storing tiles.

  FIG. 4 shows a method for storing encoded 3D video. FIG. 4 illustrates a scenario where streaming 3D video is pre-encoded and stored for future streaming. As shown in FIG. 4, at step S405, at least one preferred view perspective for the 3D video is determined. For example, a data store (eg, view perspective data store 815) may be queried or filtered based on information associated with the view perspective. The data store may be queried or filtered based on the latitude and longitude position on the spherical video in the view perspective. In an exemplary implementation, at least one preferred view perspective may be based on a previous view perspective. Therefore, the data table includes the previous view perspective. Preferences may be displayed depending on how many times the view perspective has been requested. Thus, a query or filter may include removing results that are less than a threshold counter value. In other words, the parameters set for the query of the data table containing the previous view perspective may include the value for the counter. Here, the result of the query must be above the threshold for the counter. The results of a query of the data table containing the previous view perspective may be set as at least one preferred view perspective.

  In addition, a default preferred view perspective (or multiple such view perspectives) may be associated with the 3D video. Default preferred view perspectives may be director's cuts, points of interest (eg, horizon, moving objects, priority objects), and so on. For example, the purpose of some games may be to destroy objects (eg, buildings or vehicles). This object may be labeled as a priority object. The view perspective that includes the priority object may be displayed as a preferred view perspective. A default preferred view perspective may be included in addition to, or instead of, a previous view perspective. Other factors can be used in determining at least one preferred view perspective. For example, the at least one preferred view perspective may be a previous view perspective that is within range of the current view perspective (eg, close to the current view perspective). For example, at least one preferred view perspective is one that is within (eg, has approached) the previous view perspective of the current user or of the group (type or category) to which the current user belongs. View perspective. The default preferred view perspective may be stored in the data store containing the previous view perspective, or in a separate (eg, additional) data table.

  In step S410, the 3D video is encoded using at least one encoding parameter based on at least one preferred view perspective. For example, a frame of the 3D video may be selected and a portion of the frame may be selected as a tile based on a view perspective. The tile is then encoded. In an exemplary embodiment, tiles associated with at least one preferred view perspective may be encoded with a higher QoS. Tiles associated with at least one preferred view perspective may be encoded with a higher QoS than tiles associated with the rest of the 3D video.

  In alternative implementations (and / or additional implementations), an encoder is used to generate a tile associated with at least one preferred view perspective to generate a remaining 2D representation of the 3D video frame. The projection can be performed using a different projection technique or algorithm. Some projections may have distortion in certain areas of the frame. Thus, projecting tiles differently than spherical frames can improve the quality of the final image and / or use pixels more efficiently. In one example implementation, the spherical image can be rotated before projecting the tiles to direct the tiles to locations where distortion is minimal based on the projection algorithm. In another example implementation, the tile may use (and / or modify) a projection algorithm based on the location of the tile. For example, projecting a spherical video frame into a 2D representation can use an equirectangular projection, while projecting a spherical video frame onto a representation that includes a portion to be selected as a tile can use a cubic projection.

  In step S415, the encoded 3D video is stored. For example, each of the plurality of tiles may be stored in view frame storage 795. Each of the plurality of tiles associated with the 3D video is indexed such that each of the plurality of tiles can be stored with reference to a frame (eg, time-dependent) and with reference to a view (eg, view-dependent). It may be attached. Thus, each of the multiple tiles is dependent on time and view, perspective, or view perspective, and may be recalled based on time and view dependencies.

  In an exemplary implementation, 3D video (eg, tiles associated therewith) may be encoded and stored using variable encoding parameters. Thus, 3D video may be stored in different encoding states. These states may change based on QoS. For example, 3D video may be stored as multiple tiles each encoded with the same QoS. For example, 3D video may be stored as multiple tiles, each encoded with a different QoS. For example, 3D video may be stored as multiple tiles partially coded with QoS based on at least one preferred view perspective coded.

  FIG. 5 shows a method for determining a preferred view perspective for 3D video. The preferred view perspective for 3D video may be in addition to the preferred view perspective based on previous viewing of the 3D video. As shown in FIG. 6, at step S505, at least one default view perspective is determined. For example, a default preferred view perspective may be stored in a data table included in a data store (eg, view perspective data store 815). The data store may be queried or filtered based on the default display for 3D video. If the query or filter returns results, the 3D video will have an associated default view perspective. If no result is returned, the 3D video does not have a default view perspective associated with it. Default preferred view perspectives may be director's cuts, points of interest (eg, horizon, moving objects, priority objects), and so on. For example, the purpose of some games may be to destroy objects (eg, buildings or vehicles). This object may be labeled as a priority object. The view perspective that includes the priority object may be displayed as a preferred view perspective.

  In step S510, at least one view perspective based on the user's characteristics / preferences / categories is determined. For example, a user of the HMD may have characteristics based on previous use of the HMD. These characteristics may be based on statistical viewing preferences (e.g., preferring to see nearby objects rather than objects that are far away). For example, a user of the HMD may have stored user preferences associated with the HMD. These preferences may be selected by the user as part of the setup process. The preference may be general (eg, attracted to movement) or video-specific (eg, tend to focus on guitarists in music performances). For example, HMD users may belong to a group or category (eg, men aged 15-22). For example, user characteristics / preferences / categories may be stored in a data table included in a data store (eg, view perspective data store 815). The data store may be queried or filtered based on the default display for 3D video. If the query or filter returns results, the 3D video has at least one associated preferred view perspective based on the associated characteristics / favorite / category for the user. If no result is returned, the 3D video does not have an associated view perspective based on the user.

  At step S515, at least one view perspective based on the region of interest is determined. For example, the region of interest may be the current view perspective. For example, the at least one preferred view perspective may be a previous view perspective that is within range of the current view perspective (eg, close to the current view perspective). For example, at least one preferred view perspective is one that is within (eg, has approached) the previous view perspective of the current user or of the group (type or category) to which the current user belongs. View perspective.

  At step S520, at least one view perspective based on at least one system characteristic is determined. For example, an HMD may have features that can enhance the user experience. One feature may be enhanced speech. Thus, in a virtual reality environment, a user may be drawn to a particular sound (eg, a game user may be drawn to an explosion sound). Preferred view perspectives may be based on view perspectives that include these audible cues. At step S525, at least one preferred view perspective for the 3D video is determined based on each of the aforementioned view perspective determinations and / or combinations / sub-combinations thereof. For example, at least one preferred view perspective may be generated by merging or combining the results of the aforementioned queries.

  In the example of FIG. 6A, video encoder system 600 may be or include at least one computing device, and is configured to perform the methods described herein. Can represent virtually any computing device. As such, video encoder system 600 may include various components that may be utilized to implement the techniques described herein, or different or future versions thereof. By way of example, video encoder system 600 is illustrated as including at least one processor 605 and at least one memory 610 (eg, a non-transitory computer readable storage medium).

  FIG. 6A illustrates a video encoder system according to at least one example embodiment. As shown in FIG. 6A, the video encoder system 600 includes at least one processor 605, at least one memory 610, a controller 620, and a video encoder 625. At least one processor 605, at least one memory 610, controller 620, and video encoder 625 are communicatively coupled via bus 615.

  The at least one processor 605 may be utilized to execute instructions stored on at least one memory 610, such that various features and functions described herein, or additional or alternative Implement features and functions. At least one processor 605 and at least one memory 610 may be utilized for various other purposes. In particular, at least one memory 610 may represent an example of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein.

  At least one memory 610 may be configured to store data and / or information associated with video encoder system 600. For example, at least one memory 610 may be configured to store a codec associated with encoding the spherical video. For example, the at least one memory may be configured to store a code associated with selecting a portion of the frame of the spherical video as a tile to be encoded separately from the encoding of the spherical video. . At least one memory 610 may be a shared resource. As described in more detail below, a tile may be a plurality of pixels selected based on a viewer's view perspective during playback of a spherical viewer (eg, an HMD). The plurality of pixels may be a block, a plurality of blocks, or a macroblock, which may include a portion of a spherical image that may be viewed by a user. For example, video encoder system 600 may be an element of a larger system (eg, a server, personal computer, mobile device, etc.). Thus, at least one memory 610 is configured to store data and / or information associated with other elements in the larger system (eg, image / video feed, web browsing, or wired / wireless communications). May be done.

  Controller 620 may be configured to generate various control signals and communicate the control signals to various blocks in video encoder system 600. Controller 620 may be configured to generate such control signals to implement the techniques described below. The controller 620 may be configured to control the video encoder 625 to encode an image, a sequence of images, a video frame, a video sequence, etc., according to an example embodiment. For example, controller 620 may generate a control signal corresponding to a parameter for encoding a spherical video. Further details regarding the function and operation of video encoder 625 and controller 620 are described below in connection with at least FIGS. 7A, 4A, 5A, 5B, and 6-9.

  Video encoder 625 may be configured to receive video stream input 5 and output compressed (eg, encoded) video bits 10. Video encoder 625 may convert video stream input 5 into discrete video frames. The video stream input 5 may also be an image, so the compressed (eg coded) video bits 10 may also be compressed image bits. Video encoder 625 may further convert each discrete video frame (or image) into a matrix of blocks (hereinafter, referred to as blocks). For example, a video frame (or image) is organized into a 16 × 16, 16 × 8, 8 × 8, 8 × 4, 4 × 4, 4 × 2, 2 × 2, etc. matrix of blocks each having a large number of pixels. It may be converted. Although these exemplary matrices are listed, exemplary embodiments are not limited thereto.

  The compressed video bits 10 may represent the output of the video encoder system 600. For example, the compressed video bits 10 may represent an encoded video frame (or an encoded image). For example, the compressed video bits 10 may be ready for transmission to a receiving device (not shown). For example, video bits may be transmitted to a system transceiver (not shown) for transmission to a receiving device.

  At least one processor 605 may be configured to execute computer instructions associated with controller 620 and / or video encoder 625. At least one processor 605 may be a shared resource. For example, video encoder system 600 may be an element of a larger system (eg, a mobile device). Thus, at least one processor 605 may also be configured to execute computer instructions associated with other elements in the larger system (eg, image / video feed, web browsing, or wired / wireless communications). Good.

  In the example of FIG. 6B, video decoder system 650 may be at least one computing device and may represent virtually any computing device configured to perform the methods described herein. As such, video decoder system 650 may include various components that may be utilized to implement the techniques described herein, or different or future versions thereof. By way of example, video decoder system 650 is illustrated as including at least one processor 655 and at least one memory 660 (eg, a computer-readable storage medium).

  Thus, at least one processor 655 may be utilized to execute instructions stored on at least one memory 660, such that various features and functions described herein, or additional or Provide alternative features and functions. At least one processor 655 and at least one memory 660 may be utilized for various other purposes. In particular, at least one memory 660 may represent an example of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein. According to an exemplary embodiment, video encoder system 600 and video decoder system 650 may be included in the same larger system (eg, personal computer, mobile device, etc.). According to an exemplary embodiment, video decoder system 650 may be configured to implement a reverse or reverse approach to that described with respect to video encoder system 600.

  At least one memory 660 may be configured to store data and / or information associated with video decoder system 650. For example, at least one memory 610 may be configured to store a codec associated with decoding encoded spherical video data. For example, the at least one memory may include a code associated with decoding the encoded tile and the separately encoded spherical video frame, and replace pixels in the decoded spherical video frame with the decoded tile. May be configured to store the code for At least one memory 660 may be a shared resource. For example, video decoder system 650 may be an element of a larger system (eg, personal computer, mobile device, etc.). Accordingly, at least one memory 660 may be configured to store data and / or information associated with other elements in a larger system (eg, web browsing, or wireless communication).

  Controller 670 may be configured to generate various control signals and communicate the control signals to various blocks in video decoder system 650. Controller 670 may be configured to generate the control signals to implement the video decoding techniques described below. Controller 670 may be configured to control video decoder 675 to decode a video frame, according to an example embodiment. Controller 670 may be configured to generate a control signal corresponding to decoding the video. Further details regarding the function and operation of video decoder 675 and controller 670 are described below.

  Video decoder 675 may be configured to receive the compressed (eg, encoded) video bit 10 input and output video stream 5. Video decoder 675 may convert the discrete video frames of compressed video bits 10 into video stream 5. Compressed (eg, coded) video bits 10 may also be compressed image bits, and thus video stream 5 may also be an image.

  At least one processor 655 may be configured to execute computer instructions associated with controller 670 and / or video decoder 675. At least one processor 655 may be a shared resource. For example, video decoder system 650 may be an element of a larger system (eg, personal computer, mobile device, etc.). Accordingly, at least one processor 655 may be configured to execute computer instructions associated with other elements in a larger system (eg, web browsing, or wireless communication).

  7A and 7B show a flow diagram for the video encoder 625 shown in FIG. 6A and the video decoder 675 shown in FIG. 6B, respectively, according to at least one example embodiment. The video encoder 625 (described above) includes a spherical-2D representation block 705, a prediction block 710, a transform block 715, a quantization block 720, an entropy coding block 725, an inverse quantization block 730, and an inverse transform block. 735, a reconstruction block 740, a loop filter block 745, a tile expression block 790, and a view frame storage 795. Other structural variants of the video encoder 625 may be used to encode the input video stream 5. As shown in FIG. 7A, dashed lines represent reconstruction paths between some blocks, and solid lines represent forward paths between some blocks.

  Each of the foregoing blocks is stored in a memory (eg, at least one memory 610) associated with the video encoder system (eg, as shown in FIG. 6A) and includes at least one processor (eg, such as shown in FIG. 6A) associated with the video encoder system. It may be executed as software code, executed by at least one processor 605). However, alternative embodiments are conceivable, such as a video encoder embodied as a special purpose processor. For example, each of the foregoing blocks (alone and / or combined) may be an application specific integrated circuit, ie, an ASIC. For example, the ASIC may be configured as transform block 715 and / or quantization block 720.

  The spherical-2D representation block 705 may be configured to map the spherical frame or image to a 2D representation of the spherical frame or image. For example, a sphere may be projected onto a surface of another shape (eg, a square, rectangle, cylinder, and / or cube). The projection may be, for example, an equirectangular or semi- equirectangular.

  The prediction block 710 may be configured to take advantage of video frame integrity (eg, pixels that have not changed compared to previously encoded pixels). The prediction may include two types. For example, the prediction may include intra-frame prediction and inter-frame prediction. Intra-frame prediction involves predicting the pixel values in a block of an image relative to reference samples in a previously encoded neighboring block of the same image. In intra-frame prediction, the samples are placed in the same frame to reduce the residual encoded by the transform (eg, entropy coding block 725) and entropy coding (eg, entropy coding block 725) portions of the predictive transform codec. From the reconstructed pixels. Inter-frame prediction relates to predicting pixel values in blocks of an image relative to previously encoded image data.

  Transform block 715 may be configured to transform pixel values from the spatial domain to transform coefficients in the transform domain. The transform coefficients may correspond to a two-dimensional matrix of coefficients that are usually the same size as the original block. In other words, there may be as many transform coefficients as there are pixels in the original block. However, due to the transformation, some of the transform coefficients may have a value equal to zero.

  Transform block 715 may be configured to transform the remainder (from prediction block 710) into transform coefficients in the frequency domain, for example. Typically, the transformation is a Karhunen-Loeve Transform (KLT), a Discrete Cosine Transform (DCT), a Singular Value Decomposition Transform (SVD), and an asymmetric Discrete Sine Transform (Asymmetric discrete sine transform: ADST).

  Quantization block 720 may be configured to reduce data at each transform coefficient. Quantization may involve mapping values within a relatively large range to values within a relatively small range, thus reducing the amount of data required to represent the quantized transform coefficients. . The quantization block 720 may convert the transform coefficients into discrete quantized values called quantized transform coefficients or quantization levels. For example, quantization block 720 may be configured to add zero to data associated with transform coefficients. For example, a coding criterion may specify 128 quantization levels in a scalar quantization process.

  The quantized transform coefficients are entropy coded by an entropy coding block 725. The entropy coded coefficients are then output as compressed video bits 10, along with the information needed to decode the block, such as the type of prediction used, motion vectors, and quantizer values. You. Compressed video bits 10 may be formatted using various techniques, such as run-length encoding (RLE) and zero-run coding.

  The reconstruction path in FIG. 7A is such that both video encoder 625 and video decoder 675 (described below with respect to FIG. 7B) use the same reference frame to compress compressed video bits 10 (or compressed image bits). ) Is present to guarantee decoding. The reconstruction path performs functions similar to those performed during the decoding process, described in more detail below. The function is to dequantize the quantized transform coefficients in an inverse quantization block 730 to create a derivative residual block (differential residual) and to perform an inverse transform in an inverse transform block 735. Inverse transforming the quantized transform coefficients. At reconstruction block 740, the prediction block predicted at prediction block 710 may be added to the differential residual to create a reconstruction block. Next, a loop filter 745 may be applied to the reconstructed block to reduce distortion such as blocking artifacts.

  Tile representation block 790 may be configured to convert an image and / or frame into multiple tiles. One tile may be a grouping of pixels. A tile may be a plurality of pixels selected based on a view or view perspective. The plurality of pixels may be a block, a plurality of blocks, or a macroblock that may include a portion of a spherical image that may be viewed (or expected to be viewed) by a user. A part of the spherical image as a tile may have a length and a width. A portion of the spherical image may be two-dimensional, or substantially two-dimensional. Tiles may have variable sizes (eg, what percentage of a sphere a tile covers). For example, the size of a tile is coded based on, for example, how wide the viewer's field of view is, the proximity to another tile, and / or how fast the user is turning his or her head, Can be streamed. For example, if the viewer is constantly looking around, larger, lower quality tiles may be selected. However, if the viewer is looking at one perspective, smaller, more detailed tiles may be selected.

  In one implementation, the tile representation block 790 invokes instructions to the spherical-2D representation block 705 to cause the spherical-2D representation block 705 to generate tiles. In another implementation, the tile representation block 790 generates the tile. In either implementation, each tile is then individually encoded. In yet another implementation, the tile representation block 790 triggers instructions to the view frame storage 795 to cause the view frame storage 795 to store the encoded image and / or video frames as tiles. The tile representation block 790 can trigger an instruction to the view frame storage 795 to cause the view frame storage 795 to store the tile along with information or metadata about the tile. For example, information or metadata about a tile may include an indication of tile position within an image or frame, information associated with tile encoding (eg, resolution, bandwidth, and / or 3D-2D projection algorithms), one The association with the above-mentioned region of interest may be included.

  According to an example implementation, encoder 625 may encode a frame, a portion of a frame, and / or a tile with different qualities (or Quality of Service (QoS)). According to an exemplary embodiment, encoder 625 may encode a frame, a portion of a frame, and / or a tile multiple times, each with a different QoS. Thus, view frame storage 795 can store frames, portions of frames, and / or tiles that represent the same location in an image or frame with different QoS. As such, the aforementioned information or metadata about the tile may include a frame, a portion of the frame, and / or an indication of the QoS at which the tile was encoded.

  QoS may be based on compression algorithm, resolution, transmission rate, and / or coding scheme. Accordingly, encoder 625 may use different compression algorithms and / or encoding schemes for each frame, portion of a frame, and / or tile. For example, the encoded tile may have a higher QoS than the frame encoded by the encoder 625 (associated with the tile). As described above, encoder 625 may be configured to encode a 2D representation of the spherical video frame. Thus, a tile (as a visible perspective that includes a portion of a spherical video frame) may be encoded with a higher QoS than a 2D representation of the spherical video frame. QoS may affect the resolution of a frame when decoded. Thus, the tile (as a visible perspective that includes a portion of the spherical video frame) is such that when the tile is decoded, it has a higher resolution of the frame compared to the decoded 2D representation of the spherical video frame. , Can be encoded. Tile representation block 790 may indicate the QoS at which the tile is to be encoded. The tile representation block 790 selects a QoS based on whether the frame, a portion of the frame, and / or the tile is, is within, is associated with a seed region, or the like. Is also good. The region of interest and the seed region are described in more detail below.

  Video encoder 625 described above with respect to FIG. 7A includes the illustrated blocks. However, the exemplary embodiments are not so limited. Additional blocks may be added based on different video coding configurations and / or techniques used. Also, each of the blocks shown in video encoder 625 described above with respect to FIG. 7A may be optional blocks based on the different video coding configurations and / or techniques used.

  FIG. 7B is a schematic block diagram of a decoder 675 configured to decode the compressed video bits 10 (or the compressed image bits). The decoder 675 includes an entropy decoding block 750, an inverse quantization block 755, an inverse transformation block 760, a reconstruction block 765, a loop filter block 770, and a prediction block similar to the reconstruction path of the encoder 625 described above. 775, a deblocking filter block 780, and a 2D representation-spherical block 785.

  The data elements in the compressed video bits 10 are used to generate a set of quantized transform coefficients (eg, using Context Adaptive Binary Arithmetic Decoding). It may be decoded by the entropy decoding block 750. An inverse quantization block 755 inversely quantizes the quantized transform coefficients, and an inverse transform block 760 inversely transforms the inversely quantized transform coefficients (using ADST) and reconstructs at the encoder 625. Produces a differential residual that can be identical to that produced by

  Using the header information decoded from the compressed video bits 10, the decoder 675 can use the prediction block 775 to create the same prediction block as created at the encoder 675. The prediction block may be added to the differential residual to create a reconstructed block by the reconstructed block 765. To reduce blocking artifacts, a loop filter block 770 may be applied to the reconstructed block. To reduce blocking distortion, a deblocking filter block 780 may be applied to the reconstructed block. The result is output as a video stream 5.

  The 2D representation-spherical block 785 may be configured to map a 2D representation of the spherical frame or image to the spherical frame or image. For example, mapping a 2D representation of a spherical frame or image to a spherical frame or image may be the inverse of the 3D-2D mapping performed by encoder 625.

  Video decoder 675 described above with respect to FIG. 7B includes the illustrated blocks. However, the exemplary embodiments are not so limited. Additional blocks may be added based on different video coding configurations and / or techniques used. Also, each of the blocks shown in video decoder 675 described above with respect to FIG. 7B may be optional blocks based on the different video coding configurations and / or techniques used.

  Encoder 625 and decoder 675 may be configured to encode spherical video and / or images, and to decode spherical video and / or images, respectively. The spherical image is an image including a plurality of pixels organized in a spherical shape. In other words, the spherical image is an image that is continuous in all directions. Thus, a viewer of a spherical image changes position or orientation (eg, moves his or her head or eyes) in any direction (eg, up, down, left, right, or any combination thereof). ) And part of the image can be viewed continuously.

  In an exemplary implementation, the parameters used at and / or determined by encoder 625 may be used by other elements of encoder 405. For example, a motion vector used to encode a 2D representation (eg, as used in prediction) may be used to encode a tile. Also used in the prediction block 710, transform block 715, quantization block 720, entropy coding block 725, inverse quantization block 730, inverse transform block 735, reconstruction block 740, and loop filter block 745, and / or The parameters determined by the blocks may be shared between encoder 625 and encoder 405.

  Part of the spherical video frame or image may be processed as an image. Thus, a portion of the spherical video frame may be transformed (or decomposed) into a C × R matrix of blocks (hereinafter referred to as blocks). For example, a portion of a spherical video frame may be a 16 × 16, 16 × 8, 8 × 8, 8 × 4, 4 × 4, 4 × 2, 2 × 2, etc. matrix of blocks each having a large number of pixels. It may be converted to a C × R matrix.

  FIG. 8 illustrates a system 800 according to at least one example embodiment. As shown in FIG. 8, the system 700 includes a controller 620, a controller 670, a video encoder 625, a view frame storage 795, and an orientation sensor 835. Controller 620 further includes a view position control module 805, a tile control module 810, and a view perspective data store 815. The controller 670 further includes a view position determination module 820, a tile request module 825, and a buffer 830.

  According to an exemplary implementation, orientation sensor 835 detects the orientation (or change in orientation) of the viewer's eyes (or head), and view position determination module 820 determines the orientation based on the detected orientation. Having determined the view, perspective, or view perspective, the tile request module 825 communicates the view, perspective, or view perspective (in addition to the spherical video) as part of the request for the tile or tiles. According to another example implementation, orientation sensor 835 detects an orientation (or a change in orientation) based on the image panning orientation as rendered on the HMD or display. For example, a user of the HMD may change the depth of focus. In other words, the user of the HMD may change his or her focus from an object that was far away to an object that is close, with or without a change in orientation, and vice versa. For example, a user may use a mouse, trackpad, or gesture (eg, on a touch-sensitive display) to select, move, drag, magnify, etc., a portion of a spherical video or image as rendered on a display. May be used.

  Requests for tiles may be communicated with requests for frames of the spherical video. Requests for tiles may be communicated separately from requests for frames of the spherical video. For example, a request for a tile may respond to a modified view, perspective, or view perspective, resulting in a need to replace a previously requested and / or queued tile.

  View position control module 805 receives and processes requests for tiles. For example, view position control module 805 may determine a frame and a position of a tile or tiles in the frame based on the view. Next, view position control module 805 may instruct tile control module 810 to select a tile or multiple tiles. Selecting the tile or tiles may include passing parameters to video encoder 625. The parameters may be used by video encoder 625 during encoding of the spherical video and / or tile. Alternatively, selecting a tile or multiple tiles may include selecting a tile or multiple tiles from view frame storage 795.

  Accordingly, the tile control module 810 may be configured to select a tile (or multiple tiles) based on the view or perspective or view perspective of the user watching the spherical video. A tile may be a plurality of pixels selected based on a view. The plurality of pixels may be a block, a plurality of blocks, or a macroblock, which may include a portion of a spherical image that may be viewed by a user. Part of the spherical image may have a length and a width. A portion of the spherical image may be two-dimensional, or substantially two-dimensional. Tiles may have variable sizes (eg, what percentage of a sphere a tile covers). For example, the size of the tiles may be encoded and streamed, for example, based on how wide the viewer's field of view is and / or how fast the user is turning his or her head. For example, if the viewer is constantly looking around, larger, lower quality tiles may be selected. However, if the viewer is looking at one perspective, smaller, more detailed tiles may be selected.

  Accordingly, orientation sensor 835 may be configured to detect the orientation (or change in orientation) of the viewer's eyes (or head). For example, orientation sensor 835 may include an accelerometer to detect movement and a gyroscope to detect orientation. Alternatively or additionally, orientation sensor 835 may include a camera or infrared sensor focused on the viewer's eyes or head to determine the orientation of the viewer's eyes or head. Alternatively or additionally, orientation sensor 835 may determine a portion of the spherical video or image as rendered on a display to detect the orientation of the spherical video or image. The orientation sensor 835 may be configured to communicate orientation and orientation change information to the view position determination module 820.

  The view location determination module 820 may be configured to determine a view or perspective view (eg, a portion of the spherical video that the viewer is currently viewing) with respect to the spherical video. A view, perspective, or view perspective may be determined as a position, point, or focus on a spherical video. For example, the view may be a latitude and longitude position on the spherical video. A view, perspective, or view perspective may be determined as a cube edge based on the spherical video. The views (eg, latitude and longitude locations, or sides) may be communicated to the view location control module 805 using, for example, Hypertext Transfer Protocol (HTTP).

  The view position control module 805 may be configured to determine a view position (eg, a frame and a position within the frame) of a tile or multiple tiles in the spherical video. For example, view position control module 805 may select a rectangle centered on a view position, point, or focus (eg, a latitude and longitude position, or a side). Tile control module 810 may be configured to select the rectangle as a tile or multiple tiles. Tile control module 810 may be configured to instruct video encoder 625 (eg, via parameters or configuration settings) to encode the selected tile or tiles. And / or the tile control module 810 can be configured to select a tile or multiple tiles from the view frame storage 795.

  As will be appreciated, the system 600 shown in FIG. 6A and the system 650 shown in FIG. 6B, and / or the system 800 shown in FIG. It may be implemented as 950 elements and / or extensions. Alternatively or additionally, the system 600 shown in FIG. 6A and the system 650 shown in FIG. 6B and / or the system 800 shown in FIG. May be implemented in a system separate from the general-purpose computing device 900 and / or the general-purpose mobile computing device 950, having some or all of the features described in.

  FIG. 9 is a schematic block diagram of a computing device and a mobile computing device that may be used to implement the techniques described herein. FIG. 9 is an example of a general purpose computing device 900 and a general purpose mobile computing device 950 that can be used with the techniques described herein. Computing device 900 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cell phones, smartphones, and other similar computing devices. The components shown, their connections and relationships, and their functions, are intended to be merely exemplary, and should not be construed as limiting the implementations of the invention described in this document and / or claimed. Not intended.

  The computing device 900 includes a processor 902, a memory 904, a storage device 906, a high-speed interface 908 connected to the memory 904 and the high-speed expansion port 910, and a low-speed interface connected to the low-speed bus 914 and the storage device 906. 912. Each of components 902, 904, 906, 908, 910, and 912 are interconnected using various buses and may be mounted on a common motherboard or otherwise. Processor 902 is capable of processing instructions executing within computing device 900, which in turn render graphics information for the GUI on an external input / output device, such as display 916, coupled to high speed interface 908. Includes instructions stored in memory 904 or on storage device 906 for display. In other implementations, multiple processors and / or multiple buses may be used with multiple memories and multiple types of memory as appropriate. Also, multiple computing devices 900 may be connected, each device providing a portion of the required operation (eg, as a server bank, a group of blade servers, or a multi-processor system).

  Memory 904 stores information in computing device 900. In one implementation, memory 904 is one or more volatile memory units. In another implementation, memory 904 is one or more non-volatile memory units. Memory 904 may also be another form of computer-readable medium, such as a magnetic or optical disk.

  The storage 906 can provide mass storage for the computing device 900. In one implementation, the storage device 906 is a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or a storage area network or other configuration. Or a computer readable medium, such as an array of devices, including the devices in. A computer program product can be tangibly embodied in an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable or machine-readable medium, such as memory 904, storage 906, or memory on processor 902.

  High speed controller 908 manages bandwidth intensive operations for computing device 900, while low speed controller 912 manages lower bandwidth intensive operations. Such an assignment of functions is merely exemplary. In one implementation, high-speed controller 908 couples to memory 904, display 916 (eg, via a graphics processor or accelerator), and a high-speed expansion port 910 that can accept various expansion cards (not shown). Is done. In this implementation, low speed controller 912 is coupled to storage 906 and low speed expansion port 914. The low speed expansion port, which may include various communication ports (eg, USB, Bluetooth, Ethernet, wireless Ethernet), may be connected to one or more input / output devices such as a keyboard, pointing device, scanner, or , A switch or a router, for example, via a network adapter.

  Computing device 900 may be implemented in many different forms as shown. For example, it may be implemented multiple times as a standard server 920 or in a group of such servers. It may also be implemented as part of the rack server system 924. In addition, it may be implemented on a personal computer such as a laptop computer 922. Alternatively, components from computing device 900 may be combined with other components on a mobile device (not shown), such as device 950. Each such device may include one or more of the computing devices 900, 950, and the entire system may be made up of multiple computing devices 900, 950 communicating with one another.

  The computing device 950 includes a processor 952, memory 964, input / output devices such as a display 954, a communication interface 966, and a transceiver 968, among other components. Device 950 may also be provided with storage, such as a microdrive or other device, to provide additional storage. Each of the components 950, 952, 964, 954, 966, and 968 are interconnected using various buses, some of which may be on a common motherboard or otherwise. It may be mounted.

  Processor 952 is capable of executing instructions in computing device 950, including instructions stored in memory 964. The processor may be implemented as a chipset of chips including separate analog and digital processors. The processor may provide coordination between other components of the device 950, for example, a user interface, applications executed by the device 950, and control of wireless communication by the device 950.

  Processor 952 may communicate with a user via a control interface 958 coupled to a display 954 and a display interface 956. Display 954 may be, for example, a thin-film-transistor liquid crystal display (TFT LCD), or an organic light-emitting diode (OLED) display, or other suitable display technology. Display interface 956 may include appropriate circuitry for driving display 954 to present graphical and other information to a user. Control interface 958 may receive commands from the user and convert them for delivery to processor 952. In addition, an external interface 962 may be provided in communication with the processor 952 to allow near-area communication between the device 950 and another device. External interface 962 may provide, for example, wired communication in one implementation and wireless communication in other implementations, and multiple interfaces may also be used.

  Memory 964 stores information in computing device 950. Memory 964 may be implemented as one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An extension memory 974 is also provided and may be connected to the device 950 via an extension interface 972, which may include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory 974 may provide extra storage space for device 950, or may also store applications or other information for device 950. Specifically, expansion memory 974 may include instructions for performing or supplementing the processes described above, and may also include secure information. Thus, for example, the extension memory 974 may be provided as a security module for the device 950 and may be programmed with instructions permitting secure use of the device 950. In addition, secure applications may be provided via the SIMM card with additional information, such as placing identification information on the SIMM card in a non-hackable manner.

  The memory may include, for example, flash memory and / or NVRAM memory as described below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable or machine-readable medium, such as memory 964, expanded memory 974, or memory on processor 952, and may be received, for example, through transceiver 968 or external interface 962.

  Device 950 may communicate wirelessly via communication interface 966, which may optionally include digital signal processing circuitry. Communication interface 966 may communicate under various modes or protocols, such as GSM voice communication, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. May be provided. Such communication may occur, for example, via radio frequency transceiver 968. In addition, short-range communication may occur, such as using Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 970 may provide additional navigation-related and location-related wireless data to the device 950, where the data is transmitted to an application running on the device 950. May be used as appropriate.

  Device 950 may also communicate audio using a speech codec 960 that may receive verbal information from the user and convert it into usable digital information. Audio codec 960 may also generate a sound that can be heard by the user, such as through a speaker, for example, in a handset of device 950. Such sounds may include sounds from a voice call, may include recorded sounds (eg, voice messages, music files, etc.), and may be generated by applications running on device 950. May also be included.

  Computing device 950 may be implemented in many different forms as shown. For example, it may be implemented as a mobile phone 980. It may also be implemented as part of a smartphone 982, personal digital assistant, or other similar mobile device.

  Some of the exemplary embodiments described above are described as processes or methods illustrated as flowcharts. Although these flowcharts describe operations as sequential processes, many of the operations may be performed in parallel, simultaneously, or all at once. In addition, the order of the operations may be rearranged. The process may be terminated when those operations are completed, but may have additional steps not included in the figure. These processes may correspond to methods, functions, procedures, subroutines, subprograms, etc.

  The methods described above, some of which are illustrated by flowcharts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks may be stored on machine-readable or computer-readable media, such as storage media. The processor may perform the necessary tasks.

  The specific structural and functional details disclosed herein are merely representative for describing example embodiments. However, the exemplary embodiments may be embodied in many alternative forms and should not be construed as limited to only the embodiments set forth herein.

  Terms such as first, second, etc. may be used herein to describe various elements, but it will be understood that these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

  When one element is referred to as being connected or coupled to another element, it will be understood that one element may be directly connected or coupled to another element or that there may be intervening elements. . In contrast, when one element is referred to as being directly connected to or directly coupled to another element, there are no intervening elements. Other language used to describe relationships between elements should be construed in a similar manner (eg, "between" and "directly between", "adjacent" and "directly adjacent", etc.). is there.

  The terms used in the description are intended to describe certain embodiments only, and are not intended to be limiting of the exemplary embodiments. As used herein, the singular is also intended to include the plural unless the context clearly indicates otherwise. The terms "comprises, comprising" and / or "includes, including" as used herein identify the presence of the feature, integer, step, operation, element and / or component mentioned. It will be further understood, however, that this does not preclude the presence or addition of one or more other features, integers, steps, acts, elements, components and / or groups thereof.

  Also, in some alternative implementations, the functions / acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may, in fact, be performed simultaneously, or sometimes in the reverse order, depending on the functionality / acts involved.

  Unless defined otherwise, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the exemplary embodiment belongs. Having. Furthermore, terms such as those defined in commonly used dictionaries should be construed as having meanings consistent with their meanings in the context of the relevant art, and are not explicitly so defined herein. It will be understood that, as long as it is not interpreted in an idealized or overly formal sense.

  With respect to software or algorithms and symbolic representations of operations on data bits within a computer memory, portions of the illustrative embodiments described above and corresponding detailed description are presented. These descriptions and expressions enable those skilled in the art to effectively convey the substance of their work to others skilled in the art. An algorithm, as the term is used herein, and as it is commonly used, is considered to be a consistent sequence of steps leading to a desired result. These steps require physical manipulation of physical quantities. Usually, but not necessarily, these quantities take the form of optical, electrical, or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated. Referencing these signals as bits, values, elements, symbols, characters, terms, or numbers, etc., has proven convenient at times, primarily for reasons of common usage.

  In the above-described exemplary embodiments, references to symbolic representations of acts and acts (eg, in the form of flowcharts) that can be implemented as program modules or functional processes perform particular tasks, or implement particular abstract data Includes routines, programs, objects, components, data structures, etc. that implement the type and can be described and / or implemented with existing hardware using existing hardware. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, or field programmable gate array (FPGA) computers.

  However, it should be kept in mind that all of these and similar terms should be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise stated, or as will be apparent from the description, terms such as processing, computing, calculating, or determining display manipulate data represented as physical quantities of electrons in registers and memories of computer systems; A computer system or similar electronic computing device that converts the data into computer system memory, registers, or other such information storage devices, transmitting devices or other data that are also represented as physical quantities in a display device. Refers to actions and processes.

  Also, aspects implemented by the software of the exemplary embodiments are typically encoded on some form of non-transitory program storage media, or implemented on some type of transmission medium. . The program storage medium may be magnetic (eg, a floppy disk or hard drive) or optical (eg, a compact disk read-only memory, ie, CD ROM), read-only or random-access. Is also good. Similarly, the transmission medium may be a twisted pair wire, coaxial cable, optical fiber, or any other suitable transmission medium known in the art. The illustrative embodiments are not limited by these aspects of a given implementation.

  Finally, while the appended claims set forth certain combinations of the features described herein, the scope of the disclosure is not limited to the specific combinations claimed, but instead includes those specific combinations. Whether or not the combinations are specifically recited in the appended claims at this time, they extend to encompass any combination of the features or embodiments disclosed herein.

Claims (21)

The method
Determining at least one preferred view perspective associated with a three-dimensional (3D) video, wherein the view perspective is a region selected based on at least one reference point viewed by a viewer of the 3D video. Corresponding to
Encoding a first portion of the 3D video corresponding to the at least one preferred view perspective with a first quality;
Encoding a second portion of the 3D video at a second quality, wherein the first quality is higher quality than the second quality;
Further comprising receiving a request for streaming video, wherein the request includes displaying a user view perspective associated with the 3D video, the method further comprising:
Determining whether the user view perspective is stored in a view perspective data store;
Upon determining that the user view perspective is stored in the view perspective data store, incrementing a counter associated with the user view perspective;
Determining that the user view perspective is not stored in the view perspective data store, adding the user view perspective to the view perspective data store, and setting the counter associated with the user view perspective to 1; Including, methods. Determining at least one preferred view perspective associated with a three-dimensional (3D) video, wherein the view perspective is a region selected based on at least one reference point viewed by a viewer of the 3D video. Corresponding to
Encoding a first portion of the 3D video corresponding to the at least one preferred view perspective with a first quality;
Encoding a second portion of the 3D video at a second quality, wherein the first quality is higher quality than the second quality;
Repeatedly encoding at least a portion of the second portion of the 3D video with the first quality;
Streaming said at least a portion of said second portion of said 3D video. Storing the view perspective as one of a plurality of historical view perspectives for the three-dimensional (3D) video based on the previously viewed view perspective associated with the three-dimensional (3D) video; The view perspective corresponds to an area selected based on at least one reference point viewed by a viewer of the 3D video;
Determining whether the ranking value associated with the plurality of history view perspectives is greater than a threshold and satisfies a condition for the rest of the plurality of history view perspectives,
The ranking value is greater than the threshold value, in response to determining that meets the above conditions, as at least one preferred view perspectives associated et a in the 3D video, selecting one of the plurality of history view perspectives Steps to
Encoding a first portion of the 3D video corresponding to the at least one preferred view perspective with a first quality;
Encoding a second portion of the 3D video at a second quality, wherein the first quality is higher quality than the second quality. Storing the first portion of the 3D video in a data store;
Storing the second portion of the 3D video in the data store;
Receiving a request for streaming video;
The method of any of claims 1 to 3, further comprising: streaming the first portion of the 3D video and the second portion of the 3D video from the data store as the streaming video. Further comprising receiving a request for streaming video, wherein the request comprises displaying a user view perspective, the method further comprising:
Selecting a 3D video corresponding to the user view perspective as the encoded first portion of the 3D video;
4. The method of any of the preceding claims, comprising streaming the selected first portion of the 3D video and the second portion of the 3D video as the streaming video. Further comprising receiving a request for streaming video, wherein the request includes displaying a user view perspective associated with the 3D video, the method further comprising:
Determining whether the user view perspective is stored in a view perspective data store;
Upon determining that the user view perspective is stored in the view perspective data store, incrementing a counter associated with the user view perspective;
Determining that the user view perspective is not stored in the view perspective data store, adding the user view perspective to the view perspective data store, and setting the counter associated with the user view perspective to 1; The method according to claim 1, comprising: Encoding the second portion of the 3D video includes using at least one first quality of service (QoS) parameter in a first pass encoding operation;
7. The method of claim 1, wherein encoding the first portion of the 3D video comprises using at least one second Quality of Service (QoS) parameter in a second pass encoding operation. A method according to any one of the preceding claims.   The at least one preferred view perspective associated with the 3D video is at least one of an orientation of a viewer of the 3D video, a location of a viewer of the 3D video, and a focus of a viewer of the 3D video. 4. The method according to claim 1, wherein the method is based on: Determining the at least one preferred view perspective associated with the 3D video is based on a default view perspective;
The default view perspective is
Display device user characteristics,
Characteristics of a group associated with the user of the display device;
Director's cut, and
The method according to any of the preceding claims, wherein the method is based on at least one of the characteristics of the 3D video. A streaming server,
A controller configured to determine at least one preferred view perspective associated with the three-dimensional (3D) video;
An encoder, wherein the view perspective corresponds to an area selected based on at least one reference point viewed by a viewer of the 3D video;
The encoder is
Encoding a first portion of the 3D video corresponding to the at least one preferred view perspective with a first quality;
Configured to encode a second portion of the 3D video at a second quality, wherein the first quality is higher quality than the second quality;
The controller further comprises:
The controller is configured to receive a request for streaming video, the request including displaying a user view perspective associated with the 3D video, the controller further comprising:
Determining whether the user view perspective is stored in a view perspective data store,
Upon determining that the user view perspective is stored in the view perspective data store, incrementing a counter associated with the user view perspective,
Upon determining that the user view perspective is not stored in the view perspective data store, adding the user view perspective to the view perspective data store and setting the counter associated with the user view perspective to one. Configured streaming server. A streaming server,
A controller configured to determine at least one preferred view perspective associated with the three-dimensional (3D) video;
An encoder, wherein the view perspective corresponds to an area selected based on at least one reference point viewed by a viewer of the 3D video;
The encoder is
Encoding a first portion of the 3D video corresponding to the at least one preferred view perspective with a first quality;
Configured to encode a second portion of the 3D video at a second quality, wherein the first quality is higher quality than the second quality;
The controller further comprises:
Iteratively encoding at least a portion of the second portion of the 3D video with the first quality;
A streaming server configured to stream the at least a portion of the second portion of the 3D video. A streaming server,
Based on a view perspective previously viewed and associated with the three-dimensional (3D) video, configured to store the view perspective as one of a plurality of historical view perspectives for the three-dimensional (3D) video. Wherein the plurality of historical view perspectives are associated with a plurality of users of the three-dimensional (3D) video, wherein the view perspective includes at least one reference point viewed by a viewer of the 3D video Corresponding to the region selected based on
Comprising a controller, wherein the controller comprises:
The ranking value associated with the plurality of history view perspectives is greater than a threshold, and determines whether or not a condition for the rest of the plurality of history view perspectives is satisfied,
The ranking value is greater than the threshold value, in response to determining that meets the above conditions, as at least one preferred view perspectives associated et a in the 3D video, selecting one of the plurality of history view perspectives Is configured to
Comprising an encoder, wherein the encoder comprises:
Encoding a first portion of the 3D video corresponding to the at least one preferred view perspective with a first quality;
The method, wherein the method is configured to encode a second portion of the 3D video at a second quality, wherein the first quality is higher quality than the second quality. The controller further comprises:
Storing the first portion of the 3D video in a data store;
Storing the second portion of the 3D video in the data store;
Receiving a request for streaming video,
The streaming server according to any of claims 10 to 12, wherein the streaming server is configured to stream the first part of the 3D video and the second part of the 3D video as the streaming video from the data store. . The controller further comprises:
Configured to receive a request for streaming video, wherein the request includes displaying a user view perspective, the controller further comprising:
Selecting a 3D video corresponding to the user view perspective as the encoded first portion of the 3D video;
The streaming server according to any of claims 10 to 12, wherein the streaming server is configured to stream the selected first part of the 3D video and the second part of the 3D video as the streaming video. Encoding the second portion of the 3D video includes using at least one first quality of service (QoS) parameter in a first pass encoding operation;
13. The method of claim 10, wherein encoding the first portion of the 3D video comprises using at least one second Quality of Service (QoS) parameter in a second pass encoding operation. A streaming server according to any one of the preceding claims.   The at least one preferred view perspective associated with the 3D video is at least one of an orientation of a viewer of the 3D video, a location of a viewer of the 3D video, and a focus of a viewer of the 3D video. The streaming server according to any of claims 10 to 12, which is based on: Determining the at least one preferred view perspective associated with the 3D video is based on a default view perspective,
The default view perspective is
Display device user characteristics,
Characteristics of a group associated with the user of the display device;
Director's cut, and
The streaming server according to claim 10, wherein the streaming server is based on at least one of the characteristics of the 3D video. The controller further comprises:
Iteratively encoding at least a portion of the second portion of the 3D video with the first quality;
The streaming server according to claim 10 or 12, wherein the streaming server is configured to cause the at least a portion of the second portion of the 3D video to be streamed. The method
Receiving a request for streaming video, wherein the request includes displaying a user view perspective associated with the three-dimensional (3D) video, wherein the user view perspective includes at least Corresponding to the area selected based on one reference point,
The method further comprises:
Determining whether the user view perspective is stored in a view perspective data store;
When determining that the user view perspective is stored in the view perspective data store, incrementing a ranking value associated with the user view perspective,
Determining that the user view perspective is not stored in the view perspective data store, adding the user view perspective to the view perspective data store and setting the ranking value associated with the user view perspective to 1; And a method comprising: Determining at least one preferred view perspective associated with the 3D video based on the ranking value associated with the stored user view perspective;
Encoding a first portion of the 3D video corresponding to the at least one preferred view perspective with a first quality;
Encoding the second portion of the 3D video at a second quality, wherein the first quality is a higher quality compared to the second quality. . A program executed by a computer,
The program causing the computer to execute the method according to any one of claims 1 to 9, 19, and 20.

Images