GB2559127A

GB2559127A - Media viewing method and apparatus

Info

Publication number: GB2559127A
Application number: GB1701237.8A
Authority: GB
Inventors: Kavallierou Michael
Original assignee: Sony Interactive Entertainment Inc
Current assignee: Sony Interactive Entertainment Inc
Priority date: 2017-01-25
Filing date: 2017-01-25
Publication date: 2018-08-01
Also published as: GB201701237D0

Abstract

An image display system such as a head mounted display (HMD) 700 with a view obtaining unit which obtains a main view and one or more secondary views 710, a motion information capturing unit or sensor 720 which captures information about the head movement of a user and an image display unit 730 which displays an image to the user in a portion of the main view dependent on the position and orientation of the users head. If the users head motion exceeds a threshold amount, then at least a portion of a secondary view is shown. The threshold amount may be in a direction perpendicular to that of the main axis of the main view. The system may have a view management unit 740 which defines relative positions of the main and secondary views (for example based on content or user settings) in a virtual space. The secondary views may display statistics, additional media or contextual information related to the main content on the main view. The secondary views may be displayed at the same time as or instead of the main view and may be an overlay of the main view or displayed in a split screen format.

Description

(71) Applicant(s):

Sony Interactive Entertainment Inc.

1-7-1 Konan, Minato-Ku 108-8270, Tokyo, Japan (72) Inventor(s):

Michael Kavallierou (74) Agent and/or Address for Service:

D Young & Co LLP

120 Holborn, LONDON, EC1N 2DY, United Kingdom (51) INT CL:

G06F 3/01 (2006.01) G02B 27/00 (2006.01) G02B 27/01 (2006.01) (56) Documents Cited:

EP 3179290 A2 US 9392212 B1

US 20140267400 A1 (58) Field of Search:

INT CL G02B, G06F, G06T, H04N Other: EPODOC, WPI (54) Title of the Invention: Media viewing method and apparatus

Abstract Title: Viewing media in a virtual environment with primary and secondary views.

(57) An image display system such as a head mounted display (HMD) 700 with a view obtaining unit which obtains a main view and one or more secondary views 710, a motion information capturing unit or sensor 720 which captures information about the head movement of a user and an image display unit 730 which displays an image to the user in a portion of the main view dependent on the position and orientation of the users head. If the user’s head motion exceeds a threshold amount, then at least a portion of a secondary view is shown. The threshold amount may be in a direction perpendicular to that of the main axis of the main view. The system may have a view management unit 740 which defines relative positions of the main and secondary views (for example based on content or user settings) in a virtual space. The secondary views may display statistics, additional media or contextual information related to the main content on the main view. The secondary views may be displayed at the same time as or instead of the main view and may be an overlay of the main view or displayed in a split screen format.


	View Generating Unit	Motion Information Capturing Unit
	710	720

	Image Display Unit	View Management Unit
	730	740
Image Display System 700

Figure 7

Iof4

Secondary View

110

Main View

100

Figure 1

Secondary View

110

	1---- 1 1 \|	123	1 1 1
Figure 2	1 l____		1 1

of 4

Secondary

View

110

Main View

100

Secondary

View

110

Figure 3

Figure 4

of 4

Figure 5

Secondary View

Secondary' View

Secondary View

Figure 6 of 4

Figure 7

	View Generating Unit 710	Motion Information Capturing Unit 720

	Image Display Unit	View Management Unit
	730	740
Image Display System 700

MEDIA VIEWING METHOD AND APPARATUS

This disclosure relates to a media viewing method and apparatus.

In conventional display methods, such as television broadcasts displayed on a television, information is often overlaid on video content in order to provide a viewer with details about what they are watching, related content or the like. One example of this is when watching sports; it is common to provide a scoreboard, and have statistics displayed on-screen throughout a game. This overlay hides a portion of the content that the viewer wishes to watch, so in view of this the overlaid content is often positioned in an unobtrusive area of the screen so as to not cause too much of a disturbance to the user’s viewing experience.

In recent years there has been an increase in the popularity of head-mountable displays (HMDs) as display devices. Such devices offer a wide field of view and a 3D display, and as a result HMDs may be used to provide a more immersive viewing experience to a user. This enables the user to believe that they are ‘in the action’, and thus a greater sense of enjoyment may be experienced. For example, a viewer may wish to use an HMD to view a sporting event as it more closely resembles the experience of being in the crowd at the live event.

However, conventional display methods may be incompatible with the desire for an immersive experience with an HMD; the presence of overlaid content, while desirable to keep a viewer informed, is a clear indication that the user is not experiencing the event in person. This is because overlays are simply not present in a user’s in-person viewing experiences. Therefore there is a need to provide a display method that provides a greater immersiveness, whilst maintaining the functionality of a conventional display method.

This disclosure is defined by claims 1 and 14, with further respective aspects and features of the disclosure being defined in the appended claims.

Embodiments of the disclosure will now be described with reference to the accompanying drawings, in which:

Figure 1 schematically Figure 2 schematically Figure 3 schematically Figure 4 schematically Figure 5 schematically Figure 6 schematically Figure 7 schematically illustrates a horizontal display arrangement; illustrates a plurality of user views; illustrates a vertical display arrangement; illustrates a circular display arrangement; illustrates an alternative circular display arrangement; illustrates an arrangement of views; illustrates an image display system.

Figure 1 schematically illustrates a horizontal viewing arrangement represented in a virtual space. This arrangement comprises a main view 100, which corresponds to the content that the user wishes to view, and secondary views 110 which correspond to additional content that may be viewed. While represented as rectangles, the views may in fact have a threel dimensional shape - for example, if they are mapped to the inner surface of a sphere for display. It would be apparent to the skilled person that the views also need not be rectangular, but that any suitable shape may be used, such as squares. The main view 100 may be a panoramic image that the user is able to navigate by means of their head motion or controller inputs or the like.

Further secondary views to those above and below the main view 100 may be provided to the left and right of the main view 100, although a greater head motion would be required by the user to access them. This requirement may be advantageous, as information that is sensitive or rarely required may be positioned in these secondary views.

We note that many of the embodiments described below are provided in the context of viewing sporting events. This should not be seen as limiting, however, as the described embodiments may be suitable for use with any media type. For example, information about actors in a movie or links to purchase items seen on-screen could be provided as additional content, or chat functionality or walkthrough information could be provided for a game as additional content. The description with respect to sports is merely to provide clear examples of the implementation of the present arrangement.

In one example, the main view 100 may comprise a video of a sporting event and the secondary views are operable to provide relevant statistics (such as a scoreline or elapsed time), additional media (such as replays of earlier events, tracking of individual players, or alternate viewing angles or positions), contextual content (such as information about other games, or a league table), and/or any other suitable content. In some embodiments, the main view 100 represents at least a substantial portion of the environment in which the event takes place; for example, a whole pitch in a sports match or a whole room in another programme type. At any given time, the user may view at least a subset of the main view 100.

By providing such extra content to a user via secondary views, the main view remains largely unobscured and as such a more immersive viewing experience may be provided to a user. In addition, by allowing replays and the like to be displayed as secondary content ondemand there may be a reduction in the number of interruptions to a user’s viewing, as replays could be omitted from the main view content given that the user can access them on-demand via secondary views. This further improves the sense of immersion that a user experiences.

When using an HMD, the user is able to change their view by moving their head; for example, a user may be provided with a view of a virtual environment that exists behind them if they turn around to face that direction. This is implemented by performing a tracking process on the HMD so as to monitor its position and orientation, with the user’s viewpoint in the virtual environment being updated in dependence upon this tracking.

Figure 2 schematically illustrates a plurality of possible user viewpoints 120 to 123.

In this Figure, it is assumed that the viewpoint 120 acts as a default position when the user initiates a viewing operation - this is because it represents a central portion of the main view 100. Of course, any position could be selected as a default or starting position within the main view 100. By moving their head, the user is able to navigate within the main view 100; for example, the user could follow the action in a sports game as it moves towards a goal from the centre of the pitch. For example, to view a portion of the environment in the direction of viewpoint 121 a user would usually move their head up and to the right. It will be appreciated from this figure that the main view (and indeed potentially one or more secondary views) may be larger than the user’s field of view as presented through the HMD, so that it is necessary for the user to move their head to see other parts of the same view.

However, if the user moves their head such that the displayed viewpoint is outside of the boundaries of the main view 100, such as to the viewpoint 121, then the viewpoint may no longer be occupied entirely by a portion of the main view 100. A number of possible implementations for such a case exist.

In a first example, the user is presented with a view that comprises a portion of the main view 100, a portion of the secondary view 110, and some form of border in between the two to represent a discontinuity between the views. Of course, such a border may not exist and the main view 100 and secondary view 110 may be adjacent; in such a case the viewpoint 121 would comprise only a portion of the main view 100 and a portion of the secondary view 110. In Figure 2, there is an empty part of the viewpoint 121 that is a result of the main view 100 and secondary view 110 being different sizes. In order to avoid such a gap, several methods may be implemented. For example the secondary view 110 may be scaled so as to be the same size as the main view 100, or may move so as to align with the main view 100 so as to eliminate the empty space.

In a second example, the viewpoint 121 is not able to be accessed by a user. Should the user move their head so as to indicate that such a viewpoint is desired, the viewpoint is bounded so as to not move outside of the main view 100. Although this restricts the ability of the user to explore the environment, it reduces the chance of a break in immersion by avoiding the need to display boundary/border imagery or the like. In this example, if the user moves their head more than a threshold amount outside of the main view 100, then the view could be switched to display only content from the secondary view 110.

In a third example, a combination of the above examples is considered. A user may have to provide a greater-than-threshold amount of head motion in order to obtain a view of the secondary view 110; however, once this threshold is exceeded a viewpoint is provided that comprises a portion of the main view 100 and the secondary view 110.

If the user moves their head in order to obtain the viewpoint 122, they are presented with a portion of the secondary view 110 only. In Figure 2, the viewpoint 122 is shown to be a subset of the secondary view 110, however in other embodiments the viewpoint 122 could correspond to the whole of the secondary view 110 so as to enable a viewer to see all of the presented information at once.

The viewpoint 123 has similar considerations to that of the viewpoint 121 in that it extends over the boundary of a view (in this case, the secondary view 110). As above, the user may be presented with a view that is part secondary view 110 and part background/border, or the viewpoint 123 may be limited to the area of the secondary view 110.

Figure 3 schematically illustrates an alternative arrangement of the main view 100 and the secondary views 110. In this arrangement, the views are arranged to be navigated primarily in a vertical direction as opposed to Figure 1 in which they are navigated primarily in a horizontal direction. The choice of which arrangement to use may be dependent on the content that is being shown; for example, the horizontal arrangement may be more suited to a football match and a vertical arrangement to a diving competition, as these are the respective planes in which action occurs. For instance, in a football match most of the action is within 2 metres of the ground, but a pitch is approximately 100 metres long - therefore the viewer would be expected to move their heads side-to-side a lot more often (and to a greater extent) than up and down when attempting to follow the action. Hence more generally, the main view may be oriented to have a major axis parallel to the predominant direction of play or action in the presented content.

The configurations of the views may be determined by a user, content provider or any other relevant party. For example, information could be provided with a stream that specifies that a particular number of secondary views should be provided and the location and/or size of each of these secondary views with respect to the main view.

Figure 4 schematically illustrates a further arrangement of the main view 100 and secondary views 110.

In this arrangement, the main view 100 has a circular profile, with a plurality of secondary views 110 arranged in an annular region surrounding the main view 100. This may be more suited for video content in which the action is relatively stationary or confined to a small area and so the user is unlikely to move their head so much during normal viewing; for example a boxing match, in which the boxing itself is confined to the ring. As in the view arrangements described above, there may be a gap between the main view 100 and secondary views 110 such that a plurality of annular regions is provided.

Figure 5 schematically illustrates an arrangement that is a modified version of that found in Figure 4. In this Figure, a plurality of secondary views are supplied such that there are large secondary views 110, smaller secondary views 111 and secondary views 112 that are smaller still. The manner in which different content is assigned to different sized secondary views may be determined freely by the skilled person.

In a first example, the sizes are able to be set by the user; a user may be most interested in replays of a previous goal or knockout, and information about the scores of matches that are being played concurrently. This content would therefore be provided as secondary views 110. Of lesser importance could be a relevant league table and a team’s recent performance information (such as a win/loss record), which could be provided as secondary views 111. Of an even lesser importance could be information such as future fixtures and live text information describing events in the current game; these would be assigned to the smallest secondary views 112.

In a second example, the sizes are determined in a similar manner but by the content provider or another appropriate party (such as the content producer).

In a third example, each of the secondary views may relate to replays of in-game action. The secondary views 110 may be assigned to the most important (such as goals) or most recent replays, with the smaller sizes being used for less important and/or less recent content. A determination of the importance of different clips may be performed in any suitable manner, including by the content provider or user.

It will be appreciated that the circular views of Figures 4 and 5 may similarly be hemispherical views, with the secondary views optionally placed beyond the user’s peripheral vision when facing in a default position. This approach can be taken further to place the main view in a first hemisphere and the secondary view or views in the second hemisphere.

When providing secondary views, it is apparent that it may be desirable to provide a large number of options to the user. In such a case, the views themselves may be too small to be useful to the user initially. In this case, when a user has focussed on a secondary view it is enlarged so as to fill the user’s field of view. It is also noted that the varying size of secondary views may also be implemented with other-shaped arrangements, and not just the circular arrangement.

Alternatively, or in addition, secondary views may be provided in ‘tiers’ about the main view. For example, in the arrangement of Figure 4 a second annular region could be provided that surrounds the first. This second region could comprise another group of potential views. The tiers could be determined by importance or relevance to the content in the main view, for instance, such that more important or relevant content appears closer to the main view. This may enable a greater range of content to be provided without needing to reduce the size of individual secondary views.

As a further alternative or addition, a user may provide an input so as to vary which secondary screens are available at a given time. For instance, a user may be able to access match stats in secondary screens when moving their head normally, but if they press/hold a button, perform a gesture, or (for example) move their head with one eye closed then they are able to access replays of match content instead. This may be advantageous both in the number of secondary views that are able to be conveniently provided and in that textual and video content may be separated - it would be more disruptive to a user’s viewing experience if they moved to a secondary view for video content than if they moved to a secondary view that comprised written information. Of course, any other categorisation of content may be used in an implementation of this.

Upon indicating that a secondary view is desired, the system may provide the content corresponding to this view in a number of manners.

In one example, as described above, the secondary view may be video or image content that is able to be viewed in the same manner as the content of the main view, and appears adjacent to the main view. The user is therefore able to look in the direction of the secondary view and view the content as they would in the main view.

In another example, the user’s head motion is used to ‘select’ a second view. In response to this, a split-screen, picture-in-picture or overlay may be provided so that the user is able to see content from both the main view and the secondary view simultaneously. While an overlay or split-screen may not be desirable in the long term, it allows a user to view secondary content without missing out on any action from the main view.

Figure 6 schematically illustrates an arrangement of a plurality of main views 200, 210 and 220. These main views may be linked, such as different matches being played in the same league simultaneously, or they may be completely independent, such as corresponding to programmes on neighbouring channels.

A user may be able to access the secondary content for the other main views, in some embodiments. For example, a user may be viewing main view 200 but wish to see a replay that is in the secondary content of main view 210. The user is then able to look up, and then right, to view this content (or any other appropriate operation). This may be advantageous in that a user is able to keep up with many matches simultaneously, whilst also achieving an immersive experience when viewing the desired content. In other embodiments, such an action may be prohibited - for example, if the secondary content of another stream is deemed unrelated or undesirable.

A user may be able to switch between main views in the same manner as between a main view and a secondary view, although a greater threshold for switching views may be applied as it may be a great disruption to a viewer to switch main views unintentionally.

After switching to a new main view, the relative positions of the main views may be adapted so as to allow a user to resume their natural viewing position rather than looking off to the side indefinitely. For example, if a user wished to view main stream 220 they would look to the left for a predetermined amount of time (or whatever the ‘switch’ action is chosen to be). Once the switching is performed, the main view 220 moves to the right so as to take the central position previously occupied by main view 200. This can improve the comfort of the user, and thus the level of immersiveness that is experienced.

The content that is provided in the secondary views may be obtained in any of a number of appropriate manners.

For example, the content may be provided in a stream with the content for the main view. Multiple video streams could be packaged together in a manner that identifies a main view and any secondary views.

Alternatively, or in addition, content may be identified in metadata associated with the content for the main view, and the display apparatus (or an associated processing device) may be operable to obtain the content using this identifying information (for example, hyperlinks identifying online resources). Alternatively, or in addition, contextual data may be provided with the content for the main display that is used to find appropriate content; for example, the contextual data could name the teams that are playing and state which competition they are playing in. This could then be used to find the league table for the appropriate competition, or any other relevant information.

Notably, the present invention therefore provides at least the following advantage; a user is able to enjoy an immersive experience as a spectator at a sports match or other event, in which they are free to look around within a typically static main view (i.e. the main view is captured from a specific location, such as beside the pitch). This allows for comfortable sustained viewing. However, they can also access secondary views that may include content such as more conventional sports coverage, with unnatural or inconsistent viewpoints caused by panning and/or zooming shots and/or changes of viewpoint (i.e. different cameras) that are used to track a particular event (such as the progress of a football player up the pitch towards the goal). Such a secondary view provides more detail and insight than is possible from the spectator position, but could easily cause the user discomfort over a sustained period because they are not in control of the view, and unnatural and/or unbidden changes in view can cause nausea. Consequently by separating these modes of viewing the event In this way, the user can enjoy the immersive atmosphere of the event in comfort and freely look around within the main view, which is oriented to track the main axis or direction(s) of action in the event, whilst also being able to easily access close-up views and different perspectives, just by looking sufficiently far in a direction other than (e.g. perpendicular or orthogonal to) the main axis of action, or by turning their head a sufficient angle away from a static scene (such as a boxing ring).

Figure 7 schematically illustrates an image display system 700 comprising a view obtaining unit 710, a motion information capturing unit 720, an image display unit 730 and 740. While Figure 7 shows these each being present in a single unit, it should be understood that any processing may be performed at a remote device (such as at an associated processing device) and the results transmitted to the image display system 700. In one example, the image display system 700 comprises a games console and an associated HMD that are together operable to generate and display images in the described manner.

The view obtaining unit 710 is operable to obtain a main view and one or more secondary views; as noted above, the main view may be a panoramic video which the user is able to navigate using head motion. The secondary views comprise additional content that may be relevant to the main view.

The view obtaining unit 710 may obtain a main view from any appropriate location; for example, a television broadcast, a server or a local storage medium. The secondary views may obtained in association with the main view, such as in the same file or stream, or the secondary views may be generated or accessed in response to metadata obtained with the main view. In some embodiments, the view obtaining unit 710 obtains a plurality of main views, each with associated secondary views.

The motion information capturing unit 720 is operable to capture information about the head motion of a user; this information may relate to the position and/or orientation of the user’s head, as is useful for the application. For example, this may be performed by analysing images obtained by a camera arrangement, or using accelerometers or the like to directly track motion.

The image display unit 730 is operable to display an image to the user, the image being a portion ofthe main view. The content ofthe image is determined in response to the position and orientation of the user’s head. If the user’s head motion exceeds a threshold amount, for example so as to correspond to generating an image that it outside of the main view, the image display unit is operable to display at least a portion of a secondary view.

The threshold amount of motion may be different in different directions, for example in Figure 1 a smaller motion is required to access the secondary views 110 above and below than any present to the left and right of the main view 100. The threshold amount of motion may be determined in dependence upon the content ofthe main view.

The view management unit 740 is operable to define relative positions of the main view and the one or more secondary views in a virtual space. This may be performed in dependence upon the content that is being displayed in the main view, the content that is to be shown in the secondary views, user settings and/or any other appropriate factors.

The techniques described above may be implemented in hardware, software or combinations of the two. In the case that a software-controlled data processing apparatus is employed to implement one or more features of the embodiments, it will be appreciated that such software, and a storage or transmission medium such as a non-transitory machinereadable storage medium by which such software is provided, are also considered as embodiments of the disclosure.

Claims

Claims:

1. An image display system comprising:

a view obtaining unit operable to obtain a main view and one or more secondary views; a motion information capturing unit operable to capture information about the head motion of a user; and an image display unit operable to display an image to the user, wherein the image is a portion of the main view that is determined in response to the position and orientation of the user’s head, wherein if the user’s head motion exceeds a threshold amount, the image display unit is operable to display at least a portion of a secondary view.

2. An image display system according to claim 1, wherein the user’s head motion exceeds a threshold amount in a direction perpendicular to the direction of the main axis of the main view.

3. An image display system according to claim 1 or claim 2, comprising a view management unit operable to define relative positions of the main view and the one or more secondary views in a virtual space.

4. An image display system according to claim 3, wherein the relative positions are defined in dependence upon one or more of:

a) the content that is being displayed in the main view;

b) the content that is to be shown in the secondary views; and/or

c) user settings.

5. An image display system according to claim 1, wherein the one or more secondary views may comprise any of:

a) statistics about the events shown as the main content;

b) additional media content related to the main content; and/or

c) contextual information related to the events shown as the main content.

6. An image display system according to claim 1, wherein the secondary view that is displayed is dependent upon the direction in which the user is looking.

7. An image display system according to claim 1, wherein the secondary view is displayed to the user at the same time as the main view.

8. An image display unit according to claim 7, wherein the secondary view is displayed to the user in a mode selected from the list consisting of:

i. a split-screen format or as an image overlay for the main view; and ii. instead of the main view.

9. An image display unit according to claim 1, wherein the threshold amount of motion is one or more selected from the list consisting of:

i. different in different directions; and ii. determined in dependence upon the content of the main view.

10. An image display unit according to claim 1, wherein the secondary views are generated in response to metadata obtained with the main view.

11. An image display unit according to claim 1, wherein the secondary views are obtained in association with the main view.

12. An image display unit according to claim 1, wherein a plurality of main views is obtained, each with associated secondary views.

13. An image display method comprising:

obtaining a main view and one or more secondary views; capturing information about the head motion of a user; and displaying an image to the user, wherein the image is a portion of the main view that is determined in response to the position and orientation of the user’s head, wherein if the user’s head motion exceeds a threshold amount, at least a portion of a secondary view is displayed.

14. An image display method according to claim 13, wherein the user’s head motion exceeds a threshold amount in a direction perpendicular to the direction of the main axis of the main view.

15. A computer program which, when executed by a computer, causes a computer to perform the method of any one of claims 13 and 14.

io

Application No: GB1701237.8