JP6703627B2 - Device and related methods - Google Patents

Description

The present disclosure relates to the field of virtual reality, and more particularly to displaying missed event content from virtual reality content. In particular, the present invention relates to the display of missed event content from virtual reality content whose temporal starting point is based on audio events. Related methods, computer programs, and apparatus are also disclosed. Certain disclosed aspects/examples relate to portable electronic devices.

background

Virtual reality may provide a user with an immersive virtual experience using a headset such as glasses or goggles or one or more displays surrounding the user. The virtual reality device may also present multimedia virtual reality content representing the virtual reality space to the user to simulate the user's presence in the virtual reality space. Virtual reality space may also be provided by panoramic video, such as video with a wide field of view or a 360° field of view (which may include above and/or below the horizontal field of view). In addition, spatial audio with sounds that appear to be occurring at a particular location may be provided as part of virtual reality consumption. The fact that the video images that make up the virtual reality space are larger than the user can see at one time results in the user not being able to hear and see the event.

Any listing or description of a previously published document or background in this specification is not necessarily to be construed as an admission that such document or background is part of the state of the art or is general knowledge. I won't. One or more aspects/examples of the present disclosure may or may not address one or more of the background issues.

Summary

In a first exemplary aspect, there is provided an apparatus as follows, comprising at least one processor and at least one memory containing computer program code. The apparatus comprises a virtual image comprising a video image configured to, when the computer program code is executed by the at least one processor, provide the apparatus with at least a virtual reality space for viewing in virtual reality. Configured to cause providing a display of missed event content based on a user's change of the viewing direction of the virtual reality view from the first viewing direction to the second viewing direction for the real content. ..
Here, in the virtual reality content, a virtual reality view presented to a user provides a view of the virtual reality content, and the virtual reality view comprises a spatial part of a video image forming a virtual reality space. The spatial extent is smaller than the spatial extent of the video image of the space, the virtual reality content includes spatial audio, and the perceived direction to the audio event related to the visual event in the video image is in the virtual reality space. Corresponds to the visual position of the presented visual event.
Also, the second viewing direction matches the direction of a particular audio event whose start has occurred before the change of the viewing direction within a direction threshold.
Further, the missed event content includes at least a visual image extracted from a video image of the virtual reality content. The visual image shows a visual event corresponding to a specific audio event, and the temporal starting point of the missed event content is the temporal point in the virtual reality content in which the change in the viewing direction occurs. Before and based on a specific audio event.

As a result, the user may be able to see missed visual events that the user may have heard by listening to the relevant audio event and reacting to it by changing the viewing direction. Spatial audio, otherwise known as three-dimensional audio, is used to make audio perceived by a virtual reality user (consumer) so that the user perceives the audio events that make up the overall audio environment from a particular direction. The user may also hear an audio event whose corresponding visual event is outside the user's current virtual reality view. By the time the user sees the sensed direction of an audio event, for example, if the user reacts by moving his/her head and thus moving the virtual reality view in order to see a visual event related to the audio event just heard. The user may have missed some or all of the relevant video images (visual events) associated with the audio event. Thus, the device may "rewind" the virtual reality content so that the missed event content is visible to the user and, consequently, the visual event content associated with the audio event heard. For example, a user is looking ahead (ie, a first view direction) to a particular group of football players and then behind it (ie, outside the user's current view, in a second view direction). If the user hears a sound (audio event) that is perceived as coming from, for example, a player yells "give me a pass", the user may want to see why the player is yelling. Therefore, the user turns his/her head to change the viewing direction from the first viewing direction to the second viewing direction. Not surprisingly, as time has progressed since the user decided to look in the second viewing direction, the user missed the visual content associated with the player yelling "give me a pass". .. However, the missed event content may allow the user to see unseen visual events associated with audio events that the user has indeed heard. Thus, the device provides at least a visual image display and optionally an accompanying audio from a time based on the audio event in the virtual reality content so that the user can see the missed visual event. provide.

In one or more embodiments, the visual event associated with the particular audio event was not visible in the virtual reality view when oriented in the first viewing direction. In one or more embodiments, in the virtual reality view when oriented in the first viewing direction, the visual event is not focused on by the user when the corresponding specific audio event occurs. It was

In one or more embodiments, the temporal starting point for missed event content is based on the time a particular audio event is considered to have started based on one or more of the following:
Analysis of a specific audio event for any other audio presented,
Predetermined audio event data that at least defines the analysis of a specific audio event and the timing of the audio event in the virtual reality content.

In one or more other examples, the time at which a particular audio event is considered to have begun may be determined by any suitable method.

In one or more embodiments, the provision of the missed event content is further based on a degree of change in the viewing direction between the first viewing direction and the second viewing direction exceeding a direction change threshold. This is because the redirection threshold can be used to distinguish between a user casually looking around the virtual reality space and looking at the video image presented therein, and a user who is audible, i.e. responding to an audio event. , May be advantageous. The direction change threshold is more than 0.05 degree/millisecond, more than 0.06 degree/millisecond, more than 0.07 degree/millisecond, more than 0.08 degree/millisecond, more than 0.09 degree/millisecond, Or it may be more.

In one or more embodiments, the provision of missed event content is specific to an audio threshold (which may be an absolute audio threshold or a relative threshold to any other audio presented to the user). It is further based on one or more of the volume or frequency components of the audio event. This may be advantageous as the volume or frequency components may be used to identify audio events that the user may have responded to, and correspondingly provide appropriate missed event content. Changing the direction of a virtual reality (VR) view may provide identification of one or more audio event candidates using an audio event threshold, of which a second viewing direction is audio specific. A particular audio event is identified based on matching the direction of the event within a direction threshold.

In one or more embodiments, the determination of a particular audio event is based on one or more of the following:
a) predetermined audio event data with directions to one or more audio events occurring in the virtual reality content, and a second viewing direction within a direction threshold, and one of the one or more audio events. Interaction with direction,
b) An audio analysis of the spatial audio in response to the change of viewing direction to identify a particular audio event in the viewing direction of the user in the second viewing direction.

In one or more examples, the audio analysis is based on one or more of the volume and frequency components of spatial audio relative to the audio threshold.

In one or more embodiments, the predetermined audio event data is based on one or more of the following:
Audio analysis, which may include volume-based, frequency-based, and/or voice-recognition-based analysis,
Manually provided audio event data,
Sensor data obtained from sensors associated with objects captured in VR content,
Video analysis, which may include lip-reading analysis and/or action identification algorithms looking for localized video images that change rapidly above a threshold.

Therefore, in one or more examples, the device may have a predetermined knowledge of audio events, and in order to provide missed event content, a second viewing direction of the VR view and a known audio event may be used. It may be configured to match one. In one or more examples, the device may have no knowledge of the audio event and may be a loud part (more generally a volume characteristic) that may have prompted the user to change the viewing direction, or Spatial audio may be analyzed for portions (more generally frequency characteristics) that include a particular frequency. The spatial audio analysis may be constrained to a direction of the second viewing direction that is within the direction threshold. In response to the audio event being identified by the analysis, the missed event content may be provided to indicate a visual event associated with the identified audio event. The temporal starting point of missed event content may be based on audio analysis or audio event data. For example, the starting point may be based on when a loud part of the spatial audio occurs, or on the part containing a particular audio frequency. Of course, the temporal starting point may start a predetermined period before the start of the audio event, which may help the user understand the cause of the audio event.

Thus, in one or more examples, one or more audio events may be determined on-the-fly, or may be predetermined (audio event data provided with virtual reality content), and the device may It may be configured to determine a user's interest in a particular audio event based on a VR viewing direction matching one of the audio event candidates. Alternatively, or in addition, the device may be configured so that a change in viewing direction by a user provides an identification process that identifies a particular audio event that the user may have responded to. In summary, in response to a change in viewing direction, the device provides an analysis of spatial audio within a direction threshold of a second direction to identify the audio event as a particular audio event that the user may have responded to. You may. As a result, appropriate missed event content may be provided.

In one or more embodiments, displaying the missed event content includes providing a temporary stop of the virtual reality content and displaying the missed event content in a virtual reality view.

In one or more embodiments, in response to a change in viewing direction towards a particular audio event, a point in time at which the virtual reality content appears to the user to rewind until the point at which the missed event content begins. On the contrary, it provides a display of virtual reality content.

In one or more embodiments, displaying the virtual reality content conversely in time may begin when the user stops in the second viewing direction. In one or more embodiments, displaying virtual reality content conversely in terms of time causes the user to initiate a viewing direction change based on a second viewing direction prediction that may cause the user to stop. It may start with. For example, it may be possible for the audio event data to indicate an audio event in the general direction in which the user modifies the VR view, so that the user may turn around towards the audio event.

In one or more embodiments, the missed event content comprises one or more of the following:
i) Missed event video images extracted from virtual reality content video images,
ii) Missed event still images extracted from the virtual reality content video images,
iii) virtual reality content of a missed event extracted from a video image of the virtual reality content, wherein at least an initial viewing direction of the virtual reality content of the missed event matches the direction of the particular audio event, Virtual reality content for missed events,
iv) Virtual reality content of the missed event extracted from the video image of the virtual reality content, the virtual reality content of the missed event having a spatial extent less than the spatial extent of the virtual reality content, Virtual reality content for missed events that focuses on the direction of the audio event.

In one or more embodiments, missed event content is provided for display as picture-in-picture in a virtual reality view along with virtual reality content. In this example, the virtual reality content is not interrupted in time, so that the virtual reality content continues to be displayed while at the same time indicating a temporally previous missed visual event.

In one or more embodiments, when the display of the missed event content is complete, the virtual reality content is displayed from a point in time that coincides with the time of changing the viewing direction from the first viewing direction to the second viewing direction. It will be restarted. Resuming the display of the virtual reality content may use the current viewing direction of the virtual reality view, which may be approximately the second viewing direction.

In one or more embodiments, upon completion of display of missed event content, virtual reality content is provided for display at a playback speed that is at least temporarily higher than real time. This may provide playback of virtual reality content to catch up to where the missed event content would have been played if not provided for display. This may be particularly useful if the virtual reality content comprises raw virtual reality content rather than pre-recorded virtual reality content.

In one or more embodiments, the orientation threshold may be less than 90°, less than 70°, less than 45°, or less than 30°. In one or more embodiments, the user may not be able to accurately see the source of sound heard only for a short time, where the directional threshold provides a tolerance for identifying a particular audio event. Therefore, the direction threshold may be advantageous.

In a second aspect, a method is provided. The method is
Virtual reality content comprising video images configured to provide a virtual reality space for viewing in virtual reality, wherein a virtual reality view presented to a user provides viewing of the virtual reality content, The view comprises a spatial portion of the video images that make up the virtual reality space, has a smaller spatial extent than the spatial extent of the video images in the virtual reality space, and the virtual reality content includes spatial audio and Of virtual reality content, wherein the perceived direction to the audio event associated with the visual event corresponds to the visual position of said visual event presented in virtual reality space,
A change in a viewing direction of a virtual reality view from a first viewing direction to a second viewing direction by a user, wherein the second viewing direction is a specific audio whose start occurs before the viewing direction change. Based on the changes that match the direction of the event within the direction threshold,
Providing a display of missed event content, the missed event content having at least a visual image extracted from a video image of the virtual reality content, the visual image being at a particular audio event. It shows the corresponding visual event, and the missed event content has a temporal start point before the temporal point in the virtual reality content where the viewing direction change occurred and is based on the particular audio event. To provide,
including.

In a third aspect, there is provided a computer readable medium comprising computer program code stored on the computer readable medium, the computer readable medium and the computer program code executing on at least one processor,
Virtual reality content comprising video images configured to provide a virtual reality space for viewing in virtual reality, wherein a virtual reality view presented to a user provides viewing of the virtual reality content, The view comprises a spatial portion of the video images that make up the virtual reality space, has a smaller spatial extent than the spatial extent of the video images in the virtual reality space, and the virtual reality content includes spatial audio and Of virtual reality content, wherein the perceived direction to the audio event associated with the visual event corresponds to the visual position of said visual event presented in virtual reality space,
A change in a viewing direction of a virtual reality view from a first viewing direction to a second viewing direction by a user, wherein the second viewing direction is a specific audio whose start occurs before the viewing direction change. Based on the changes that match the direction of the event within the direction threshold,
Providing a display of missed event content, the missed event content having at least a visual image extracted from a video image of the virtual reality content, the visual image corresponding to a particular audio event. Presents a visual event, and the missed event content has a temporal start point before the temporal point in the virtual reality content where the viewing direction change occurred, and is based on a specific audio event. do,
The method is configured to perform.

In a further aspect, a device is provided, the device being virtual reality content comprising video images configured to provide a virtual reality space for viewing in virtual reality, the virtual reality view being presented to a user. Provides a view of the virtual reality content, and the virtual reality view comprises a spatial portion of the video images that make up the virtual reality space and has a smaller spatial extent than the spatial extent of the video images in the virtual reality space. The content includes spatial audio, wherein the perceived direction to the audio event related to the visual event in the video image corresponds to the visual location of the presented visual event in virtual reality space, Regarding virtual reality content,
A change in a viewing direction of a virtual reality view from a first viewing direction to a second viewing direction by a user, wherein the second viewing direction is a specific audio whose start occurs before the viewing direction change. Based on the changes that match the direction of the event within the direction threshold,
Providing a display of missed event content, the missed event content having at least a visual image extracted from a video image of the virtual reality content, the visual image corresponding to a particular audio event. Presents a visual event, and the missed event content has a temporal start point before the temporal point in the virtual reality content where the viewing direction change occurred, and is based on a specific audio event. do,
And means configured as described above.

This disclosure includes one or more corresponding aspects, examples, or features, alone or in various combinations, whether or not specifically described (including claimed) in that combination or alone. Associated means and associated functional units for performing one or more of the functions described (eg functional enabler, video image extractor, video image compiler, view direction measuring device, view direction changer, video player, direction sensor). ) Is also within the scope of the present disclosure.

Corresponding computer programs implementing one or more of the disclosed methods are also within the scope of the present disclosure and included in one or more of the described examples.

The above summary is intended as illustrative only and not limiting.

The following description is given, by way of example only, with reference to the accompanying drawings.

An exemplary embodiment of a device is shown with a virtual reality device. FIG. 3 shows an exemplary plan view showing a user looking in a first viewing direction in a virtual reality space and a position of occurrence of an audio event. FIG. 6 illustrates an exemplary plan view showing a user in a virtual reality space looking at a second viewing direction that has responded to an audio event but missed a related visual event. 4 illustrates example missed event content. FIG. 7 illustrates continuation of virtual reality content after display of missed event content. 3 illustrates exemplary audio event data. An example of audio analysis is shown. 3 shows a flow chart illustrating an exemplary method. 3 illustrates a computer-readable medium.

Description of example aspects

Virtual reality (VR) may provide a user with an immersive virtual experience using a headset such as glasses or goggles or one or more displays surrounding the user. The virtual reality device may also present multimedia virtual reality content representing the virtual reality space to the user to simulate the user's presence in the virtual reality space. The virtual reality space may reproduce the real world environment to simulate the physical presence of the user in the real world position. Alternatively, the virtual reality space may be computer-generated or a combination of computer-generated content and real-world multimedia content. Virtual reality space may also be provided by panoramic video, such as video with a wide field of view or a 360° field of view (which may include above and/or below the horizontal field of view). In essence, virtual reality space provides a virtual three-dimensional environment in which video images of VR content are provided for display. The user is provided with a spatial subset of the virtual reality space as a viewing window for viewing the VR view, ie the video image of the VR space. The virtual reality device may provide user interaction with the displayed virtual reality space. The virtual reality content provided to the user may comprise a real world live or recorded image captured by the virtual reality content capture device, eg, a panoramic video capture device or a virtual reality content capture device. .. An example of a virtual reality content capture device is a Nokia OZO camera. The virtual reality space may provide a field of view of 360° or more, and may provide pan/rotation of the field of view based on the movement of the VR user's head or eyes. A virtual reality view of the virtual reality space may be provided to the user by the virtual reality device via the display of the headset. The virtual reality space may be seen by the user of the VR device as a three-dimensional space created from the image of the virtual reality content. In essence, VR content may also comprise images obtained from multiple viewing directions that can be displayed and arranged together to form a (continuous and uninterrupted) wrap-around view.

By its very nature, virtual reality content may be immersive and, as a result, may include large amounts of data. Virtual reality content may also include video images (ie, moving images) that have a large spatial extent, such as surrounding a user. The virtual reality view presented to the user may only cover the area of the video image that is smaller than the total area, and the user may be required to view the entire spatial extent of the virtual reality space covered by the video image in order to You may need to move the view around. Of course, for VR video images that progress over time, it is not possible for the user to see all of the spatial portion of the video image at any given time, resulting in the event being essentially missed.

VR content may also be presented with spatial audio. Spatial audio, otherwise known as 3D audio or directional audio, presents audio to the virtual reality user (consumer) so that the user perceives each element of audio as originating from a particular direction. To do. Therefore, the audio presented to a user consuming VR content is said to have originated from a particular direction, as well as peripheral or omnidirectional audio such as background music that is perceived as not coming from a particular direction. And directional audio sensed by. Not surprisingly, the events in the virtual reality content produce accompanying audio, such as spoken by a character, the car speeds up the engine, the hockey puck is hit, and so on. Thus, the voice of the speaking character, which visually appears to the left of the user's virtual reality view, may be presented as directional audio originating from the left hand side. Here, the audio event may comprise at least a time-based portion of the directional audio associated with the visual event occurring in the virtual reality content. Accordingly, the spoken monologue audio event may be associated with the visual event of the character speaking the monologue in the visual image of the VR content. A “bashing” audio event may be associated with a visual event where the hockey puck is struck by a hockey player, because the hockey puck's striking produced that sound. Of course, the spatial audio presented to the user may also include numerous directional sounds, or audio events, that are perceived as originating from different directions.

As mentioned above, the virtual reality view comprises only part of the total spatial extent of the video image of the VR content, but the audio presented to the user, like reality, occurs in a direction outside the spatial extent of the virtual reality view. It may also include directional audio that is perceived as being present. Thus, a user may hear directional audio but not see the associated visual event because he was looking in a different direction when the audio event occurred.

The user may react as soon as the audio event is heard, changing the direction of the VR view to see the sensed direction of the audio event, but missed the associated visual event in the video image. It may be. This can be confusing to the user and can make it difficult for the user to understand the VR content, or to view the content several times in different viewing directions to fully understand the event that occurred. It may take.

FIG. 1 is configured to provide a display of missed event content comprising at least a visual image extracted from a video image of virtual reality content showing a visual event corresponding to a particular audio event heard by a user. 1 shows a device 100. Thus, the user may be able to see a visual event that has heard the associated audio event.

The apparatus 100 may be part of a virtual reality device 101 configured to present virtual reality content to a user. Virtual reality device 101 may present a VR view of VR space to a user via VR display 102, which may include a headset. The VR view provided to the user on the VR display 102 may be based on the orientation of the VR display so that the user can look around the VR space by moving his head. Of course, other types of displays and other means for "looking around" the virtual reality space may be provided. In some examples, the device 100 is functionally provided by a computer, a virtual reality device 101, or a computer server, each of which may include a memory 101a and at least one processor 101b, while in other examples the device is a computer, It may be an electronic device such as a mobile phone or other device listed below that communicates with the VR device 101. In this example, virtual reality device 101 is configured to receive virtual reality content from virtual reality content store 103, where virtual reality content may be stored (which may include transient or temporary storage). Thus, the virtual reality content may be raw content and the store 103 may be a memory or a buffer of the transmission path of the display or beyond. The device 100 provides an indication of where in the virtual reality space the user is looking, as well as the location of the audio event for the audio presented to the user, the audio itself, and audio analysis information. The instructions may be received. Audio event data may be predetermined or may be generated by device 100 or another device 105.

VR content may be provided by virtual reality content capture device 104. Another device 105 may be configured to create audio event data, which may be stored in VR content store 103 along with VR content. In another one or more examples, another device 105 is not provided and the occurrence and direction of the audio event may be determined by the device 100 or related devices.

In one or more examples, audio event data may be provided manually based on human analysis of virtual reality content during capture or after production, thus another device 105 represents a data entry of audio event data. You may. In one or more examples, audio event data may be automatically generated based on computer-based audio analysis of spatial audio, which may be provided during the production of VR content, or post-production. May be provided. In one or more examples, the audio event data can be provided by the device 100 and/or VR device 101, etc., as part of preparing virtual reality content for display to a user, or while displaying virtual reality content to a user. Generated on the fly. Of course, the content capture device 104 and another device 105 are shown connected to the VR content store, but this is only for understanding the entire system, and device 100 is another device 105. The recorded VR content may be used regardless of whether or not the operation is performed.

In this embodiment, the apparatus 100 (or other electronic device) described above may have only one processor 101b and one memory 101a, although, of course, other embodiments include more than one. A processor and/or more than one memory (eg, the same or different processor/memory types) may be utilized. Further, the device 100 may be an application specific integrated circuit (ASIC).

The processor may be a general purpose processor that only executes/processes information received from the content store 103 and other components such as VR device 101 in accordance with instructions stored in the form of computer program code in memory. The output signaling generated by such operation of the processor is provided to further components, such as VR device 101, for display of missed event content.

Memory (not necessarily a single memory unit) is a computer-readable medium that stores computer program code (solid state memory in this example, but is not limited to a hard drive, ROM, RAM, flash, or the like). It may be another type of memory). The computer program code stores instructions executable by the processor when the program code is executed on the processor. An interconnect between the memory and the processor, in one or more exemplary embodiments, provides an active coupling between the processor and the memory to allow the processor to access computer program code stored in the memory. I can understand.

In this example, the processor and memory are all electrically connected to each other internally, allowing electrical communication between the individual components. In this example, the components are all located in close proximity to each other such that they are configured together as an ASIC, in other words integrated together as a single chip/circuit that can be installed in an electronic device. In some examples, one or more or all of the components may be located separately from each other.

FIG. 2 shows an exemplary plan view showing a user 200 wearing a VR display 102, which in this example comprises a headset. The user 200 is watching the VR content 202. The VR content 202 is depicted graphically as a ring that surrounds the user, similar to how a video image of the VR content is displayed in multiple viewing directions so as to surround the user in virtual reality space. In FIG. 2, a user 200 is looking at a portion of a video image of VR content defined by lines of sight 204, 205 in a first viewing direction 203 in virtual reality space.

FIG. 2 further illustrates the occurrence of audio event 206. The audio event 206 comprises spatial audio and is therefore perceived by the user 200 as originating from a particular direction. Audio event 206 is related to visual event 207. Since the user 200 is looking in the first direction and the visual event 207 occurs in the second direction, it can be seen that the user 200 does not see the visual event 207. It is shown. The audio event comprises the sound of the hockey stick 208 hitting the puck 210. Accordingly, the visual event 207 associated with the audio event 206 comprises a visual image of the source of the sound that comprises the audio event 206. In this example, the source of sound for the audio event 206 is that the player 211 hits the puck 210 with his hockey stick 208.

FIG. 3 shows the user 200 turning his head to see the sensed direction of the audio event 206, the second direction 300, perhaps in response to the audio event 206. Therefore, the viewing direction of the virtual reality view is changed from the first direction 203 to the second direction 300. In this example, the change in the viewing direction of the VR view is performed by turning the head of the user 200, but in another example, the control of the VR view may be achieved by another method.

As shown in FIG. 3, by the time the user 200 changes the VR view in response to the audio event 206, he has missed at least some of the visual events 207 associated with the audio event 206. In this example, the audio event 206 was relatively short, so the VR view 301 presented to the user 200 is now later in time than the visual event 207 and is happy to happen after the player 211 hits the puck 210. It seems to be there. Therefore, the user 200 misses the visual event 207 related to the audio event 206 that was heard.

As described with respect to FIG. 4, in this example the device 100 is within the direction threshold from the first viewing direction 203 to the second viewing direction 300 to the direction 300 of the particular audio event 206 heard by the user. Is configured to provide a display of missed event content based on a change in the viewing direction of the virtual reality view by the user 200 in the second viewing direction 300 that is consistent with. Thus, depending on the length of time of the audio event 206 and how quickly the user 200 reacts to it, the second direction 300 may also be the direction of the audio event 206 in progress, or recently or immediately before. May be the direction of the audio event 206 that has occurred. In determining the audio event 206 that the user 200 has responded to, a time window extending from the current time to the past by a near time threshold may be used. The near threshold is the last 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, 8 seconds, 9 seconds, 10 when determining the particular audio event that the user 200 turned around. It may be set to include any audio event that occurred in seconds or more. Moreover, it should be appreciated that the user does not have to look closely at the direction of the audio event 206 that the user heard. Therefore, the second direction 300 may not exactly match the direction of the audio event 206. A direction threshold may be used to provide a tolerance between the second direction 300 and the direction of the audio event 206. The direction threshold may be set to less than 5°, less than 10°, less than 15°, less than 20°, less than 30°, less than 40° or more. The direction threshold value may have different values in the horizontal plane and the vertical plane, for example, to resemble the recognition accuracy of the human listening direction.

The user action of changing the viewing direction from the first viewing direction 203 to the second viewing direction 300 may prompt the device 100 to provide a display of missed event content. The matching of the second viewing direction with the recent audio event prompts the device 100 to identify this recent audio event as the particular audio event the user turned to and provide a display of the missed event content 400. May be. The change in viewing direction may prompt the device to identify the particular audio event that the user responded to and then provide an indication of the missed event content. The device responds to the user confirming that he/she wants to view the missed event content after the device provides a prompt to the user that the missed event content is available. May be provided. Thus, the device 100 may be configured to identify the particular audio event that the user has responded to by the change in viewing direction, and then present the user with a prompt or option to view the missed event content 400. May be provided. Since the matching between the second viewing direction and the direction of the recent audio event may occur frequently, the display of the missed event content or the prompt regarding the display may be troublesome. Therefore, when the device 100 determines whether to provide a display or prompt for displaying the missed event content, the device 100 makes that determination in the viewing direction between the first viewing direction 203 and the second viewing direction 300. May be configured to make a change based on a direction change threshold exceeding a direction change threshold. This may allow the device to distinguish between a user 200 casually looking around the video image presented in the virtual reality space and a user 200 reacting to a particular audio event 206. The direction change thresholds (direction changes are, for example, toward the audio event 206) are greater than 0.05 degrees/milliseconds, greater than 0.06 degrees/milliseconds, greater than 0.07 degrees/milliseconds, 0. It may be greater than 08 degrees/millisecond, greater than 0.09 degree/millisecond, or higher. Information about the degree of change may be provided by VR device 101 and/or VR display 102. If the rate of change is relatively fast, the user 200 may be presumed to have responded to the audio event, thus providing a display or prompt for the missed event content 400. If the degree of change is relatively slow, it may be assumed that the user did not react to the audio event and the missed event content is not provided for display. Alternatively, or additionally, a prompt asking the user if they want to see the missed event content may not be provided for display.

FIG. 4 shows three screenshots of example missed event content 400. Of course, the three exemplary frames 401, 402, 403 shown in FIG. 4 represent continuous video with missed event content. In this example, the missed event content 400 comprises a video with a visual image of a visual event 207 associated with an audio event heard by the user 200. In other examples, the missed event content may be a series of still images, a flat video (ie, you cannot look around virtual reality space to see different parts of the video), or virtual reality content. In the present example, the missed event content comprises at least a visual image showing a visual event 207 corresponding to a particular audio event 206 and extracted from a video image of the virtual reality content. Prior to displaying the missed event content 400, or as part of the missed event content 400, the device 100 may provide rewind of virtual reality content. In essence, the device may provide the user with an indication of the virtual reality content being played back to indicate that the content that the user is about to see has occurred in the past. Thus, the virtual reality content may appear to the user to be "rewound" to the point where the missed event content begins.

Thus, the temporal start point of the missed event content 400 is before the temporal point in the virtual reality content in which the change in the viewing directions 203 and 300 has occurred. Moreover, the point in time at which the missed event content begins is based on the particular audio event 206. In one or more examples, the temporal starting point of the missed event content may be the time when the start of the audio event 206 occurs. Therefore, the temporal starting point may be immediately before the sound of the audio event 206. In one or more examples, the temporal start point of missed event content for virtual reality content is a predetermined period before the start of audio event 206, which helps the user understand the cause of the audio event. Can be. Thus, missed event content may start a few seconds before the audio event 206 rather than starting when the audio event 206 occurs. The maximum period is 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, 8 seconds, 9 seconds, 10 seconds, 15 seconds, 20 seconds, 30 seconds, 60 seconds. As a result, the temporal starting point of the missed event content begins this predetermined period before the beginning of the audio event 206.

In the example of FIG. 4, the first frame 401 of the missed event content video (ie, visual image and accompanying audio) begins seconds before the occurrence of audio event 206. The first frame 401 shows a player 211 ready to hit the puck 210. The second frame 402 shows the player 211 hitting the puck 210 and the puck moving to the right. The third frame 403 shows the end of the missed event content 400 and comprises a visual image of the approximate time when the user saw the second direction 300, which is referenced 301 in FIG.

Thus, the missed event content comprises at least a visual image containing the visual event 207. The missed event content 400 may further include at least a visual image a predetermined time before the occurrence of the audio event 206. The missed event content 400 may include at least a visual image after the audio event 206, at least up to the point in time when the user looked at the second direction 300.

Thus, a user who hears the directional audio of the audio event 206 from the second direction 300 is automatically provided with missed event content including at least a visual image of VR content associated with the audio event. Good (this may be a visual image that occurs in the second direction, depending on how accurately the user can identify the direction of the audio event). As a result, the missed event content allows the user to view missed visual events 207 that the user has responded to by listening to the associated audio event 206 and changing viewing directions 203, 300 thereto. To do.

An indication of missed event content may be provided in response to one or more of the following:
i) automatically tracking changes in viewing direction, and identification of audio events,
ii) A user request to view missed event content,
iii) The user's confirmation that he/she wants to see the missed event content in response to a prompt provided to the user based on the change of the viewing direction by the user that the missed event content is available.

The prompt may be a visual graphic provided for display in the VR view provided by the VR display 102. The prompt may be an audible prompt or a tactile prompt. The user request or confirmation of the user may be one or more of a visual command, a voice command, and a command via a user interface, such as a mobile phone or other electronic device in communication with the device.

FIG. 5 shows two exemplary frames showing that virtual reality content continues as missed event content 400 is provided for display. In some cases, such as in the case of pre-recorded content or raw content, missed event content may be provided for display, and then virtual reality content may be displayed as the missed event content. You may continue from the point you left off. However, especially in the case of raw content, this means that the user is no longer watching the raw content and is delayed by the length of the missed event content (buffered by device 100 or VR device 101). May also mean that you are looking at). Thus, FIG. 5 shows that the device 100 has a VR content from a) when the virtual reality content is interrupted to display the missed event content to b) when the VR content is displayed live to the user. 7 illustrates how virtual reality content display may be provided at an increased playback speed relative to the normal playback speed.

Thus, FIG. 5 shows the last frame 403 of the missed event content 400 and the continuation of VR content displayed in virtual reality, but at an increased playback speed. Frame 500 shows a sliding player away 211 with an increased playback speed. The virtual reality content is then for display to the user when the VR content displayed to the user "catch up" with the raw VR content received by the VR device 101, at its normal, eg real-time playback speed. Provided to.

To display the appropriate visual content as part of the missed event content, the device may identify the particular audio event 206 that the user 200 reacted to. This can be achieved in a number of ways.

For example, the device 100 may have access to predetermined audio event data that identifies the direction and timing of the audio event that occurs in the VR content. Therefore, in response to the change of the viewing direction, the device 100 may identify the specific audio event to which the user has reacted based on the audio event data, the second viewing direction, and the time based on the VR content. Good. Audio event data, which may include start times and end times of audio events and their directions, may be used to extract suitable visual images from VR content to create missed event content.

The audio event data may be manually created after the VR content is created. Audio event data may be automatically created from audio analysis of spatial audio of VR content or visual analysis of visual images.

In one or more examples, audio event data may be generated from sensor data, which is obtained from a sensor (not shown) associated with an object captured in VR content. Thus, for example, sensors such as acceleration sensors or sound sensors may be placed in the puck during VR content capture of an ice hockey game. Thus, the acceleration of the puck may be detected by the sensor, which can then estimate when there will be an audio event that the user may react to. Audio event data may be created using such an in-VR content capture scene sensor. The duration of acceleration (or other sensed amount) may be used to determine the start and end points of an audio event. The orientation of the sensor with respect to the content capture device 104 may be used to specify the orientation of the audio event.

In one or more examples, the device may analyze spatial audio of VR content to identify audio events. Such analysis may be performed to create audio event data. The analyzes described herein may be performed on the fly. Thus, in response to the change of the viewing direction, the device is directed to the audio of the spatial audio in a direction within a threshold direction of the second direction and at least within the current time, and optionally within a near time threshold of the current time. Analysis may be provided to identify audio events. Audio events may be identified by absolute volume or relative volume threshold audio analysis. Thus, the louder sound of a particular part of directional audio may help to identify it as a particular audio event, and the time when the audio event becomes louder is the start of missed event content. The points may be defined at least partially. Audio events may be identified by audio analysis of frequencies against a threshold. Thus, the occurrence of directional audio for sounds in a particular frequency band (eg, low and fat voice of a character) may help identify it as a particular audio event. The time that audio in that frequency band is present may at least partially define the starting point of the missed event content. Audio events may be identified by audio analysis using voice recognition to identify the voice of a particular character (ie, actor/actress) appearing in VR content. Thus, it may be helpful to identify a particular voice as a particular audio event, as the particular voice is part of the directional audio in the second direction, and the time the voice is heard may be missed by the missed event. The starting point for the content may be defined at least in part.

In one or more examples, the device may provide video analysis of video images of VR content to identify actions in the video in which potential audio events may occur.

FIG. 6 shows exemplary audio event data 600. Of course, the tabular format of the audio event data 600 is for illustration only and may be stored in any number of ways. As an example, the audio event data 600 comprises two audio events 601 and 602. These audio events 601, 602 may have been determined by sensor data, such as a content capture scene sensor in VR, or by audio analysis, or by any of the methods described herein.

For each audio event 601, 602, the audio event data 600 may include a sensed direction 603 of the audio event to be presented to the user 200, an audio event start time 604, and an end time 605. Therefore, it can be seen that the “audio event 1” 601 occurs in the direction of 135°, starts at the time of 30 seconds with respect to the VR content, and is up to 35 seconds.

Therefore, when the user 200 changes the viewing direction of the VR view to 140° when the reproduction of the VR content has progressed to approximately 32 seconds, the device uses the audio event data 600 and outputs the “audio event 1” 601. Identification of a particular audio event may be provided. As such, 140° may be considered within the directional threshold and thus sufficiently close to 135°. Further, 32 seconds may be determined to be during the occurrence of “audio event 1” 601. Thus, the device should use the audio event data 600 to ensure that the missed event content uses a visual image of the VR content from a viewing direction of about 135°, and the missed event content temporally. It may be determined that the starting point may be based on the starting time 604 of “Audio Event 1” 601. Thus, if the predetermined period used to determine how long in time the missed event content starts before the audio event 601 is 5 seconds, the starting point for the missed event content is , VR content may be reached up to the execution time of 25 seconds.

FIG. 7 illustrates an exemplary graph of a plot 700 of audio volume for a particular viewing direction, such as viewing direction 300, or range of viewing directions. The graph of Figure 7 provides a diagram of an audio analysis of spatial audio. Of course, the graphical form of audio analysis is for illustration only and may be implemented in any number of ways. In this example, the change of the viewing direction to the second direction 300 by the user occurs at the time t _reaction . The device 100 may use the threshold 701 in providing the audio analysis described herein to identify that a sound above the threshold occurred between time t ₁ and time t ₂ . Consequently, the sound may be identified as a particular audio event 206. Thus, the time point t ₁ may be used to determine the start time of the missed event content. In one or more examples, the time t _response may occur after time t ₂ . The device may be configured to identify the audio event between times t ₁ and t ₂ as an audio event that occurred in the past close enough to be an audio event that the user has responded to.

In another example, the plot 700 may represent a particular frequency and the frequency range of a particular portion of audio may provide its identification as a particular audio event, without being distinguished in terms of volume. Good.

The embodiments described above may advantageously use audio events to determine the start time of missed event content. As mentioned above, the identification and timing of audio events may be determined by one or more of the following:
Predetermined audio event data, which may be further based on audio analysis, video analysis, sensor data,
It may include audio analysis, which may include volume-based, frequency-based, and/or voice-recognition-based analysis, or lip reading analysis and/or action identification algorithms looking for localized video images that change rapidly above a threshold. , Video analysis.

FIG. 8 shows a change 800 of a viewing direction of a virtual reality view from a first viewing direction to a second viewing direction, wherein the second viewing direction is in the direction of a particular audio event. 801 providing a display of missed event content based on changes that match within a threshold, wherein the missed event content is at least a visual image extracted from a video image of the virtual reality content. And the visual image shows a visual event corresponding to a particular audio event, and the temporal starting point of the missed event content is the temporal point in the virtual reality content where the viewing direction change occurred. Prior to the point and based on a particular audio event, which allows the user to view missed visual events that the user has responded to by listening to the relevant audio event and changing the viewing direction relative to it. Figure 4 shows a flow diagram showing the steps that enable viewing.

FIG. 9 schematically illustrates a computer/processor readable medium 900 that provides a program according to an example. In this example, the computer/processor readable medium is a disc such as a digital versatile disc (DVD) or a compact disc (CD). In some examples, a computer-readable medium may be any medium programmed to perform the functions of the invention. The computer program code may be distributed among multiple memories of the same type or multiple memories of different types such as ROM, RAM, flash, hard disk, solid state.

User input is tap, swipe, slide, press, hold, rotate gesture, stationary hover gesture near the user interface of the device, moving hover gesture near the device, bending at least a portion of the device, at least a portion of the device The gesture may comprise one or more of clasping, multi-finger gesture, tilting the device, or flipping the control device. Further, the gesture may be the user's body, such as the user's arm, or any free space user gesture using a stylus or other element suitable for performing a free space user gesture.

The device shown in the above example is a portable electronic device, laptop computer, mobile phone, smartphone, tablet computer, personal digital assistant, digital camera, smart watch, smart eyewear, pen-based computer, not portable It may be an electronic device, desktop computer, monitor, consumer electronics, smart TV, server, wearable device, virtual reality device, or one or more of these modules/circuitry.

Any device mentioned and/or other features of the particular device mentioned are to be adapted such that they are configured to perform a desired action only when activated, eg when switched on. May be provided by the device prepared in. In such cases, it may not be necessary to load the appropriate software into active memory when deactivating (eg switched off), but only when activated (eg switched on). May be. The device may comprise hardware circuitry and/or firmware. The device may comprise software loaded on memory. Such software/computer programs may be recorded on the same memory/processor/functional unit and/or on one or more memory/processor/functional unit.

In some examples, the particular device referred to may be pre-programmed with suitable software to perform the desired operations, which may include, for example, that software and its associated functionality. May be made available to the user downloading the "key" in order to unlock/enable. An advantage associated with such an example may be that it reduces the need to download data when the device needs more functionality, which is a feature that the user may not enable. It may be useful in instances where the device is known to have sufficient capacity to store such pre-programmed software for sex.

Any device/circuitry/element/processor mentioned may have other functions in addition to those mentioned, and these functions may be performed by the same device/circuitry/element/processor. Good. One or more aspects disclosed may include electronic distribution of the computer program (which may be source/transport encoded) recorded on an associated computer program and a suitable carrier (eg, memory, signal). Good.

Any "computer" described herein may or may not be located on the same circuit board, in the same area/position of the circuit board, or in the same device. / Can comprise a set of processing elements. In some examples, one or more of any of the mentioned processors may be distributed across multiple devices. The same or different processors/processing elements may perform one or more of the functions described herein.

The term "signaling" may refer to one or more signals transmitted as a series of electrical/optical signals that are transmitted and/or received. The series of signals may comprise 1, 2, 3, 4 or even 5 or more individual signal components or separate signals making up said signaling. Some or all of these individual signals may be transmitted/received by wireless or wired communication simultaneously, sequentially and/or in time overlapping with each other.

With respect to any description of any computer and/or processor and memory (eg, including ROM, CD-ROM, etc.) referred to, these may include computer processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs). : Field-programmable gate array), and/or other hardware components programmed to perform the functions of the present invention.

Applicants disclose herein each individual feature described herein alone, as well as any combination of two or more such features. This disclosure has been made in light of the general knowledge of those skilled in the art to the extent that such features or combinations are generally feasible herein, as such features or combinations of features are referred to herein. It does not limit the scope of the claims, whether or not they solve any disclosed problems. Applicants point out that the disclosed aspects/examples may consist of any such individual feature or combination of features. Those skilled in the art will appreciate that various modifications can be made within the scope of the present disclosure in view of the above description.

While the basic novel features have been shown, described and pointed out in the examples as applied, it should be understood that various omissions and substitutions and changes in the form and detail of the devices and methods described may occur. It can be done by a person skilled in the art without departing from the scope of the present disclosure. For example, all combinations of such elements and/or method steps that perform substantially the same function in substantially the same manner to achieve the same result are explicitly intended to be within the scope of the present disclosure. It Moreover, it should be understood that the structures and/or elements and/or method steps shown and/or described in connection with any of the disclosed forms or examples may be of other general design choice. May be incorporated into any disclosed or described or suggested form or example of. Further, in the claims, the means-plus-function clause is intended to cover structures described herein as performing the recited functions, as well as structural equivalents, as well as equivalent structures. Intended to do. In short, nails and screws are structurally equivalent in that in the context of fastening wooden parts, nails use cylindrical surfaces to secure wooden parts together, while screws use helical surfaces. Although not physical, nails and screws can be equivalent structures.

Claims (15)

An apparatus comprising at least one processor and at least one memory containing computer program code, wherein the computer program code, when executed by the at least one processor, causes the apparatus to at least:
Virtual reality content comprising a video image configured to provide a virtual reality space for viewing in virtual reality, wherein a virtual reality view presented to a user provides viewing of the virtual reality content, The virtual reality view comprises a spatial portion of the video images that make up the virtual reality space, has a smaller spatial extent than the spatial extent of the video images in the virtual reality space, and the virtual reality content is spatial audio. The virtual reality, wherein the sensed direction to an audio event related to a visual event in the video image corresponds to a visual position of the visual event presented in the virtual reality space. Regarding content,
The change of the viewing direction of the virtual reality view from the first viewing direction to the second viewing direction by the user, wherein the second viewing direction starts before the change of the viewing direction. Based on said change, matching the direction of the particular audio event that occurred within a direction threshold,
Providing a display of missed event content, wherein the missed event content comprises at least a visual image extracted from the video image of the virtual reality content, the visual image comprising: Indicating the visual event corresponding to a particular audio event, the temporal starting point of the missed event content is greater than the temporal point in the virtual reality content at which the change of the viewing direction occurred. An apparatus that is prior and configured to cause the providing based on the particular audio event. The visual event related to the particular audio event was not visible in the virtual reality view when oriented in the first viewing direction or was oriented in the first viewing direction. 2. The apparatus of claim 1, wherein in the virtual reality view, if present, the user's attention was not placed on the visual event when the corresponding particular audio event occurred. The temporal starting point of the missed event content is a predetermined at least defining the analysis of the particular audio event relative to any other presented audio and the timing of the audio event in the virtual reality content. 3. An apparatus according to claim 1 or claim 2, based on a time at which the particular audio event is considered to have started based on one or more of audio event data. The provision of the missed event content is further based on a degree of change in a viewing direction between the first viewing direction and the second viewing direction exceeding a direction change threshold. The device according to any one of 1. The apparatus of any of claims 1 to 4, wherein the providing of the missed event content is further based on one or more of the volume or frequency components of the particular audio event with respect to an audio threshold. The determination of the specific audio event is
a) predetermined audio event data comprising the direction to one or more audio events occurring in the virtual reality content, the second viewing direction within the direction threshold and the one or more audio events. Interrelation with one of the above directions,
b) Any one of claims 1 to 5 based on one or more of an audio analysis of the spatial audio in response to the change in viewing direction to identify the particular audio event to which the user responded. The device according to claim 1. 7. The providing of a display of the missed event content comprises providing a temporary stop of the virtual reality content and a display of the missed event content in the virtual reality view. The device according to any one of 1. In response to the change in the viewing direction towards the direction of the particular audio event, the time is reversed so that the virtual reality content appears to be rewound to the user until the point at which the missed event content begins. 8. The apparatus according to any of claims 1-7, which provides a display of the virtual reality content to a. The missed event content is
i) a missed event video image extracted from the video image of the virtual reality content,
ii) a missed event still image extracted from the video image of the virtual reality content,
iii) virtual reality content of a missed event extracted from the video image of the virtual reality content, wherein at least an initial viewing direction of the virtual reality content of the missed event is the virtual audio content of the particular audio event. Virtual reality content of the missed event, matching the direction,
iv) Virtual reality content of the missed event extracted from the video image of the virtual reality content, wherein the virtual reality content of the missed event has a spatial range less than the spatial range of the virtual reality content. Virtual reality content of the missed event, having the focus on the direction of the particular audio event,
9. The device according to any of claims 1-8, comprising one or more of: 10. The apparatus of any of claims 1-9, wherein the missed event content is provided for display as a picture-in-picture in the virtual reality view with the virtual reality content. When the display of the missed event content is completed, the display of the virtual reality content is restarted from a time point corresponding to the time of the change of the viewing direction from the first viewing direction to the second viewing direction. 10. A device according to any one of claims 1 to 9 which is provided. 12. Upon completion of the display of the missed event content, the virtual reality content is provided for display at a playback speed that is at least temporarily higher than real time. Equipment. 13. The apparatus of any of claims 1-12, wherein the orientation threshold may be less than 90°, less than 70°, less than 45°, or less than 30°. Virtual reality content comprising a video image configured to provide a virtual reality space for viewing in virtual reality, wherein a virtual reality view presented to a user provides viewing of the virtual reality content, The virtual reality view comprises a spatial portion of the video images that make up the virtual reality space, has a smaller spatial extent than the spatial extent of the video images in the virtual reality space, and the virtual reality content is spatial audio. The virtual reality, wherein the sensed direction to an audio event related to a visual event in the video image corresponds to a visual position of the visual event presented in the virtual reality space. Regarding content,
The change of the viewing direction of the virtual reality view from the first viewing direction to the second viewing direction by the user, wherein the second viewing direction starts before the change of the viewing direction. Based on said change, matching the direction of the particular audio event that occurred within a direction threshold,
Providing a display of missed event content, wherein the missed event content comprises at least a visual image extracted from the video image of the virtual reality content, the visual image comprising: Indicating the visual event corresponding to a particular audio event, the temporal starting point of the missed event content is greater than the temporal point in the virtual reality content at which the change of the viewing direction occurred. A method comprising: providing, based on the particular audio event, which was before. The device, when executed on at least one processor of the device,
Virtual reality content comprising a video image configured to provide a virtual reality space for viewing in virtual reality, wherein a virtual reality view presented to a user provides viewing of the virtual reality content, The virtual reality view comprises a spatial portion of the video images that make up the virtual reality space, has a smaller spatial extent than the spatial extent of the video images in the virtual reality space, and the virtual reality content is spatial audio. The virtual reality, wherein the sensed direction to an audio event related to a visual event in the video image corresponds to a visual position of the visual event presented in the virtual reality space. Regarding content,
The change of the viewing direction of the virtual reality view from the first viewing direction to the second viewing direction by the user, wherein the second viewing direction starts before the change of the viewing direction. Based on said change, matching the direction of the particular audio event that occurred within a direction threshold,
Providing a display of missed event content, wherein the missed event content comprises at least a visual image extracted from the video image of the virtual reality content, the visual image being the specific image. Showing the visual event corresponding to an audio event, the temporal starting point of the missed event content is before the temporal point in the virtual reality content where the change of the viewing direction occurs. Yes, based on the specific audio event, causing the provision,
A computer program comprising program instructions configured to:

Images