WO2018200734A1

WO2018200734A1 - Field-of-view prediction method based on non-invasive eeg data for vr video streaming services

Info

Publication number: WO2018200734A1
Application number: PCT/US2018/029453
Authority: WO
Inventors: Juhyung Son; Jin Sam Kwak; Hyun Oh Oh
Original assignee: Pcms Holdings, Inc.
Priority date: 2017-04-28
Filing date: 2018-04-25
Publication date: 2018-11-01

Abstract

Some embodiments of systems and methods disclosed herein include displaying, on a display of a head-mounted display (HMD), a first plurality of tiles of a multi-tile video; measuring an electroencephalography (EEG) signal of the user while the multi-tile video is displayed; predicting a head movement of the user based on the EEG signal; retrieving a second plurality of tiles of the multi-tile video based on the predicted head movement of the user; and rendering one or more of the second plurality of tiles of the multi-tile video.

Description

FIELD-OF-VIEW PREDICTION METHOD BASED ON NON-INVASIVE EEG DATA FOR VR VIDEO

STREAMING SERVICES

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application is a non-provisional filing of and claims benefit under 35 U.S.C. §119(e) from, U.S. Provisional Patent Application Serial No. 62/491 ,586, entitled "Field-Of-View Prediction Method Based on Non-Invasive EEG Data for VR Video Streaming Services," filed April 28, 2017, the entirety of which is incorporated herein by reference.

BACKGROUND

[0002] Virtual reality (VR) refers to computer technologies that use software to render the realistic images, sounds and other sensations that replicate a real environment. The rendering is designed to mimic the visual and audio sensory stimuli of the real world as naturally as possible to a user as they move within the limits defined by the application. Virtual reality usually requires a user to wear a head mounted display (HMD) to completely replace the user's field of view with a simulated visual component and to wear headphones to provide the user with the accompanying audio.

SUMMARY

[0003] Some embodiments of a method may include: displaying, on a display of a head-mounted display (HMD), a first plurality of tiles of a multi-tile video; measuring, while the multi-tile video is displayed, an electroencephalography (EEG) signal of the user to generate a measured EEG signal; determining a predicted head movement of the user based on the measured EEG signal; retrieving a second plurality of tiles of the multi-tile video based on the predicted head movement of the user; and rendering one or more of the second plurality of tiles of the multi-tile video.

[0004] For some embodiments, determining the predicted head movement of the user may include: detecting a zero crossing of an EEG-derived signal derived from the measured EEG signal; and determining the predicted head movement of the user to occur proximate to the detected zero crossing.

[0005] For some embodiments, detecting the zero crossing of the EEG-derived signal may include detecting that the EEG-derived signal is less than a zero crossing threshold. Luuuu] oun ie embodiments of a method further may include generating a frequency-band power signal from the measured EEG signal, wherein the EEG-derived signal is the frequency-band power signal.

[0007] Some embodiments of a method further may include: performing a training procedure that matches a plurality of head movements of the user to a plurality of EEG signal patterns, wherein determining the predicted head movement of the user may include: identifying, within the measured EEG signal, an EEG signal pattern selected from the plurality of EEG signal patterns; and determining the predicted head movement to be the head movement of the user matched, by the training procedure from the plurality of EEG signal patterns, to the identified EEG signal pattern selected from the plurality of EEG signal patterns.

[0008] For some embodiments, performing the training procedure comprises performing a supervised training procedure that matches induced FOV changes to the plurality of EEG signal patterns.

[0009] For some embodiments, performing the training procedure comprises performing an unsupervised training procedure that matches detected FOV changes to the plurality of EEG signal patterns.

[0010] Some embodiments of a method further may include: selecting a plurality of resolutions based on the predicted head movement of the user, wherein each of the plurality of resolutions corresponds to a respective one of the second plurality of tiles, and wherein the second plurality of tiles are rendered at the corresponding resolution.

[0011] Some embodiments of a method further may include: determining a predicted field of view (FOV) based on the predicted head movement; and matching the predicted FOV to one or more tiles in the multi- tile video, wherein selecting the plurality of resolutions selects an increased resolution for tiles matched to the predicted FOV.

[0012] For some embodiments, determining the predicted head movement of the user may include determining a predicted direction, wherein determining the predicted FOV may include: determining a current FOV to be a first selection of one or more tiles in the multi-tile video; and determining the predicted FOV to be a second selection of one or more tiles in the multi-tile video, wherein the second selection is a shift of the first selection in the predicted direction.

[0013] For some embodiments, each of the second plurality of tiles may be retrieved at the respective selected resolution.

[0014] For some embodiments, determining the predicted head movement may include determining a predicted direction.

[0015] For some embodiments, measuring the EEG signal may include measuring the EEG signal to generate a plurality of measured EEG signals; and determining the predicted direction may include: selecting a selected EEG signal from the plurality of measured EEG signals; determining the predicted direction as a unewiuii ui neau muvement of the user associated with the selected EEG signal, wherein detecting the zero crossing of the EEG-derived signal may include detecting the zero crossing of an EEG signal derived from the selected EEG signal.

[0016] For some embodiments, retrieving the second plurality of tiles of the multi-tile video may include: identifying the first plurality of tiles as a first selection of tiles of the multi-tile video; and selecting the second plurality of tiles as a second selection of tiles of the multi-tile video, wherein the second selection may be a shift of the first selection in the predicted direction.

[0017] Some embodiments of a device may include: a processor; and a non-transitory computer- readable medium storing instructions that are operative, when executed on the processor, to perform the functions of: displaying, on a display of a head-mounted display (HMD), a first plurality of tiles of a multi-tile video; measuring an electroencephalography (EEG) signal of the user while the multi-tile video is displayed; determining a predicted head movement of the user based on the EEG signal; retrieving a second plurality of tiles of the multi-tile video based on the predicted head movement of the user; and rendering one or more of the second plurality of tiles of the multi-tile video.

[0018] Some embodiments of a method may include: retrieving, by a head-mounted display (HMD), a plurality of tiles of a multi-tile video based on a viewing direction of a user; displaying, on a display of the HMD, one or more of the plurality of tiles; measuring an electroencephalography (EEG) signal of the user while the multi-tile video is displayed; determining whether a head movement of the user is predicted based on the measured EEG signal; and updating which tiles of the multi-tile video to retrieve if a head movement is predicted.

[0019] For some embodiments, determining the predicted head movement of the user may include: detecting a zero crossing of an EEG-derived signal derived from the measured EEG signal; and determining the predicted head movement of the user to occur proximate to the detected zero crossing.

[0020] Some embodiments of a method further may include: reducing noise from the measured EEG signal to generate a noise-reduced EEG signal; bandpass filtering the noise-reduced EEG signal to generate a bandpass-filtered signal; squaring the bandpass-filtered signal to generate a current power signal; and subtracting a previous power signal from the current power signal to generate a difference power signal, wherein the EEG-derived signal is the difference power signal.

[0021] Some embodiments of a method further may include: performing a training procedure that matches a plurality of head movements of the user to a plurality of EEG signal patterns, wherein determining the predicted head movement of the user may include: identifying, within the measured EEG signal, an EEG signal pattern selected from the plurality of EEG signal patterns; and determining the predicted head iiiuveiiieni ιυ ue me iiead movement of the user matched, by the training procedure from the plurality of EEG signal patterns, to the identified EEG signal pattern selected from the plurality of EEG signal patterns.

[0022] For some embodiments, performing the training procedure may include: displaying, on the display of the HMD, a plurality of tiles of a multi-tile training procedure video that are configured to induce a user head movement and a change to a field of view (FOV) of the user; measuring a training procedure EEG signal while the multi-tile training procedure video is displayed; extracting an observed EEG signal pattern from the training procedure EEG signal to generate an extracted EEG signal pattern; and matching the extracted EEG signal pattern to the head movement corresponding to the change to the FOV of the user.

[0023] For some embodiments, performing the training procedure may include: displaying, on the display of the HMD, a plurality of tiles of a multi-tile training procedure video; measuring a training procedure EEG signal while the multi-tile training procedure video is displayed; detecting a head movement corresponding to a change to a field of view (FOV) of the user; extracting an observed EEG signal pattern from the training procedure EEG signal to generate an extracted EEG signal pattern; and matching the extracted EEG signal pattern to the head movement corresponding to the change to the FOV of the user.

[0024] Some embodiments of a device may include: a processor; and a non-transitory computer- readable medium storing instructions that are operative, when executed on the processor, to perform the functions of: retrieving, by a head-mounted display (HMD), a plurality of tiles of a multi-tile video based on a viewing direction of a user; displaying, on a display of the HMD, one or more of the plurality of retrieved tiles; measuring an electroencephalography (EEG) signal of the user while the multi-tile video is displayed; determining whether a head movement of the user is predicted based on the measured EEG signal; and updating which tiles of the multi-tile video to retrieve if a head movement is predicted.

BRIEF DESCRIPTION OF THE DRAWINGS

[0025] A more detailed understanding may be had from the following description, presented by way of example in conjunction with the accompanying drawings.

[0026] FIG. 1 is a picture of a user wearing a Head Mounted Display (HMD) Electroencephalography (EEG) test setup.

[0027] FIG. 2 is a picture of a 5-channel non-invasive EEG headset.

[0028] FIG. 3 is a perspective view diagram showing the viewing angle for a field of view in an HMD.

[0029] FIG. 4 is a series of example graphs for human EEG data according to some embodiments.

[0030] FIGs. 5A and 5B are side and top view schematics, respectively, of example EEG electrodes placed on a user's scalp according to some embodiments. uu I j n b. on to 6C are example front view virtual reality field of views showing tile-based VR video streaming according to some embodiments.

[0032] FIG. 7 is a system diagram of an example system illustrating a potential discrepancy due to Motion-to-Photon latency if a user's HMD moves according to some embodiments.

[0033] FIG. 8 is a system diagram of an example system for a user connecting to a VR video server according to some embodiments.

[0034] FIG. 9 is a linear axis chart showing the frequency bands used for EEG signals according to some embodiments.

[0035] FIG. 10A is a graph of three example EEG signals for the alpha, beta, and gamma frequency bands.

[0036] FIG. 10B is a graph of three example EEG signals (alpha, beta, and gamma frequency bands) with an indication of zero cross times (ZCT) for each of the signals for some embodiments.

[0037] FIG. 11 A is a flowchart of an example process for feature extraction procedure according to some embodiments.

[0038] FIG. 11 B is a pictorial flowchart of an example process of an example extraction procedure for translating raw data into a format used by a prediction algorithm according to some embodiments.

[0039] FIG. 12 is a perspective view of a VR HMD with indications of pitch, yaw, and roll movements in a coordinate system according to some embodiments.

[0040] FIG. 13 is a spreadsheet example that matches HMD movements to related EEG features according to some embodiments.

[0041] FIG. 14 is a message sequence diagram of an example process for a movement prediction system using a VR video streaming application according to some embodiments.

[0042] FIGs. 15A and 15B are system configurations for some embodiments for showing an example supervised training sequence to a user according to some embodiments.

[0043] FIG. 16 is a video frame layout diagram indicating a prediction for the direction of FOV movement and selection of video tile resolutions according to some embodiments.

[0044] FIG. 17 is a flowchart of an example process for a user performing FOV prediction and EEG matching with a content provider platform according to some embodiments.

[0045] FIG. 18 is a flowchart of an example process for a server performing FOV prediction and EEG matching with a content provider platform according to some embodiments. Luuwj no. i s is a flowchart of an example process for retrieving and rendering multi-tile video tiles based on a predicted head movement according to some embodiments.

[0047] FIG. 20 is a flowchart of an example process for retrieving multi-tile video tiles based on a predicted head movement according to some embodiments.

[0048] FIG. 21 is a system diagram of an example system illustrating an example communications system according to some embodiments.

[0049] FIG. 22 is a system diagram of an example system depicting an example wireless transmit/receive unit (WTRU) that may be used as an HMD according to some embodiments.

[0050] FIG. 23 is a system diagram of an example system depicting an exemplary network entity that may be used by a content provider server according to some embodiments.

[0051] FIG. 24 is a system diagram of an example system illustrating an example radio access network (RAN) and an example core network (CN) that may be used within the communications system illustrated in FIG. 21 according to some embodiments.

[0052] FIG. 25 is a system diagram illustrating a further example RAN and a further example CN that may be used within the communications system illustrated in FIG. 21 for some embodiments.

[0053] The entities, connections, arrangements, and the like that are depicted in— and described in connection with— the various figures are presented by way of example and not by way of limitation. As such, any and all statements or other indications as to what a particular figure "depicts," what a particular element or entity in a particular figure "is" or "has," and any and all similar statements— that may in isolation and out of context be read as absolute and therefore limiting— may only properly be read as being constructively preceded by a clause such as "In at least one embodiment,...." For brevity and clarity of presentation, this implied leading clause is not repeated ad nauseum in the detailed description of the drawings.

DETAILED DESCRIPTION

[0054] Virtual reality (VR) refers to computer technologies that use software to render the realistic images, sounds and other sensations that replicate a real environment. The rendering is designed to mimic the visual and audio sensory stimuli of the real world as naturally as possible to a user as they move within the limits defined by the application. Virtual reality usually requires a user to wear a head mounted display (HMD) to completely replace the user's field of view with a simulated visual component and to wear headphones to provide the user with the accompanying audio.

[0055] Some form of head and motion tracking of the user in VR may be used to enable updating of the simulated visual and audio components to ensure, from a user's perspective, that items and sound sources remain consistent with the user's movements. ΐυυυυ] no. I shows a picture 150 of a user 154 wearing an EEG test setup 152 according to a Facebook IQ article, How Virtual Reality Facilitates Social Connection, FACEBOOK IQ (Jan. 9, 2017), https://insights.fb.com/2017/01/09/how-virtual-reality-facilitates-social-connection/ [last visited April 10, 2018). FIG. 2 shows a picture of another headset 200, a five-channel, non-invasive EEG headset, according to the website Insight, EMOTIV, http://www.emotiv.com/insight/ {last visited April 10, 2018).

[0057] FIG. 3 is a picture 300 of a field of view in virtual reality for an HMD 302. A user's field of view (FOV) is the area of vision at a given moment. The FOV is the angle of visible field expressed in degrees measured from the focal point. The field of view in VR refers to the part of a virtual world a user sees at a given moment. For the limited FOVs of existing VR displays, only part of a video frame needs to be displayed to the user. As shown in the example of FIG. 3, the FOV range 304 of most currently available HMDs range between 100°-110°. Because the actually displayed video covers only a portion of the scene, the renderer may receive metadata (motion data) related to the current user viewport so that the renderer may render only the required FOV. If the received metadata is incorrect or out of date, the VR user may see the unmatched viewport compared to his or her current viewing direction, and this situation may cause the user to, e.g., suffer VR sickness. Therefore, in some embodiments, the VR user's motion information should be reflected in the rendered VR video images with a short latency, called the Motion-to-Photon Latency, to be unnoticeable by a user.

[0058] The main driver on the performance of VR video interaction latency, referred to as motion-to- photon (MTP) latency, comes from the requirement that gaze changes should be matched with orientation changes detected by a vestibular system. Although research has shown that sensitivity to virtual reality sickness depends on gender, general health, and other factors, it is desirable for VR systems to mimic a real- world experience by adapting the display as fast as possible.

[0059] The latency of action of the angular or rotational vestibulo-ocular reflex ranges from 7-15 milliseconds, and this reflex may represent a performance goal for VR systems. The frame rate from the renderer to the viewer for VR video is usually at least 60 frames per second but more recent systems have been reporting frame rates up to 90 frames per second (~11 ms frame interval). They are more consistent with the motion-to-photon (MTP) latency requirements, albeit without any allowance for the detection of user movement and image processing times. When such detection times and image processing delays are taken into account, a requirement of 20 ms may be set. When a VR HMD is connected by an HDMI/USB cable from a host PC, those MTP latency requirements may be met due to the low latency of cables and powerful rendering performance of a PC's graphics card. However, when VR contents are streamed from a remote server, network latencies may make meeting the MTP latency requirement difficult. Therefore, a typical VR video streaming server transmits the whole 360-degree VR video tiles irrespective of the user's FOV, thus consuming a lot of network bandwidth. Luuuu] ividn οοθ-degree VR video streaming services offer a limited user experience because the resolution (the visual quality) of the user's viewport may not be on par with traditional video streaming services. Multiple times UHD resolution is generally needed to cover the full-360-degree surroundings in a visually-sufficient resolution. This poses a major challenge to the established video processing chain and to the available end devices. One problem for VR streaming is the huge bandwidth that may be required by VR video.

[0061] FIG. 4 is a series 400 of graph traces for example human EEG data. Electroencephalography (EEG) is an electrophysiological monitoring method to record electrical activity of the brain. EEG is the most prevalent method of signal acquisition for Brain Computer Interfaces (BCI), according to a book by Bin He, Neural Engineering, (Springer: 2^nd Ed., 2013). An EEG is typically noninvasive, with the electrodes placed along the scalp. EEG measures voltage fluctuations resulting from ionic current within the neurons of the brain. Many studies have shown that although non-invasive EEG is less accurate in comparison with invasive EEG, non-invasive EEG still contains enough real-time information to be used as a source for different BCI applications. EEG recording has high temporal resolution and is capable of measuring changes in brain activity that occur within a few milliseconds. The spatial resolution of EEG may be less than invasive methods, but signals from up to 256 electrode sites may be measured at the same time. EEG recording equipment is often portable and the electrodes may be placed on a subject's scalp by, e.g., donning a cap.

[0062] FIGs. 5A and 5B show a side view 500 and a top view 550, respectively, of EEG electrodes placed on a user's scalp. "A" stands for auricular. "C" stands for central. "FP" stands for prefrontal. "F" stands for frontal. "0" stands for occipital lobe. "P" stands for parietal. "T" stands for temporal. "Z" stands for zero. Orientation markers are shown on FIGs. 5A and 5B. The nasion marker 502 points to just above a user's nose. The inion marker 504 points to the top of the neck at the back of the head. The preauricular point marker 506 indicates the entrance to the ear canal. The vertex marker 508 indicates the half-way point between nasion marker 502 and the inion marker 504 and the half-way point between each ear.

[0063] Many EEG-based BCI systems use an electrode placement strategy based on the international system shown in FIGs. 5A and 5B. EEG-based BCI control with several degrees of freedom may be achieved with just a few electrodes. Many noninvasive BCIs are based on classification of different mental states rather than decoding parameters, which is typically done in invasive BCIs. Various investigators have attempted to directly decode the kinematic information related to movement or motor imagery and have reported success in revealing information about the (imagined) movement direction and speed from the spatiotemporal profiles of EEG signals.

[0064] Tile-based VR video streaming may greatly reduce bandwidth consumption but at the risk of low resolution tiles being displayed when there is a discrepancy between predicted high-resolution tiles and the actual FOV region at the time of play. This discrepancy may generally degrade the quality of user experiences υι rv viueu bii ecu I iing. Motion-to-photon latency is the time needed for a user movement to be fully reflected on the user's display screen. As the latency increases, the chance of discrepancy increases from network and buffering delays. The server selects and sends a subset of tiles with high resolution according to the reported FOV, but the user's FOV has moved away from the selected tiles when the user actually views the tiles.

[0065] Systems and methods described herein in accordance with some embodiments use bio signals (such as electroencephalography (EEG) or Electromyography (EMG)) data measured from a VR user's HMD to predict his/her future FOV (Field of View) by using the observation that EEG data may characterize pre- movements and pre-motor imageries of a human. An exemplary HMD (Head Mounted Display) may use several non-invasive EEG measuring probes to measure EEG data. The brain status captured in EEG data before a voluntary movement may indicate a user's future movement, which may be used in VR applications to predict a user's future physical movements. Using an EEG's pre-movement or pre-motor imagery potentials, the onset and direction of upcoming movement may be predicted. This information may be used with more efficient VR video streaming systems that more correctly match high-resolution tiles displayed to a VR user.

[0066] As described herein, systems and methods in accordance with some embodiments save bandwidth usage (and may lead to a high satisfaction user experience under a bandwidth-limited streaming environment) by streaming an entire view of VR video while only part of the view is streamed as high quality (or high resolution) encoding.

[0067] FIGs. 6A to 6C show one example of tile-based VR video streaming. FIG. 6A is an example FOV 600 where frames 602 have equally high resolution. Irrespective of a user's FOV, there may be high user satisfaction but with high bandwidth (BW) usage in total. FIG. 6B shows a FOV 630 where all frames 632 have low resolution with low BW usage in total. As a result, there may be low user satisfaction. FIG. 6C is a FOV 660 with an unequal distribution of high-resolution frames 662 and low-resolution frames 664. The four video frames (tiles) in the center matching with the user's FOV have high resolution, while the other twelve frames (tiles) located around the edge of the FOV have low resolution. This method results in likely higher user satisfaction and a middle level BW usage in total, resulting in an improved compromise between rendered image quality and resolution in the user experience versus bandwidth usage.

[0068] Some embodiments use video tiles or separate video streams for multi-tile (or 360-degree) VR video delivery that allows emphasizing the current user viewport through transmitting non-viewport samples with decreased resolution (selecting the tiles for the viewport at a high-resolution version and the tiles that do not belong to the viewport at a lower resolution version). Hence, the full 360° surroundings are available on the end device but the quality of video tiles that lie outside the user's FOV is reduced. For this purpose, di Li ic ei iuuuei siue mil 360-degree video is projected into a frame, mapped, and encoded into several tiles at different resolutions.

[0069] Systems and methods described herein in accordance with some embodiments may use EEG data (an alpha, beta, or gamma frequency band's unique ERD/ERS features) measured before movement onset about N seconds (e.g., 3 seconds) for predicting the future movement of VR HMD. This time value may be called a predicted user movement onset time. Based on predicted future movement and direction, relevant video tiles may be preemptively refreshed.

[0070] FIG. 7 is a system diagram of an example system 700 for some embodiments that illustrates a potential discrepancy when a user 706 moves between when the time motion information 704 is reported and VR video frames are displayed. FIG. 7 illustrates the inconsistency between the transferred high- resolution tiles, and the actual FOV at the time of display (or play). One challenge is predicting future FOV tiles when a VR content server 702 streams 360-degree VR video 718, and the time difference between the current and the future point is in the order of seconds, which may result in the future FOV location being uncorrelated with the current FOV location. When VR contents are transmitted by means of wireless communication, the latency problem may be worsened due to the wireless access delay. Motion-to-Photon delay is the time needed for a user movement (or motion information) to be fully reflected on a HMD's display. Any timely clue to predict a user's future movement may be valuable. At time t=0, a user 706 may be looking at a true focal point 716. A system may generate a prediction of a predicted focal point at time t=T 708 with a predicted FOV 710. A true focal point at time t=T 712 may have a true FOV 714.

[0071] FIG. 8 is a system diagram 800 of some embodiments of a VR user (or a user wearing an HMD) 802 connected to a VR video streaming server 812. EEG 804 and motion 806 data is sent to a server 812, e.g., a VR Video Streaming Server 812, while VR video tiles 810 are sent to a user's HMD 802. For a human's motor tasks, motor commands create an internal copy which reflects the predicted movement and resulting sensations. Through EEG data acquisition, a brain's current state may be analyzed in real time. Part of this current state is a prediction of the next motor task. Therefore, by understanding and analyzing brain rhythms and EEGs, future motor commands may be predicted. This may lead to better estimation of a user's future FOV prediction in 360-degree VR video streaming. For some embodiments, a future FOV prediction process 816 uses motion and EEG data 814 as an input to select VR video tiles 818.

[0072] A "pre-movement" phenomenon refers to when no muscle movement is detectable, but the subject is aware of the action that he or she is going to perform in the near future. The "pre-movement" phenomenon also refers to planning and preparation for movements. In this time interval, which ranges from 500 ms to 3 s before movement, the brain prepares for action execution. There is a noticeable change in brain waves that only happens before movement. Luu/ j oysienis and methods described herein in accordance with some embodiments use electroencephalography (EEG) data measured for a VR user to predict his/her future FOV (Field of View) by using EEG data to characterize the pre-movements and pre-motor imageries of a human. The brain status captured in an EEG before a voluntary movement indicates a user's future movement and the EEG data may be used (which may be in VR applications) to predict a user's future physical movements. Using the EEG's pre-movement or pre-motor imagery potentials, the onset and direction of an upcoming movement may be predicted. As shown in FIG. 13, the onset and direction of a VR user's movement may be analyzed for a training period by determining a relationship between a user's pre-onset EEG signal and onset inertial measurement unit (IMU) sensor data of the user's VR HMD. By using a data analysis method, such as PCA (Principal Component Analysis), a determination may be made of which EEG signals to use as predicting components for each IMU sensor data (e.g., for a ττ/4 yaw rotation). This information may be used, in some embodiments, to design more efficient VR video streaming systems with more correctly matched high- resolution tiles displayed to a VR user, which appear more natural to users.

[0074] Systems and methods described herein in accordance with some embodiments reduce risk of inconsistency between transferred high resolution tiles and the FOV at the time of play (or display), while still transferring a subset of tiles to reduce bandwidth consumption. A VR video streaming server may analyze a VR user's current EEG information to calculate a prediction of a user's future FOV when the related VR video stream is played, thus providing the best quality for VR video streaming services and minimizing use of network resources.

[0075] For some embodiments, other bio signals, such as Electromyography (EMG) signals, may be used. EMG is an electrodiagnostic medicine technique for evaluating and recording the electrical activity produced by skeletal muscles. EMG is performed using an instrument called an electromyograph to produce a record called an electromyogram. An electromyograph detects the electric potential generated by muscle cells when these cells are electrically or neurologically activated. EMG data measured from a VR (or HMD) user may be used to predict his or her future FoV (Field of View).

[0076] For some embodiments, EMG sensors are placed around the VR user's neck skin, and pre- movement or pre-motor imagery potentials of the user's head may be measured. EMG signals may be used for more accurate predictions for short prediction periods (e.g., less than 100 msec), and EEG signals may be used for more accurate predictions for longer prediction period (e.g., larger than 100 msec). Physiological data may be selected based on prediction requirements of the VR system. Because EEG-based predictions may be made earlier than EMG-based predictions, EEG may be more suitable for a VR server to prepare video tiles with a large server-to-client delay. Moreover, a multimodal analysis may improve absolute prediction performance by using a combination of multimodal data, such as EEG and EMG data. Luui I j no. a ib a linear axis chart 900 showing the frequency bands 902, 904, 906, 908, 910 used for EEG signals. Components in the alpha, beta, and gamma bands are used. Electromagnetic recordings from a brain at rest exhibits internal oscillatory activity that is widespread across the entire brain. As shown in FIG. 9, this oscillatory activity may be split into several bands. Spontaneous activity comprises oscillations mainly in the alpha-frequency band (8-13 Hz) (called mu rhythm) 906, beta frequency band (14-30 Hz) 908, and gamma frequency band (>30 Hz) 910 when focused over the sensorimotor cortex. Oscillations recorded over the sensorimotor cortex are called sensorimotor rhythms (SMRs).

[0078] FIG. 10A is a graph 1000 of three EEG signals 1002, 1004, 1006 off the alpha, beta, and gamma frequency bands. For the normal state, if a human is constantly focusing on a certain object without significant movements, the alpha-rhythm may be most prominent. An alpha-rhythm consists of sinusoidal-like waves with frequencies in the 8-12 Hz range more prominent in posterior sites. In the similar manner, when a VR user is stationary and watching VR contents without prominent movement, certain EEG characteristics of a "Stationary State" may be determined from an EEG data set. During this stationary state, a VR user has a very low probability of changing FOV direction. Therefore, the VR streaming server may optimize video streaming quality by maximizing the resolution of the current FOV. FIG. 10A shows Event-Related Desynchronization (ERD) 1002 and Event-Related Synchronization (ERS) 1004, 1006 features measured in three frequency bands before the movement onset. FIG. 10A is a graph found in da Silva, Fernando Henrique Lopes, EEG: Origin and Measurement (2010), available at: https://pdfs.semanticscholar.org/3769/c67d11c746d1 a495ed00f522a5930d95cff4.pdf (last visited April 10, 2018) ("da Silva"). According to da Silva (p. 29), the three frequency bands shown in FIG. 10A are measured with the C3 electrode (as shown in FIGs. 5A and 5B).

[0079] FIG. 10B is a graph 1000B that shows, for example purposes in accordance with some embodiments, e.g., the addition of the identification of zero cross times (ZCTs) for the alpha, beta, and gamma signals. FIG. 10B shows a solid dot for a zero cross point for each EEG signal before the movement onset. Those zero cross points may be used as references for time durations [ZCT_alpha, ZCT_beta, ZCT_gamma) before the onset of actual head movement for each feature. The graphs also show ERD/ERS features.

[0080] For the movement state, task-related modulation in sensorimotor rhythms is usually manifested as an amplitude decrease in the alpha/beta band-frequency components that is called an Event-Related Desynchronization (ERD). In contrast, an amplitude increase in a frequency band is called an Event-Related Synchronization (ERS). ERD/ERS events are time-specific event-related potentials (ERP) associated with sensory stimulation or mental imagery tasks. ERD is the result of a decrease in the synchronization of neurons, which causes a decrease of power in specific frequency bands. ERS is the result of an increase in the synchronization of neurons, which causes an increase of power in specific frequency bands. For example, Li ie picti II in ly ai iu execution of movements leads to predictable decreases (ERD) in the alpha and beta frequency bands and increases (ERS) in gamma frequency bands. Such characteristic changes in EEG rhythms may be used to classify brain states relating to the planning/imagining of different types of physical movement.

[0081] Also, for timing relationships, zero cross point refers to the last time epoch where ERD/ERS signals cross the x-axis before a movement onset, and a ZCT (e.g., ZCT_alpha, ZCT_beta, ZCT_gamma) is measured as a negative time value (e.g., -2.8 sec.) relative to the HMD movement onset 1010 (0 sec). The value tieft is the time duration left before the predicted movement onset when the zero crossing epoch (or ZCT point) is already passed and the value of ZCT 1012 is determined a priori. For the example shown in FIG 10B, the alpha frequency band is used to determine ZCT 1012, which is equal to ZCT_alpha. For some embodiments, a zero cross point 1008 is detected (which may not include calculation of a ZCT 1012, which is a time duration) and used for predicting HMD movement onset 1010.

[0082] In FIG. 10B, time epochs of alpha/beta/gamma ERD/ERS feature's zero crossing times are identified by analyzing a user's EEG signals. By using those time reference points, a VR server may estimate actual HMD movement onset time points ( ) relative to the principal ERD/ERS feature's zero cross point. The y-axis is a normalized ERD or ERS signal times 100%, and the x-axis is time. For some embodiments, the zero cross time (ZCT) 1012 is the time (about -2.8 seconds for the example shown in FIG. 10B) between a zero cross point 1008 when the Alpha ERD signal goes negative and the HMD movement onset 1010. As depicted in FIG.10B, before a certain HMD movement (e.g., a ^ yaw rotation) onset, there may be multiple

ZCT epochs for multiple EEG band signals (e.g., alpha, beta, gamma). If a band's ERD/ERS feature is consistently recorded for a certain HMD movement with a stable ZCT value, the band's ERD/ERS feature may be identified as the principal predictor for an HMD movement, and the ZCT value (or zero cross point) may be used to estimate the actual HMD movement onset (a point in time). In addition to the principal predictor, there may be second and third (or more) predictors for the same HMD movement from different EEG bands. Those multiple predictors and related multiple ZCT values may be used in combinations to predict HMD movement and onset timing (or HMD movement onset). For example, if a VR server takes 500ms to prepare high-resolution VR video tiles and to deliver the tiles to the client VR HMD, the server may use multiple ERD/ERS features that have ZCT values greater than 500ms to predict the user's head movement and to prepare and deliver the video tiles on time (before actual HMD movement onset).

[0083] FIG. 11 A is a flowchart 1100 of an example feature extraction procedure in accordance with some embodiments. FIG. 11 B is a pictorial flowchart 1150 of an example extraction procedure, in accordance with some embodiments, as the data is translated from raw form into a format used by a prediction algorithm.

EEG data signal processing may be used to extract features from acquired EEG signals and to translate them into unique features for VR HMD future movement prediction applications. A feature in an EEG signal may ue vieweu as a reflection of a specific intention of a certain future movement and related internal anatomy of the nervous system. Feature extraction for VR movement prediction applications obtain features that may accurately and reliably reflect the specific movement of VR user's head movement. In addition, even though a VR user may be able to generate detectable signals that convey his intent, signal acquisition methods may also capture noise generated by other unrelated activity in or outside of the brain. Thus, feature extraction may maximize the signal-to-noise ratio.

[0084] Many embodiments use noise removal or reduction as part of an EEG-based VR head prediction application. Because signals are often captured across several electrodes over a series of time points, some methods have concentrated on spatial-domain processing, temporal-domain processing, or both. Noise may be captured from neural sources if brain signals not related to the target signal, such as facial muscular movements, are recorded. Mathematical operations, such as linear transformations, may be used for artifact removal.

[0085] Signal processing techniques may be used to extract ERD and ERS from raw EEG signals. For one embodiment, a raw EEG signal from each trial may be bandpass filtered for an alpha 1152, beta 1154, or gamma 1156 frequency band. Positive/negative voltage-based EEG amplitude samples are squared to obtain power samples. Power samples are averaged across multiple samples. For a certain VR HMD movement detected, related EEG signals are calculated and averaged to minimize variance noises. Variability is reduced and the graph is smoothed by averaging over time samples. For some embodiments, data also is normalized.

[0086] For EEG signals, spatial resolution/specificity is low and acquired signals may indicate activity in larger regions. For these methods, features may be defined both by spatial location and by temporal/spectral characteristics. To optimize spatial information, channels used for VR head prediction may be a selected subset of a few electrode channels. For a certain VR HMD movement, alpha/beta/gamma features and related prominent electrode location may be selected with methods such as principal components analysis (PCA) or independent component analysis (ICA).

[0087] Alpha/beta/gamma rhythm components may be significant frequency characteristics reflecting motion-related tasks. Optimal frequency bands, however, may vary from subject to subject. A training procedure may be used to grade the contribution of each frequency band to the classification performance. For some embodiments, the ERD/ERS features 1158, 1160, 1162 may be derived by squaring temporally- filtered EEGs at the three alpha/beta/gamma frequency bands and time-averaging to smooth the waveforms. The smoothed waveforms may be normalized and features 1158, 1160, 1162 may be extracted from the normalized waveforms.

[0088] For some embodiments, temporally-filtered EEGs are squared and a corresponding band power from a reference period a few seconds prior is subtracted from the temporally-filtered and squared EEGs. I lie leieieiiue ϋΐιυά may be, for example, set as a time duration when there are no significant user movements measured from VR HMD's inertial sensors. The selected EEG signals from the reference period may be used for reducing background noise level from those EEG signals. For some embodiments, a nonzero threshold is used to distinguish between a zero crossing and a random fluctuation. If an alpha/beta/gamma frequency power signal crosses the non-zero threshold after crossing zero, a zero cross is detected. The zero cross threshold may be set to, e.g., 10% of the reference band power, for example. Voltage fluctuations of EEG measurements may be recorded from multiple electrodes placed on the scalp. EEG measurements may have plus or minus values compared to a ground voltage (OV) measured on the user's body (e.g., an ear's voltage level). EEG data of specific frequency bands (e.g., alpha, beta, and gamma) may be extracted using spectral analysis methods. The ZCT (Zero Crossing Time) of each spectral EEG data may be used as a feature to predict user movement.

[0089] Some embodiments of processes may include squaring of EEG data to obtain power signals and determining an adjusted power signal by subtracting a non-zero threshold from the power signals (thereby enabling the adjusted power signals to be plus and minus signals). However, for some embodiments, the above procedures may be performed by alternative data handling methods that also may utilize ZCT. For some embodiments, if an ERD zero crossing is detected, the zero cross threshold may be a negative value that is 10% of the reference band power, for example. If an ERS zero crossing is detected, the zero cross threshold may be a positive value that is 10% of the reference band power, for example.

[0090] As explained in Pfurtscheller, G. and da Silva, F.H. Lopes, Event-Related EEG/MEG Synchronization and Desynchronization: Basic Principles, 1 10 CLINICAL NEUROPHYSIOLOGY 1845 (1999) 'Pfurtscheller"), an ERD signal may be calculated by bandpass filtering an EEG signal, squaring the filtered signal to obtain a power signal, averaging the power signals across all trials, and averaging the signals over time. Page 1845 of Pfurtscheller states an ERD % signal may be obtained as shown in Eq. 1 :

ERD% =— x 100% Eq. 1

R ^

The power within a frequency band (such as the alpha frequency band) after an ERD event is given by A. The power in a reference period for a few second prior is given by R.

[0091] FIG. 12 is a perspective view 1200 for some embodiments of a VR HMD with indications of pitch 1202, yaw 1204, and roll 1206 movements in a coordinate system. For one embodiment, VR HMD hardware may contain a number of micro-electrical-mechanical (MEMS) sensors, such as a gyroscope, accelerometer, and magnetometer. There also may be a sensor to track headset position. Information from each of these sensors is combined through the sensor fusion process to determine the motion of the user's head in the real world and synchronize the user's view in real-time. For FIG. 12, the coordinate system uses the following uuiiveiiuuiib. me λ-axis is positive to the user's right; the y-axis is positive going up; and the z-axis is positive heading backwards from the user.

[0092] Rotation is maintained as a unit quaternion, but also may be reported in pitch-yaw-roll form. Positive rotation is counter-clockwise (CCW, direction of the rotation arrows in FIG. 12) when looking in the negative direction of each axis. Pitch is rotation around the x-axis, with positive values when looking up. Yaw is rotation around the y-axis, with positive values when turning left. Roll is rotation around the z-axis, with positive values when tilting to the left in the X-Y plane.

[0093] For example, a user tilts his or her head about the z-axis. The roll is a CCW rotation of γ degrees about the z-axis. The rotation matrix is given by Eq. 2:

cos y —sin 7 0

R_z(y) = sin 7 cos 7 0 Eq. 2

0 0 1

The upper left of the matrix matches a 2D rotation matrix, and this reduces roll to a 2D rotation in the X-Y plane. The remainder of Rz{y) is similar to the identity matrix, which causes the z coordinate to remain unchanged after a roll.

[0094] Similarly, a user pitches his or her head in a counterclockwise rotation of β degree about the x- axis, as shown in Eq. 3:

sin β cos /?

In this case, the points are rotated with respect to y and z, while the x coordinate is left unchanged.

[0095] Finally, a user rotates his or her head in a counterclockwise rotation of a degree about the y- axis, as shown in Eq. 4:

In this case, rotation occurs with respect to x and z while leaving the y coordinate unchanged.

[0096] The yaw, pitch, and roll rotations are combined sequentially in Eq. 5 to describe any 3D rotation:

α, β,γ) = Ry{a)R_x )R_z(y) Eq. 5

In this case, the ranges of a and y are from 0 to 2ττ; however, the pitch β needs to range only from -ττ/2 to TT/2 while nevertheless reaching all 3D rotations.

[0097] FIG. 13 is an example spreadsheet 1300 that matches HMD movements 1302 to related EEG features 1304. Within each frequency band's ERD/ERS signatures, a zero cross time and maximum value υιι ιΐϋ in nr\uinr\ graphs may be used as unique features for each matched VR HMD movement. As disclosed herein, ZCT is used for some embodiments to predict HMD movement onset. However, for some embodiments, the length of time from a frequency band signal maximum (or minimum) value and HMD movement onset (epoch) may be used to predict HMD movement. A server may derive unique ERD/ERS feature sets for each movement index. Because the ZCTs 1314, 1320, 1326 from each feature set 1312, 1318, 1324 represent the time duration between the ERD/ERS signal's zero crossing epochs (or zero cross point) and actual movement onset, the current EEG's zero cross points and pre-measured ZCT values may be used to estimate actual movement epochs of the VR user.

[0098] FIG. 13 shows an example set of data for an HMD movement onset for some embodiments. The table 1300 lists principal 1310, second 1316, and third 1322 components of EEG features 1304 for movement type 1306 and angle 1308. For principal 1310, second 1316, and third 1322 components of EEG features 1304, ERD/ERS features 1312, 1318, 1324 and zero cross times 1314, 1320, 1326 are listed. For a stationary state 1328, no angle is associated. For a yaw movement 1330, angles are divided into four regions: -ττ/2 to -TT/4, TT/4 to 0, 0 to TT/4, and ττ/4 to ττ/2. For a roll movement 1332, angles are divided into two regions: -ττ/4 to 0 and 0 to TT/4. For a pitch movement 1334, angles are divided into two regions: -TT/4 to 0 and 0 to TT/4. For each scenario, a frequency band 1336, feature ID 1338, and electrode 1340 are listed.

[0099] For some embodiments, determining a predicted direction may include selecting an EEG signal from the one or more measured EEG signals and determining the predicted direction as a direction of head movement of the user associated with the selected EEG signal, electrode, or frequency band.

[0100] Systems and methods described herein in accordance with some embodiments predict future movements of a VR HMD by using an alpha/beta/gamma frequency band's unique ERD/ERS features measured about N seconds (e.g., 3 seconds) before movement onset. For some embodiments, within each frequency band's ERD/ERS signatures, the zero crossing time and maximum value points in ERD/ERS graphs may be used as unique features for each matched VR HMD movement. In some embodiments, among the EEG signals from multiple electrodes worn on VR users, the most prominent features may be selected among the multiple electrodes by using principal components analysis (PCA) or independent component analysis (ICA) methods.

[0101] For each reported data of prominent positive/negative-degree movements of Pitch/Yaw/Roll, a server may examine EEG data set that is received before a certain point in time. By using the signal processing procedures and feature selection algorithms described above, a server may derive unique ERD/ERS feature sets for each movement index. FIG. 13 shows examples in accordance with some embodiments for correlation of these features. lu i u j ru[ example, Yaw movements ranging from 0 to π/4 have the feature of [alpha₁ (C₄), garnrna₂ (P_z), beta₁ (C_z)] as listed in FIG. 13. The alpha frequency feature ID #1 measured by the C4 electrode is a principal component for this movement prediction. The gamma frequency feature ID #2 measured by the Pz electrode and the beta frequency feature ID #1 measured by the Cz electrode are second and third components, respectively, in predicting the Yaw 0 to π/4 movement. The frequency feature ID is an increasing index that differentiates various numerical features for each electrode and frequency, respectively. For example, during a training period, by analyzing HMD movement and past EEG signals for a frequency band, a determination may be made of which electrode consistently has a zero crossing before a movement, such as a yaw movement with a rotation angle between 0 to π/4. Based on the coupling strength between the feature and the movement, principal, second, and third (or more) components may be identified. With features (e.g., a/p ia₁ (C₄)) identified, a feature's zero crossing time before a matching movement may be calculated and averaged, enabling a VR server to prepare for a user's future movement, such as a yaw movement with a rotation angle between 0 to π/4.

[0103] However, there may be several EEG signal patterns before and after the zero crossing, such as, for example, a swing from a large positive signal to a large negative signal or a swing from a large positive signal to a small negative signal. If those differences in signal patterns are individual predictors of different head movements that should be identified separately, multiple feature IDs 1338 may be created in the same frequency band (e.g., alpha) and the same electrode (e.g., C4), such as: a/p ia₁ (C₄) and alpha₂ (C₄).

[0104] For some embodiments, the zero crossing time may be used in a VR video streaming application. When a certain movement is predicted by analyzing-and-extracting features from the current EEG data, the VR video server estimates when the predicted movement will actually happen at the user by using zero crossing time data. Because the ZCTs from each feature set represents the time duration between the ERD/ERS signals' zero crossing epochs (zero cross point) and actual movement onset, by using the current EEG's zero cross points and pre-measured ZCT values, an actual movement epoch of the VR user may be estimated. For some embodiments, the existence of a maximum or minimum EEG value before a certain movement may be used as a prediction feature for a head movement. For example, if there is tends to be a peak value event in the alpha frequency band from the electrode C4 at an average of 2.1 seconds before a yaw movement with a rotation angle between 0 to π/4, the peak value event may be detected to prepare for a movement that is predicted to occur 2.1 seconds after the peak value event.

[0105] If a VR user's movement is interpreted as a part of the motor preparation procedure, several phenomena may be measured. For Alpha/Beta/Gamma frequency event-related desynchronization/synchronization, a short-lasting decrease of frequency power (ERD) in the alpha band

(about 8-12 Hz) and in the central beta band (about 16-24 Hz) may begin about 2 seconds before self-paced movement or motor imageries. A sharp increase in frequency power for the gamma band (30~40Hz) (ERS) may aisu uuuui . event-related potentials recorded by EEG may occur not only when a movement is performed by the subject, but also when the movement is imagined. Therefore, ERD/ERS of the EEG may be treated as a clue of future user movement.

[0106] FIG. 14 is an example message sequence diagram in accordance with an example process 1400 for some embodiments for a movement prediction system using a VR video streaming application. The server 1404 uses the current pre-movement phenomenon to predict future movement of the user's head (and HMD 1402) and to deliver video tiles corresponding to the predicted future head movement. A VR user requests 1406 VR video streaming service, such as by clicking on VR video content in a video content service application while wearing a VR HMD 1402. The VR server 1404 determines that the VR HMD is equipped with EEG sensor data and decides to use EEG sensor data to predict VR HMD movements.

[0107] Using feature extraction, the direction of a moving limb may be determined prior to movement onset. Therefore, a future viewing direction of a user also may be extracted from an EEG signal. The future viewing direction of the user is more specific information than the intention of the user. Moreover, each person (user) may have a different EEG record for the same movement. Therefore, an additional training phase or a calibration phase for each VR user may be used to correctly predict future FOV changes. The training phase may be used to correlate an HMD movement to an EEG feature, as shown in FIG. 13. The VR content sever 1404 picks supervised 1426 and/or unsupervised 1428 training based on application requirements. For explanatory purposes, FIG. 14 shows an example in which both a supervised 1426 and unsupervised 1428 training phase may be used in addition to an application phase 1430 shown on the bottom half of FIG. 14. During the training phase, the VR server 1404 collects an EEG signal corresponding to the motion information of the user and matches the EEG signal and the motion information.

[0108] For supervised training 1426, a server 1404 transmits 1408 to an HMD 1402 (worn by the user) VR content that may induce a FOV change by a user. To extract 1412 unique EEG features for a specific HMD movement, the contents having a single video/audio stimulus may be displayed to the user in sequential order. If this supervised training phase 1426 is included in an introduction of 360-degree movie content, called a visual queue, a specific character or a text (for example, a name of a producer or an actor) may appear on a screen at a certain position. An HMD 1402 (and the user) responds to the FOV change-inducing content, and the VR server 1404 receives 1410 information describing motion x and corresponding EEG data before x.

[0109] After receiving an EEG signal and related motion information, a VR server 1404 may match them to make predictions of future FOV movements of the user. For some embodiments of movement pattern matching, supervised training may be repeated multiple times. In addition, ZCT calculation may be based on the time difference between the actual head movement and the related EEG feature's zero crossing epoch (or zero cross point). iu I luj rui unsupervised training 1428, a server 1404 transmits 1414 to an HMD 1402 (worn by the user) "normal" VR content. For unsupervised training 1428, a VR user's current HMD motion data and the related past EEG data may be collected and analyzed without displaying a visual cue that induces a motion. This unsupervised training may be used if introduction of a visual cue-based training phase in a VR streaming session is not feasible (or is not performed).

[0111] A VR user may change his or her viewing direction during an initial part of the VR video streaming. Before a direction change occurs, a pre-movement EEG signal may be transmitted to a VR server, and the corresponding motion information sensed by the HMD may be transmitted thereto. An HMD responds to the VR content with information describing motion x and EEG data before motion x occurs. After receiving 1416 the EEG signal and the related motion information, the VR server 1404 may match them to calculate a prediction of future FOV movement of the user. Also, unsupervised training 1428 may be performed repeatedly to extract features for various HMD movements. A server 1404 may perform 1418 feature extraction for motion x.

[0112] Before a user moves, EEG data may be measured by HMDs in non-invasive manner. Alpha/beta/gamma ERD/ERS may indicate a future movement or intention of a user. An HMD sends EEG data containing alpha/beta/gamma ERD/ERS features to a VR server.

[0113] After receiving EEG data 1420 from the HMD 1402, a VR server 1404 may select 1422 video tiles corresponding to the EEG signal-based movement prediction. This selection 1422 may use the set of mappings that map an EEG signal pattern to a motion x generated by the supervised training 1426 or unsupervised training 1428. The location and the number of selected tiles may depend on the content of the EEG signal (the intention or movement of the user captured in the received EEG signal). If a certain movement is predicted by analyzing-and-extracting features from the current EEG data, the VR video server estimates when the predicted movement will happen by the user by using the zero crossing time data of the related features. Because the ZCTs from each feature set represents the time duration between a ERD/ERS signal's zero crossing epoch (or zero cross point) and actual HMD movement onset, in some embodiments, an actual movement epoch of the VR user may be estimated by using a current EEG's zero cross point and pre-measured ZCT values.

[0114] During an application phase 1430, EEG data may be transmitted 1420 from an HMD 1402 to a server 1404. A server 1404 may select 1422 video tiles based on EEG data. For some embodiments, video tiles may be transmitted 1424 within t_left seconds or less from a server 1404 to an HMD 1402, where t_left may be calculated as shown in FIG. 10B. For example, if a zero crossing of an alpha band EEG signal (which may be measured by the C4 electrode attached to a VR user) was detected 1 second ago, a yaw movement with a rotation angle between 0 to π/4 is predicted to happen 1.5 seconds later [t_left), because the leaiuie is the principal predictor for a yaw movement with a rotation angle between 0 to π/4 and the ZCT value is 2.5 seconds, as indicated in FIG.13. Equation 6 shows the relationship of ZCT and t_left:

eft = ZCT— t_elapsed Eq. 6 teiapsed ^is the ^len9^{tn of} time between the zero cross point and the current time. t_left is the length of time between the current time and HMD movement onset. ZCT is the length of time between the zero cross point and HMD movement onset.

[0115] FIGs. 15A and 15B are system configurations 1500, 1550 for some embodiments showing an example of a supervised training sequence to a user 1502, 1552. FIG. 15A is a multi-view drawing with a front view of the display 1504 and a corresponding top view of the user 1502. The display 1504 shows a text stimulus 1506 that induces yaw movement EEG data that is accumulated internally. For some embodiments, a plurality of video tiles may be displayed that induce a user field of view (FOV) change. FIG. 15B is a multi- view drawing with a front view of the display 1554 and a corresponding top view of the user 1552. The display 1554 shows a text stimulus 1556 that induces an actual yaw head motion, which may be used for EEG feature extraction, zero crossing detection, and ZCT calculations. The actual ^ yaw head motion is an example of a training procedure user movement that may be used for the induced FOV change. At the start of a movie, the HMD displays a blank screen to the user, and there is no stimulus for the user. As shown in FIG. 15A, the text stimulus 1506 "Micha" (which is the first five letters of the name "Michael Bay") appears on the right side of the screen, and the user 1502 wants to see the text precisely. Before the user's FOV changes, an EEG signal related to the FOV change is acquired by the HMD. For some embodiments, the HMD transmits the EEG signal to a VR server. As shown in FIG. 15B, after a certain time, the user 1552 may actually turn his or her head to see the whole text ("Michael Bay"), and the HMD may transmit the motion information to the VR server. The VR server may perform feature extraction for motion x by estimating the time and matching an EEG pattern with final motion information. Supervised training may be performed several times to acquire various EEG features for multiple sets of motion data. A supervised training procedure may include extracting of an EEG pattern (or feature) from an EEG signal for an induced FOV change. A supervised training procedure may match a training procedure user movement to an extracted EEG pattern.

[0116] For example, as depicted in FIGs. 15A and 15B, while the current FOV for the VR HMD is in the center direction (FIG. 15A), the VR HMD's EEG data indicates that the user will have a "Yaw 0-π/4 degree" movement (FIG. 15B) in fieft time later. The fiefttime is calculated based on the timing relationship diagram in FIG. 10B. Therefore, the VR server prepares and sends related high resolution tiles matched to the future direction before the movement happens. Because there is considerable network/buffering latency on VR client side, the predicted movement information gives the VR server time to prepare for the future optimal tile seiewiuii aiiu

For one embodiment, the corresponding tiles may be selected before the user moves, and the tiles may be delivered to the user when the user executes the predicted movement.

[0117] FIG. 16 is a video frame layout diagram 1600 for some embodiments indicating a prediction for the direction of FOV movement and selection of video tile resolutions. "L" indicates low resolution, "M" indicates middle resolution, and "H" indicates high resolution. Using the example shown in FIGs. 15A and 15B, the HMD is predicted to have a yaw rotation the right. FIG. 16 shows the twelve tiles 1622, 1624, 1626, 1628, 1638, 1640, 1642, 1644, 1654, 1656, 1658, 1660 marked with an "H" used for the FOV at time t = At time t = t₁ + t_left, the user's HMD is predicted to have a ^ yaw rotation to the right. The affected video tiles 1630, 1632, 1646, 1648, 1662, 1664 are marked as "M or H" to indicate that the resolution of the video tiles may be increased to middle or high resolutions. The video tiles 1602, 1604, 1606, 1608, 1610, 1612, 1614, 1616, 1618, 1620, 1634, 1636, 1650, 1652 outside the current FOV at time t = t_x and the predicted FOV at time t = t₁ + t_left are marked with an "L" to indicate that those frames are low resolution.

[0118] For some embodiments, a predicted field of view (FOV) may be determined based on the predicted head movement, and the predicted FOV may be matched to one or more tiles in the multi-tile video. The resolution of retrieved tiles corresponding to the predicted FOV may be increased. Determining a predicted FOV may include determining a current FOV as a first selection of tiles and determining the predicted FOV as a second selection of tiles, where the second selection of tiles may be a shift in the predicted direction of the first selection. Resolution of tiles retrieved corresponding to the predicted FOV may be retrieved for a higher resolution.

[0119] FIG. 17 is a flowchart for an example process 1700 in accordance with some embodiments for a user performing FOV prediction and EEG matching with a content provider platform. Several online contents providers may deliver VR content through HMDs. With the popularity of high-resolution content, problems with bandwidth may be an issue for a content provider.

[0120] For example, a VR content provider may offer personalized service to customers. The content provider may analyze characteristics and preferences of each customer and suggest content optimized for each customer. Personalized information, like analyzed characteristics, user preferences, user selections, user play control, and user playlists may be stored on a content provider server. Using systems and methods described herein, a content provider server may store personalized EEG matching information (calibration/training information) for each customer.

[0121] If a user (or client device) connects 1702 to a content provider server, he or she may login 1704 to a content provider platform. The user may wear an HMD that has electrodes or sensors to collect EEG data from the user. If the user wears an HMD, the HMD may transmit a signal to a content server to communicate to the server that the user is wearing an HMD. A process may be performed to determine 1706 II Βΐ ϋυι ceo matching data exists. If the content provider server has stored user personalized EEG matching information, the user may view the VR contents using the HMD without a calibration/training phase. If not, the user (or an HMD) may perform 1708 an EEG matching phase (calibration/training phase) that a content provider server offers. An EEG matching phase may be incorporated into part of the VR content (such as, the introduction part of the content). VR content may be rendered 1710 for video tiles selected based on EEG signals.

[0122] FIG. 18 is a flowchart of an example process 1800 for some embodiments for a server performing FOV prediction and EEG matching with a content provider platform. A check may be performed to determine 1802 if VR content uses specific EEG matching information. For example, in certain VR content, a user may move his or her head frequently or use an upside-down FOV while watching the VR content. A check may be performed 1804 to determine if personalized EEG matching data exists. If a content provider server does not have content-related specific EEG matching information in an EEG database, a content provider may perform 1806 an EEG matching phase for the content. During a matching phase, older EEG matching information may be updated 1808 with acquired specific EEG matching information or specific EEG matching information may be stored separately. A content server may select tiles based on specific EEG signals. VR content may be delivered 1810 to a user (or HMD). An HMD may render the received tiles for some embodiments.

[0123] FIG. 19 is a flowchart of an example process 1900 for some embodiments for retrieving and rendering a plurality of tiles of a multi-tile video based on an EEG signal. For some embodiments, a method may include: displaying 1902, on a display of a head-mounted display (HMD), a first plurality of tiles of a multi-tile video; measuring 1904, while the multi-tile video is displayed, an electroencephalography (EEG) signal of the user; determining 1906 a predicted head movement of the user based on the measured EEG signal; retrieving 1908 a second plurality of tiles of a multi-tile video based on the predicted head movement of the user; and rendering 1910 one or more of the second plurality of tiles of the multi-tile video.

[0124] FIG. 20 is a flowchart of an example process 2000 for some embodiments for retrieving a plurality of tiles of a multi-tile video based on an EEG signal. For some embodiments, a method may include retrieving 2002, by a head-mounted display (HMD), a plurality of tiles of a multi-tile video based on a viewing direction of a user; displaying 2004, on a display of the HMD, one or more of the plurality of tiles; measuring 2006 an electroencephalography (EEG) signal of the user while the multi-tile video is displayed; determining 2008 whether a head movement of the user is predicted based on the measured EEG signal; and updating 2010 which tiles of the multi-tile video to retrieve if a head movement is predicted.

[0125] Some embodiments of a method may include: measuring an electroencephalography (EEG) signal for each of a plurality of electrodes; performing a training procedure to match an EEG signal selected from the plurality of EEG signals to a user movement of a Head Mounted Display (HMD); calculating a ΐϋυι ϋΐ] usei niuvement onset time based on a zero crossing time in relation to a frequency band and an electrode; calculating a predicted direction of user movement; and retrieving relevant video tiles based on the predicted direction of user movement.

[0126] For some embodiments, performing the training procedure may include performing a supervised training procedure, which may include: displaying a plurality of video tiles on the HMD that induce a user field of view (FOV) change; determining a training procedure user movement for the induced FOV change; extracting an EEG pattern from the selected EEG signal for the induced FOV change; and matching the training procedure user movement to the extracted EEG pattern.

[0127] For some embodiments, performing the training procedure may include performing an unsupervised training procedure, which may include: displaying a plurality of video tiles on the HMD; detecting a change to a user field of view (FOV); determining a training procedure user movement for the detected FOV change; extracting an EEG pattern from the selected EEG signal for the detected FOV change; and matching the training procedure user movement to the extracted EEG pattern.

[0128] Some embodiments of a method further may include: calculating a future location of a user field of view (FOV) based on the predicted direction of user movement; and adjusting resolution of the relevant video tiles based on the calculated future location of the user FOV.

[0129] For some embodiments, adjusting resolution of the retrieved video tiles may include: matching the future location of the user FOV to a plurality of relevant video tiles; and increasing resolution of the plurality of relevant video tiles.

[0130] For some embodiments, calculating a predicted user movement onset time may include: selecting an EEG onset signal from the plurality of measured EEG signals; removing noise from the EEG onset signal to generate a noise-reduced EEG signal; bandpass filtering the noise-reduced EEG signal to generate a bandpass-filtered signal; squaring the bandpass-filtered signal to generate a power signal; and matching the power signal to a prediction of a user movement onset time.

[0131] For some embodiments, the training procedure user movement may include an onset and a direction.

[0132] For some embodiments, calculating a predicted direction of user movement may include: selecting an EEG direction signal from the plurality of measured EEG signals; removing noise from the EEG direction to generate a noise-reduced EEG signal; bandpass filtering the noise-reduced EEG signal to generate a bandpass-filtered signal; squaring the bandpass-filtered signal to generate a power signal; and matching the power signal to a prediction of a direction of user movement.

[0133] Some embodiments of a method further may include time-averaging and normalizing the power signal prior to matching the power signal. Lu i ^j oun ie embodiments of a method may include: measuring an electroencephalography (EEG) signal for each of a plurality of electrodes; calculating a predicted user movement onset time; calculating a predicted direction of user movement; selecting relevant video tiles from a plurality of video tiles based on the predicted user movement onset time and the predicted direction of user movement; retrieving the relevant video tiles from a server; and displaying the relevant video tiles on a head mounted display.

[0135] Some embodiments of a method further may include: calculating a future location of a user field of view (FOV) based on the predicted direction of user movement predicted and predicted user movement onset time; and adjusting resolution of the relevant video tiles based on the calculated future location of the user FOV.

[0136] For some embodiments, adjusting resolution of the retrieved video tiles may include: matching the future location of the user FOV to a plurality of relevant video tiles; and increasing resolution of the plurality of relevant video tiles.

[0137] For some embodiments, calculating a predicted direction of user movement may include: selecting an EEG direction signal from the plurality of measured EEG signals; removing noise from the EEG direction signal to generate a noise-reduced EEG signal; bandpass filtering the noise-reduced EEG signal to generate a bandpass-filtered signal; squaring the bandpass-filtered signal to generate a power signal; and matching the power signal to a prediction of a direction of user movement.

[0138] For some embodiments, selecting an EEG direction signal from the plurality of measured EEG signals is based on a principal components analysis of the plurality of measured EEG signals.

[0139] For some embodiments, selecting an EEG direction signal from the plurality of measured EEG signals is based on an independent component analysis of the plurality of measured EEG signals.

[0140] For some embodiments, calculating a predicted user movement onset time may include: selecting an EEG onset signal from a plurality of EEG signals; removing noise from the selected EEG onset signal to generate a noise-reduced EEG signal; bandpass filtering the noise-reduced EEG signal to generate a bandpass-filtered signal; squaring the bandpass-filtered signal to generate a power signal; and matching the power signal to a prediction of a user movement onset time based on the zero cross time for the selected EEG onset signal.

[0141] Some embodiments of a method further may include: determining a zero cross time for the selected EEG onset signal, wherein matching the power signal to a prediction of a user movement onset time is based on the zero cross time for the selected EEG onset signal.

[0142] For some embodiments, matching the power signal to a prediction of a user movement onset time may include: detecting an event-related synchronization (ERS) feature in the selected EEG onset signal; and calculating the prediction of the user movement onset time based on the detected ERS feature. [u ifo] rui buine embodiments, matching the power signal to a prediction of a user movement onset time may include: detecting an event-related Desynchronization (ERD) feature in the selected EEG onset signal; and calculating the prediction of the user movement onset time based on the detected ERD feature.

[0144] Some embodiments of a device may include: a processor; and a non-transitory computer- readable medium storing instructions that are operative, when executed on the processor, to perform the functions of: measuring an electroencephalography (EEG) signal; performing a training procedure to match an EEG signal to user movement of a Head Mounted Display (HMD); determining a time duration of user movement based on a zero crossing time in relation to a frequency band and an electrode used to measure the EEG signal; determining a direction of user movement; and retrieving relevant video tiles based on the determined direction.

[0145] Some embodiments of a method may include: at a head-mounted display (HMD), retrieving a plurality of tiles of a multi-tile video and displaying at least some of the retrieved tiles of video, wherein a resolution of each of the retrieved tiles is selected based at least in part on head tracking of the user; collecting EEG data from the user during display of the video; detecting an event-related desynchronization in the EEG data; and changing the selected the resolution of the retrieved tiles based at least in part on detection of the event-related desynchronization.

[0146] Some embodiments of a method may include: at a head-mounted display (HMD), retrieving a plurality of tiles of a multi-tile video and displaying at least some of the retrieved tiles of video, wherein a resolution of each of the retrieved tiles is selected based at least in part on (i) head tracking of the user and (ii) EEG data of the user.

[0147] Some embodiments of a method may include: displaying, on a display of a head-mounted display (HMD), a first plurality of tiles of a multi-tile video; measuring, while the multi-tile video is displayed, an electroencephalography (EEG) signal of the user to generate a measured EEG signal; determining a predicted head movement of the user based on the measured EEG signal; retrieving a second plurality of tiles of the multi-tile video based on the predicted head movement of the user; and rendering one or more of the second plurality of tiles of the multi-tile video.

[0148] For some embodiments, determining the predicted head movement of the user may include: detecting a zero crossing of an EEG-derived signal derived from the measured EEG signal; and determining the predicted head movement of the user to occur proximate to the detected zero crossing.

[0149] For some embodiments, detecting the zero crossing of the EEG-derived signal may include detecting that the EEG-derived signal is less than a zero crossing threshold.

[0150] Some embodiments of a method further may include generating a frequency-band power signal from the measured EEG signal, wherein the EEG-derived signal is the frequency-band power signal. ΐυ ι υ ΐ] ounie embodiments of a method further may include: performing a training procedure that matches a plurality of head movements of the user to a plurality of EEG signal patterns, wherein determining the predicted head movement of the user may include: identifying, within the measured EEG signal, an EEG signal pattern selected from the plurality of EEG signal patterns; and determining the predicted head movement to be the head movement of the user matched, by the training procedure from the plurality of EEG signal patterns, to the identified EEG signal pattern selected from the plurality of EEG signal patterns.

[0152] For some embodiments, performing the training procedure comprises performing a supervised training procedure that matches induced FOV changes to the plurality of EEG signal patterns.

[0153] For some embodiments, performing the training procedure comprises performing an unsupervised training procedure that matches detected FOV changes to the plurality of EEG signal patterns.

[0154] Some embodiments of a method further may include: selecting a plurality of resolutions based on the predicted head movement of the user, wherein each of the plurality of resolutions corresponds to a respective one of the second plurality of tiles, and wherein the second plurality of tiles are rendered at the corresponding resolution.

[0155] Some embodiments of a method further may include: determining a predicted field of view (FOV) based on the predicted head movement; and matching the predicted FOV to one or more tiles in the multi- tile video, wherein selecting the plurality of resolutions selects an increased resolution for tiles matched to the predicted FOV.

[0156] For some embodiments, determining the predicted head movement of the user may include determining a predicted direction, wherein determining the predicted FOV may include: determining a current FOV to be a first selection of one or more tiles in the multi-tile video; and determining the predicted FOV to be a second selection of one or more tiles in the multi-tile video, wherein the second selection is a shift of the first selection in the predicted direction.

[0157] For some embodiments, each of the second plurality of tiles may be retrieved at the respective selected resolution.

[0158] For some embodiments, determining the predicted head movement may include determining a predicted direction.

[0159] For some embodiments, measuring the EEG signal may include measuring the EEG signal to generate a plurality of measured EEG signals; and determining the predicted direction may include: selecting a selected EEG signal from the plurality of measured EEG signals; and determining the predicted direction as a direction of head movement of the user associated with the selected EEG signal, wherein detecting the zero crossing of the EEG-derived signal may include detecting the zero crossing of an EEG signal derived from the selected EEG signal. [u l ouj rui buine embodiments, retrieving the second plurality of tiles of the multi-tile video may include: identifying the first plurality of tiles as a first selection of tiles of the multi-tile video; and selecting the second plurality of tiles as a second selection of tiles of the multi-tile video, wherein the second selection may be a shift of the first selection in the predicted direction.

[0161] Some embodiments of a device may include: a processor; and a non-transitory computer- readable medium storing instructions that are operative, when executed on the processor, to perform the functions of: displaying, on a display of a head-mounted display (HMD), a first plurality of tiles of a multi-tile video; measuring an electroencephalography (EEG) signal of the user while the multi-tile video is displayed; determining a predicted head movement of the user based on the EEG signal; retrieving a second plurality of tiles of the multi-tile video based on the predicted head movement of the user; and rendering one or more of the second plurality of tiles of the multi-tile video.

[0162] Some embodiments of a method may include: retrieving, by a head-mounted display (HMD), a plurality of tiles of a multi-tile video based on a viewing direction of a user; displaying, on a display of the HMD, one or more of the plurality of tiles; measuring an electroencephalography (EEG) signal of the user while the multi-tile video is displayed; determining whether a head movement of the user is predicted based on the measured EEG signal; and updating which tiles of the multi-tile video to retrieve if a head movement is predicted.

[0163] For some embodiments, determining the predicted head movement of the user may include: detecting a zero crossing of an EEG-derived signal derived from the measured EEG signal; and determining the predicted head movement of the user to occur proximate to the detected zero crossing.

[0164] Some embodiments of a method further may include: reducing noise from the measured EEG signal to generate a noise-reduced EEG signal; bandpass filtering the noise-reduced EEG signal to generate a bandpass-filtered signal; squaring the bandpass-filtered signal to generate a current power signal; and subtracting a previous power signal from the current power signal to generate a difference power signal, wherein the EEG-derived signal is the difference power signal.

[0165] Some embodiments of a method further may include: performing a training procedure that matches a plurality of head movements of the user to a plurality of EEG signal patterns, wherein determining the predicted head movement of the user may include: identifying, within the measured EEG signal, an EEG signal pattern selected from the plurality of EEG signal patterns; and determining the predicted head movement to be the head movement of the user matched, by the training procedure from the plurality of EEG signal patterns, to the identified EEG signal pattern selected from the plurality of EEG signal patterns.

[0166] For some embodiments, performing the training procedure may include: displaying, on the display of the HMD, a plurality of tiles of a multi-tile training procedure video that are configured to induce a usei i ieau i i iuvei i ienL and a change to a field of view (FOV) of the user; measuring a training procedure EEG signal while the multi-tile training procedure video is displayed; extracting an observed EEG signal pattern from the training procedure EEG signal to generate an extracted EEG signal pattern; and matching the extracted EEG signal pattern to the head movement corresponding to the change to the FOV of the user.

[0167] For some embodiments, performing the training procedure may include: displaying, on the display of the HMD, a plurality of tiles of a multi-tile training procedure video; measuring a training procedure EEG signal while the multi-tile training procedure video is displayed; detecting a head movement corresponding to a change to a field of view (FOV) of the user; extracting an observed EEG signal pattern from the training procedure EEG signal to generate an extracted EEG signal pattern; and matching the extracted EEG signal pattern to the head movement corresponding to the change to the FOV of the user.

[0168] Some embodiments of a device may include: a processor; and a non-transitory computer- readable medium storing instructions that are operative, when executed on the processor, to perform the functions of: retrieving, by a head-mounted display (HMD), a plurality of tiles of a multi-tile video based on a viewing direction of a user; displaying, on a display of the HMD, one or more of the plurality of retrieved tiles; measuring an electroencephalography (EEG) signal of the user while the multi-tile video is displayed; determining whether a head movement of the user is predicted based on the measured EEG signal; and updating which tiles of the multi-tile video to retrieve if a head movement is predicted.

NETWORK ARCHITECTURE

[0169] A wireless transmit/receive unit (WTRU) may be used as a Head Mounted Display (HMD) in embodiments described herein.

[0170] FIG. 21 is a diagram illustrating an example communications system 100 in which one or more disclosed embodiments may be implemented. The communications system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communications system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communications systems 100 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), and the like.

[0171 ] As shown in FIG. 21 , the communications system 100 may include wireless transmit/receive units ( TRUs) 102a, 102b, 102c, 102d, a RAN 104/113, a CN 106/115, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and/or network elements. Each υι Li ic w I rvub I I CI, 102b, 102c, 102d may be any type of device configured to operate and/or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a "station" and/or a "STA", may be configured to transmit and/or receive wireless signals and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (loT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and/or other wireless devices operating in an industrial and/or an automated processing chain contexts), a consumer electronics device, a device operating on commercial and/or industrial wireless networks, and the like. Any of the WTRUs 102a, 102b, 102c and 102d may be interchangeably referred to as a UE.

[0172] The communications systems 100 may also include a base station 114a and/or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106/115, the Internet 110, and/or the other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and/or network elements.

[0173] The base station 114a may be part of the RAN 104/113, which may also include other base stations and/or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and/or the base station 114b may be configured to transmit and/or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for a wireless service to a specific geographical area that may be relatively fixed or that may change over time. The cell may further be divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one for each sector of the cell. In an embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and/or receive signals in desired spatial directions.

[0174] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio neijueiiuy \Τ\Γ), nii owave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0175] More specifically, as noted above, the communications system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 104/113 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 115/116/117 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and/or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink (DL) Packet Access (HSDPA) and/or High-Speed UL Packet Access (HSUPA).

[0176] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and/or LTE-Advanced (LTE-A) and/or LTE-Advanced Pro (LTE-A Pro).

[0177] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR Radio Access, which may establish the air interface 116 using New Radio (NR).

[0178] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement LTE radio access and NR radio access together, for instance using dual connectivity (DC) principles. Thus, the air interface utilized by WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and/or transmissions sent to/from multiple types of base stations (e.g., a eNB and a gNB).

[0179] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0180] The base station 114b in FIG. 21 may be a wireless router, Home Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a roadway, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network iw Lrti . ill an eniuodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR etc.) to establish a picocell or femtocell. As shown in FIG. 21, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not be required to access the Internet 110 via the CN 106/115.

[0181] The RAN 104/113 may be in communication with the CN 106/115, which may be any type of network configured to provide voice, data, applications, and/or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have varying quality of service (QoS) requirements, such as differing throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. The CN 106/115 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and/or perform high-level security functions, such as user authentication. Although not shown in FIG. 21, it will be appreciated that the RAN 104/113 and/or the CN 106/115 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104/113 or a different RAT. For example, in addition to being connected to the RAN 104/113, which may be utilizing a NR radio technology, the CN 106/115 may also be in communication with another RAN (not shown) employing a GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0182] The CN 106/115 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and/or the other networks 112. The PSTN 108 may include circuit-switched telephone networks that provide plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and/or the internet protocol (IP) in the TCP/IP internet protocol suite. The networks 112 may include wired and/or wireless communications networks owned and/or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 104/113 or a different RAT.

[0183] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU 102c shown in FIG. 21 may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology. Lu no. is a system diagram of an example WTRU 2202. As shown in FIG. 22, the WTRU 2202 may include a processor 2218, a transceiver 2220, a transmit/receive element 2222, a speaker/microphone 2224, a keypad 2226, a display/touchpad 2228, a non-removable memory 2230, a removable memory 2232, a power source 2234, a global positioning system (GPS) chipset 2236, and other peripherals 2238. The transceiver 2220 may be implemented as a component of decoder logic 2219. For example, the transceiver 2220 and decoder logic 2219 may be implemented on a single LTE or LTE-A chip. The decoder logic may include a processor operative to perform instructions stored in a non-transitory computer-readable medium. As an alternative, or in addition, the decoder logic may be implemented using custom and/or programmable digital logic circuitry.

[0185] The processor 2218 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 2218 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 2202 to operate in a wireless environment. The processor 2218 may be coupled to the transceiver 2220, which may be coupled to the transmit/receive element 2222. While FIG. 22 depicts the processor 2218 and the transceiver 2220 as separate components, the processor 2218 and the transceiver 2220 may be integrated together in an electronic package or chip.

[0186] The transmit/receive element 2222 may be configured to transmit signals to, or receive signals from, a base station (or other WTRU 2202 for some embodiments) over the air interface 2216. For example, in one embodiment, the transmit/receive element 2222 may be an antenna configured to transmit and/or receive RF signals. In another embodiment, the transmit/receive element 2222 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, as examples. In yet another embodiment, the transmit/receive element 2222 may be configured to transmit and receive both RF and light signals. The transmit/receive element 2222 may be configured to transmit and/or receive any combination of wireless signals.

[0187] In addition, although the transmit/receive element 2222 is depicted in FIG. 22 as a single element, the WTRU 2202 may include any number of transmit/receive elements 2222. More specifically, the WTRU 2202 may employ MIMO technology. Thus, in one embodiment, the WTRU 2202 may include two or more transmit/receive elements 2222 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 2216.

[0188] The transceiver 2220 may be configured to modulate the signals that are to be transmitted by the transmit/receive element 2222 and to demodulate the signals that are received by the transmit/receive eieiiieni LLLL. rts noted above, the WTRU 2202 may have multi-mode capabilities. Thus, the transceiver 2220 may include multiple transceivers for enabling the WTRU 2202 to communicate via multiple RATs, such as UTRA and IEEE 802.1 1 , as examples.

[0189] The processor 2218 of the WTRU 2202 may be coupled to, and may receive user input data from, the speaker/microphone 2224, the keypad 2226, and/or the display/touchpad 2228 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 2218 may also output user data to the speaker/microphone 2224, the keypad 2226, and/or the display/touchpad 2228. In addition, the processor 2218 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 2230 and/or the removable memory 2232. The non-removable memory 2230 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 2232 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 2218 may access information from, and store data in, memory that is not physically located on the WTRU 2202, such as on a server or a home computer (not shown).

[0190] The processor 2218 may receive power from the power source 2234, and may be configured to distribute and/or control the power to the other components in the WTRU 2202. The power source 2234 may be any suitable device for powering the WTRU 2202. As examples, the power source 2234 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), and the like), solar cells, fuel cells, and the like.

[0191] The processor 2218 may also be coupled to the GPS chipset 2236, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 2202. In addition to, or in lieu of, the information from the GPS chipset 2236, the WTRU 2202 may receive location information over the air interface 2216 from a base station and/or determine its location based on the timing of the signals being received from two or more nearby base stations. The WTRU 2202 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0192] The processor 2218 may further be coupled to other peripherals 2238, which may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity. For example, the peripherals 2238 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a Virtual Reality and/or Augmented Reality (VR/AR) device, an activity tracker, and the like. The peripherals 138 may include one or more sensors, the sensors may be one or more of a gyroscope, an accelerometer, a hall effect seiisui, a niayiieiuiiieter, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and/or a humidity sensor.

[0193] The WTRU 102 may include a full duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for both the UL (e.g., for transmission) and downlink (e.g., for reception) may be concurrent and/or simultaneous. The full duplex radio may include an interference management unit to reduce and or substantially eliminate self-interference via either hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In an embodiment, the WRTU 102 may include a half-duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or the downlink (e.g., for reception)).

[0194] FIG. 23 depicts an example network entity 2390 that may be used by a content provider server. As depicted in FIG. 23, network entity 2390 includes a communication interface 2392, a processor 2394, and non-transitory data storage 2396, all of which are communicatively linked by a bus, network, or other communication path 2398.

[0195] Communication interface 2392 may include one or more wired communication interfaces and/or one or more wireless-communication interfaces. With respect to wired communication, communication interface 2392 may include one or more interfaces such as Ethernet interfaces, as an example. With respect to wireless communication, communication interface 2392 may include components such as one or more antennae, one or more transceivers/chipsets designed and configured for one or more types of wireless (e.g., LTE) communication, and/or any other components deemed suitable by those of skill in the relevant art. And further with respect to wireless communication, communication interface 2392 may be equipped at a scale and with a configuration appropriate for acting on the network side— as opposed to the client side— of wireless communications (e.g., LTE communications, Wi-Fi communications, and the like). Thus, communication interface 2392 may include the appropriate equipment and circuitry (including multiple transceivers) for serving multiple mobile stations, UEs, or other access terminals in a coverage area.

[0196] Processor 2394 may include one or more processors of any type deemed suitable by those of skill in the relevant art, some examples including a general-purpose microprocessor and a dedicated DSP.

[0197] Data storage 2396 may take the form of any non-transitory computer-readable medium or combination of such media, some examples including flash memory, read-only memory (ROM), and random- access memory (RAM) to name but a few, as any one or more types of non-transitory data storage deemed suitable by those of skill in the relevant art may be used. As depicted in FIG. 23, data storage 2396 contains program instructions 2397 executable by processor 2394 for carrying out various combinations of the various network-entity functions described herein. [u 1 30] in sunie embodiments, the network-entity functions described herein are carried out by a network entity having a structure similar to that of network entity 2390 of FIG. 23. In some embodiments, one or more of such functions are carried out by a set of multiple network entities in combination, where each network entity has a structure similar to that of network entity 2390 of FIG. 23. And certainly other network entities and/or combinations of network entities may be used in various embodiments for carrying out the network- entity functions described herein, as the foregoing list is provided by way of example and not by way of limitation.

[0199] FIG. 24 is a system diagram illustrating the RAN 104 and the CN 106 according to an embodiment. As noted above, the RAN 104 may employ an E-UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also be in communication with the CN 106.

[0200] The RAN 104 may include eNode-Bs 160a, 160b, 160c, though it will be appreciated that the RAN 104 may include any number of eNode-Bs while remaining consistent with an embodiment. The eNode- Bs 160a, 160b, 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the eNode-Bs 160a, 160b, 160c may implement MIMO technology. Thus, the eNode-B 160a, for example, may use multiple antennas to transmit wireless signals to, and/or receive wireless signals from, the WTRU 102a.

[0201] Each of the eNode-Bs 160a, 160b, 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and/or DL, and the like. As shown in FIG. 24, the eNode-Bs 160a, 160b, 160c may communicate with one another over an X2 interface.

[0202] The CN 106 shown in FIG. 24 may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (or PGW) 166. While each of the foregoing elements are depicted as part of the CN 106, it will be appreciated that any of these elements may be owned and/or operated by an entity other than the CN operator.

[0203] The MME 162 may be connected to each of the eNode-Bs 162a, 162b, 162c in the RAN 104 via an S1 interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, bearer activation/deactivation, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, and the like. The MME 162 may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM and/or WCDMA.

[0204] The SGW 164 may be connected to each of the eNode Bs 160a, 160b, 160c in the RAN 104 via the S1 interface. The SGW 164 may generally route and forward user data packets to/from the WTRUs i i d, ne SGW 164 may perform other functions, such as anchoring user planes during inter- eNode B handovers, triggering paging when DL data is available for the WTRUs 102a, 102b, 102c, managing and storing contexts of the WTRUs 102a, 102b, 102c, and the like.

[0205] The SGW 164 may be connected to the PGW 166, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0206] The CN 106 may facilitate communications with other networks. For example, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. For example, the CN 106 may include, or may communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the CN 106 and the PSTN 108. In addition, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to the other networks 112, which may include other wired and/or wireless networks that are owned and/or operated by other service providers.

[0207] Although the WTRU is described in FIGs. 22-25 as a wireless terminal, it is contemplated that in certain representative embodiments that such a terminal may use (e.g., temporarily or permanently) wired communication interfaces with the communication network.

[0208] In representative embodiments, the other network 112 may be a WLAN.

[0209] A WLAN in Infrastructure Basic Service Set (BSS) mode may have an Access Point (AP) for the BSS and one or more stations (STAs) associated with the AP. The AP may have an access or an interface to a Distribution System (DS) or another type of wired/wireless network that carries traffic in to and/or out of the BSS. Traffic to STAs that originates from outside the BSS may arrive through the AP and may be delivered to the STAs. Traffic originating from STAs to destinations outside the BSS may be sent to the AP to be delivered to respective destinations. Traffic between STAs within the BSS may be sent through the AP, for example, where the source STA may send traffic to the AP and the AP may deliver the traffic to the destination STA. The traffic between STAs within a BSS may be considered and/or referred to as peer-to- peer traffic. The peer-to-peer traffic may be sent between (e.g., directly between) the source and destination STAs with a direct link setup (DLS). In certain representative embodiments, the DLS may use an 802.11e DLS or an 802.11z tunneled DLS (TDLS). A WLAN using an Independent BSS (IBSS) mode may not have an AP, and the STAs (e.g., all of the STAs) within or using the IBSS may communicate directly with each other. The IBSS mode of communication may sometimes be referred to herein as an "ad-hoc" mode of communication.

[0210] When using the 802. 1 ac infrastructure mode of operation or a similar mode of operations, the AP may transmit a beacon on a fixed channel, such as a primary channel. The primary channel may be a iiAeu wiuLi i ve.y., L< MHz wide bandwidth) or a dynamically set width via signaling. The primary channel may be the operating channel of the BSS and may be used by the STAs to establish a connection with the AP. In certain representative embodiments, Carrier Sense Multiple Access with Collision Avoidance (CSMA/CA) may be implemented, for example in in 802.11 systems. For CSMA/CA, the STAs (e.g., every STA), including the AP, may sense the primary channel. If the primary channel is sensed/detected and/or determined to be busy by a particular STA, the particular STA may back off. One STA (e.g., only one station) may transmit at any given time in a given BSS.

[0211] High Throughput (HT) STAs may use a 40 MHz wide channel for communication, for example, via a combination of the primary 20 MHz channel with an adjacent or nonadjacent 20 MHz channel to form a 40 MHz wide channel.

[0212] Very High Throughput (VHT) STAs may support 20 MHz, 40 MHz, 80 MHz, and/or 160 MHz wide channels. The 40 MHz, and/or 80 MHz, channels may be formed by combining contiguous 20 MHz channels. A 160 MHz channel may be formed by combining 8 contiguous 20 MHz channels, or by combining two non-contiguous 80 MHz channels, which may be referred to as an 80+80 configuration. For the 80+80 configuration, the data, after channel encoding, may be passed through a segment parser that may divide the data into two streams. Inverse Fast Fourier Transform (IFFT) processing, and time domain processing, may be done on each stream separately. The streams may be mapped on to the two 80 MHz channels, and the data may be transmitted by a transmitting STA. At the receiver of the receiving STA, the above described operation for the 80+80 configuration may be reversed, and the combined data may be sent to the Medium Access Control (MAC).

[0213] Sub 1 GHz modes of operation are supported by 802.11 af and 802.11 ah. The channel operating bandwidths, and carriers, are reduced in 802.11af and 802.11 ah relative to those used in 802.11 η, and 802.11 ac. 802.11 af supports 5 MHz, 10 MHz and 20 MHz bandwidths in the TV White Space (TVWS) spectrum, and 802.11 ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz bandwidths using non-TVWS spectrum. According to a representative embodiment, 802.11 ah may support Meter Type Control/Machine- Type Communications, such as MTC devices in a macro coverage area. MTC devices may have certain capabilities, for example, limited capabilities including support for (e.g., only support for) certain and/or limited bandwidths. The MTC devices may include a battery with a battery life above a threshold (e.g., to maintain a very long battery life).

[0214] WLAN systems, which may support multiple channels, and channel bandwidths, such as 802.11 η, 802.11ac, 802.11 af, and 802.11 ah, include a channel which may be designated as the primary channel. The primary channel may have a bandwidth equal to the largest common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel may be set and/or limited by a STA, from among all STAs in operating in a BSS, which supports the smallest bandwidth operating mode. In the ϋλΰΐ ι ι ΐϋ υι ΟΙ . 1 1 en I, the primary channel may be 1 MHz wide for STAs (e.g., MTC type devices) that support (e.g., only support) a 1 MHz mode, even if the AP, and other STAs in the BSS support 2 MHz, 4 MHz, 8 MHz, 16 MHz, and/or other channel bandwidth operating modes. Carrier sensing and/or Network Allocation Vector (NAV) settings may depend on the status of the primary channel. If the primary channel is busy, for example, due to a STA (which supports only a 1 MHz operating mode), transmitting to the AP, the entire available frequency bands may be considered busy even though a majority of the frequency bands remains idle and may be available.

[0215] In the United States, the available frequency bands, which may be used by 802.11 ah, are from 902 MHz to 928 MHz. In Korea, the available frequency bands are from 917.5 MHz to 923.5 MHz. In Japan, the available frequency bands are from 916.5 MHz to 927.5 MHz. The total bandwidth available for 802.11 ah is 6 MHz to 26 MHz depending on the country code.

[0216] FIG. 25 is a system diagram illustrating the RAN 113 and the CN 115 according to an embodiment. As noted above, the RAN 113 may employ an NR radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 113 may also be in communication with the CN 115.

[0217] The RAN 113 may include gNBs 180a, 180b, 180c, though it will be appreciated that the RAN 113 may include any number of gNBs while remaining consistent with an embodiment. The gNBs 180a, 180b, 180c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the gNBs 180a, 180b, 180c may implement MIMO technology. For example, gNBs 180a, 108b may utilize beamforming to transmit signals to and/or receive signals from the gNBs 180a, 180b, 180c. Thus, the gNB 180a, for example, may use multiple antennas to transmit wireless signals to, and/or receive wireless signals from, the WTRU 102a. In an embodiment, the gNBs 180a, 180b, 180c may implement carrier aggregation technology. For example, the gNB 180a may transmit multiple component carriers to the WTRU 102a (not shown). A subset of these component carriers may be on unlicensed spectrum while the remaining component carriers may be on licensed spectrum. In an embodiment, the gNBs 180a, 180b, 180c may implement Coordinated Multi-Point (CoMP) technology. For example, WTRU 102a may receive coordinated transmissions from gNB 180a and gNB 180b (and/or gNB 180c).

[0218] The WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c using transmissions associated with a scalable numerology. For example, the OFDM symbol spacing and/or OFDM subcarrier spacing may vary for different transmissions, different cells, and/or different portions of the wireless transmission spectrum. The WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c using subframe or transmission time intervals (TTIs) of various or scalable lengths (e.g., containing varying number of OFDM symbols and/or lasting varying lengths of absolute time). ΐϋί ΐ

180a, 180b, 180c may be configured to communicate with the WTRUs 102a, 102b, 102c in a standalone configuration and/or a non-standalone configuration. In the standalone configuration, WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c without also accessing other RANs (e.g., such as eNode-Bs 160a, 160b, 160c). In the standalone configuration, WTRUs 102a, 102b, 102c may utilize one or more of gNBs 180a, 180b, 180c as a mobility anchor point. In the standalone configuration, WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c using signals in an unlicensed band. In a non-standalone configuration WTRUs 102a, 102b, 102c may communicate with/connect to gNBs 180a, 180b, 180c while also communicating with/connecting to another RAN such as eNode-Bs 160a, 160b, 160c. For example, WTRUs 102a, 102b, 102c may implement DC principles to communicate with one or more gNBs 180a, 180b, 180c and one or more eNode-Bs 160a, 160b, 160c substantially simultaneously. In the non-standalone configuration, eNode-Bs 160a, 160b, 160c may serve as a mobility anchor for WTRUs 102a, 102b, 102c and gNBs 180a, 180b, 180c may provide additional coverage and/or throughput for servicing WTRUs 102a, 102b, 102c.

[0220] Each of the gNBs 180a, 180b, 180c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and/or DL, support of network slicing, dual connectivity, interworking between NR and E-UTRA, routing of user plane data towards User Plane Function (UPF) 184a, 184b, routing of control plane information towards Access and Mobility Management Function (AMF) 182a, 182b and the like. As shown in FIG. 25, the gNBs 180a, 180b, 180c may communicate with one another over an Xn interface.

[0221] The CN 115 shown in FIG. 25 may include at least one AMF 182a, 182b, at least one UPF 184a,184b, at least one Session Management Function (SMF) 183a, 183b, and possibly a Data Network (DN) 185a, 185b. While each of the foregoing elements are depicted as part of the CN 115, it will be appreciated that any of these elements may be owned and/or operated by an entity other than the CN operator.

[0222] The AMF 182a, 182b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via an N2 interface and may serve as a control node. For example, the AMF 182a, 182b may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, support for network slicing (e.g., handling of different PDU sessions with different requirements), selecting a particular SMF 183a, 183b, management of the registration area, termination of NAS signaling, mobility management, and the like. Network slicing may be used by the AMF 182a, 182b in order to customize CN support for WTRUs 102a, 102b, 102c based on the types of services being utilized WTRUs 102a, 102b, 102c. For example, different network slices may be established for different use cases such as services relying on ultra-reliable low latency (URLLC) access, services relying on enhanced massive mobile broadband (eMBB) access, services for machine type communication (MTC) access, and/or the like. The AMF 162 may provide a control plane luiiuiiun lui swii iing between the RAN 113 and other RANs (not shown) that employ other radio technologies, such as LTE, LTE-A, LTE-A Pro, and/or non-3GPP access technologies such as WiFi.

[0223] The SMF 183a, 183b may be connected to an AMF 182a, 182b in the CN 115 via an N11 interface. The SMF 183a, 183b may also be connected to a UPF 184a, 184b in the CN 115 via an N4 interface. The SMF 183a, 183b may select and control the UPF 184a, 184b and configure the routing of traffic through the UPF 184a, 184b. The SMF 183a, 183b may perform other functions, such as managing and allocating UE IP address, managing PDU sessions, controlling policy enforcement and QoS, providing downlink data notifications, and the like. A PDU session type may be IP-based, non-IP based, Ethernet- based, and the like.

[0224] The UPF 184a, 184b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via an N3 interface, which may provide the WTRUs 102a, 102b, 102c with access to packet- switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. The UPF 184, 184b may perform other functions, such as routing and forwarding packets, enforcing user plane policies, supporting multi-homed PDU sessions, handling user plane QoS, buffering downlink packets, providing mobility anchoring, and the like.

[0225] The CN 115 may facilitate communications with other networks. For example, the CN 115 may include, or may communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the CN 115 and the PSTN 108. In addition, the CN 115 may provide the WTRUs 102a, 102b, 102c with access to the other networks 112, which may include other wired and/or wireless networks that are owned and/or operated by other service providers. In one embodiment, the WTRUs 102a, 102b, 102c may be connected to a local Data Network (DN) 185a, 185b through the UPF 184a, 184b via the N3 interface to the UPF 184a, 184b and an N6 interface between the UPF 184a, 184b and the DN 185a, 185b.

[0226] In view of FIGs. 22-25, and the corresponding description of FIGs. 22-25, one or more, or all, of the functions described herein with regard to one or more of: WTRU 102a-d, Base Station 114a-b, eNode-B 160a-c, MME 162, SGW 164, PGW 166, gNB 180a-c, AMF 182a-b, UPF 184a-b, SMF 183a-b, DN 185a-b, and/or any other device(s) described herein, may be performed by one or more emulation devices (not shown). The emulation devices may be one or more devices configured to emulate one or more, or all, of the functions described herein. For example, the emulation devices may be used to test other devices and/or to simulate network and/or WTRU functions.

[0227] The emulation devices may be designed to implement one or more tests of other devices in a lab environment and/or in an operator network environment. For example, the one or more emulation devices may perform the one or more, or all, functions while being fully or partially implemented and/or deployed as part of a wired and/or wireless communication network in order to test other devices within the communication i ieiwui Fv. I l ie ui ie ur more emulation devices may perform the one or more, or all, functions while being temporarily implemented/deployed as part of a wired and/or wireless communication network. The emulation device may be directly coupled to another device for purposes of testing and/or may performing testing using over-the-air wireless communications.

[0228] The one or more emulation devices may perform the one or more, including all, functions while not being implemented/deployed as part of a wired and/or wireless communication network. For example, the emulation devices may be utilized in a testing scenario in a testing laboratory and/or a non-deployed (e.g., testing) wired and/or wireless communication network in order to implement testing of one or more components. The one or more emulation devices may be test equipment. Direct RF coupling and/or wireless communications via RF circuitry (e.g., which may include one or more antennas) may be used by the emulation devices to transmit and/or receive data.

[0229] Note that various hardware elements of one or more of the described embodiments are referred to as "modules" that carry out (perform or execute) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by those of skill in the relevant art for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and those instructions may take the form of or include hardware (hardwired) instructions, firmware instructions, software instructions, and/or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM or ROM.

[0230] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element may be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

What is Claimed:

1. A method comprising:

displaying, on a display of a head-mounted display (HMD), a first plurality of tiles of a multi-tile video;

measuring, while the multi-tile video is displayed, an electroencephalography (EEG) signal of the user to generate a measured EEG signal;

determining a predicted head movement of the user based on the measured EEG signal;

retrieving a second plurality of tiles of the multi-tile video based on the predicted head movement of the user; and

rendering one or more of the second plurality of tiles of the multi-tile video.

2. The method of claim 1 , wherein determining the predicted head movement of the user comprises: detecting a zero crossing of an EEG-derived signal derived from the measured EEG signal; and determining the predicted head movement of the user to occur proximate to the detected zero crossing.

3. The method of claim 2, wherein detecting the zero crossing of the EEG-derived signal comprises detecting that the EEG-derived signal is less than a zero crossing threshold.

4. The method of claim 2, further comprising generating a frequency-band power signal from the measured EEG signal, wherein the EEG-derived signal is the frequency-band power signal.

5. The method of claims 1 to 4, further comprising:

performing a training procedure that matches a plurality of head movements of the user to a plurality of EEG signal patterns,

wherein determining the predicted head movement of the user comprises:

identifying, within the measured EEG signal, an EEG signal pattern selected from the plurality of EEG signal patterns; and

determining the predicted head movement to be the head movement of the user matched, by the training procedure from the plurality of EEG signal patterns, to the identified EEG signal pattern selected from the plurality of EEG signal patterns.

6. The method of claim 5, wherein performing the training procedure comprises performing a supervised training procedure that matches induced FOV changes to the plurality of EEG signal patterns. / . I lie nieiiiuu of claim 5, wherein performing the training procedure comprises performing an unsupervised training procedure that matches detected FOV changes to the plurality of EEG signal patterns.

8. The method of claims 1 to 4, further comprising:

selecting a plurality of resolutions based on the predicted head movement of the user, wherein each of the plurality of resolutions corresponds to a respective one of the second plurality of tiles, and

wherein the second plurality of tiles are rendered at the corresponding resolution.

9. The method of claim 8, further comprising:

determining a predicted field of view (FOV) based on the predicted head movement; and matching the predicted FOV to one or more tiles in the multi-tile video,

wherein selecting the plurality of resolutions selects an increased resolution for tiles matched to the predicted FOV.

10. The method of claim 9,

wherein determining the predicted head movement of the user comprises determining a predicted direction, and

wherein determining the predicted FOV comprises:

determining a current FOV to be a first selection of one or more tiles in the multi-tile video; and determining the predicted FOV to be a second selection of one or more tiles in the multi-tile video, wherein the second selection is a shift of the first selection in the predicted direction.

11. The method of claim 9, wherein each of the second plurality of tiles is retrieved at the respective selected resolution.

12. The method of claims 2 to 4, wherein determining the predicted head movement comprises determining a predicted direction.

13. The method of claim 12,

wherein measuring the EEG signal comprises measuring the EEG signal to generate a plurality of measured EEG signals; and

wherein determining the predicted direction comprises:

selecting a selected EEG signal from the plurality of measured EEG signals;

determining the predicted direction as a direction of head movement of the user associated with the selected EEG signal, wiieiein ueieuuiig the zero crossing of the EEG-derived signal comprises detecting the zero crossing of an EEG signal derived from the selected EEG signal.

14. The method of claims 12 or 13, wherein retrieving the second plurality of tiles of the multi-tile video comprises:

identifying the first plurality of tiles as a first selection of tiles of the multi-tile video;

selecting the second plurality of tiles as a second selection of tiles of the multi-tile video, wherein the second selection is a shift of the first selection in the predicted direction.

15. A device, comprising:

a processor; and

a non-transitory computer-readable medium storing instructions that are operative, when executed on the processor, to perform the functions of:

measuring an electroencephalography (EEG) signal of the user while the multi-tile video is displayed;

determining a predicted head movement of the user based on the EEG signal;

rendering one or more of the second plurality of tiles of the multi-tile video.

16. A method comprising:

retrieving, by a head-mounted display (HMD), a plurality of tiles of a multi-tile video based on a viewing direction of a user;

displaying, on a display of the HMD, one or more of the plurality of tiles;

determining whether a head movement of the user is predicted based on the measured EEG signal; and

updating which tiles of the multi-tile video to retrieve if a head movement is predicted.

17. The method of claim 16, wherein determining the predicted head movement of the user comprises: detecting a zero crossing of an EEG-derived signal derived from the measured EEG signal; and determining the predicted head movement of the user to occur proximate to the detected zero crossing. ι ο. I lie I iieu luu of claim 17, further comprising:

reducing noise from the measured EEG signal to generate a noise-reduced EEG signal;

bandpass filtering the noise-reduced EEG signal to generate a bandpass-filtered signal;

squaring the bandpass-filtered signal to generate a current power signal; and

subtracting a previous power signal from the current power signal to generate a difference power signal,

wherein the EEG-derived signal is the difference power signal.

19. The method of claims 16 to 18, further comprising:

wherein determining the predicted head movement of the user comprises:

20. The method of claim 19, wherein performing the training procedure comprises:

displaying, on the display of the HMD, a plurality of tiles of a multi-tile training procedure video that are configured to induce a user head movement and a change to a field of view (FOV) of the user; measuring a training procedure EEG signal while the multi-tile training procedure video is displayed;

extracting an observed EEG signal pattern from the training procedure EEG signal to generate an extracted EEG signal pattern; and

matching the extracted EEG signal pattern to the head movement corresponding to the change to the FOV of the user.

21. The method of claim 19, wherein performing the training procedure comprises:

displaying, on the display of the HMD, a plurality of tiles of a multi-tile training procedure video; measuring a training procedure EEG signal while the multi-tile training procedure video is displayed;

detecting a head movement corresponding to a change to a field of view (FOV) of the user; extracting an observed EEG signal pattern from the training procedure EEG signal to generate an extracted EEG signal pattern; and niai iiiiy me extracted EEG signal pattern to the head movement corresponding to the change to the FOV of the user.

22. A device, comprising:

a processor; and

displaying, on a display of the HMD, one or more of the plurality of retrieved tiles;