WO2018171084A1 - 一种虚拟现实图像发送方法及装置 - Google Patents

一种虚拟现实图像发送方法及装置 Download PDF

Info

Publication number
WO2018171084A1
WO2018171084A1 PCT/CN2017/092285 CN2017092285W WO2018171084A1 WO 2018171084 A1 WO2018171084 A1 WO 2018171084A1 CN 2017092285 W CN2017092285 W CN 2017092285W WO 2018171084 A1 WO2018171084 A1 WO 2018171084A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual reality
motion information
modulation
channel quality
compression ratio
Prior art date
Application number
PCT/CN2017/092285
Other languages
English (en)
French (fr)
Inventor
罗毅
郑方舟
徐彧
邱文轶
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to US16/495,858 priority Critical patent/US11402894B2/en
Priority to AU2017405914A priority patent/AU2017405914B2/en
Priority to CA3057180A priority patent/CA3057180C/en
Priority to KR1020197030873A priority patent/KR102296139B1/ko
Priority to CN201780012081.9A priority patent/CN109074152B/zh
Priority to JP2019552177A priority patent/JP6873268B2/ja
Priority to EP17901675.3A priority patent/EP3591502B1/en
Publication of WO2018171084A1 publication Critical patent/WO2018171084A1/zh

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00Image coding
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/194Transmission of image signals
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/20Input arrangements for video game devices
    • A63F13/21Input arrangements for video game devices characterised by their sensors, purposes or types
    • A63F13/211Input arrangements for video game devices characterised by their sensors, purposes or types using inertial sensors, e.g. accelerometers or gyroscopes
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/20Input arrangements for video game devices
    • A63F13/21Input arrangements for video game devices characterised by their sensors, purposes or types
    • A63F13/212Input arrangements for video game devices characterised by their sensors, purposes or types using sensors worn by the player, e.g. for measuring heart beat or leg activity
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01Head-up displays
    • G02B27/017Head mounted
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017Gesture based interaction, e.g. based on a set of recognized hand gestures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/038Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00Manipulating 3D models or images for computer graphics
    • G06T19/006Mixed reality
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00Manipulating 3D models or images for computer graphics
    • G06T19/20Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/20Scenes; Scene-specific elements in augmented reality scenes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00Arrangements for detecting or preventing errors in the information received
    • H04L1/0001Systems modifying transmission characteristics according to link quality, e.g. power backoff
    • H04L1/0002Systems modifying transmission characteristics according to link quality, e.g. power backoff by adapting the transmission rate
    • H04L1/0003Systems modifying transmission characteristics according to link quality, e.g. power backoff by adapting the transmission rate by switching between different modulation schemes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00Arrangements for detecting or preventing errors in the information received
    • H04L1/0001Systems modifying transmission characteristics according to link quality, e.g. power backoff
    • H04L1/0023Systems modifying transmission characteristics according to link quality, e.g. power backoff characterised by the signalling
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • H04N13/161Encoding, multiplexing or demultiplexing different image signal components
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/115Selection of the code volume for a coding unit prior to coding
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136Incoming video signal characteristics or properties
    • H04N19/137Motion inside a coding unit, e.g. average field, frame or block difference
    • H04N19/139Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/164Feedback from the receiver or from the transmission channel
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/177Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a group of pictures [GOP]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/20Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding
    • H04N19/27Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC]
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/80Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game specially adapted for executing a specific type of game
    • A63F2300/8082Virtual reality
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01Head-up displays
    • G02B27/0101Head-up displays characterised by optical features
    • G02B2027/014Head-up displays characterised by optical features comprising information/image processing systems

Definitions

  • the present application relates to the field of virtual reality technologies, and in particular, to a method and an apparatus for transmitting a virtual reality image.
  • VR technology is an emerging multimedia technology in recent years.
  • VR technology is a computer simulation technology that can create and experience virtual worlds.
  • the user typically wears a virtual reality device to experience a VR scene that enables the user to obtain an interactive immersive experience through an integrated graphics system, optical system, and gesture tracking system.
  • virtual reality devices have limited processing power, and virtual reality devices may not be affordable if some large-scale games or video applications are to be implemented.
  • the industry has proposed to set up a processor for performing a large number of calculations on other devices such as a desktop computer, a mobile phone, or a game console.
  • These devices are called virtual reality hosts, and virtual reality hosts generate virtual reality images, and The virtual reality image is wirelessly transmitted to the virtual reality device, and the virtual reality device is only used to receive the virtual reality image sent by the virtual reality host and display the virtual reality image, thereby helping to reduce the weight and volume of the virtual reality device.
  • the current VR system is generally composed of a virtual reality device and a virtual reality host, and the virtual reality host is used to create a virtual world, and the user experiences the virtual world by wearing a virtual reality device.
  • the number of virtual reality images transmitted in the VR system is very large, that is, the amount of data is very large, and the transmission rate of the wireless channel between the virtual reality host and the virtual reality device is usually required to be 5 Gbps or more.
  • Existing commercial wireless channels have been unable to meet this demand.
  • the industry proposes a method of compressing virtual reality images before transmission to reduce the amount of data transmitted to meet bandwidth requirements.
  • the image compression ratio used for compressing the virtual reality image is determined according to the channel quality of the wireless channel. For example, if the current channel quality is poor, in order to reduce the amount of data transmitted, one can be determined. Larger image compression ratio. However, the image compression ratio determined in this way is not accurate enough. For example, since the switching speed of the virtual reality image displayed by the virtual reality device is fast and slow, for a slow motion or even a static virtual reality image, the user tends to be more critical of the image quality, so in this case, in order to ensure image quality, Virtual reality images cannot be compressed with a large image compression ratio. However, if the channel quality is always poor, the virtual reality host will always compress the virtual reality image with a larger image compression ratio, resulting in poor image quality. It can be seen that the current method of determining the image compression ratio is less accurate.
  • the present application provides a virtual reality image transmitting method and apparatus for improving the accuracy of an image compression ratio determined in a VR system.
  • the present application provides a virtual reality image transmitting method.
  • the method can be implemented by a virtual reality host in a VR system.
  • the method includes: receiving, by the virtual reality host, first motion information sent by the virtual reality device, where the first motion information is used to indicate a motion state of the virtual reality device.
  • the virtual reality host determines an image compression ratio based on the first motion information, and compresses the virtual reality image to be transmitted according to the first image compression.
  • the virtual reality host sends the compressed virtual reality image to the virtual reality device.
  • the virtual reality host determines an image compression ratio according to the motion information of the virtual reality device. For example, when the virtual reality device moves at a slow rate, that is, the virtual reality image displayed by the virtual reality device has a slower switching rate, in order to ensure virtual The quality of the actual image determines a smaller image compression ratio. When the virtual reality device moves at a faster rate, that is, the virtual reality image displayed by the virtual reality device has a faster switching rate, and a larger image compression ratio is determined in order to ensure the continuity of the virtual reality image. In this way, the accuracy of the determined image compression ratio is improved, and the user's needs can be better met.
  • the method before the virtual reality host determines the first image compression ratio according to the first motion information, the method further includes: the virtual reality host detecting the current channel quality of the wireless channel, and according to the channel quality and image of the wireless channel Determining, by the mapping relationship between the compression ratios, an image compression ratio corresponding to the detected channel quality; the virtual reality host determining the first image compression ratio according to the first motion information, including: the virtual reality host determining, according to the first motion information
  • the image compression ratio corresponding to the detected channel quality is a first image compression ratio, or the virtual reality host adjusts an image compression ratio corresponding to the detected channel quality according to the first motion information, and compresses the adjusted image compression ratio as the first image compression ratio.
  • the virtual reality host initially determines an image compression ratio according to the current channel quality of the wireless channel, and then the virtual reality host determines, according to the first motion information, whether the initially determined image compression ratio can be used as the first image compression ratio.
  • the virtual reality image is compressed. For example, when the virtual reality device is in a static state, the virtual reality host may directly determine that the initially determined image compression ratio is a first image compression ratio, and when the virtual reality device is in a motion state, the virtual reality host may adjust the initially determined according to the first motion information.
  • the image compression ratio is used as the first image compression ratio. In this way, it helps to improve the accuracy of the determined image compression ratio.
  • the virtual reality host determines, according to the first motion information, an image compression ratio corresponding to the detected channel quality as the first image compression ratio, including: the virtual reality host according to the first motion information and the history of the virtual reality device The motion information determines that the image compression ratio corresponding to the detected channel quality is the first image compression ratio if it is determined that the virtual reality device is in a stationary state for the first time period in the future.
  • the channel quality of the wireless channel between the virtual reality device and the virtual reality host has a certain correlation with the motion information of the virtual reality device. For example, when the virtual reality device moves at a faster rate, the channel quality is often unstable, and the virtual reality device motion rate is When it is slow, the channel quality is relatively stable. Therefore, in the present application, the virtual reality host can predict the motion state of the virtual reality device in the future first time period, and if it is determined that the virtual reality device is in a static state for the first time in the future, that is, the virtual reality device and the virtual reality The channel quality of the wireless channel between the hosts is relatively stable, so the virtual reality host can directly determine the image compression ratio corresponding to the channel quality as the first image compression ratio. In this way, the accuracy of the determined image compression ratio is improved.
  • the virtual reality host adjusts an image compression ratio corresponding to the detected channel quality according to the first motion information, including: the virtual reality host according to the first motion information and the historical motion information of the virtual reality device, if determined in the future The virtual reality device is in a motion state within the first time period, and the image compression ratio corresponding to the detected channel quality is adjusted according to the first motion information.
  • the virtual reality host can predict the motion state of the virtual reality device in the future for the first time period. If it is determined that the virtual reality device is in the motion state for the first time in the future, then the virtual reality host directly The image compression ratio corresponding to the detected channel quality is inaccurate as the first image compression ratio, so the virtual reality host can adjust the image compression ratio corresponding to the detected channel quality according to the first motion information. In this way, the accuracy of the determined image compression ratio is improved.
  • the virtual reality host adjusts an image compression ratio corresponding to the detected channel quality according to the first motion information, including: if the first motion information is used to indicate that the moving rate of the virtual reality device is less than or equal to the first rate, The virtual reality host reduces an image compression ratio corresponding to the detected channel quality; or if the first motion information is used to indicate that the moving rate of the virtual reality device is greater than the second rate, the virtual reality host increases image compression corresponding to the detected channel quality Ratio; wherein the first rate is less than or equal to the second rate.
  • the virtual reality host may adjust an image compression ratio corresponding to the detected channel quality according to the moving rate of the virtual reality device. For example, when the moving rate of the virtual reality device is large, the image compression ratio corresponding to the detected channel quality is increased to ensure the continuity of the virtual reality image. When the moving rate of the virtual reality device is small, the image compression ratio corresponding to the detected channel quality is reduced to ensure the quality of the virtual reality image. In this way, the accuracy of the determined image compression ratio is improved.
  • the method further includes: determining, by the virtual reality host, the channel quality according to the mapping relationship between the channel quality of the wireless channel and the modulation and coding mode. Corresponding modulation coding mode; the virtual reality host determines, according to the first motion information, a modulation coding mode corresponding to the detected channel quality as a first modulation coding mode, or a channel quality adjusted and detected by the virtual reality host according to the first motion information.
  • the method further includes: a virtual reality host
  • the compressed virtual reality image is modulated and encoded according to the first modulation and coding mode; the virtual reality host sends the compressed virtual reality image to the virtual reality device, and the virtual reality host sends the modulated and encoded compressed virtual reality image to the virtual reality host.
  • the virtual reality host initially determines a modulation and coding mode according to the channel quality of the wireless channel, and then the virtual reality host determines, according to the first motion information, whether the initially determined modulation and coding mode can be compressed as the first modulation and coding mode.
  • the virtual reality image is modulated and encoded. For example, when the virtual reality device is in a static state, the virtual reality host can directly determine that the initially determined modulation and coding mode is the first modulation and coding mode, and when the virtual reality device is in the motion state, the virtual reality host can adjust the initially determined according to the first motion information.
  • the modulation coding method uses the adjusted modulation and coding method as the first modulation and coding method. In this way, the accuracy of the determined modulation coding method is improved.
  • the virtual reality host determines, according to the first motion information, a modulation coding mode corresponding to the detected channel quality as the first modulation and coding mode, including: the virtual motion host according to the first motion information and the historical motion of the virtual reality device The information, if it is determined that the virtual reality device is in a static state for the first time period in the future, determines that the modulation and coding mode corresponding to the detected channel quality is the first modulation and coding mode.
  • the channel quality of the wireless channel between the virtual reality device and the virtual reality host has a certain correlation with the motion information of the virtual reality device. For example, when the virtual reality device moves at a faster rate, the channel quality is often unstable, and the virtual reality device motion rate is When it is slow, the channel quality is relatively stable. Therefore, in the present application, the virtual reality host can predict the motion state of the virtual reality device in the future first time period, and if the virtual reality device is determined to be in a static state for the first time in the future, that is, the virtual reality device and the virtual reality Wireless mail between hosts The channel quality of the channel is relatively stable, so the virtual reality host can directly determine the modulation and coding mode corresponding to the detected channel quality as the first modulation and coding mode. In this way, the accuracy of the determined modulation coding method is improved.
  • the virtual reality host adjusts a modulation and coding manner corresponding to the detected channel quality according to the first motion information, including: the virtual reality host according to the first motion information and the historical motion information of the virtual reality device, if determined in the future The virtual reality device is in a motion state within the first time period, and then the modulation and coding mode corresponding to the detected channel quality is adjusted according to the first motion information.
  • the virtual reality host can predict the motion state of the virtual reality device in the future for the first time period. If it is determined that the virtual reality device is in the motion state for the first time in the future, then the virtual reality host directly The modulation and coding scheme corresponding to the detected channel quality is inaccurate as the first modulation and coding scheme, so the virtual reality host adjusts the modulation and coding scheme corresponding to the detected channel quality according to the first motion information. In this way, the accuracy of the determined modulation coding method is improved.
  • the virtual reality host adjusts the modulation and coding mode corresponding to the detected channel quality according to the first motion information, including: if the first motion information is used to indicate that the moving rate of the virtual reality device is less than or equal to the first rate, The virtual reality host adjusts the modulation and coding mode corresponding to the detected channel quality to the second modulation and coding mode, and the modulation and coding rate of the second modulation and coding mode is greater than the modulation and coding rate of the modulation and coding mode corresponding to the detected channel quality; The motion information is used to indicate that the moving rate of the virtual reality device is greater than the second rate, and the virtual reality host adjusts the modulation and coding mode corresponding to the detected channel quality to the third modulation and coding mode, and the modulation and coding rate of the third modulation and coding mode is less than and detection.
  • the channel coding quality corresponds to a modulation coding rate of the modulation coding mode; wherein the first rate is smaller than the second rate.
  • the virtual reality host may adjust the adjustment coding mode corresponding to the detected channel quality according to the moving rate of the virtual reality device. For example, when the moving rate of the virtual reality device is large, an adjustment coding mode with a small modulation coding rate is determined to ensure the continuity of the virtual reality image. When the moving rate of the virtual reality device is small, an adjustment coding mode with a large modulation coding rate is determined to ensure the quality of the virtual reality image. In this way, the accuracy of the determined modulation coding method is improved.
  • the first motion information is used to indicate current location information of the virtual reality device, and the virtual reality host detects the current channel quality of the wireless channel, including: the virtual reality host according to the location information in the first angle range from the virtual reality Determining, by the device, the first wireless transmission direction in at least one wireless transmission direction distributed within a range of 360 degrees; the first angle is greater than 0 degrees and less than 360 degrees; the virtual reality host detects a current channel of the wireless channel in the first wireless transmission direction quality.
  • the virtual reality host may determine the first wireless transmission direction from the at least one wireless transmission direction according to the location information of the virtual reality device, and the wireless channel in the wireless transmission direction is a virtual reality.
  • the communication channel between the host and the virtual reality device In this way, the virtual reality host does not need to find the wireless transmission direction within a range of 360 degrees, thereby improving the transmission efficiency of the virtual reality image.
  • the first motion information includes one or more of speed information, acceleration information, angular velocity information, and angular acceleration information of the virtual reality device.
  • the first motion information may also be other information, and the above several are only examples, and the present application does not specifically limit this.
  • the present application also provides a virtual reality image transmitting apparatus.
  • the virtual reality image transmitting apparatus has a function of realizing a virtual reality host in the above method design.
  • These functions can be implemented in hardware or in software by executing the corresponding software.
  • the hardware or software includes one or more of the functions corresponding to the above functions unit.
  • the specific structure of the virtual reality image transmitting apparatus may include a receiving unit, a processing unit, and a transmitting unit. These units may perform the respective functions of the methods provided by any of the possible aspects of the first aspect or the first aspect described above.
  • the present application further provides a virtual reality image transmitting apparatus.
  • the virtual reality image transmitting apparatus has a function of realizing a virtual reality host in the above method design. These features can be implemented in hardware.
  • the virtual reality image transmitting apparatus includes a memory for storing computer executable program code, a transceiver, and a processor coupled to the memory and the transceiver.
  • the program code stored in the memory includes instructions that, when executed by the processor, cause the virtual reality image transmitting device to perform the virtual reality host executed by any one of the first aspect or the first aspect of the first aspect of the design Methods.
  • the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores instructions, when executed on a computer, causing the computer to execute the virtual reality image transmitting method according to the first aspect described above .
  • the present application further provides a computer program product comprising instructions for causing a computer to execute the virtual reality image transmitting method of the first aspect described above when it is run on a computer.
  • FIG. 1 is an architectural diagram of a VR system provided by the present application
  • FIG. 2 is a flowchart of a method for transmitting a virtual reality image provided by the present application
  • FIG. 3 is a schematic diagram of beamforming of a virtual reality host and a virtual reality device provided by the present application
  • FIG. 4 is a schematic structural diagram of a virtual reality image transmitting apparatus provided by the present application.
  • FIG. 5 is a schematic structural diagram of another virtual reality image transmitting apparatus provided by the present application.
  • This application describes a VR system including a virtual reality host and a virtual reality device as an example.
  • a VR system including a virtual reality host and a virtual reality device as an example.
  • a virtual reality host is an important device in a VR system for generating a virtual reality image, or acquiring a virtual reality image from a local, other electronic device or the cloud, and transmitting the virtual reality image to the virtual reality device.
  • the virtual reality host can be an electronic device such as a television, a notebook computer, or a smart phone.
  • a virtual reality host can also be referred to as a virtual reality host or the like.
  • a virtual reality image which may be a three-dimensional stereoscopic image, such as a virtual reality host generated from a geometric model or transformed from a solid model in a two-dimensional image.
  • the virtual reality device is a Head Mounted Display (HMD).
  • HMD Head Mounted Display
  • the virtual reality display is disposed directly in front of the user's eyes, and the virtual reality display usually includes two left and right glasses, each of which is separately separated.
  • the left and right eye screens respectively display different images of the left and right eyes, and the human eye obtains the information with the difference and generates a stereoscopic effect in the mind.
  • Beam is a communication resource.
  • the beam can be a wide beam, or a narrow beam, or Other types of beams. Different beams can be considered as different communication resources.
  • the same information or different information can be transmitted through different beams.
  • Beamforming is to form a specific beam in a specified direction when transmitting a wireless signal through multiple antennas, thereby obtaining a large signal gain in a desired direction, thereby reducing power consumption and ensuring signal quality.
  • the transmit beams generated by the transmitting end may be distributed in different directions of 360 degrees in the space, and the receiving beam generated at the receiving end may also be distributed in different directions of 360 degrees in the space.
  • the beam training may be a process of beam alignment between a transmit beam generated by a transmitting end and a receive beam generated by a receiving end, from which an optimal transmit-receive beam is determined, and the transmit-receive beam is between a transmitting end and a receiving end.
  • Wireless channel may be a process of beam alignment between a transmit beam generated by a transmitting end and a receive beam generated by a receiving end, from which an optimal transmit-receive beam is determined, and the transmit-receive beam is between a transmitting end and a receiving end.
  • the "beam” may also be referred to as a "transmission resource” or the like. That is to say, the name of the "beam” is not limited herein, as long as the above concept is expressed.
  • the VR system includes a virtual reality host and a virtual reality device, wherein the virtual reality device takes a head mounted device as an example.
  • the virtual reality host generates a virtual reality image, and then the virtual reality host sends the generated virtual reality image to the HMD.
  • the HMD receives the virtual reality image, the virtual reality image is displayed, so the user wearing the HMD can view the virtual reality image.
  • a virtual reality host generally transmits a virtual reality image to a virtual reality device through a wireless channel, but the amount of data of the virtual reality image that needs to be transmitted is very large.
  • the bandwidth of the wireless channel is 4M, then the transmission of the VR image requires a minimum of 360 hours, that is, 15 days. It can be seen that the virtual reality image needs to be performed due to the limitation of the bandwidth of the wireless channel and the huge amount of data. Compression to reduce the amount of data transmitted and improve transmission efficiency.
  • the virtual reality host needs to determine the image compression ratio.
  • the following describes the current VR system to determine the image compression ratio.
  • a virtual reality host determines an image compression ratio based on the channel quality of the wireless channel.
  • the virtual reality host detects the current channel quality of the wireless channel, for example, the current channel quality is poor, and in order to reduce the amount of data transmitted, a larger image compression ratio is determined.
  • the image compression ratio determined in this way is not accurate enough.
  • the user due to the switching rate of the virtual reality image, that is, the content changes quickly and slowly, for slow motion or even static virtual reality images, the user tends to be more critical about the image quality, so in this case, in order to ensure image quality, it cannot be used.
  • a larger image compression ratio compresses the virtual reality image.
  • the virtual reality host will always compress the virtual reality image with a larger image compression ratio, resulting in poor image quality. It can be seen that the current method of determining the image compression ratio is less accurate.
  • the present application provides a virtual reality image transmitting method. That is, the virtual reality host determines an image compression ratio according to the motion information of the virtual reality device, and by using the method, the determined image pressure is improved. The accuracy of the reduction ratio.
  • FIG. 2 is a flowchart of a method for transmitting a virtual reality image provided by the present application.
  • FIG. 2 can also be understood as the information interaction between the virtual reality host and the HMD in the VR architecture shown in FIG. 1 .
  • schematic diagram. The process of this method is described as follows:
  • the motion information is used to indicate the current state of motion of the virtual reality device.
  • S201 may be completed by a virtual reality device or by a virtual reality host.
  • a motion sensor such as a gyroscope or an accelerometer can be set on the virtual reality device to detect the motion information.
  • the camera or other optical sensor may be set on the virtual reality host, and the positional movement size of the virtual reality device per unit time is detected by the camera or the optical sensor, thereby determining the motion information of the virtual reality device.
  • the motion information is detected by using a virtual reality device as an example, and the content described below will continue to use the virtual reality device to detect motion information as an example.
  • the motion information may include one or more of position information, speed information, acceleration information, angular velocity information, and angular acceleration information of the virtual reality device.
  • the virtual reality device can detect motion information periodically or in real time.
  • the virtual reality device may also detect the motion information only after receiving the request for detecting the motion information sent by the virtual reality host, and does not detect the request without receiving the request.
  • the virtual reality host may send a request for requesting detection of motion information to the virtual reality host, and the virtual reality device detects the motion information after receiving the request.
  • the virtual reality host may send a request for requesting detection of the motion information to the virtual reality device each time the image compression ratio is determined, and the virtual reality device detects the motion after receiving the request. information.
  • the virtual reality device sends the motion information to the virtual reality host, and correspondingly, the virtual reality host receives the motion information.
  • the virtual reality device may detect the motion information periodically or in real time, or may detect the motion information after receiving the request for detecting the motion information sent by the virtual reality host. Then, if the virtual reality device detects the motion information after receiving the request for detecting the motion information sent by the virtual reality host, the virtual reality device may send the motion information to the virtual reality host after detecting the motion information. If the virtual reality device detects the motion information periodically or in real time, the virtual reality device may send the motion information to the virtual reality host every time the motion information is detected, or may receive the motion information sent by the virtual reality host. The newly detected motion information or all motion information detected during the most recent time period is sent to the virtual reality host after the request.
  • the virtual reality device when transmitting the motion information, may simultaneously send the time information of the virtual reality device to generate the motion information and the device identifier of the virtual reality device.
  • the virtual reality device may simultaneously send the time information of the virtual reality device to generate the motion information and the device identifier of the virtual reality device.
  • the virtual reality host usually one virtual reality host needs to communicate with multiple HMDs, that is, the virtual reality host needs to send virtual reality images to multiple HMDs, and the virtual reality host can also receive motion information sent by multiple HMDs.
  • the HMD device ID of the HMD that generates the motion information may be sent together by the HMD to identify the motion information that is sent by the HMD of the plurality of HMDs.
  • the virtual reality host may receive multiple pieces of motion information sent by the same HMD, so each HMD may send the time information of the motion information generated by the HMD together when transmitting its own motion information, so as to be virtualized.
  • the real host recognizes the generation time of each of the plurality of pieces of motion information.
  • the virtual reality host Take the virtual reality host as an example of communication with only one virtual reality device. If the virtual reality device sends the motion information together with the time information for generating the motion information, the virtual reality host may store the motion information and the time information correspondingly after receiving the motion information and the time information. If the virtual reality host receives the plurality of motion information, the generation time of each motion information is different. When the virtual reality host needs to use the motion information, the latest time motion information may be selected from the plurality of motion information, or may be taken within a certain period of time. The average of all the motion information, and the average is used as the required motion information.
  • the three motion information sent by the virtual reality host to the virtual reality device is a rate of 3 m/s, a rate of 4 m/s, and a rate of 3 m/s, and the time information for generating the three pieces of motion information is separately received, respectively. It is 00:10, 00:15, and 00:20.
  • the time information of the latest time can be directly selected, that is, the virtual reality device moves at a speed of 3 m/s at 00:20, and can also take a period of time, for example, 00:15-00:20.
  • the average of the motion information that is, an average of 3 m/s and 4 m/s, that is, 3.5 m/s
  • the virtual reality host determines 3.5 m/s as the required motion information.
  • the period of time mentioned here may be determined according to actual conditions, and the application is not limited.
  • the virtual reality host determines an image compression ratio according to the motion information.
  • a possible implementation manner of S203 is: the virtual reality host firstly determines an image compression ratio according to the channel quality of the wireless channel, and then determines a final image compression ratio according to the motion information and the initially determined image compression ratio. . In this way, considering two different factors, namely channel quality and motion information, the determined image compression ratio is more accurate.
  • the implementation process of this method is specifically described below.
  • the virtual reality host determines the final image compression ratio, which can be done in three steps.
  • the first step is to determine a wireless channel between the virtual reality host and the virtual reality device; in the second step, the image compression ratio is initially determined according to the channel quality of the wireless channel; and the third step is to determine the final based on the motion information and the initially determined image compression ratio.
  • Image compression ratio is to determine a wireless channel between the virtual reality host and the virtual reality device.
  • the current VR system can be roughly divided into two categories: the first type, a VR system that uses beamforming technology for data transmission, for example, a VR system that implements data transmission using a high frequency band (for example, 60 GHz) communication method.
  • the second category VR systems that do not utilize beamforming techniques for data transmission, such as VR systems that implement low-band (eg, 30 GHz) communication techniques for data transmission.
  • the wireless channel between the virtual reality host and the virtual reality device may vary.
  • the virtual reality host and the virtual reality device can perform beam training before establishing communication. Since the virtual reality host in the current VR system is not clear about the location of the virtual reality device. Therefore, beam training between the virtual reality host and the virtual reality device requires a 360-degree search, that is, a transmit beam generated by the virtual reality host (distributed at different angles within 360 degrees) and a receive beam generated by the virtual reality device.
  • the alignment process distributed at different angles within 360 degrees
  • the transmit-receive beam formed after the alignment is the wireless channel between the virtual reality host and the virtual reality device. It can be seen that the beam training between the virtual reality host and the virtual reality device in the current first type of VR system takes a long time, resulting in an excessive image transmission delay.
  • the virtual reality host may determine the optimal wireless transmission direction from the virtual reality device to generate at least one wireless transmission direction distributed within a range of 360 degrees according to the location information of the virtual reality device. It is only necessary to detect the channel quality of the wireless channel in the optimal wireless transmission direction, and it is not necessary to search within a range of 360 degrees, which shortens the delay caused by beam training.
  • the first angle range may be determined according to the moving distance between the location information of the current time and the location information of the previous moment by the virtual reality device, or may be based on the historical location information of the virtual reality device and the current
  • the location information predicts location information of the virtual reality device at the next moment, and is determined according to the predicted moving distance between the location information of the next moment and the location information of the current moment.
  • the transmit beam generated by the virtual reality host is divided into 8 directions within a range of 360 degrees.
  • the virtual reality device moves from the first position to the second position, and the first The moving distance from the position to the second position, that is, the moving range is in two directions of the transmitting beam generated by the virtual reality host, that is, the direction t1-t2.
  • the virtual reality host only needs to search for the receiving beam generated by the virtual reality device in the direction t1-t2.
  • the optimal wireless channel determined by the search result for the virtual reality host is the transmit-receive beam (t2, r8).
  • the VR system since the VR system does not adopt beamforming technology, there is no need to search for the beamforming direction.
  • the wireless channel between the virtual reality host and the virtual reality device is often set in advance. Therefore, the virtual reality host can directly execute the second step of S203.
  • the virtual reality host After determining the wireless channel, the virtual reality host proceeds to the second step, that is, the virtual reality host initially determines the image compression ratio according to the channel quality of the wireless channel.
  • the virtual reality host needs to detect the channel quality of the wireless channel.
  • the specific implementation manner may be multiple.
  • the virtual reality host detects the channel quality in real time or periodically, and the virtual reality host can store the detected channel quality.
  • the virtual reality host can determine the channel quality of the last detection in the stored channel quality. This method is relatively simple, and the virtual reality host does not need to specifically detect the channel quality, which helps improve efficiency; or the virtual reality host can directly The current channel quality is detected so that the determined channel quality is more accurate.
  • the virtual reality host initially determines the image compression ratio according to the mapping relationship between the previously stored channel quality and the image compression ratio.
  • mapping relationships between channel quality and image compression ratio of a wireless channel there are various mapping relationships between channel quality and image compression ratio of a wireless channel, and Table 1 is an example.
  • the channel quality of the wireless channel in Table 1 is exemplified by the transmission rate of the wireless channel.
  • the image compression ratio can be initially determined to be 4:1 according to Table 1.
  • the virtual reality host determines the final image compression ratio based on the motion information and the initially determined image compression ratio.
  • the virtual reality host may first predict the motion state of the virtual reality device in the future first time period according to the received motion information. The reason for the virtual reality host to predict the motion state of the virtual reality device within the first time period in the future will be introduced later. The virtual reality host then determines the final image compression ratio based on the predicted motion state of the virtual reality device within the first time period of the future and the initially determined image compression ratio.
  • the state of motion of a virtual reality device can usually be divided into two situations, static or moving.
  • the implementation process of the third step is divided into two cases: In the first case, how the virtual reality host determines the final image when the virtual reality device is in a stationary state for the first time period in the future. Compression ratio; second, how the virtual reality host determines the final image compression ratio when the virtual reality device is in motion for the first time period in the future.
  • the user's motion state changes at any time, that is, the motion state of the virtual reality device changes at any time.
  • the virtual reality device detects that the current state is static, that is, the rate is 0 m/s
  • the virtual reality device sends the motion information with the rate of 0 m/s to the virtual reality host, and the virtual reality device receives the compression sent by the virtual reality host.
  • This process up to the virtual reality image may take a certain amount of time after which the virtual reality device may have changed from a stationary state to a motion state. Therefore, in order to improve the accuracy of the determined image compression ratio, the virtual reality host can predict the motion state within the first duration of the virtual reality device in the future.
  • the virtual reality host receives the motion information sent by the virtual reality device and the time information of the virtual reality device to generate the motion information at 00:10, specifically, the rate of the virtual reality device is 0 m/s at 00:08. . Then the virtual reality host can only determine that the virtual reality device is still at 00:08. If the virtual reality host compresses the virtual reality image, and the process of transmitting the compressed virtual reality image to the virtual reality device takes 5 seconds, the virtual reality device receives the compressed virtual reality image at 00:15, but the virtual reality at 00:15. The device may have changed from a stationary state to a motion state. Therefore, in order to improve the accuracy of the image compression ratio, the virtual reality host may predict the motion state of the virtual reality device within the first time duration, for example, within 5 seconds, according to the received motion information.
  • the first duration may be determined empirically or may be determined by calculation. If it is determined by calculation, the reason for predicting the motion state of the virtual reality device in the future first time period by the foregoing virtual reality host is that the first time length may be that the virtual reality device starts from the time when the certain motion information is generated to the receiving virtual
  • the duration can be specifically divided into three sums of time.
  • the first duration may be the length of time required for the virtual reality device to send the motion information to the virtual reality host
  • the second duration may be the duration required for the virtual reality host to compress the virtual reality image
  • the third duration may be the virtual reality host. The length of time it takes to send a compressed virtual reality image to a virtual reality device.
  • the first duration may be a time difference between a time when the virtual reality host receives the motion information and a time when the virtual reality device generates the motion information.
  • the second duration may be that the total number of frames of the virtual reality image that needs to be transmitted is multiplied by the length of time required to compress one frame of the virtual reality image.
  • the length of time required to compress one frame of virtual reality image can be obtained through testing.
  • the virtual reality host can record the first time before compressing one frame of the virtual reality image and the second time when the compression is completed, and the difference between the second time and the first time is the length of time required to compress one frame of the virtual reality image. .
  • the third duration may be that the amount of data transmitted at a time is divided by the transmission rate of the wireless channel.
  • the amount of data transmitted at one time can be determined by the total number of frames of the transmitted virtual reality image and the amount of data included in each frame of the virtual reality image. After determining the three durations, the three durations are added to obtain the first duration.
  • the first duration may be other algorithms for the first duration, which is not specifically limited in this application.
  • the virtual reality host may determine, according to the received motion information and the historical motion information of the virtual reality device, whether the virtual reality device is in a stationary state or in a motion state in the future first time period.
  • the virtual reality device sends the motion information together, the time information for generating the motion information is also sent, and the virtual reality host can receive the motion information and generate the time information of the motion information.
  • the motion information and the time information are stored correspondingly, thereby obtaining information that the virtual reality device motion information changes with time, that is, obtaining historical motion information of the virtual reality device.
  • the historical motion information of the virtual reality device stored in the virtual reality host is: at time 00:00, the rate of the virtual reality device is zero.
  • the rate of the virtual reality device is 1 m/s.
  • the virtual reality device rate is 2 m/s.
  • the rate of the virtual reality device is 3 m/s.
  • the rate of the virtual reality device is 2 m/s.
  • the rate of the virtual reality device is 1 m/s.
  • the rate of the virtual reality device is 0 m/s.
  • the rate of the virtual reality device is 0 m/s.
  • the rate of the virtual reality device is 0 m/s.
  • the rate of the virtual reality device is 0 m/s.
  • the rate of the virtual reality device is 0 m/s.
  • the rate of the virtual reality device is 0 m/s and so on.
  • the virtual reality host receives the motion information sent by the virtual reality device and the time information of the motion information at 00:14, specifically, the speed of the virtual reality device is 2 m/ at time 00:12. s. If the first duration is 3s, the virtual reality host according to the historical motion information of the virtual reality device, it can be known that the virtual reality host rate is 3 m/s in the next 3 s, that is, at 00:15, that is, the virtual reality device is determined to be in motion. . For another example, the virtual reality host receives the motion information sent by the virtual reality host and the time information of the motion information at 00:30. Specifically, the virtual reality device has a rate of 0 m/s at 00:28.
  • the virtual reality host according to the historical motion information of the virtual reality device, it can be seen that in the next 3s, that is, at 00:15, the virtual reality host rate is 0 m/s, that is, the virtual reality device is in a static state.
  • the channel quality of the wireless channel between the virtual reality host and the virtual reality device has a certain correlation with the motion state of the virtual reality device. For example, when the virtual reality device moves rapidly, the channel stability is poor, and the channel quality is poor. The change will be relatively large, and when the virtual reality device is stationary, the channel quality is generally stable and the change is not large. Therefore, when the virtual reality host determines that the virtual reality device is in a static state for the first time period in the future, it can be determined that the channel quality is relatively stable, and the virtual reality host can directly determine that the image compression ratio corresponding to the detected channel quality of the wireless channel is The final image compression ratio, that is, the initially determined image compression ratio is directly used as the final image compression ratio.
  • the motion information may be used according to the motion information (the motion information here is used to indicate the current motion state of the virtual reality device, not the aforementioned Historical motion information) adjusts the initially determined image compression ratio. For example, if the motion information is used to indicate that the moving speed of the virtual reality device is slow, the virtual reality host appropriately reduces the image compression ratio based on the initially determined image compression ratio. If the motion information is used to indicate that the moving speed of the virtual reality device is faster, the virtual reality host appropriately increases the image compression ratio based on the initially determined image compression ratio.
  • the moving speed of the virtual reality device is fast, that is, the switching rate of the virtual reality image displayed by the virtual reality device, that is, the content changes rapidly.
  • the switching rate of the virtual reality image displayed by the virtual reality device that is, the content changes rapidly.
  • the content of the virtual reality image displayed by the virtual reality device changes slowly.
  • the user often has high quality requirements for the image. For example, the image clarity, resolution, and the like are required to be high. At this time, the image compression ratio is reduced, and the image quality can be improved, thereby improving the user experience.
  • the virtual reality host may also consider the trend of the channel quality of the wireless channel in the future first time period when adjusting the initially determined image compression ratio according to the motion information. That is, the virtual reality host can adjust the initially determined image compression ratio based on the motion information and the predicted change trend of the channel quality in the first time duration in the future. details as follows:
  • the virtual reality host If the motion information is used to indicate that the moving rate of the virtual reality device is slow, and the channel quality in the first time period is good, the virtual reality host appropriately reduces the image compression ratio based on the initially determined image compression ratio. If the motion information is used to indicate that the moving speed of the virtual reality device is faster, and the channel quality deteriorates in the first time period in the future, the virtual reality host appropriately increases the image compression ratio based on the initially determined image compression ratio.
  • the virtual reality host increases the image compression ratio based on the initially determined image compression ratio, but increasing the image compression ratio will greatly reduce the amount of data transmitted. The transmission requirement can be satisfied without being too good, so if the channel quality is deteriorated, the transmission of the virtual reality image will not be affected.
  • the moving speed of the virtual reality device is slow, that is, the content of the virtual reality image displayed by the virtual reality device changes slowly.
  • the virtual reality host reduces the image compression ratio based on the initially determined image compression ratio, but after reducing the image compression ratio, the amount of data transmitted may be greatly increased, if the channel of the wireless channel Better quality will not affect the transmission of increased data volume. That is, the virtual reality host adjusts the image compression ratio according to the trend of the motion information and the channel quality in the first time period in the future, so that the accuracy of determining the image compression ratio can be further improved.
  • the virtual reality host may choose to adjust the image compression ratio only according to the motion information, or may select to adjust the image compression ratio according to the motion information and the trend of the channel quality of the wireless channel in the future first time period.
  • S204 The virtual reality host compresses the virtual reality image to be sent according to the image compression.
  • the virtual reality host may compress each frame of the virtual reality image according to the determined final image compression ratio, or may compress the multi-frame virtual reality image package together each time.
  • the virtual reality host determines the image compression ratio according to the motion information of the virtual reality device. For example, when the virtual reality device moves at a slow rate, that is, the content of the virtual reality image displayed by the virtual reality device changes slowly, in order to ensure virtual reality. The quality of the image determines a smaller image compression ratio. When the virtual reality device moves at a faster rate, that is, the content of the virtual reality image displayed by the virtual reality device changes rapidly, and a large image compression ratio is determined in order to ensure the consistency of the virtual reality image. In this way, the accuracy of the determined image compression ratio is improved, and the user's needs can be better met.
  • the virtual reality host needs to determine the modulation and coding mode before transmitting the virtual reality image, and then modulates and encodes the virtual reality image according to the modulation and coding mode.
  • the following describes how the current VR system determines the modulation and coding method.
  • the virtual reality host determines the modulation and coding mode according to the channel quality of the wireless channel. For example, if the channel quality of the wireless channel is good, a modulation and coding mode with a fast modulation and coding rate is determined.
  • the modulation coding method determined in this way is not accurate enough. This is because, in fact, each modulation coding mode corresponds to a respective modulation coding rate, and the modulation coding rate is related to the frame error rate generated by the virtual reality image during transmission, for example, when the modulation coding rate is fast, although transmission The efficiency is higher, but the probability of frame error and frame loss is higher. Big.
  • the moving speed of the virtual reality device is fast, the content of the virtual reality image displayed by the virtual reality device changes rapidly.
  • Modulation coding method with a coding rate too fast If the channel quality of the wireless channel is always good, the virtual reality host will always adopt the modulation and coding mode with a faster modulation and coding rate, that is, the probability of frame loss and frame error is large, and once the frame is dropped, the user sees the virtual Realistic images will be incoherent. It can be seen that at present, in the VR system, the modulation and coding mode determined by the virtual reality host according to the channel quality is not accurate enough. The process of determining the modulation coding method in this application is described below.
  • the virtual reality host determines a modulation and coding manner according to the motion information.
  • S204 is not limited as long as it occurs after S202.
  • FIG. 2 is an example in which S205 occurs after S204.
  • a possible implementation manner of S205 is that the virtual reality host firstly determines a modulation and coding mode according to the channel quality of the wireless channel, and then determines the final modulation and coding mode according to the motion information and the initially determined modulation and coding mode. In this way, considering two different factors, namely channel quality and motion information, the determined modulation and coding method will be more accurate. The implementation process of this method is specifically described below.
  • the virtual reality host determines the final modulation and coding scheme, which can be done in two steps.
  • the first step is to determine a wireless channel between the virtual reality host and the virtual reality device; in the second step, the modulation and coding mode is initially determined according to the channel quality of the wireless channel; and the third step is to determine the final according to the motion information and the initially determined modulation and coding mode.
  • Modulation coding method is to determine a wireless channel between the virtual reality host and the virtual reality device.
  • the virtual reality host first needs to determine the mapping relationship between channel quality and channel quality and modulation and coding mode.
  • the virtual reality host can be pre-stored. Therefore, the main requirement of the virtual reality host is the channel quality.
  • the manner in which the virtual reality host determines the channel quality has been previously described.
  • the virtual reality host initially determines the modulation and coding mode according to the mapping relationship between the stored channel quality and the modulation and coding mode.
  • Table 2 is an example.
  • Table 2 is only an example of the mapping relationship between the channel quality and the modulation and coding scheme, and is not a limitation of the mapping relationship between the channel quality and the modulation and coding scheme.
  • the channel quality of the wireless channel in Table 1 is exemplified by the transmission rate of the wireless channel.
  • the mapping relationship between the channel quality of the wireless channel and the modulation and coding mode may be a modulation and coding scheme (MCS) index table in the prior art, or may be a user-defined mapping relationship. .
  • MCS modulation and coding scheme
  • the modulation and coding mode is 16-QAM.
  • the implementation process of the third step is described below, that is, the virtual reality host determines the final modulation and coding mode according to the motion information and the initially determined modulation and coding mode. Similar to S203, the virtual reality host predicts the motion state of the virtual reality device in the future first time period according to the received motion information. The final modulation and coding scheme is then determined based on the motion state and the initially determined modulation and coding scheme. Therefore, the implementation process of the third step is also divided into two cases: In the first case, how the virtual reality host determines the final modulation and coding mode when the virtual reality device is in a stationary state for the first time period in the future.
  • the virtual reality host determines the final modulation and coding mode when the virtual reality device is in motion for the first time period in the future.
  • the reason and implementation manner of the virtual reality host predicting the motion state of the virtual reality device in the first time period in the future has been introduced, and will not be repeatedly described herein. Therefore, the following two cases of the implementation process of the second step are directly introduced.
  • the channel quality of the wireless channel between the virtual reality host and the virtual reality device has a certain correlation with the motion state of the virtual reality device, and the virtual reality device determines that the virtual reality device is in the first time period in the future. In the quiescent state, it can be determined that the channel quality is relatively stable, and the virtual reality host can directly determine the modulation and coding mode corresponding to the channel quality of the detected wireless channel as the final modulation and coding mode.
  • the second case is how the virtual reality host determines the final modulation and coding mode when the virtual reality device is in motion for the first time in the future.
  • the initially determined modulation and coding mode may be adjusted according to the motion information. For example, if the motion information is used to indicate that the moving rate of the virtual reality device is slow, the virtual reality host adjusts the initially determined modulation and coding mode to a first modulation and coding mode, and the modulation and coding rate of the first modulation code is greater than or equal to a preliminary determination. The modulation coding rate of the modulation coding mode.
  • the virtual reality host modulates the initially determined modulation and coding mode into a second modulation and coding mode, where the modulation coding rate of the second modulation and coding mode is smaller than the initially determined modulation code.
  • the modulation coding rate of the mode is smaller than the initially determined modulation code.
  • the virtual reality host can determine the modulation and coding mode with a lower modulation and coding rate.
  • the moving rate of the virtual reality device is slow, that is, the content of the virtual reality image displayed by the virtual reality device changes slowly, that is, the content of the virtual reality image transmitted each time changes less, and the number of frames of the virtual reality image that needs to be transmitted each time is smaller.
  • a modulation and coding method that does not require a modulation coding rate to be too low at this time can also be implemented, so the virtual reality host can determine a modulation coding mode with a higher modulation coding rate, thereby improving transmission efficiency.
  • the moving rate of the virtual reality device is slow, since the wireless channel changes slowly, the detected channel quality is relatively accurate, so the virtual reality host can directly determine the modulation and coding mode corresponding to the detected channel quality as the final modulation. Encoding.
  • the virtual reality host modulates and encodes the compressed virtual reality image according to a modulation and coding manner.
  • the virtual reality host may modulate and encode each frame of the compressed virtual reality image according to the final modulation and coding manner. If the virtual reality host is packaged and compressed with multi-frame virtual reality images, then the virtual reality host can be based on the most The final modulation coding method modulates and encodes the virtual reality image compression package together.
  • the virtual reality host sends the modulated and encoded compressed virtual reality image to the virtual reality device, and correspondingly, the virtual reality device receives the modulated and encoded virtual reality image.
  • the virtual reality device After receiving the modulated encoded virtual reality image, the virtual reality device demodulates and decodes the virtual reality image to obtain a compressed virtual reality image, and then decompresses the compressed virtual reality image, and finally displays the virtual reality image.
  • the virtual reality host determines the modulation and coding mode according to the motion information of the virtual reality device. For example, when the moving speed of the virtual reality device is fast, that is, the switching speed of the virtual reality image displayed by the virtual reality device is fast, in order to ensure the consistency and fluency of the virtual reality image, the probability of occurrence of frame error and frame loss is reduced, and the probability of avoiding frame loss is avoided.
  • the image is retransmitted, so the virtual reality host can determine the modulation coding mode with a lower modulation coding rate.
  • the virtual reality host may determine a modulation coding mode with a higher modulation coding rate or directly determine a modulation and coding mode according to the channel quality of the wireless channel. , thereby improving transmission efficiency. In this way, the accuracy of the determined modulation and coding method is improved, and the user's needs can be better met.
  • the virtual reality host determines an image compression ratio and a modulation and coding manner according to motion information of the virtual reality device. For example, when the virtual reality device moves at a slow rate, that is, the virtual reality image content displayed by the virtual reality device changes slowly, a smaller image compression ratio is determined, and the modulation and coding mode can be directly determined according to the current channel quality. When the virtual reality device moves at a faster rate, that is, the virtual reality image displayed by the virtual reality device changes rapidly, and for the flash lens, the image quality of the human eye is reduced, so in order to ensure the continuity of the virtual reality image, A larger image compression ratio can be determined.
  • the virtual reality host determines the modulation coding mode with a lower modulation coding rate, which double ensures the continuity of the virtual reality image. Therefore, in the method for transmitting a virtual reality image provided by the present application, the virtual reality host can determine the image compression ratio and the modulation and coding mode according to the motion information of the virtual reality device, thereby ensuring the quality of the virtual reality image, improving the transmission efficiency, and being better. To meet the needs of users.
  • FIG. 4 shows a schematic structural diagram of a virtual reality image transmitting apparatus 400.
  • the virtual reality image transmitting apparatus 400 can implement the functions of the virtual reality host referred to above.
  • the virtual reality image transmitting apparatus 400 may include a receiving unit 401, a processing unit 402, and a transmitting unit 403.
  • the receiving unit 401 can be used to execute S202 in the embodiment shown in FIG. 2, and/or other processes for supporting the techniques described herein.
  • Processing unit 402 may be used to perform S203-S206 in the embodiment shown in FIG. 2, and/or other processes for supporting the techniques described herein.
  • Transmitting unit 403 can be used to perform S207 in the embodiment shown in FIG. 2, and/or other processes for supporting the techniques described herein. All the related content of the steps involved in the foregoing method embodiments may be referred to the functional descriptions of the corresponding functional modules, and details are not described herein again.
  • the above-described virtual reality image transmitting apparatus is displayed in the form of a functional unit.
  • the term "unit” as used herein, without limitation, may refer to an application-specific integrated circuit (ASIC), electronic circuit, (shared, dedicated, or group) that executes one or more software or firmware programs.
  • a processor and memory, combinational logic, and/or other suitable components that provide the described functionality.
  • the virtual reality image transmitting apparatus 400 can also be implemented by the structure as shown in FIG.
  • the virtual reality image transmitting apparatus 500 may include: a memory 501, a processor 502, and a connection. Receiver 503, transmitter 504 and bus 505.
  • the memory 501, the processor 502, the receiver 503, and the transmitter 504 can be connected by a bus 505.
  • the receiver 503 is configured to receive first motion information that is sent by the virtual reality device, where the first motion information is used to indicate a current motion state of the virtual reality device, and the processor 502 is configured to determine a first image compression ratio according to the first motion information. And compressing the virtual reality image to be sent according to the first image compression; the transmitter 504 is configured to send the compressed virtual reality image to the virtual reality device.
  • the operations performed by the processor 502, the receiver 503, and the transmitter 504 reference may be made to the corresponding steps performed by the virtual reality host in the embodiment of FIG. 2, and details are not described herein again.
  • the memory 501 is configured to store a computer execution instruction.
  • the processor 502 executes a computer execution instruction stored in the memory 501 to implement the foregoing functions, thereby implementing the virtuality provided by the embodiment shown in FIG. 2.
  • Realistic image transmission method For a specific image capturing method, reference may be made to the related descriptions in the above and the drawings, and details are not described herein again.
  • the processor 502 may be a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processing unit. (central processor unit, CPU), network processor (NP), digital signal processor (DSP), microcontroller (micro controller unit (MCU), can also use programmable controller (programmable Logic device, PLD) or other integrated chip.
  • the bus 505 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus.
  • PCI peripheral component interconnect
  • EISA extended industry standard architecture
  • the bus 505 can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is shown in Figure 5, but it does not mean that there is only one bus or one type of bus.
  • the invention may be implemented in whole or in part by software, hardware, firmware or any combination thereof.
  • software it may be implemented in whole or in part in the form of a computer program product.
  • the computer program product includes one or more computer instructions.
  • the computer program instructions When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with the present application are generated in whole or in part.
  • the computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
  • the computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another readable storage medium, for example, the computer instructions can be passed from a website site, computer, server or data center Wired (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.) to another website site, computer, server, or data center.
  • the computer readable storage medium can be any available media that can be accessed by a computer or a data storage device such as a server, data center, or the like that includes one or more available media.
  • the usable medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)) or the like.
  • a magnetic medium eg, a floppy disk, a hard disk, a magnetic tape
  • an optical medium eg, a DVD
  • a semiconductor medium eg, a solid state disk (SSD)

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • Optics & Photonics (AREA)
  • Computer Graphics (AREA)
  • Quality & Reliability (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Architecture (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Cardiology (AREA)
  • General Health & Medical Sciences (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • User Interface Of Digital Computer (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
  • Detection And Prevention Of Errors In Transmission (AREA)
  • Mobile Radio Communication Systems (AREA)

Abstract

本申请公开了一种虚拟现实图像发送方法及装置,用于提高在发送虚拟现实图像时确定的图像压缩比的准确性。该方法包括:虚拟现实主机接收虚拟现实设备发送的第一运动信息;该第一运动信息用于指示虚拟现实设备的当前的运动状态;虚拟现实主机根据运动信息确定第一图像压缩比;虚拟现实主机根据该第一图像压缩比对待发送的虚拟现实图像进行压缩;虚拟现实主机将压缩的虚拟现实图像发送给虚拟现实设备。

Description

一种虚拟现实图像发送方法及装置
本申请要求于2017年3月22日提交中国专利局、申请号201710175242.8、申请名称为“应用于无线虚拟现实中的图像压缩与传输的方法和设备”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
技术领域
本申请涉及虚拟现实技术领域,尤其涉及一种虚拟现实图像发送方法及装置。
背景技术
虚拟现实(Virtual Reality,VR)技术是近年来新兴的多媒体技术。VR技术是一种可以创建和体验虚拟世界的计算机仿真技术。使用者通常佩戴虚拟现实设备体验VR场景,该虚拟现实设备通过集成的图形系统、光学系统和姿态追踪系统,使用者能够获得一种可交互的沉浸式体验。
目前,虚拟现实设备的处理能力有限,如果要实现一些大型游戏或视频类应用的话,虚拟现实设备就可能无法负担。为此,目前业界提出将用来进行大量计算的处理器设置在台式机、手机、或游戏控制台等其它设备上,这些设备被称为虚拟现实主机,虚拟现实主机生成虚拟现实图像,并将虚拟现实图像通过无线方式发送给虚拟现实设备,虚拟现实设备只用于接收虚拟现实主机发送的虚拟现实图像并显示该虚拟现实图像即可,从而有助于减小虚拟现实设备的重量和体积,给用户带来更好的体验。因此,目前的VR系统一般都是由虚拟现实设备和虚拟现实主机构成,虚拟现实主机用来创建虚拟世界,而用户通过佩戴虚拟现实设备体验该虚拟世界。
为了呈现更为丰富的虚拟世界,VR系统中传输的虚拟现实图像的数量很多,即数据量非常大,通常要求虚拟现实主机和虚拟现实设备之间的无线信道的传输速率在5Gbps甚至以上,而现有的已商用的无线信道已无法满足这一需求。为了解决这一问题,业界提出了在传输前对虚拟现实图像进行压缩的方法来降低传输的数据量,以满足带宽需求。
现有技术中,对虚拟现实图像进行压缩所采用的图像压缩比是根据无线信道的信道质量来确定的,例如,当前的信道质量较差,则为了减小传输的数据量,就可以确定一个较大的图像压缩比。但通过这种方式确定的图像压缩比是不够准确的。例如,由于虚拟现实设备显示的虚拟现实图像的切换速率时快时慢,对于慢镜头甚至静态的虚拟现实图像来说,用户往往对图像质量比较挑剔,所以这种情况下,为了保证图像质量,不能用较大的图像压缩比对虚拟现实图像进行压缩。但是如果信道质量始终较差,那么虚拟现实主机将一直采用较大的图像压缩比对虚拟现实图像进行压缩,最终导致图像质量较差。可见,目前的确定图像压缩比的方式准确性较低。
发明内容
本申请提供一种虚拟现实图像发送方法及装置,用以提高VR系统中确定的图像压缩比的准确性。
第一方面,本申请提供一种虚拟现实图像发送方法。该方法可通过VR系统中的虚拟现实主机实现。该方法包括:虚拟现实主机接收虚拟现实设备发送的第一运动信息,该第一运动信息用于指示虚拟现实设备的运动状态。虚拟现实主机根据该第一运动信息确定图像压缩比,并且根据该第一图像压缩比对待发送的虚拟现实图像进行压缩。虚拟现实主机将压缩后的虚拟现实图像发送给虚拟现实设备。
在本申请中,虚拟现实主机根据虚拟现实设备的运动信息确定图像压缩比,例如,当虚拟现实设备移动速率较慢时,即虚拟现实设备显示的虚拟现实图像的切换速率较慢,为了保证虚拟现实图像的质量,确定一个较小的图像压缩比。当虚拟现实设备运动速率较快时,即虚拟现实设备显示的虚拟现实图像的切换速率较快,为了保证虚拟现实图像的连贯性,确定一个较大的图像压缩比。通过这种方式,提高了确定的图像压缩比的准确性,能够更好地满足用户的需求。
在一个可能的设计中,在虚拟现实主机根据第一运动信息确定第一图像压缩比之前,该方法还包括:虚拟现实主机检测无线信道当前的信道质量,并根据该无线信道的信道质量与图像压缩比之间的映射关系,确定与检测的信道质量对应的图像压缩比;虚拟现实主机根据所述第一运动信息确定第一图像压缩比,包括:虚拟现实主机根据第一运动信息,确定与检测的信道质量对应的图像压缩比为第一图像压缩比,或虚拟现实主机根据第一运动信息调整与检测的信道质量对应的图像压缩比,并将调整后的图像压缩比作为第一图像压缩比。
在本申请中,虚拟现实主机根据无线信道当前的信道质量初步确定一个图像压缩比,然后虚拟现实主机根据第一运动信息来决定该初步确定的图像压缩比是否可作为第一图像压缩比来对虚拟现实图像进行压缩。例如虚拟现实设备处于静止状态时,虚拟现实主机可以直接确定该初步确定的图像压缩比为第一图像压缩比,虚拟现实设备处于运动状态时,虚拟现实主机可以根据第一运动信息调整初步确定的图像压缩比,将调整后的图像压缩比作为第一图像压缩比。通过这种方式,有助于提高确定的图像压缩比的准确性。
在一个可能的设计中,虚拟现实主机根据第一运动信息,确定与检测的信道质量对应的图像压缩比为第一图像压缩比,包括:虚拟现实主机根据第一运动信息和虚拟现实设备的历史运动信息,若确定在未来的第一时长内虚拟现实设备处于静止状态,则确定与检测的信道质量对应的图像压缩比为第一图像压缩比。
虚拟现实设备与虚拟现实主机之间的无线信道的信道质量与虚拟现实设备的运动信息具有一定的相关性,例如虚拟现实设备运动速率较快时,信道质量往往不稳定,而虚拟现实设备运动速率较慢时,信道质量较为稳定。因此,在本申请中,虚拟现实主机可以对未来的第一时长内虚拟现实设备的运动状态进行预测,如果确定虚拟现实设备在未来的第一时长内处于静止状态,即虚拟现实设备与虚拟现实主机之间的无线信道的信道质量相对稳定,所以虚拟现实主机可以直接确定与信道质量对应的图像压缩比为第一图像压缩比。通过这种方式,提高了确定的图像压缩比的准确性。
在一个可能的设计中,虚拟现实主机根据第一运动信息调整与检测的信道质量对应的图像压缩比,包括:虚拟现实主机根据第一运动信息和虚拟现实设备的历史运动信息,若确定在未来的第一时长内虚拟现实设备处于运动状态,则根据第一运动信息调整与检测的信道质量对应的图像压缩比。
在本申请中,虚拟现实主机可以对未来的第一时长内虚拟现实设备的运动状态进行预测,若确定虚拟现实设备在未来的第一时长内处于运动状态,这时,虚拟现实主机如果直接将与检测的信道质量对应的图像压缩比作为第一图像压缩比是不准确的,所以虚拟现实主机可以根据第一运动信息调整与检测的信道质量对应的图像压缩比。通过这种方式,提高了确定的图像压缩比的准确性。
在一个可能的设计中,虚拟现实主机根据第一运动信息调整与检测的信道质量对应的图像压缩比,包括:若第一运动信息用于指示虚拟现实设备的移动速率小于或等于第一速率,虚拟现实主机减小与检测的信道质量对应的图像压缩比;或若第一运动信息用于指示虚拟现实设备的移动速率大于第二速率,虚拟现实主机增大与检测的信道质量对应的图像压缩比;其中,第一速率小于或等于第二速率。
在本申请中,虚拟现实主机可以根据虚拟现实设备的移动速率来调整与检测到的信道质量对应的图像压缩比。例如虚拟现实设备移动速率较大时,增大与检测的信道质量对应的图像压缩比,以保证虚拟现实图像的连贯性。虚拟现实设备移动速率较小时,减小与检测的信道质量对应的图像压缩比,以保证虚拟现实图像的质量。通过这种方式,提高了确定的图像压缩比的准确性。
在一个可能的设计中,在虚拟现实主机检测无线信道当前的信道质量之后,该方法还包括:虚拟现实主机根据无线信道的信道质量与调制编码方式之间的映射关系,确定与检测的信道质量对应的调制编码方式;虚拟现实主机根据第一运动信息确定与检测的信道质量对应的调制编码方式为第一调制编码方式,或,虚拟现实主机根据所述第一运动信息调整与检测的信道质量对应的调制编码方式,并将调整后的调制编码方式作为第一调制编码方式;在虚拟现实主机将压缩的所述虚拟现实图像发送给所述虚拟现实设备之前,该方法还包括:虚拟现实主机根据第一调制编码方式对压缩的虚拟现实图像进行调制编码;虚拟现实主机将压缩后的虚拟现实图像发送给虚拟现实设备,包括:虚拟现实主机将调制编码后的压缩的虚拟现实图像发送给虚拟现实设备。
在本申请中,虚拟现实主机根据无线信道的信道质量初步确定一个调制编码方式,然后虚拟现实主机根据第一运动信息决定该初步确定的调制编码方式是否可以作为第一调制编码方式来对压缩的虚拟现实图像进行调制编码。例如虚拟现实设备处于静止状态时,虚拟现实主机可以直接确定该初步确定的调制编码方式为第一调制编码方式,虚拟现实设备处于运动状态时,虚拟现实主机可以根据第一运动信息调整初步确定的调制编码方式,将调整后的调制编码方式作为第一调制编码方式。通过这种方式,提高确定的调制编码方式的准确性。
在一个可能的设计中,虚拟现实主机根据第一运动信息确定与检测的信道质量对应的调制编码方式为第一调制编码方式,包括:虚拟现实主机根据第一运动信息和虚拟现实设备的历史运动信息,若确定在未来的第一时长内虚拟现实设备处于静止状态,则确定与检测的信道质量对应的调制编码方式为第一调制编码方式。
虚拟现实设备与虚拟现实主机之间的无线信道的信道质量与虚拟现实设备的运动信息具有一定的相关性,例如虚拟现实设备运动速率较快时,信道质量往往不稳定,而虚拟现实设备运动速率较慢时,信道质量较为稳定。因此,在本申请中,虚拟现实主机可以对未来的第一时长内虚拟现实设备的运动状态进行预测,若确定虚拟现实设备在未来的第一时长内处于静止状态,即虚拟现实设备与虚拟现实主机之间的无线信 道的信道质量相对稳定,所以虚拟现实主机可以直接确定与检测的信道质量对应的调制编码方式为第一调制编码方式。通过这种方式,提高确定的调制编码方式的准确性。
在一个可能的设计中,虚拟现实主机根据第一运动信息调整与检测的信道质量对应的调制编码方式,包括:虚拟现实主机根据第一运动信息和虚拟现实设备的历史运动信息,若确定在未来的第一时长内虚拟现实设备处于运动状态,则根据第一运动信息调整与检测的信道质量对应的调制编码方式。
在本申请中,虚拟现实主机可以对未来的第一时长内虚拟现实设备的运动状态进行预测,若确定虚拟现实设备在未来的第一时长内处于运动状态,这时,虚拟现实主机如果直接将与检测的信道质量对应的调制编码方式作为第一调制编码方式是不准确的,所以虚拟现实主机根据第一运动信息调整与检测的信道质量对应的调制编码方式。通过这种方式,提高确定的调制编码方式的准确性。
在一个可能的设计中,虚拟现实主机根据第一运动信息调整与检测的信道质量对应的调制编码方式,包括:若第一运动信息用于指示虚拟现实设备的移动速率小于或等于第一速率,虚拟现实主机将与检测的信道质量对应的调制编码方式调整为第二调制编码方式,第二调制编码方式的调制编码速率大于与检测的信道质量对应的调制编码方式的调制编码速率;若第一运动信息用于指示虚拟现实设备的移动速率大于第二速率,虚拟现实主机将与检测的信道质量对应的调制编码方式调整为第三调制编码方式,第三调制编码方式的调制编码速率小于与检测的信道质量对应的调制编码方式的调制编码速率;其中,第一速率小于第二速率。
在本申请中,虚拟现实主机可以根据虚拟现实设备的移动速率来调整与检测到的信道质量对应的调整编码方式。例如虚拟现实设备移动速率较大时,确定一个调制编码速率较小的调整编码方式,以保证虚拟现实图像的连贯性。虚拟现实设备移动速率较小时,确定一个调制编码速率较大的调整编码方式,以保证虚拟现实图像的质量。通过这种方式,提高确定的调制编码方式的准确性。
在一个可能的设计中,第一运动信息用于指示虚拟现实设备的当前位置信息,虚拟现实主机检测无线信道当前的信道质量,包括:虚拟现实主机根据位置信息在第一角度范围内从虚拟现实设备产生的分布于360度范围内的至少一个无线传输方向中确定第一无线传输方向;第一角度大于0度小于360度;虚拟现实主机检测第一无线传输方向上的无线信道的当前的信道质量。
在本申请中,虚拟现实主机可以根据虚拟现实设备的位置信息在一个较小的角度范围内从至少一个无线传输方向中确定第一无线传输方向,该无线传输方向上的无线信道便是虚拟现实主机和虚拟现实设备之间的通信信道。通过这种方式,虚拟现实主机不需要在360度的范围内寻找无线传输方向,提高了虚拟现实图像的传输效率。
在一个可能的设计中,第一运动信息包括虚拟现实设备的速度信息、加速度信息、角速度信息以及角加速度信息中的一种或多种。
在本申请中,第一运动信息还可能是其它信息,以上几种只是举例,本申请对此不作具体的限定。
第二方面,本申请还提供一种虚拟现实图像发送装置。该虚拟现实图像发送装置具有实现上述方法设计中虚拟现实主机的功能。这些功能可以通过硬件实现,也可以通过硬件执行相应的软件实现。所述硬件或软件包括一个或多个与上述功能相对应的 单元。
在一个可能的设计中,虚拟现实图像发送装置的具体结构可包括接收单元、处理单元、发送单元。这些单元可执行上述第一方面或第一方面的任意一种可能的设计所提供的方法中的相应功能。
第三方面,本申请还提供一种虚拟现实图像发送装置。该虚拟现实图像发送装置具有实现上述方法设计中虚拟现实主机的功能。这些功能可以通过硬件实现。该虚拟现实图像发送装置包括:存储器,用于存储计算机可执行程序代码;收发器,以及处理器,处理器与存储器、收发器耦合。其中存储器所存储的程序代码包括指令,当处理器执行所述指令时,所述指令使虚拟现实图像发送装置执行上述第一方面或第一方面的任意一种可能的设计中虚拟现实主机所执行的方法。
第四方面,本申请还提供一种计算机可读存储介质,该计算机可读存储介质中存储有指令,当其在计算机上运行时,使得计算机执行上述第一方面所述的虚拟现实图像发送方法。
第五方面,本申请还提供一种包含指令的计算机程序产品,当其在计算机上运行时,使得计算机执行上述第一方面所述的虚拟现实图像发送方法。
附图说明
图1为本申请提供的一种VR系统的架构图;
图2为本申请提供的一种虚拟现实图像发送方法的流程图;
图3为本申请提供的一种虚拟现实主机和虚拟现实设备的波束成形示意图;
图4为本申请提供的一种虚拟现实图像发送装置的结构示意图;
图5为本申请提供的另一种虚拟现实图像发送装置的结构示意图。
具体实施方式
下面将结合本申请中的附图,对本申请提供的技术方案进行清楚、完整地描述。
本申请以VR系统包括虚拟现实主机和虚拟现实设备为例进行说明。以下,对本申请中的部分用语进行解释说明,以便于本领域技术人员理解。
(1)虚拟现实主机,是VR系统中重要的设备,用来生成虚拟现实图像,或者是从本地、其它电子设备或云端获取虚拟现实图像,并将虚拟现实图像发送于虚拟现实设备。虚拟现实主机可以是电视机、笔记本电脑、智能手机等电子设备。虚拟现实主机也可以被称为虚拟现实主机等。
(2)虚拟现实图像,可以是三维立体图像,例如虚拟现实主机根据几何模型生成,或者由二维图像中的实体模型转化而成。
(3)虚拟现实设备,包括虚拟现实显示器。例如虚拟现实设备为头戴式设备(Head Mounted Display,HMD),用户佩戴该头戴式设备时,将虚拟现实显示器设置于用户眼睛正前方,虚拟现实显示器通常包括左右两个眼镜,各自独立分开,使得人的左右眼获取信息有差异,例如左右眼屏幕分别显示左右眼的不同的图像,人眼获取这种带有差异的信息后在脑海中产生立体感。
(4)波束(beam),是一种通信资源。波束可以是宽波束,或者窄波束,或者 其他类型波束。不同的波束可以认为是不同的通信资源。通过不同的波束可以发送相同的信息或者不同的信息。波束成形技术(Beamforming)是将无线信号通过多天线发送时在指定的方向上形成特定的波束,从而在期望方向上获得较大的信号增益,从而降低了功耗、保证了信号质量。通常,发射端产生的发射波束可以是分布于空间360度不同方向,接收端产生接收波束也可以分布于空间360度不同方向上。波束训练可以是发射端产生的发射波束和接收端产生的接收波束之间的波束对准的过程,从中确定最佳的发射-接收波束,该发射-接收波束即为发射端和接收端之间的无线通道。
本申请中,也可以将“波束”称为“传输资源”等。也就是说,本文对于“波束”的名称不作限制,只要表达的是如上的概念即可。
(5)另外,本文中术语“和/或”,仅仅是一种描述关联对象的关联关系,表示可以存在三种关系,例如,A和/或B,可以表示:单独存在A,同时存在A和B,单独存在B这三种情况。另外,本文中字符“/”,如无特殊说明,一般表示前后关联对象是一种“或”的关系。且在本申请的描述中,“第一”、“第二”等词汇,仅用于区分描述的目的,而不能理解为指示或暗示相对重要性,也不能理解为指示或暗示顺序。
为更好地介绍本申请中的虚拟现实图像的发送方法,下面首先介绍本申请的一种应用场景,即VR系统的系统架构,请参见图1。如图1所示,VR系统包括虚拟现实主机和虚拟现实设备,其中虚拟现实设备以头戴式设备为例。虚拟现实主机生成虚拟现实图像,之后虚拟现实主机将生成的虚拟现实图像发送给HMD。HMD接收虚拟现实图像后,显示该虚拟现实图像,因此佩戴了该HMD的用户便可观看该虚拟现实图像。
目前,虚拟现实主机一般是通过无线信道将虚拟现实图像传输给虚拟现实设备,但是需要传输的虚拟现实图像的数据量非常庞大。例如一段VR影像:画面大小为1000×1000pixel,24位真彩色,每秒30帧,时长2小时。若不进行任何压缩,存储这段视频需要1000*1000*24*30*60*120=5.184×1012bit≈648GB的空间。例如无线信道的带宽为4M,那么传输完该VR影像,最少需要360小时,即15天的时间,可见,由于受到无线信道的带宽的限制和巨大的数据量的影响,需要将虚拟现实图像进行压缩,以减小传输的数据量,提高传输效率。
若要对虚拟现实图像进行压缩,虚拟现实主机就需确定图像压缩比。下面介绍目前的VR系统确定图像压缩比的方式。
目前,虚拟现实主机是根据无线信道的信道质量来确定图像压缩比。虚拟现实主机检测无线信道当前的信道质量,例如当前的信道质量较差,为了减小传输的数据量,便确定一个较大的图像压缩比。
然而,通过这种方式确定的图像压缩比是不够准确的。例如,由于虚拟现实图像的切换速率即内容变化时快时慢,对于慢镜头甚至静态的虚拟现实图像来说,用户往往对图像质量比较挑剔,所以这种情况下,为了保证图像质量,不能用较大的图像压缩比对虚拟现实图像进行压缩。但是如果信道质量始终较差,那么虚拟现实主机将一直采用较大的图像压缩比对虚拟现实图像进行压缩,最终导致图像质量较差。可见,目前的确定图像压缩比的方式准确性较低。
为了解决该技术问题,本申请提供一种虚拟现实图像发送方法。即,虚拟现实主机根据虚拟现实设备的运动信息确定图像压缩比,通过该方法,提高了确定的图像压 缩比的准确性。具体请参考图2,为本申请提供的虚拟现实图像发送方法的流程图。
在下文的介绍过程中,以将该方法应用在图1所示的VR架构为例,因此,图2也可以理解为图1所示的VR架构中的虚拟现实主机与HMD之间的信息交互示意图。该方法的流程描述如下:
S201、检测虚拟现实设备的运动信息。该运动信息用于指示虚拟现实设备的当前的运动状态。
在本申请中,S201可以是由虚拟现实设备完成,也可以是由虚拟现实主机完成。若由虚拟现实设备完成,则可以在虚拟现实设备上设置陀螺仪,加速计等运动传感器来检测运动信息。若由虚拟现实主机来完成,则可以在虚拟现实主机上设置摄像头或其它光学传感器,通过摄像头或光学传感器检测单位时间内虚拟现实设备的位置移动大小,从而确定虚拟现实设备的运动信息。当然还可以有其它的检测方式,本申请不作具体限定。其中,图2中是以通过虚拟现实设备来检测运动信息为例,以下介绍的内容也将继续以虚拟现实设备检测运动信息为例进行说明。
在本申请中,运动信息可以包括虚拟现实设备的位置信息、速度信息、加速度信息、角速度信息以及角加速度信息中的一种或多种。虚拟现实设备可以周期性地或实时地检测运动信息。当然,为了减小负担,虚拟现实设备也可以是只在接收虚拟现实主机发送的检测运动信息的请求后才检测运动信息,没接受到该请求便不检测。例如,在VR应用刚被打开时,虚拟现实主机可以向虚拟现实主机发送用于请求检测运动信息的请求,虚拟现实设备在接收该请求后检测运动信息。或者,在用户体验VR应用的过程中,虚拟现实主机可以在每次确定图像压缩比之前,都向虚拟现实设备发送用于请求检测运动信息的请求,虚拟现实设备在接收该请求后再检测运动信息。
S202、虚拟现实设备将运动信息发送给虚拟现实主机,相应的,虚拟现实主机接收该运动信息。
前面提到过,虚拟现实设备可以周期性地或实时地检测运动信息,也可以在接收虚拟现实主机发送的检测运动信息的请求后再检测运动信息。那么,如果虚拟现实设备是在接收虚拟现实主机发送的检测运动信息的请求后再检测运动信息,则虚拟现实设备可以在检测到运动信息后将运动信息发给虚拟现实主机。如果虚拟现实设备是周期性地或者实时地检测运动信息,那么虚拟现实设备可以在每次检测到运动信息之后即向虚拟现实主机发送运动信息,也可以在接收虚拟现实主机发送的检测运动信息的请求之后再将最新检测的运动信息或者在最近的时间段内检测的所有运动信息发送给虚拟现实主机。
在本申请中,虚拟现实设备在发送运动信息时,可以一并发送虚拟现实设备生成该运动信息的时间信息以及该虚拟现实设备的设备标识等信息。举例来说,在多人VR互动游戏中,多个使用者通过佩戴各自的HMD体验同一个虚拟游戏。在这种情况下,通常一个虚拟现实主机需要与多个HMD通信,即,虚拟现实主机要向多个HMD发送虚拟现实图像,而该虚拟现实主机也可以接收多个HMD发送的运动信息。其中,每个HMD在发送自身的运动信息时,可以一并发送生成该运动信息的HMD的设备标识,以便虚拟现实主机识别接收的运动信息是多个HMD中的哪一个HMD发送的运动信息。再例如,虚拟现实主机可能接收同一个HMD发送的多个运动信息,所以每个HMD在发送自身的运动信息时,可以一并发送HMD生成该运动信息的时间信息,以便虚拟 现实主机识别多个运动信息中每个运动信息的生成时间。
以虚拟现实主机只和一个虚拟现实设备通信为例。如果虚拟现实设备在发送运动信息时,还一并发送了生成该运动信息的时间信息,那么虚拟现实主机在接收该运动信息和该时间信息后,可以将该运动信息和该时间信息对应存储。如果虚拟现实主机接收多个运动信息,每个运动信息的生成时间不同,当虚拟现实主机需要使用运动信息时,可以从多个运动信息中选择时间最新的运动信息,也可以是取一段时长内的所有运动信息的平均值,并将该平均值作为所需的运动信息。例如,虚拟现实主机接收虚拟现实设备发送的三个运动信息分别为速率为3m/s、速率为4m/s和速率为3m/s,并且一并接收生成这三个运动信息的时间信息,分别为00:10时刻,00:15时刻,和00:20时刻。当虚拟现实主机需要使用运动信息时,可以直接选择时间最新的运动信息,即虚拟现实设备在00:20时刻移动速率为3m/s,也可以取一段时长例如00:15-00:20时长内的运动信息的平均值,即取3m/s和4m/s的平均值即3.5m/s,虚拟现实主机将3.5m/s确定为需要的运动信息。这里提到的一段时长可以根据实际情况而定,本申请不作限定。
S203、虚拟现实主机根据运动信息确定图像压缩比。
在本申请中,S203的一种可能的实现方式为:虚拟现实主机首先根据无线信道的信道质量初步确定一个图像压缩比,然后根据运动信息和该初步确定的图像压缩比确定最终的图像压缩比。这种方式,综合考虑了两种不同的因素,即信道质量和运动信息这两种因素,确定的图像压缩比会更为准确。下面具体介绍该方式的实现过程。
在这种方式下,虚拟现实主机确定最终的图像压缩比,具体可以分三步完成。第一步,确定虚拟现实主机和虚拟现实设备之间的无线信道;第二步,根据无线信道的信道质量初步确定图像压缩比;第三步,根据运动信息和初步确定的图像压缩比确定最终的图像压缩比。
由于目前的VR系统可大致分为两类:第一类,利用波束成形技术进行数据传输的VR系统,例如采用高频段(例如60GHz)通信方式来实现数据传输的VR系统。第二类,未利用波束成形技术进行数据传输的VR系统,例如采用低频段(例如30GHz)通信技术来实现数据传输的VR系统。
对于第一类VR系统,虚拟现实主机和虚拟现实设备之间的无线信道可能是变化的。虚拟现实主机和虚拟现实设备在建立通信之前可以进行波束训练。由于目前的VR系统中虚拟现实主机并不清楚虚拟现实设备的所处的位置。因此,虚拟现实主机和虚拟现实设备之间的波束训练需要各自进行360度范围内搜索,即虚拟现实主机产生的发射波束(分布于360度内的不同角度上)和虚拟现实设备产生的接收波束(分布于360度内的不同角度上)的对准过程,对准之后形成的发射-接收波束即为虚拟现实主机和虚拟现实设备之间的无线信道。可见,目前的第一类VR系统中虚拟现实主机和虚拟现实设备之间的波束训练所需时间较长,从而导致图像传输时延过大。
通常,虚拟现实主机自身的位置较为固定。所以,在本申请中,虚拟现实主机可以根据虚拟现实设备的位置信息,在第一角度范围内从虚拟现实设备产生分布于360度范围内的至少一个无线传输方向中确定最佳的无线传输方向,只需检测该最佳的无线传输方向上的无线信道的信道质量即可,无需在360度的范围内进行搜索,缩短了波束训练带来的时延。
在本申请中,第一角度范围可以根据虚拟现实设备根据当前时刻的位置信息与前一时刻的位置信息之间的移动距离来确定,或者,也可以根据虚拟现实设备的历史位置信息和当前的位置信息预测下一时刻虚拟现实设备所处的位置信息,并根据预测出的下一时刻的位置信息和当前时刻的位置信息之间的移动距离来确定。
举例来说,请参考图3所示,虚拟现实主机产生的发射波束在360度范围内分为8个方向,如图所示,虚拟现实设备从第一位置移动到第二位置,而第一位置到第二位置的移动距离即移动范围处于虚拟现实主机产生的发射波束的2个方向内,即方向t1-t2。那么虚拟现实主机只需在该方向t1-t2内搜索虚拟现实设备产生的接收波束即可,例如搜索结果为虚拟现实主机确定的最佳的无线信道为发射-接收波束(t2,r8)。
对于第二类VR系统,由于该类VR系统未采用波束成形技术,因而也无需进行波束成形方向的搜索,该类VR系统中虚拟现实主机和虚拟现实设备之间的无线信道往往是事先设置好的,所以虚拟现实主机可直接执行S203的第二步。
虚拟现实主机在确定无线信道之后,进入第二步,即虚拟现实主机根据无线信道的信道质量初步确定图像压缩比。在第二步中,虚拟现实主机需要检测无线信道的信道质量,具体的实现方式可以有多种,例如虚拟现实主机实时地或周期性地检测信道质量,且虚拟现实主机可以存储检测的信道质量,则虚拟现实主机可以在已存储的信道质量中确定最近一次检测的信道质量,这种方式较为简单,虚拟现实主机无需专门再检测信道质量,有助于提高效率;或者,虚拟现实主机可以直接检测当前的信道质量,这样确定的信道质量更为准确。然后,虚拟现实主机根据事先存储的信道质量与图像压缩比之间的映射关系,初步确定图像压缩比。无线信道的信道质量与图像压缩比之间的映射关系有多种,表1为一种示例。表1中无线信道的信道质量以无线信道的传输速率为例。
表1
Figure PCTCN2017092285-appb-000001
例如,虚拟现实主机确定的无线信道的传输速率是4Gbps,则根据表1就能初步确定图像压缩比为4:1。
下面介绍第三步的实现过程,即虚拟现实主机根据运动信息和初步确定的图像压缩比确定最终的图像压缩比。为了提高确定的图像压缩比的准确性,虚拟现实主机可以首先根据接收的运动信息预测未来的第一时长内虚拟现实设备的运动状态。关于虚拟现实主机预测未来的第一时长内虚拟现实设备的运动状态的原因,将在后续介绍。然后虚拟现实主机根据预测出的未来的第一时长内虚拟现实设备的运动状态和初步确定的图像压缩比来确定最终的图像压缩比。而虚拟现实设备的运动状态通常又可以分为静止或运动两种情况。因此,第三步的实现过程又分为两种情况:第一种情况,当未来的第一时长内虚拟现实设备处于静止状态时,虚拟现实主机如何确定最终的图像 压缩比;第二种,当未来的第一时长内虚拟现实设备处于运动状态时,虚拟现实主机如何确定最终的图像压缩比。
在介绍这两种情况之前,首先介绍虚拟现实主机预测未来的第一时长内虚拟现实设备的运动状态的原因以及一种实现方式。具体如下:
由于用户在佩戴虚拟现实设备体验虚拟世界的过程中,用户运动状态随时发生变化,即虚拟现实设备的运动状态随时发生变化。例如虚拟现实设备检测到当前是静止状态即速率为0m/s,那么从虚拟现实设备将该速率为0m/s的运动信息发送给虚拟现实主机开始,到虚拟现实设备接收虚拟现实主机发送的压缩的虚拟现实图像为止的这一过程可能需要一定的时长,在该时长后虚拟现实设备可能已经由静止状态改变为运动状态。因此,那么为了提高确定的图像压缩比的准确性,虚拟现实主机可以预测虚拟现实设备未来的第一时长内的运动状态。
举例来讲,如果虚拟现实主机在00:10时刻接收虚拟现实设备发送的运动信息以及虚拟现实设备生成该运动信息的时间信息,具体为,在00:08时刻虚拟现实设备的速率为0m/s。那么虚拟现实主机只能确定虚拟现实设备在00:08时刻是静止的。如果虚拟现实主机压缩虚拟现实图像、将压缩的虚拟现实图像发送给虚拟现实设备等过程需要5s,那么虚拟现实设备会在00:15时刻接收压缩的虚拟现实图像,但在00:15时刻虚拟现实设备可能已经由静止状态变为运动状态。因此,为了提高图像压缩比的准确性,虚拟现实主机可以根据接收的运动信息预测未来的第一时长内例如5s内虚拟现实设备的运动状态。
在本申请中,第一时长可以是根据经验确定的,也可以是通过计算确定的。如果是通过计算确定的,由前述的虚拟现实主机预测未来的第一时长内虚拟现实设备的运动状态的原因可知,第一时长可以是虚拟现实设备从生成某个运动信息的时刻开始到接收虚拟现实主机发送的压缩的虚拟现实图像为止所需的时长。该时长具体可以分为三个时长之和。其中,第一个时长可以是虚拟现实设备将运动信息发送给虚拟现实主机所需的时长,第二个时长可以是虚拟现实主机压缩虚拟现实图像所需时长,第三个时长可以是虚拟现实主机将压缩的虚拟现实图像发送给虚拟现实设备所需时长。以下将分别这三个时长的计算方式。第一个时长可以是虚拟现实主机接收运动信息的时间与虚拟现实设备生成该运动信息的时间的时间差。第二个时长可以是一次需要传输的虚拟现实图像的总帧数与压缩一帧虚拟现实图像所需的时长相乘。其中,压缩一帧虚拟现实图像所需的时长可以通过测试获得。例如虚拟现实主机可以记录对一帧虚拟现实图像进行压缩前的第一时刻,以及完成压缩时的第二时刻,第二时刻与第一时刻之差即为压缩一帧虚拟现实图像所需的时长。第三个时长可以是一次传输的数据量与无线信道的传输速率相除。而一次传输的数据量可以由传输的虚拟现实图像的总帧数以及每帧虚拟现实图像包括的数据量确定。当确定了三个时长之后,将该三个时长相加即可得到第一时长。当然了,第一时长的还可以有其它的算法,本申请不作具体的限定。
在本申请中,虚拟现实主机可以根据接收的运动信息和虚拟现实设备的历史运动信息,确定在未来的第一时长内虚拟现实设备是处于静止状态还是处于运动状态。
前面提到过,如果虚拟现实设备在发送运动信息时,还一并发送了生成该运动信息的时间信息,虚拟现实主机在接收运动信息和生成该运动信息的时间信息时,可以 将该运动信息和该时间信息对应存储,进而得到虚拟现实设备运动信息随时间变化的信息,即得到虚拟现实设备的历史运动信息。
以VR游戏为例,虚拟现实主机中存储的虚拟现实设备的历史运动信息为:在00:00时刻时,虚拟现实设备的速率为0。在00:05时刻时,虚拟现实设备的速率为1m/s。在00:10时刻时,虚拟现实设备速率为2m/s。在00:15时刻时,虚拟现实设备的速率为3m/s。在00:20时刻时,虚拟现实设备的速率为2m/s。在00:25时刻时,虚拟现实设备的速率为1m/s。在00:30时刻时,虚拟现实设备的速率为0m/s。在00:35时刻时,虚拟现实设备的速率为0m/s。在00:40时刻时,虚拟现实设备的速率为0m/s等等。
继续以该VR游戏为例,如果虚拟现实主机在00:14时刻接收虚拟现实设备发送的运动信息以及生成该运动信息的时间信息,具体为,在00:12时刻虚拟现实设备的速率为2m/s。如果第一时长为3s,那么虚拟现实主机根据虚拟现实设备的历史运动信息,可知在未来的3s内即在00:15时刻,虚拟现实主机速率为3m/s,即确定虚拟现实设备处于运动状态。再例如,虚拟现实主机是在00:30时刻接收虚拟现实主机发送的运动信息以及生成该运动信息的时间信息,具体为,虚拟现实设备在00:28时刻速率为0m/s。如果第一时长为3s,虚拟现实主机根据虚拟现实设备的历史运动信息,可知在未来的3s内即在00:15时刻,虚拟现实主机速率为0m/s,即虚拟现实设备处于静止状态。
以下分别介绍在第三步实现过程中的两种情况。
第一种情况,即当未来的第一时长内虚拟现实设备处于静止状态时,虚拟现实主机如何确定最终的图像压缩比。
需要说明的是,虚拟现实主机与虚拟现实设备之间的无线信道的信道质量和虚拟现实设备的运动状态具有一定的相关性,例如虚拟现实设备快速移动时,信道的稳定性较差,信道质量的变化会比较大,而虚拟现实设备静止时,信道质量一般都比较稳定,变化不大。因此,当虚拟现实主机确定在未来的第一时长内虚拟现实设备处于静止状态时,可以确定信道质量比较稳定,那么虚拟现实主机可以直接确定与检测的无线信道的信道质量对应的图像压缩比为最终的图像压缩比,即直接将初步确定的图像压缩比作为最终的图像压缩比。
第二种情况,即当未来的第一时长内虚拟现实设备处于运动状态时,虚拟现实主机如何确定最终的图像压缩比。
在本申请中,当虚拟现实设备确定未来的第一时长内虚拟现实设备处于运动状态时,可以根据运动信息(这里的运动信息用于指示虚拟现实设备的当前的运动状态,并非前面提到的历史运动信息)调整初步确定的图像压缩比。例如,如果运动信息用于指示虚拟现实设备的移动速率较慢,则虚拟现实主机在初步确定的图像压缩比的基础上适当减小图像压缩比。如果运动信息用于指示虚拟现实设备的移动速率较快,虚拟现实主机在初步确定的图像压缩比的基础上适当的增大图像压缩比。
这是因为,当虚拟现实设备的移动速率较快时,即虚拟现实设备显示的虚拟现实图像的切换速率即内容变化较快。对于快闪镜头,用户往往会忽略图像的质量,而注重图像的连贯性和流畅性。因此增大图像压缩比,可以保证图像的连贯性,进而提高用户体验。而当虚拟现实设备的移动速率较慢时,虚拟现实设备显示的虚拟现实图像的内容变化较慢,对于慢镜头或者静止的镜头,用户往往对于图像的质量要求较高, 例如对图像的清晰度、分辨率等要求较高,此时减小图像压缩比,可以提高图像的质量,进而提高用户体验。
对于第二种情况,为了提高确定图像压缩比的准确性,虚拟现实主机在根据运动信息调整初步确定的图像压缩比时,也可以考虑未来的第一时长内无线信道的信道质量的变化趋势。即,虚拟现实主机可以根据运动信息和预测出的未来的第一时长内信道质量的变化趋势一起调整初步确定的图像压缩比。具体如下:
如果运动信息用于指示虚拟现实设备的移动速率较慢,且未来第一时长内信道质量变好,则虚拟现实主机在初步确定的图像压缩比的基础上适当减小图像压缩比。如果运动信息用于指示虚拟现实设备的移动速率较快,且未来第一时长内信道质量变差,则虚拟现实主机在初步确定的图像压缩比的基础上适当的增大图像压缩比。
这是因为,当虚拟现实设备的移动速率较快时,即虚拟现实设备显示的虚拟现实图像的内容变化较快。为了保证虚拟现实图像的流畅性,虚拟现实主机在初步确定的图像压缩比的基础上增大图像压缩比,但是增大图像压缩比之后会使得传输的数据量大幅度减小,此时信道质量无需太好也可以满足传输需求,所以如果信道质量变差也不会影响虚拟现实图像的传输。当虚拟现实设备的移动速率较慢时,即虚拟现实设备显示的虚拟现实图像的内容变化较慢。为了保证虚拟现实图像的质量,虚拟现实主机在初步确定的图像压缩比的基础上减小图像压缩比,但是减小图像压缩比之后可能会使得传输的数据量大幅度增加,如果无线信道的信道质量变好便不会影响增加的数据量的传输。即虚拟现实主机在根据运动信息和未来的第一时长内信道质量的变化趋势一起调整图像压缩比,可以更加提高确定图像压缩比准确性。
在实际应用中,虚拟现实主机可以选择只根据运动信息调整图像压缩比,也可以选择根据运动信息和未来第一时长内无线信道的信道质量的变化趋势一起调整图像压缩比。
S204:虚拟现实主机根据图像压缩比对待发送的虚拟现实图像进行压缩。
在本申请中,虚拟现实主机可以根据确定的最终的图像压缩比对每一帧虚拟现实图像进行压缩,也可以是每次对多帧虚拟现实图像打包一起进行压缩。
本申请中,虚拟现实主机根据虚拟现实设备的运动信息确定图像压缩比,例如,当虚拟现实设备移动速率较慢时,即虚拟现实设备显示的虚拟现实图像的内容变化较慢,为了保证虚拟现实图像的质量,确定一个较小的图像压缩比。当虚拟现实设备运动速率较快时,即虚拟现实设备显示的虚拟现实图像的内容变化较快,为了保证虚拟现实图像的连贯性,确定一个较大的图像压缩比。通过这种方式,提高了确定的图像压缩比的准确性,能够更好地满足用户的需求。
在实际操作过程中,虚拟现实主机在发送虚拟现实图像之前,需要确定调制编码方式,然后根据该调制编码方式对虚拟现实图像进行调制编码。下面介绍目前的VR系统如何确定调制编码方式。
目前,VR系统中,虚拟现实主机根据无线信道的信道质量确定调制编码方式,例如无线信道的信道质量较好,便确定一个调制编码速率较快的调制编码方式。但通过这种方式确定的调制编码方式不够准确。这是因为,实际上,每种调制编码方式都对应有各自的调制编码速率,而调制编码速率与虚拟现实图像在传输过程中产生的误帧率相关,例如调制编码速率较快时,虽然传输效率较高,但出现误帧、丢帧的概率较 大。如果虚拟现实设备的移动速率较快,则虚拟现实设备显示的虚拟现实图像的内容变化较快,为了保证虚拟现实图像的流畅性,需要减小误帧、丢帧的概率,所以不适合使用调制编码速率太快的调制编码方式。如果无线信道的信道质量始终较好,那么虚拟现实主机将一直采用调制编码速率较快的调制编码方式,即出现丢帧、误帧的概率很大,一旦出现丢帧,用户所看到的虚拟现实图像将不连贯。可见,目前,VR系统中,虚拟现实主机根据信道质量确定的调制编码方式不够准确。下面介绍本申请中确定调制编码方式的过程。
S205、虚拟现实主机根据运动信息确定调制编码方式。
S205与S203、S204之间的执行顺序不作限定,只要发生在S202之后即可。图2是以S205发生在S204之后为例。
类似于S203,S205的一种可能的实现方式为,虚拟现实主机首先根据无线信道的信道质量初步确定一个调制编码方式,然后根据运动信息和该初步确定的调制编码方式确定最终的调制编码方式。这种方式,综合考虑了两种不同的因素,即信道质量和运动信息这两种因素,确定的调制编码方式会更为准确。下面具体介绍这种方式的实现过程。
在这种方式下,虚拟现实主机确定最终的调制编码方式,可以分两步完成。第一步,确定虚拟现实主机和虚拟现实设备之间的无线信道;第二步,根据无线信道的信道质量初步确定调制编码方式;第三步,根据运动信息和初步确定的调制编码方式确定最终的调制编码方式。
关于第一步的实现过程,请参见S203中关于第一步的描述,为了说明书的简洁,在此不做赘述。
在第二步的实现过程中,虚拟现实主机首先需要确定信道质量和信道质量与调制编码方式之间的映射关系。关于信道质量与调制编码方式之间的映射关系,虚拟现实主机可以预先存储,因此,虚拟现实主机主要需要确定的就是信道质量。关于虚拟现实主机确定信道质量的方式前面已经介绍过。然后,虚拟现实主机根据存储的信道质量与调制编码方式之间的映射关系,初步确定调制编码方式。无线信道的信道质量与调制编码方式之间的映射关系有多种,表2为一种示例。并且,表2只是对信道质量与调制编码方式之间的映射关系的举例,并不是对信道质量与调制编码方式之间的映射关系的限定。表1中无线信道的信道质量以无线信道的传输速率为例。
表2
Figure PCTCN2017092285-appb-000002
在本申请中,无线信道的信道质量与调制编码方式之间的映射关系可以采用现有技术中的调制与编码策略(Modulation and Coding Scheme,MCS)索引表,也可以是用户自定义的映射关系。
例如,虚拟现实主机确定的无线信道的传输速率是4Gbps,则根据表2就能初步确定调制编码方式是16-QAM。
下面介绍第三步的实现过程,即虚拟现实主机根据运动信息和初步确定的调制编码方式确定最终的调制编码方式。与S203类似,虚拟现实主机根据接收的运动信息预测未来第一时长内虚拟现实设备的运动状态。然后根据该运动状态和初步确定的调制编码方式来确定最终的调制编码方式。因此,第三步的实现过程也分为两种情况:第一种情况,当未来的第一时长内虚拟现实设备处于静止状态时,虚拟现实主机如何确定最终的调制编码方式。第二种情况,当未来的第一时长内虚拟现实设备处于运动状态时,虚拟现实主机如何确定最终的调制编码方式。关于虚拟现实主机预测未来第一时长内虚拟现实设备的运动状态的原因和实现方式,前面已经介绍过,在此不再重复介绍。因此,下面直接介绍第二步的实现过程的两种情况。
第一种情况,即当未来第一时长内虚拟现实设备处于静止状态时,虚拟现实主机如何确定最终的调制编码方式。
前面已经提到过,虚拟现实主机与虚拟现实设备之间的无线信道的信道质量和虚拟现实设备的运动状态具有一定的相关性,当虚拟现实主机确定在未来的第一时长内虚拟现实设备处于静止状态时,可以确定信道质量比较稳定,那么虚拟现实主机可以直接确定与检测的无线信道的信道质量对应的调制编码方式为最终的调制编码方式。
第二种情况,即当未来第一时长内虚拟现实设备处于运动状态时,虚拟现实主机如何确定最终的调制编码方式。
在本申请中,当虚拟现实设备确定未来的第一时长内虚拟现实设备处于运动状态时,可以根据运动信息调整初步确定的调制编码方式。例如,如果运动信息用于指示虚拟现实设备的移动速率较慢,则虚拟现实主机将初步确定的调制编码方式调整为第一调制编码方式,该第一调制编码的调制编码速率大于或者等于初步确定的调制编码方式的调制编码速率。如果运动信息用于指示虚拟现实设备的移动速率较快,则虚拟现实主机将初步确定的调制编码方式调制为第二调制编码方式,该第二调制编码方式的调制编码速率小于初步确定的调制编码方式的调制编码速率。
这是因为,当虚拟现实设备移动速率较快时,即虚拟现实设备显示的虚拟现实图像的内容变化较快,所以每次传输的虚拟现实图像的帧数较多,为了保证虚拟现实图像的连贯性和流畅性,减小出现误帧、丢帧的概率,避免重传图像,所以虚拟现实主机可以确定调制编码速率较低的调制编码方式。当虚拟现实设备移动速率较慢时,即虚拟现实设备显示的虚拟现实图像的内容变化较慢,即每次传输的虚拟现实图像内容变化较少,每次需要传输的虚拟现实图像的帧数较少,即此时无需调制编码速率太低的调制编码方式也可以实现,所以虚拟现实主机可以确定调制编码速率较高的调制编码方式,进而提高传输效率。或者,当虚拟现实设备移动速率较慢时,由于无线信道变化也较慢,所以检测的信道质量较准确,所以虚拟现实主机也可以直接确定与检测的信道质量对应的调制编码方式为最终的调制编码方式。
S206:虚拟现实主机根据调制编码方式对压缩的虚拟现实图像进行调制编码。
在本申请中,如果虚拟现实主机对每一帧虚拟现实图像都进行压缩,那么虚拟现实主机可以根据最终的调制编码方式对每一帧压缩的虚拟现实图像进行调制编码。如果虚拟现实主机是对多帧虚拟现实图像打包一起压缩,那么虚拟现实主机可以根据最 终的调制编码方式对虚拟现实图像压缩包一起进行调制编码。
S207:虚拟现实主机将调制编码后的压缩的虚拟现实图像发送给虚拟现实设备,相应的,虚拟现实设备接收经过调制编码的虚拟现实图像。
虚拟现实设备接收调制编码后的压缩的虚拟现实图像之后,对该虚拟现实图像进行解调解码得到压缩的虚拟现实图像,然后将压缩的虚拟现实图像解压,最后将虚拟现实图像显示出来。
在本申请中,虚拟现实主机根据虚拟现实设备的运动信息确定调制编码方式。例如当虚拟现实设备移动速率较快时,即虚拟现实设备显示的虚拟现实图像的切换速率较快,为了保证虚拟现实图像的连贯性和流畅性,减小出现误帧、丢帧的概率,避免重传图像,所以虚拟现实主机可以确定调制编码速率较低的调制编码方式。当虚拟现实设备移动速率较慢时,即每次传输的虚拟现实图像的帧数较少,虚拟现实主机可以确定调制编码速率较高的调制编码方式或者直接根据无线信道的信道质量确定调制编码方式,进而提高传输效率。通过这种方式,提高了确定的调制编码方式的准确性,能够更好地满足用户的需求。
总结来说,在本申请中,虚拟现实主机根据虚拟现实设备的运动信息确定图像压缩比、调制编码方式。例如,当虚拟现实设备移动速率较慢时,即虚拟现实设备显示的虚拟现实图像内容变化较慢,此时确定一个较小的图像压缩比,同时可以直接根据当前的信道质量确定调制编码方式。当虚拟现实设备运动速率较快时,即虚拟现实设备显示的虚拟现实图像内容变化较快,对于快闪的镜头,人眼对图像质量要求降低,所以为了保证虚拟现实图像的连贯性,此时可以确定一个较大的图像压缩比。同时,虚拟现实主机确定调制编码速率较低的调制编码方式,双重保证了虚拟现实图像的连贯性。因此,通过本申请提供的虚拟现实图像的发送方法中虚拟现实主机可以根据虚拟现实设备的运动信息确定图像压缩比、调制编码方式,保证了虚拟现实图像的质量,提高了传输效率,能够更好地满足用户的需求。
下面结合附图介绍本申请提供的设备。
图4示出了一种虚拟现实图像发送装置400的结构示意图。该虚拟现实图像发送装置400可以实现上文中涉及的虚拟现实主机的功能。该虚拟现实图像发送装置400可以包括接收单元401、处理单元402、发送单元403。其中,接收单元401可以用于执行图2所示的实施例中的S202,和/或用于支持本文所描述的技术的其它过程。处理单元402可以用于执行图2所示的实施例中的S203-S206,和/或用于支持本文所描述的技术的其它过程。发送单元403可以用于执行图2所示的实施例中的S207,和/或用于支持本文所描述的技术的其它过程。其中,上述方法实施例涉及的各步骤的所有相关内容均可以援引到对应功能模块的功能描述,在此不再赘述。
需要说明的是,在本申请中,上述虚拟现实图像发送装置以功能单元的形式展示。在不受限制的情况下,本文所使用的术语“单元”可指执行一个或多个软件或固件程序的专用集成电路(application-specific integrated circuit,ASIC)、电子电路、(共享、专用或组)处理器以及存储器,组合逻辑电路,和/或提供所述功能的其它合适的部件。
在一个简单的实施例中,本领域的技术人员可以想到,还可以将虚拟现实图像发送装置400通过如图5所示的结构实现。
如图5所示,虚拟现实图像发送装置500可以包括:存储器501、处理器502、接 收器503、发送器504和总线505。存储器501、处理器502、接收器503、发送器504可以通过总线505连接。接收器503,用于接收虚拟现实设备发送的第一运动信息,第一运动信息用于指示虚拟现实设备的当前运动状态;处理器502,用于根据第一运动信息确定第一图像压缩比;并根据第一图像压缩比对待发送的虚拟现实图像进行压缩;发送器504,用于将压缩的虚拟现实图像发送给虚拟现实设备。处理器502、接收器503、发送器504所执行操作的具体实现方式可以参照图2的实施例的中由虚拟现实主机执行的对应步骤,本申请不再赘述。
其中,存储器501用于存储计算机执行指令,当虚拟现实图像发送装置500运行时,处理器502执行存储器501存储的计算机执行指令,实现上述功能,从而实现如图2所示的实施例提供的虚拟现实图像发送方法。具体的图像拍摄方法可参考上文及附图中的相关描述,此处不再赘述。
在本申请中,处理器502可以是现场可编程门阵列(field-programmable gate array,FPGA),专用集成芯片(application specific integrated circuit,ASIC),系统芯片(system on chip,SoC),中央处理器(central processor unit,CPU),网络处理器(network processor,NP),数字信号处理电路(digital signal processor,DSP),微控制器(micro controller unit,MCU),还可以采用可编程控制器(programmable logic device,PLD)或其他集成芯片。总线505可以是外设部件互连标准(peripheral component interconnect,PCI)总线或扩展工业标准结构(extended industry standard architecture,EISA)总线等。总线505可以分为地址总线、数据总线、控制总线等。为便于表示,图5中仅用一条粗线表示,但并不表示仅有一根总线或一种类型的总线。
在上述发明实施例中,可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时,可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时,全部或部分地产生按照本申请所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中,或者从一个计算机可读存储介质向另一个可读存储介质传输,例如,所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质,(例如,软盘、硬盘、磁带)、光介质(例如,DVD)、或者半导体介质(例如,固态硬盘Solid State Disk(SSD))等。
以上所述,仅为本申请的具体实施方式,但本申请的保护范围并不局限于此,任何熟悉本技术领域的技术人员在本申请揭露的技术范围内,可轻易想到变化或替换,都应涵盖在本申请的保护范围之内。因此,本申请的保护范围应以所述权利要求的保护范围为准。

Claims (25)

  1. 一种虚拟现实图像发送方法,其特征在于,包括:
    虚拟现实主机接收虚拟现实设备发送的第一运动信息,所述第一运动信息用于指示所述虚拟现实设备的当前运动状态;
    所述虚拟现实主机根据所述第一运动信息确定第一图像压缩比;
    所述虚拟现实主机根据所述第一图像压缩比对待发送的虚拟现实图像进行压缩;
    所述虚拟现实主机将压缩的所述虚拟现实图像发送给所述虚拟现实设备。
  2. 根据权利要求1所述的方法,其特征在于,在所述虚拟现实主机根据所述第一运动信息确定第一图像压缩比之前,所述方法还包括:
    所述虚拟现实主机检测无线信道当前的信道质量,其中,所述无线信道为所述虚拟现实设备与所述虚拟现实主机之间的通信信道;
    所述虚拟现实主机根据无线信道的信道质量与图像压缩比之间的映射关系,确定与检测的信道质量对应的图像压缩比;
    所述虚拟现实主机根据所述第一运动信息确定第一图像压缩比,包括:
    所述虚拟现实主机根据所述第一运动信息,确定与检测的信道质量对应的图像压缩比为所述第一图像压缩比,或所述虚拟现实主机根据所述第一运动信息调整与检测的信道质量对应的图像压缩比,并将调整后的图像压缩比作为所述第一图像压缩比。
  3. 根据权利要求2所述的方法,其特征在于,所述虚拟现实主机根据所述第一运动信息,确定与检测的信道质量对应的图像压缩比为所述第一图像压缩比,包括:
    所述虚拟现实主机根据所述第一运动信息和所述虚拟现实设备的历史运动信息,若确定在未来的第一时长内所述虚拟现实设备处于静止状态,则确定与检测的信道质量对应的图像压缩比为所述第一图像压缩比。
  4. 根据权利要求2所述的方法,其特征在于,所述虚拟现实主机根据所述第一运动信息调整与检测的信道质量对应的图像压缩比,包括:
    所述虚拟现实主机根据所述第一运动信息和所述虚拟现实设备的历史运动信息,若确定在未来的第一时长内所述虚拟现实设备处于运动状态,则根据所述第一运动信息调整与检测的信道质量对应的图像压缩比。
  5. 根据权利要求2或4所述的方法,其特征在于,所述虚拟现实主机根据所述第一运动信息调整与检测的信道质量对应的图像压缩比,包括:
    若所述第一运动信息用于指示所述虚拟现实设备的移动速率小于或等于第一速率,所述虚拟现实主机减小与检测的信道质量对应的图像压缩比;或
    若所述第一运动信息用于指示所述虚拟现实设备的移动速率大于第二速率,所述虚拟现实主机增大与检测的信道质量对应的图像压缩比;
    其中,所述第一速率小于或等于所述第二速率。
  6. 根据权利要求2-5任一所述的方法,其特征在于,在所述虚拟现实主机检测无线信道当前的信道质量之后,所述方法还包括:
    所述虚拟现实主机根据无线信道的信道质量与调制编码方式之间的映射关系,确定与检测的信道质量对应的调制编码方式;
    所述虚拟现实主机根据所述第一运动信息确定与检测的信道质量对应的调制编码方式为第一调制编码方式,或,所述虚拟现实主机根据所述第一运动信息调整与检测 的信道质量对应的调制编码方式,并将调整后的调制编码方式作为第一调制编码方式;
    在所述虚拟现实主机将压缩的所述虚拟现实图像发送给所述虚拟现实设备之前,所述方法还包括:
    所述虚拟现实主机根据所述第一调制编码方式对所述压缩的虚拟现实图像进行调制编码;
    所述虚拟现实主机将压缩后所述虚拟现实图像发送给所述虚拟现实设备,包括:
    所述虚拟现实主机将调制编码后的压缩的所述虚拟现实图像发送给所述虚拟现实设备。
  7. 根据权利要求6所述的方法,其特征在于,所述虚拟现实主机根据所述第一运动信息确定与检测的信道质量对应的调制编码方式为第一调制编码方式,包括:
    所述虚拟现实主机根据所述第一运动信息和所述虚拟现实设备的历史运动信息,若确定在未来的第一时长内所述虚拟现实设备处于静止状态,则确定与检测的信道质量对应的调制编码方式为所述第一调制编码方式。
  8. 根据权利要求6所述的方法,其特征在于,所述虚拟现实主机根据所述第一运动信息调整与检测的信道质量对应的调制编码方式,包括:
    所述虚拟现实主机根据所述第一运动信息和所述虚拟现实设备的历史运动信息,若确定在未来的第一时长内所述虚拟现实设备处于运动状态,则根据所述第一运动信息调整与检测的信道质量对应的调制编码方式。
  9. 根据权利要求8所述的方法,其特征在于,所述虚拟现实主机根据所述第一运动信息调整与检测的信道质量对应的调制编码方式,包括:
    若所述第一运动信息用于指示所述虚拟现实设备的移动速率小于或等于第一速率,所述虚拟现实主机将与检测的信道质量对应的调制编码方式调整为第二调制编码方式,所述第二调制编码方式的调制编码速率大于与检测的信道质量对应的调制编码方式的调制编码速率;
    若所述第一运动信息用于指示所述虚拟现实设备的移动速率大于第二速率,所述虚拟现实主机将所述与检测的信道质量对应的调制编码方式调整为第三调制编码方式,所述第三调制编码方式的调制编码速率小于与检测的信道质量对应的调制编码方式的调制编码速率;
    其中,所述第一速率小于所述第二速率。
  10. 根据权利要求2所述的方法,其特征在于,所述第一运动信息用于指示所述虚拟现实设备的当前位置信息,所述虚拟现实主机检测无线信道当前的信道质量,包括:
    所述虚拟现实主机根据所述位置信息在第一角度范围内从所述虚拟现实设备产生的分布于360度范围内的至少一个无线传输方向中确定第一无线传输方向;所述第一角度大于0度小于360度;
    所述虚拟现实主机检测所述第一无线传输方向上的无线信道的当前的信道质量。
  11. 根据权利要求1-9任一所述的方法,其特征在于,所述第一运动信息包括所述虚拟现实设备的速度信息、加速度信息、角速度信息以及角加速度信息中的一种或多种。
  12. 一种虚拟现实图像发送装置,其特征在于,包括:
    接收单元,用于接收虚拟现实设备发送的第一运动信息,所述第一运动信息用于指示所述虚拟现实设备的当前运动状态;
    处理单元,用于根据所述第一运动信息确定第一图像压缩比;并根据所述第一图像压缩比对待发送的虚拟现实图像进行压缩;
    发送单元,用于将压缩的所述虚拟现实图像发送给所述虚拟现实设备。
  13. 根据权利要求12所述的装置,其特征在于,所述处理单元还用于:
    检测无线信道当前的信道质量,其中,所述无线信道为所述虚拟现实设备与所述虚拟现实图像发送装置之间的通信信道;
    根据无线信道的信道质量与图像压缩比之间的映射关系,确定与检测的信道质量对应的图像压缩比;
    所述处理单元具体用于:
    根据所述第一运动信息,确定与检测的信道质量对应的图像压缩比为所述第一图像压缩比,或根据所述第一运动信息调整与检测的信道质量对应的图像压缩比,并将调整后的图像压缩比作为所述第一图像压缩比。
  14. 根据权利要求13所述的装置,其特征在于,所述处理单元具体用于:
    根据所述第一运动信息和所述虚拟现实设备的历史运动信息,若确定在未来的第一时长内所述虚拟现实设备处于静止状态,则确定与检测的信道质量对应的图像压缩比为所述第一图像压缩比。
  15. 根据权利要求13所述的装置,其特征在于,所述处理单元具体用于:
    根据所述第一运动信息和所述虚拟现实设备的历史运动信息,若确定在未来的第一时长内所述虚拟现实设备处于运动状态,则根据所述第一运动信息调整与检测的信道质量对应的图像压缩比。
  16. 根据权利要求13或15所述的装置,其特征在于,所述处理单元具体用于:
    若所述第一运动信息用于指示所述虚拟现实设备的移动速率小于或等于第一速率,则减小与检测的信道质量对应的图像压缩比;或
    若所述第一运动信息用于指示所述虚拟现实设备的移动速率大于第二速率,则增大与检测的信道质量对应的图像压缩比;
    其中,所述第一速率小于或等于所述第二速率。
  17. 根据权利要求13-16任一所述的装置,其特征在于,所述处理单元还用于:
    根据无线信道的信道质量与调制编码方式之间的映射关系,确定与检测的信道质量对应的调制编码方式;
    根据所述第一运动信息确定与检测的信道质量对应的调制编码方式为第一调制编码方式,或,根据所述第一运动信息调整与检测的信道质量对应的调制编码方式,并将调整后的调制编码方式作为第一调制编码方式;
    根据所述第一调制编码方式对所述压缩的虚拟现实图像进行调制编码;
    所述发送单元具体用于:将调制编码后的压缩的所述虚拟现实图像发送给所述虚拟现实设备。
  18. 根据权利要求17所述的装置,其特征在于,所述处理单元具体用于:
    根据所述第一运动信息和所述虚拟现实设备的历史运动信息,若确定在未来的第一时长内所述虚拟现实设备处于静止状态,则确定与检测的信道质量对应的调制编码 方式为所述第一调制编码方式。
  19. 根据权利要求17所述的装置,其特征在于,所述处理单元具体用于:
    根据所述第一运动信息和所述虚拟现实设备的历史运动信息,若确定在未来的第一时长内所述虚拟现实设备处于运动状态,则根据所述第一运动信息调整与检测的信道质量对应的调制编码方式。
  20. 根据权利要求19所述的装置,其特征在于,所述处理单元具体用于:
    若所述第一运动信息用于指示所述虚拟现实设备的移动速率小于或等于第一速率,将与检测的信道质量对应的调制编码方式调整为第二调制编码方式,所述第二调制编码方式的调制编码速率大于与检测的信道质量对应的调制编码方式的调制编码速率;
    若所述第一运动信息用于指示所述虚拟现实设备的移动速率大于第二速率,将所述与检测的信道质量对应的调制编码方式调整为第三调制编码方式,所述第三调制编码方式的调制编码速率小于与检测的信道质量对应的调制编码方式的调制编码速率;
    其中,所述第一速率小于所述第二速率。
  21. 根据权利要求13所述的装置,其特征在于,所述第一运动信息用于指示所述虚拟现实设备的当前位置信息,所述处理单元具体用于:
    根据所述位置信息在第一角度范围内从所述虚拟现实设备产生的分布于360度范围内的至少一个无线传输方向中确定第一无线传输方向;所述第一角度大于0度小于360度;
    检测所述第一无线传输方向上的无线信道的当前的信道质量。
  22. 根据权利要求12-20任一所述的装置,其特征在于,所述第一运动信息包括所述虚拟现实设备的速度信息、加速度信息、角速度信息以及角加速度信息中的一种或多种。
  23. 一种虚拟现实图像发送装置,其特征在于,包括:处理器和存储器;
    所述存储器用于存储计算机执行指令,当所述处理器执行所述指令时,使所述图像拍摄装置执行如权利要求1-11任意一项所述的方法。
  24. 一种计算机可读存储介质,其特征在于,包括指令,当其在计算机上运行时,使得计算机执行如权利要求1-11任意一项所述的方法。
  25. 一种包含指令的计算机程序产品,其特征在于,当其在计算机上运行时,使得计算机执行如权利要求1-11任意一项所述的方法。
PCT/CN2017/092285 2017-03-22 2017-07-07 一种虚拟现实图像发送方法及装置 WO2018171084A1 (zh)

Priority Applications (7)

Application Number Priority Date Filing Date Title
US16/495,858 US11402894B2 (en) 2017-03-22 2017-07-07 VR image sending method and apparatus
AU2017405914A AU2017405914B2 (en) 2017-03-22 2017-07-07 Virtual reality image sending method and apparatus
CA3057180A CA3057180C (en) 2017-03-22 2017-07-07 Method and apparatus for sending virtual reality image
KR1020197030873A KR102296139B1 (ko) 2017-03-22 2017-07-07 가상 현실 이미지를 송신하는 방법 및 장치
CN201780012081.9A CN109074152B (zh) 2017-03-22 2017-07-07 一种虚拟现实图像发送方法及装置
JP2019552177A JP6873268B2 (ja) 2017-03-22 2017-07-07 仮想現実画像を送信するための方法および装置
EP17901675.3A EP3591502B1 (en) 2017-03-22 2017-07-07 Virtual reality image sending method and apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710175242.8 2017-03-22
CN201710175242 2017-03-22

Publications (1)

Publication Number Publication Date
WO2018171084A1 true WO2018171084A1 (zh) 2018-09-27

Family

ID=63586109

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/092285 WO2018171084A1 (zh) 2017-03-22 2017-07-07 一种虚拟现实图像发送方法及装置

Country Status (8)

Country Link
US (1) US11402894B2 (zh)
EP (1) EP3591502B1 (zh)
JP (1) JP6873268B2 (zh)
KR (1) KR102296139B1 (zh)
CN (1) CN109074152B (zh)
AU (1) AU2017405914B2 (zh)
CA (1) CA3057180C (zh)
WO (1) WO2018171084A1 (zh)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110391831B (zh) * 2019-09-04 2021-09-14 联想(北京)有限公司 一种建立波束连接的方法、装置及电子设备
US20230120092A1 (en) * 2020-03-06 2023-04-20 Sony Group Corporation Information processing device and information processing method
US11652513B2 (en) * 2021-08-17 2023-05-16 Cisco Technology, Inc. Wireless reliability for devices in motion
CN113824746B (zh) * 2021-11-25 2022-02-18 山东信息职业技术学院 一种虚拟现实信息传输方法及虚拟现实系统
US11870852B1 (en) * 2023-03-31 2024-01-09 Meta Platforms Technologies, Llc Systems and methods for local data transmission

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103248883A (zh) * 2012-02-13 2013-08-14 联想(北京)有限公司 传输方法和电子设备
CN105117111A (zh) * 2015-09-23 2015-12-02 小米科技有限责任公司 虚拟现实交互画面的渲染方法和装置
US9460351B2 (en) * 2012-11-28 2016-10-04 Electronics And Telecommunications Research Institute Image processing apparatus and method using smart glass
CN106293043A (zh) * 2015-06-29 2017-01-04 北京智谷睿拓技术服务有限公司 可视化内容传输控制方法、发送方法、及其装置

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0973129A3 (en) * 1998-07-17 2005-01-12 Matsushita Electric Industrial Co., Ltd. Motion image data compression system
US6351499B1 (en) 1999-12-15 2002-02-26 Iospan Wireless, Inc. Method and wireless systems using multiple antennas and adaptive control for maximizing a communication parameter
KR100619031B1 (ko) 2004-06-11 2006-08-31 삼성전자주식회사 부가 데이터의 인터랙티브한 이용방법 및 장치, 그에 따른수신장치
US8749556B2 (en) * 2008-10-14 2014-06-10 Mixamo, Inc. Data compression for real-time streaming of deformable 3D models for 3D animation
US9897805B2 (en) 2013-06-07 2018-02-20 Sony Interactive Entertainment Inc. Image rendering responsive to user actions in head mounted display
US10585472B2 (en) * 2011-08-12 2020-03-10 Sony Interactive Entertainment Inc. Wireless head mounted display with differential rendering and sound localization
KR101546654B1 (ko) * 2012-06-26 2015-08-25 한국과학기술원 웨어러블 증강현실 환경에서 증강현실 서비스 제공 방법 및 장치
JP6397243B2 (ja) * 2014-07-22 2018-09-26 キヤノン株式会社 表示装置、制御方法、及びプログラム
KR20160092136A (ko) * 2015-01-26 2016-08-04 한국전자통신연구원 가상 데스크탑 제공 방법 및 가상 데스크탑 제공 서버
US9840003B2 (en) 2015-06-24 2017-12-12 Brain Corporation Apparatus and methods for safe navigation of robotic devices
WO2017044795A1 (en) 2015-09-10 2017-03-16 Google Inc. Playing spherical video on a limited bandwidth connection
US20170287097A1 (en) * 2016-03-29 2017-10-05 Ati Technologies Ulc Hybrid client-server rendering in a virtual reality system
US9965899B2 (en) * 2016-04-28 2018-05-08 Verizon Patent And Licensing Inc. Methods and systems for minimizing pixel data transmission in a network-based virtual reality media delivery configuration

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103248883A (zh) * 2012-02-13 2013-08-14 联想(北京)有限公司 传输方法和电子设备
US9460351B2 (en) * 2012-11-28 2016-10-04 Electronics And Telecommunications Research Institute Image processing apparatus and method using smart glass
CN106293043A (zh) * 2015-06-29 2017-01-04 北京智谷睿拓技术服务有限公司 可视化内容传输控制方法、发送方法、及其装置
CN105117111A (zh) * 2015-09-23 2015-12-02 小米科技有限责任公司 虚拟现实交互画面的渲染方法和装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3591502A4

Also Published As

Publication number Publication date
KR20190132434A (ko) 2019-11-27
JP2020520133A (ja) 2020-07-02
AU2017405914A1 (en) 2019-10-17
JP6873268B2 (ja) 2021-05-19
EP3591502B1 (en) 2024-03-13
KR102296139B1 (ko) 2021-08-30
US20200064907A1 (en) 2020-02-27
CN109074152A (zh) 2018-12-21
CA3057180A1 (en) 2018-09-27
CA3057180C (en) 2022-06-28
US11402894B2 (en) 2022-08-02
EP3591502A1 (en) 2020-01-08
CN109074152B (zh) 2021-01-05
EP3591502A4 (en) 2020-05-06
AU2017405914B2 (en) 2021-06-10

Similar Documents

Publication Publication Date Title
US10861215B2 (en) Asynchronous time and space warp with determination of region of interest
WO2018171084A1 (zh) 一种虚拟现实图像发送方法及装置
US11050810B2 (en) Method and apparatus for transmitting and receiving image data for virtual-reality streaming service
US10776992B2 (en) Asynchronous time warp with depth data
CN111052750A (zh) 用于点云流传输的方法和装置
WO2020060984A1 (en) Cross layer traffic optimization for split xr
US20160378177A1 (en) Visualized content transmission control method, sending method and apparatuses thereof
US20210240257A1 (en) Hiding latency in wireless virtual and augmented reality systems
EP2849080B1 (en) Image display method and device
US20220301184A1 (en) Accurate optical flow interpolation optimizing bi-directional consistency and temporal smoothness
US20160378178A1 (en) Visualized content transmission control method, sending method and apparatuses thereof
CN106919376B (zh) 动态画面传输方法、伺服器装置及用户装置
US20230079171A1 (en) Image content transmitting method and device using edge computing service
US20230206575A1 (en) Rendering a virtual object in spatial alignment with a pose of an electronic device
US20230215075A1 (en) Deferred rendering on extended reality (xr) devices
EP4202611A1 (en) Rendering a virtual object in spatial alignment with a pose of an electronic device
KR20240083484A (ko) Vr 모션 예측 장치
KR20240083468A (ko) Vr 모션 예측 장치

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17901675

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 3057180

Country of ref document: CA

ENP Entry into the national phase

Ref document number: 2019552177

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2017901675

Country of ref document: EP

Effective date: 20191004

ENP Entry into the national phase

Ref document number: 2017405914

Country of ref document: AU

Date of ref document: 20170707

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 20197030873

Country of ref document: KR

Kind code of ref document: A