CN106998409A - A kind of image processing method, head-mounted display and rendering apparatus - Google Patents
A kind of image processing method, head-mounted display and rendering apparatus Download PDFInfo
- Publication number
- CN106998409A CN106998409A CN201710169082.6A CN201710169082A CN106998409A CN 106998409 A CN106998409 A CN 106998409A CN 201710169082 A CN201710169082 A CN 201710169082A CN 106998409 A CN106998409 A CN 106998409A
- Authority
- CN
- China
- Prior art keywords
- moment
- information
- attitude information
- head
- mounted display
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
Abstract
The embodiment of the invention discloses a kind of image processing method, head-mounted display and rendering apparatus.One of which method includes：Attitude information of the head-mounted display at the first moment is sent to rendering apparatus；The video data from rendering apparatus is received, video data includes rendering according to attitude information obtaining view data；Attitude information of the head-mounted display at the second moment is obtained, the second moment was later than for the first moment；According to the attitude information at the second moment and the information gap of the attitude information at the first moment, the image that adjustment view data is indicated, so as to obtain the image after the adjustment for display.Using the embodiment of the present invention, the rendered image data obtained by rendering apparatus according to the attitude information at the first moment is adjusted by HMD, without the attitude information at the second moment for adjusting processing is sent to rendering apparatus, allow the view data after adjustment to match head rotation, eliminate because head rotation causes scene changes to produce pattern drafting delay.
Description
Technical field
The present invention relates to field of computer technology, more particularly to a kind of image processing method, head-mounted display and render
Equipment.
Background technology
Virtual reality (Virtual Reality, VR) technology is multimedia technology emerging in recent years.It is that one kind can be with
Create the computer simulation system with the experiencing virtual world.In VR scenes, user by wear a kind of integrated graphics system,
After the head-mounted display (Head Mounted Display, HMD) of optical system and posture tracing system, user can be allowed
Obtain a kind of immersion experience interacted.
In VR transmission of video scenes, use wirelessly is transmitted more.At present, wireless VR video transmitting procedures such as Fig. 1 institutes
Show, by being wirelessly transmitted to rendering apparatus (Application Processor, AP) after HMD collection attitude informations, and AP then roots
Corresponding view data/video data is rendered according to the attitude information received, and is wirelessly back to HMD so that HMD is to image
Data/video data is shown.When AP is being rendered, if a certain frame data render performance deficiency (such as render time is long),
Next frame data will be dropped so that HMD will be shown and previous frame data identical view data.And once head turns
Dynamic, the light of identical image can then fall different parts on the retina, so as to cause float.Now, AP sides are passed through
Graphics processor (Graphics Processing Unit, GPU) carries out secondary rendering processing, that is, the figure directly to the frame
As carrying out displacement spatially and distorting to match head rotation, reduce float.
But due in secondary rendering, it is necessary to be gathered according to the corresponding attitude information of the two field picture and current time HMD
The information gap of attitude information is shifted to match head rotation, that is to say, that need the attitude information that gathers current time without
Line is transmitted to AP sides, and attitude information has propagation delay time in transmitting procedure so that the image after adjustment still can not be matched
Head rotation, therefore, it is impossible to eliminate because of the pattern drafting delay that head rotation causes scene changes to produce.
The content of the invention
Technical problem to be solved of the embodiment of the present invention be there is provided a kind of image processing method, head-mounted display with
And rendering apparatus, the rendered image data obtained by rendering apparatus according to the attitude information at the first moment is adjusted by HMD
It is whole, without the attitude information at the second moment for adjusting processing is sent to rendering apparatus, reduce the transmission of attitude information
Time delay so that the view data after adjustment can match head rotation, is eliminated because head rotation causes what scene changes were produced
Pattern drafting is delayed.
First aspect present invention provides a kind of image processing method, including：Head-mounted display is to rendering apparatus delivery header
Attitude information of the display at the first moment is worn, and receives the video data from rendering apparatus, wherein video data includes root
Obtained view data is rendered according to the attitude information at the first moment, posture of the head-mounted display at the second moment is then obtained
Information, the second moment is later than first moment, according to the attitude information at the second moment and the letter of the attitude information at the first moment
Breath difference adjusts the image that the view data is indicated, so as to obtain the image after the adjustment for display.
In first aspect of the embodiment of the present invention, by HMD to being obtained by rendering apparatus according to the attitude information at the first moment
To rendered image data be adjusted, set without the attitude information at the second moment for adjusting processing is sent to rendering
It is standby, reduce the propagation delay time of attitude information so that the view data after adjustment can match head rotation, eliminate because of head
Rotate the pattern drafting delay for causing scene changes to produce.
With reference in a first aspect, in the first implementation of first aspect, methods described also includes：Head-mounted display to
Rendering apparatus sends the first time information for indicating for the first moment, wherein, the posture at the first time information and the first moment
Information is bound；Video data also include with video data bind the first time information when, head-mounted display by obtain with
The attitude information at the first moment of first time information binding, and the attitude information according to the second moment and first moment
Attitude information information gap adjustment view data indicate image.
With reference to the first implementation of first aspect or first aspect, in second of implementation of first aspect,
The attitude information at first moment includes the attitude angle at first moment, and the attitude information at second moment is including described
The attitude angle at the second moment.
With reference to the first implementation or second of implementation of first aspect of first aspect or first aspect,
In the third implementation of one side, the attitude information according to second moment and the posture at first moment are believed
The information gap of breath, the image that adjustment described image data are indicated, so that the image after being adjusted, including：By information gap correspondence
Transformation matrix be multiplied with the image that described image information is indicated, so as to obtain the image after the adjustment.
With reference to the first implementation of first aspect or first aspect or second of implementation of first aspect or
The third implementation of one side, in the 4th kind of implementation of first aspect, methods described also includes：Head-mounted display
Attitude information of the head-mounted display at the 3rd moment is obtained, the 3rd moment was later than for the first moment, and the second moment and the 3rd moment are not
It is synchronization, and the information gap adjustment view data of the attitude information according to the 3rd moment and the attitude information at the first moment refers to
The image shown, so as to obtain the image after another adjustment for display.Attitude information and the first moment by the 3rd moment
The information gap of attitude information and the information gap of the attitude information at the second moment and the attitude information at the first moment to rendering
One frame image data passes through adjustment of displacement different twice, realizes the interleave processing of view data, and reduces view data
The transmission bandwidth for rendering frame per second, reducing wireless image data.
Second aspect of the present invention provides a kind of image processing method, including：Rendering apparatus, which is received, comes from head-mounted display
The first moment attitude information and with the first moment attitude information binding the first time information, when described first
Carving information is used to indicate for the first moment, is then rendered according to attitude information at the first moment and obtains view data, specifically can be with
For the attitude information according to the first moment, panoramic video/image is rendered, so as to obtain view data.It is most backward described
Head-mounted display sends view data and the first time information bound with described image data, for wearing display device pair
View data is adjusted processing.
In second aspect of the embodiment of the present invention, rendering apparatus is received after the attitude information of head-mounted display, according to the appearance
State information, which is rendered, obtains video data/view data, and sends to head-mounted display, for head-mounted display to the video counts
It is adjusted and shows according to/view data, so that the view data of display can match head rotation, eliminates because of head
Rotate the pattern drafting delay for causing scene changes to produce.
Third aspect present invention provides a kind of head-mounted display, including：Wireless transmission unit, for being sent out to rendering apparatus
Send attitude information of the head-mounted display at the first moment；The wireless transmission unit, is additionally operable to reception and is rendered from described
The video data of equipment, the video data includes being rendered according to the attitude information obtaining view data；Motion sensor list
Member, for obtaining attitude information of the head-mounted display at the second moment, second moment is later than first moment；Figure
As processing unit, during for the attitude information at second moment obtained according to the movement sensor unit and described first
The information gap of the attitude information at quarter, adjusts the image for the described image data instruction that the wireless transmission unit is received, so that
Obtain the image after the adjustment for display.
In the third aspect of the embodiment of the present invention, by HMD to being obtained by rendering apparatus according to the attitude information at the first moment
To rendered image data be adjusted, set without the attitude information at the second moment for adjusting processing is sent to rendering
It is standby, reduce the propagation delay time of attitude information so that the view data after adjustment can match head rotation, eliminate because of head
Rotate the pattern drafting delay for causing scene changes to produce.
With reference to the third aspect, in the first implementation of the third aspect, the head-mounted display also includes：The nothing
Line transmission unit, is additionally operable to send the first time information for indicating first moment, described to the rendering apparatus
One time information and the attitude information at first moment are bound；The video data also includes what is bound with the video data
First time information；Described image processing unit specifically for：Described the obtained according to the movement sensor unit
The information gap of the attitude information at two moment and the attitude information at first moment, adjusts what the wireless transmission unit was received
The image that described image data are indicated.
With reference to the third aspect, in second of implementation of the third aspect, the attitude information at first moment includes
The attitude angle at first moment, the attitude information at second moment includes the attitude angle at second moment.
With reference to second of implementation of the third aspect, in the third implementation of the third aspect, at described image
Manage unit specifically for：The corresponding transformation matrix of information gap is multiplied with the image that described image information is indicated, so as to obtain institute
State the image after adjustment.
With reference to the third implementation of the third aspect, in the 4th kind of implementation of the third aspect, it is described wear it is aobvious
Show that device also includes：The movement sensor unit, is additionally operable to obtain attitude information of the head-mounted display at the 3rd moment, institute
Stating for the 3rd moment is later than first moment, and second moment and the 3rd moment are not synchronizations；At described image
Unit is managed, attitude information and first moment at the 3rd moment obtained according to the movement sensor unit is additionally operable to
Attitude information information gap, adjust the image that the described image data that the wireless transmission unit receives are indicated, thus
Image to after another adjustment for display.Pass through the attitude information and the information of the attitude information at the first moment at the 3rd moment
Difference and the attitude information at the second moment and the information gap of the attitude information at the first moment pass through to the frame image data rendered
Adjustment of displacement different twice, realizes the interleave processing of view data, and reduce view data render frame per second, reduce
The transmission bandwidth of wireless image data.
Fourth aspect present invention provides a kind of rendering apparatus, and the rendering apparatus includes：Wireless transmission unit, for connecing
Receive from head-mounted display the attitude information at the first moment and bound with the attitude information at the first moment the
One time information, first time information is used to indicate for the first moment；Display unit is rendered, for being wirelessly transferred according to
The attitude information at the first moment that unit is received, renders and obtains view data；The wireless transmission unit, is additionally operable to
To the head-mounted display send described in render described image data and the wireless transmission unit that display unit obtains and connect
The first time information bound with described image data received.
In fourth aspect of the embodiment of the present invention, rendering apparatus is received after the attitude information of head-mounted display, according to the appearance
State information, which is rendered, obtains video data/view data, and sends to head-mounted display, for head-mounted display to the video counts
It is adjusted and shows according to/view data, so that the view data of display can match head rotation, eliminates because of head
Rotate the pattern drafting delay for causing scene changes to produce.
In a possible design, the structure of head-mounted display includes processor and transceiver.The processor is used
In the image processing method for performing the offer of first aspect of the embodiment of the present invention.Optionally, memory, the storage can also be included
Device is used to store the application code for supporting head-mounted display to perform the above method, and the processor is configurable for performing
The application program stored in the memory.
In a possible design, the structure of rendering apparatus includes processor and transceiver.The processor is used for
Perform the image processing method that second aspect of the embodiment of the present invention is provided.Optionally, memory, the memory can also be included
The application code for supporting rendering apparatus to perform the above method for storing, it is described that the processor is configurable for execution
The application program stored in memory.
By implementing the embodiment of the present invention, attitude information of the head-mounted display at the first moment is gathered by head-mounted display
And send to rendering apparatus, rendered by rendering apparatus according to the attitude information and obtain video data/view data, then by video
Data/image data back is to head-mounted display, and head-mounted display gathers head-mounted display after the attitude information at the second moment,
According to the information gap of the attitude information at the second moment and the attitude information at the first moment to video data/view data for receiving
It is adjusted.Without the attitude information at the second moment is sent to rendering apparatus, the propagation delay time of attitude information is reduced so that
View data after adjustment can match head rotation, eliminate because the pattern drafting that head rotation causes scene changes to produce prolongs
When.
Brief description of the drawings
Technical scheme in order to illustrate the embodiments of the present invention more clearly or in background technology, below will be to present invention implementation
The accompanying drawing used required in example or background technology is illustrated.
Fig. 1 is a kind of logical construction schematic diagram of wireless VR transmission of video provided in an embodiment of the present invention；
Fig. 2 (a) is a kind of interface schematic diagram of ATW Rendering operations provided in an embodiment of the present invention；
Fig. 2 (b) is the interface schematic diagram that a kind of attitude information provided in an embodiment of the present invention sends and received；
Fig. 3 is a kind of schematic flow sheet of image processing method provided in an embodiment of the present invention；
Fig. 4 is a kind of interface schematic diagram of attitude angle description provided in an embodiment of the present invention；
Fig. 5 is a kind of interface schematic diagram of Image Adjusting operation provided in an embodiment of the present invention；
Fig. 6 is a kind of structural representation of head-mounted display provided in an embodiment of the present invention；
Fig. 7 is a kind of structural representation of rendering apparatus provided in an embodiment of the present invention；
Fig. 8 is a kind of structural representation for head-mounted display that another embodiment of the present invention is provided；
Fig. 9 is a kind of structural representation for rendering apparatus that another embodiment of the present invention is provided.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is described.
Term " first ", " second ", " the 3rd " in description and claims of this specification and the accompanying drawing and "
Four " etc. be to be used to distinguish different objects, rather than for describing particular order.In addition, term " comprising " and " having " and it
Any deformation, it is intended that covering non-exclusive is included.For example contain the process of series of steps or unit, method, be
The step of system, product or equipment are not limited to list or unit, but the step of alternatively also include not listing or list
Member, or alternatively also include for the intrinsic other steps of these processes, method, product or equipment or unit.
Referenced herein " embodiment " is it is meant that the special characteristic, structure or the characteristic that describe can be wrapped in conjunction with the embodiments
In at least one embodiment containing the present invention.The phrase, which occurs, in each position in the description might not each mean identical
Embodiment, nor the independent or alternative embodiment with other embodiments mutual exclusion.Those skilled in the art explicitly with it is hidden
Understand, embodiment described herein can be combined with other embodiments likes.
HMD in the embodiment of the present invention can be the display devices such as VR glasses, eye tracker, for gathering information, at calculating
Reason and display；Rendering apparatus in the embodiment of the present invention can calculate the application processor handled to provide virtual reality,
Can be other types of computing terminal (Proxy), such as mobile phone (cellphone), smart mobile phone (smartphone), calculating
Machine (computer), tablet personal computer (tablet computer), personal digital assistant (personal digital
Assistant, PDA), mobile internet device (mobile Internet device, MID) etc..
The wireless transmission method referred in the embodiment of the present invention can be ZigBee protocol (Zigbee), bluetooth
(Bluetooth), Wireless Fidelity (WIreless-Fidelity, WIFI), infrared data tissue (Infrared Data
Association, IrDA), ultra-wideband (Ultra WideBand), wireless near field communication (Near Field
Communication, NFC), it is WiMedia, global positioning system (Global Positioning System, GPS), wireless
139 and private radio systems etc..
The technical scheme of the embodiment of the present invention is applied to various communication systems, such as global system for mobile telecommunications (Global System
Of Mobile communication, GSM) system, CDMA (Code Division Multiple Access, CDMA)
System, WCDMA (Wideband Code Division Multiple Access, WCDMA) system, Long Term Evolution
(Long-Term Evolution, LTE) system, mimo system etc..
The application scenarios of the embodiment of the present invention are described below.In VR scenes, existing mobile terminal is limited to
Processor ability, some big games or video class application can not be by placing the place that processing unit completes VR in head-mounted display
Reason, thus for a part need it is a large amount of calculate or energy-conservation VR applications solution be computing unit be placed on desktop computer,
On mobile phone or game console, head-mounted display is connected by wired mode, head-mounted display weight is alleviated while to use
Family brings ultimate attainment audiovisual experience.But this kind of split type VR systems are due to needing wired connection so that when user moves extremely
Inconvenience, therefore propose to solve the puzzlement that wired connection is brought by the way of being wirelessly transferred.And existing wireless VR transmission of video
Conventional method is as shown in figure 1, HMD is gathered after attitude information by being wirelessly transmitted to rendering apparatus (such as AP), and AP is then received
The attitude information simultaneously renders corresponding video data/view data, and by video data/view data wireless backhaul to HMD,
So that the video that HMD is indicated video data is shown.In VR scenes, the renewal of image has to same with Vsync signals
Step, still, if a certain frame renders performance deficiency (such as render time is long), next frame will be dropped so that HMD will be shown
With previous frame identical image.Once and head rotation, the light of identical image can then fall different parts on the retina, from
And cause float.In order to reduce float, now, secondary rendering processing is carried out by the GPU of AP sides, such as Fig. 2 (a) institutes
Show, L1, L2 and L3 are continuous three frame image data of left eye, R1, R2 and R3 are continuous three frame image data of right eye, generally first
Render and right eye image data is rendered after left eye image data, the left eye image data of same frame and right eye image data can be made
For a frame image data.In the processing of the first frame, right and left eyes image data render is completed in time, is sent directly to HMD progress
It has been shown that, and in the processing of the second frame, render and fail completion in time, now occur that picture is trembled if doing nothing
It is dynamic, by carrying out displacement spatially to the second two field picture in AP sides and distorting to match head rotation, reduce float.But
Due in secondary rendering, it is necessary to according to the corresponding attitude information of the two field picture and current time the HMD attitude information gathered
Information gap is shifted to match head rotation, that is to say, that need the attitude information for gathering current time to be wirelessly transmitted to
AP, and there is propagation delay time in transmitting procedure in attitude information, shown in such as Fig. 2 (b), if HMD is gathered at T0, T1, T2, T3 moment
To attitude information be respectively MU0, MU1, MU2 and MU3, there is propagation delay time when being transmitted due to MU by HMD to AP so that AP
MU0, MU1 and MU2 are just received at T1, T2 and T3 moment.By taking MU0 as an example, if AP carries out wash with watercolours at the T1 moment according to MU0
Dye, and rendered image data A is obtained at the T2 moment (render there is also time delay), then will be to A because A can not match head movement
Carry out secondary rendering, that is to say, that, it is necessary to solve the attitude information MU2 attitude informations corresponding with A that T2 moment HMD is collected
MU0 information gap, the and in fact, attitude information that AP is arrived in T2 receptions is MU1, then that now AP is solved is MU1 with
MU0 information gap, therefore, although reduce the shake of picture, but still the pattern drafting of float generation can not be eliminated and prolong
When.
In embodiments of the present invention, it is concurrent in the attitude information at the first moment by head-mounted display collection head-mounted display
Rendering apparatus is delivered to, is rendered by rendering apparatus according to the attitude information and obtains video data/view data, then by video counts
Head-mounted display is back to according to/view data, head-mounted display gathers head-mounted display after the attitude information at the second moment, root
Video data/the view data received is entered with the information gap of the attitude information at the first moment according to the attitude information at the second moment
Row adjustment.Without the attitude information at the second moment is sent to rendering apparatus, the propagation delay time of attitude information is reduced so that adjust
View data after whole can match head rotation, eliminate because the pattern drafting that head rotation causes scene changes to produce prolongs
When.
Image processing method is described in detail below by way of the embodiment shown in Fig. 3-Fig. 9.
Referring to Fig. 3, Fig. 3 is a kind of schematic flow sheet of image processing method provided in an embodiment of the present invention.Below with
It is described exemplified by head-mounted display and rendering apparatus both sides, methods described includes：
Step S101：Head-mounted display sends attitude information of the head-mounted display at the first moment to rendering apparatus.
I.e. specifically, a portion image that human eye is currently seen in the panoramic picture of attitude information VR scenes
The corresponding attitude information for wearing display device.By taking FOV (angle of visual field) as an example, panoramic picture is 360 °F of OV of level, vertically
FOV in 180 °F of OV, VR scenes is usually that the image that human eye can be seen in 90 ° of level, vertical 90 °, i.e. VR scenes is panorama
1/8 (90*90/360/180) in image, in head-mounted display rotation process, this 1/8 image is also change.Namely
Say, 1/8 image of display is one-to-one with the attitude information for wearing display device.Wherein, the posture at first moment
Information includes the attitude angle at first moment.
Generally, object can be measured by Inertial Measurement Unit (Inertial measurement unit, IMU) sensor
Attitude information.Wherein, an IMU sensor contains the accelerometer of three single shafts and the gyroscope of three single shafts, accelerates
Degree meter detection object founds the acceleration signal of three axles in carrier coordinate system unification and independence, and gyroscope detects carrier relative to carrier coordinate
The angular velocity signal of system, by measuring the angular speed and acceleration of object in three dimensions, and calculates with this appearance of object
State.The calculation method can be preset algorithm, not limit specifically.
In specific implementation, by setting IMU sensors just can directly obtain the attitude information of equipment in a device.Example
Such as, a BMI160 IMU is integrated with intel chips intel Curie, three axles that it encapsulates 16 extremely low power dissipations add
Speedometer and three-axis gyroscope, therefore directly can measure using intel Curie module and resolve the attitude information of oneself.
In embodiments of the present invention, IMU sensors are arranged in head-mounted display, and posture letter can be just obtained by head-mounted display
Breath, and the attitude information of acquisition is sent to rendering apparatus.
Optionally, when head-mounted display is rotated, measured attitude information changes constantly.
In one embodiment, head-mounted display also performs step S1011：Head-mounted display sends to rendering apparatus and used
In the first time information for indicating first moment, first time information and the attitude information at first moment are tied up
It is fixed.
Specifically, the time information can be timestamp.The timestamp (timestamp) is typically a character sequence
Row, uniquely identify the time at a certain moment.Continue real-time process because head-mounted display collection attitude information is one, because
This can stab unique mark attitude information passage time.
In embodiments of the present invention, it regard information at the time of indicating for the first moment as the first time information, such as very first time
Stamp.Because head-mounted display is according to the speed continuation of collection collection attitude information, when being added by the attitude information to collection
Between stab to recognize targeted attitude information.
In one embodiment, head-mounted display also performs step S1012：Head-mounted display is by the posture of the collection
Information is with indicating that information is added in caching at the time of each attitude information.
Specifically, head-mounted display is cached the attitude information of collection with corresponding temporal information, can be with mapping
The form of table is cached, as shown in table 1.Table 1 is a kind of attitude information and the mapping table of timestamp in caching, is passed through
The acquisition time that timestamp is shown can just find corresponding attitude information.When head-mounted display receives targeted attitude information
Timestamp when being moment 1, search determine that corresponding targeted attitude information is attitude information 1 in the buffer.
Table 1
| Attitude information | Timestamp |
| Attitude information 1 | Moment 1 |
| Attitude information 2 | Moment 2 |
| Attitude information 3 | Moment 3 |
| Attitude information 4 | Moment 4 |
Step S102：Rendering apparatus receives the attitude information at the first moment from head-mounted display.
Further, rendering apparatus also performs step S1021：Rendering apparatus is received to be believed with the posture at the first moment
Cease the first time information of binding.
Step S103：Rendering apparatus renders according to the attitude information at the first moment and obtains view data.
Specifically, the attitude information described in the basis at the first moment, renders and obtains view data, it is possible to understand that
It is that rendering apparatus is rendered according to the attitude information at the first moment to panoramic video/image, so as to generate video data/figure
As data.Wherein, described render (Render) refers to attribute and method being added to corresponding component, then by component addition pair
The process for the container answered.For example, in PS (Adobe Photoshop), render and refer to color, the attribute such as size, which is added to, works as
The process of preceding painting canvas, wherein it is possible to which painting canvas is regarded as container, there are many widgets the inside, and each component is to that should have oneself
Attribute.
In specific implementation, the attitude information that rendering apparatus is collected according to IMU sensors renders right and left eyes visual angle correspondence
Image.Rendering order can first to render left-eye image, after render eye image, and adopt and rendered in a like fashion.
In one embodiment, because head-mounted display continues to gather attitude information in real time, and rendering apparatus then root
Rendered according to the frame per second that renders of itself, therefore, rendering apparatus is rendered to multiple attitude informations, so as to generate render video
Data, the render video data include multiple image data.
For example, the speed of head-mounted display collection attitude information is 1000/second, and the frame per second that renders of rendering apparatus is
100HZ, then in this 1 second, rendering apparatus selects 100 attitude informations to carry out wash with watercolours from 1000 attitude informations received
Dye, so as to generate render video data, the render video data of the generation include 100 frame image datas.Wherein, attitude information
Selection mode can be that 1 is randomly choosed from every 10 attitude informations.
Optionally, after the step S103, step S1031 is also performed：Rendering apparatus by described image data with it is described
First time information is bound.
In one embodiment, rendering apparatus is corresponding to each frame image data binding in the render video of generation
Time information.The information at the time of time information is the corresponding attitude information per frame image data.For example, rendering for generation is regarded
Frequency includes 5 frame image datas, and this 5 frame image data is according to 1-attitude information of attitude information 10 this 10 postures received
Attitude information 1, attitude information 2, attitude information 5, attitude information 8 and attitude information 10 in information are generated, and attitude information
1st, attitude information 2, attitude information 5, attitude information 8 and attitude information 10 distinguish corresponding time information for T1, T2, T5, T8
And T10, therefore, this 5 frame image data is bound into T1, T2, T5, T8 and T10 this 5 time informations respectively, to recognize these figures
As the corresponding attitude information of data.
Step S104：Rendering apparatus sends described image data to the head-mounted display and tied up with described image data
The first fixed time information.
Step S105：Head-mounted display receives the video data from the rendering apparatus, and the video data includes root
Rendered according to the attitude information and obtain view data.
In one embodiment, after the step S105, step S1051 is also performed：Head-mounted display is received and institute
State the first time information of the view data binding in video data.
Specifically, video data includes an at least frame image data, it will be used for what is adjusted in an at least frame image data
View data is as destination image data, and the corresponding attitude information of destination image data is targeted attitude information.
Wherein, it can be to obtain information at the time of the targeted attitude information is carried to obtain the targeted attitude information, such as
Timestamp, then searches the corresponding targeted attitude information of the timestamp in the buffer.
Step S106：Head-mounted display obtains attitude information of the head-mounted display at the second moment, when described second
It is later than first moment quarter.
Specifically, due to the attitude information of head-mounted display continuous collecting head-mounted display, appearance is identified by time information
State information.Wherein, the attitude information at second moment includes the attitude angle at second moment.
Wherein, the attitude angle of object in three dimensions can be represented with spin matrix, Eulerian angles, four elements etc..With
Exemplified by Eulerian angles, for a referential in three dimensions, the orientation of any coordinate system can with three Eulerian angles come
Performance.Wherein, three Eulerian angles be respectively nutational angle a, angle of precession (i.e. angle of precession) b and angle of rotation r composition, such as：(a, r, b).
For example, as shown in figure 4, xyz is the reference axis (reference axis of the attitude information before head rotation) of referential, XYZ
For the reference axis of the sample attitude information after head rotation, the intersecting lens of x/y plane and X/Y plane is the line of nodes, uses English alphabet
(N) identify.So, zxz can be defined so along the Eulerian angles of rule：A is the angle of x-axis and the line of nodes, and b is the folder of z-axis and Z axis
Angle, r is the line of nodes and the angle of X-axis, wherein, a=α, b=β, r=γ.That is, α, β, γ are the sample after head rotation
This attitude angle.
Step S107：Head-mounted display is according to the attitude information at second moment and the attitude information at first moment
Information gap, the image that adjustment described image data are indicated, so as to obtain the image after the adjustment for display.
Specifically, head-mounted display obtains the attitude information at the first moment bound with first time information, root
According to the attitude information and the information gap of the attitude information at the first moment at the second moment, the image that adjustment view data is indicated.
In specific implementation, when the attitude information includes attitude angle, by the corresponding transformation matrix of information gap and the figure
As the image that information is indicated is multiplied, so as to obtain the image after the adjustment.Wherein, during described information difference namely two differences
The posture for carving head-mounted display is poor.In a kind of feasible embodiment, by the attitude angle at second moment and described first
The attitude angle at moment is subtracted each other, so as to obtain the information gap at two moment.As shown in Figure 4, believe xyz as the posture at the first moment
The reference axis of breath, XYZ as the attitude information at the second moment reference axis, using xyz as reference axis, then, α, β, γ be two
Information gap between moment.
Using xyz reference axis as reference axis, when around x, y, the matrix that z-axis rotates θ (α=θ, β=θ, γ=θ) respectively is：
So, the transformation matrix of corresponding Eulerian angles is：
Image information indicate the corresponding source matrix of image beWherein xy is represented on the image that image information is indicated
The coordinate of each pixel, that is to say, that Matrix C 1, C2 ... the Cn of multiple pixels are contained in C, then, respectively by C1, C2 ... Cn
It is multiplied with Fix, obtains processing array A1, A2 ... An to each pixel, wherein：
Indicated by resulting each An be adjustment after image on each pixel so that the image after being adjusted.
Optionally, methods described also includes：Obtain attitude information of the head-mounted display at the 3rd moment, the described 3rd
Moment is later than first moment, and second moment and the 3rd moment are not synchronizations；
According to the attitude information at the 3rd moment and the information gap of the attitude information at first moment, the figure is adjusted
As the image that data are indicated, so as to obtain the image after another adjustment for display.
Specifically, wear display device use with step S107 identicals adjustment mode to the attitude information at the 3rd moment with
View data indicated by the information gap of the attitude information at the first moment is adjusted, and is specifically repeated no more.
As shown in figure 5, the view data that PicA, PicB are rendered by rendering apparatus, PicA0, PicA1 are head-mounted display
The two field pictures data corresponding to PicA after adjustment, PicB0, PicB1 are then to correspond to PicB after head-mounted display is adjusted
Two field pictures data.
In embodiments of the present invention, it is concurrent in the attitude information at the first moment by head-mounted display collection head-mounted display
Rendering apparatus is delivered to, is rendered by rendering apparatus according to the attitude information and obtains video data/view data, then by video counts
Head-mounted display is back to according to/view data, head-mounted display gathers head-mounted display after the attitude information at the second moment, root
Video data/the view data received is entered with the information gap of the attitude information at the first moment according to the attitude information at the second moment
Row adjustment.Without the attitude information at the second moment is sent to rendering apparatus, the propagation delay time of attitude information is reduced so that adjust
View data after whole can match head rotation, on the one hand reduce because the picture that head rotation causes scene changes to produce is painted
System delay, on the other hand, shown in such as Fig. 2 (b), due to the picture number for generating secondary rendering in T2 moment rendering apparatus (such as AP)
According to A1 transmit to head-mounted display (such as) HMD shown when, the propagation delay time based on view data/video data and cause AP
Just received at the T3 moment and display image data A1, and A1 is generated according to MU2 and MU0 information gap, and be in fact
Elimination pattern drafting delay, the view data that now HMD is shown should be obtained according to MU3 and MU0 information gap, therefore,
In the embodiment of the present invention, HMD can be adjusted using the attitude information at newest moment to rendered image data, further matching
Head rotation and the view data of display.Therefore, using the embodiment of the present invention, it can eliminate because head rotation causes scene to become
Change the pattern drafting delay produced.In addition, the attitude information and the information gap of the attitude information at the first moment that passed through for the 3rd moment
And second the moment attitude information and the information gap of the attitude information at the first moment two are passed through to the frame image data rendered
Secondary different adjustment of displacement, realizes the interleave processing of view data, and reduce view data render frame per second, reduce figure
As the transmission bandwidth of data.
The above-mentioned method for illustrating the embodiment of the present invention, is provided below the device of the embodiment of the present invention.
Referring to Fig. 6, Fig. 6 is a kind of structural representation of head-mounted display 10 provided in an embodiment of the present invention.Such as Fig. 6 institutes
Show, the equipment includes wireless transmission unit 11, movement sensor unit 12 and graphics processing unit 13, wherein, unit
It is described in detail as follows.
Wireless transmission unit 11, for sending attitude information of the head-mounted display at the first moment to rendering apparatus；
The wireless transmission unit 11, is additionally operable to receive the video data from the rendering apparatus, the video data
View data is obtained including being rendered according to the attitude information；
Movement sensor unit 12, for obtaining attitude information of the head-mounted display at the second moment, described second
Moment is later than first moment；
Graphics processing unit 13, for the posture letter at second moment obtained according to the movement sensor unit 12
The information gap of breath and the attitude information at first moment, adjusts the described image data that the wireless transmission unit 11 is received
The image of instruction, so as to obtain the image after the adjustment for display.
Optionally, the head-mounted display 10 also includes：
The wireless transmission unit 11, be additionally operable to the rendering apparatus send for indicate first moment first
Time information, first time information and the attitude information at first moment are bound；
The video data also includes first time information bound with the video data；
The movement sensor unit 12, is additionally operable to obtain described at the first moment with first time information binding
Attitude information,
Described image processing unit 13 specifically for：Second moment obtained according to the movement sensor unit 12
Attitude information and first moment attitude information information gap, adjust that the wireless transmission unit 11 receives is described
The image that view data is indicated.
Optionally, the attitude information at first moment includes the attitude angle at first moment, second moment
Attitude information includes the attitude angle at second moment.
Optionally, described image processing unit 13 specifically for：
The corresponding transformation matrix of information gap is multiplied with the image that described image information is indicated, so as to obtain after the adjustment
Image.
The optional head-mounted display 10 also includes：
The movement sensor unit 12, is additionally operable to obtain attitude information of the head-mounted display at the 3rd moment, institute
Stating for the 3rd moment is later than first moment, and second moment and the 3rd moment are not synchronizations；
Described image processing unit 13, is additionally operable to the 3rd moment obtained according to the movement sensor unit 12
The information gap of attitude information and the attitude information at first moment, adjusts the figure that the wireless transmission unit 11 is received
As the image that data are indicated, so as to obtain the image after another adjustment for display.
In embodiments of the present invention, it is concurrent in the attitude information at the first moment by head-mounted display collection head-mounted display
Rendering apparatus is delivered to, is rendered by rendering apparatus according to the attitude information and obtains video data/view data, then by video counts
Head-mounted display is back to according to/view data, head-mounted display gathers head-mounted display after the attitude information at the second moment, root
Video data/the view data received is entered with the information gap of the attitude information at the first moment according to the attitude information at the second moment
Row adjustment.Without the attitude information at the second moment is sent to rendering apparatus, the propagation delay time of attitude information is reduced so that adjust
View data after whole can match head rotation, on the one hand reduce because the picture that head rotation causes scene changes to produce is painted
System delay, on the other hand, shown in such as Fig. 2 (b), due to the picture number for generating secondary rendering in T2 moment rendering apparatus (such as AP)
According to A1 transmit to head-mounted display (such as) HMD shown when, the propagation delay time based on view data/video data and cause AP
Just received at the T3 moment and display image data A1, and A1 is generated according to MU2 and MU0 information gap, and be in fact
Elimination pattern drafting delay, the view data that now HMD is shown should be obtained according to MU3 and MU0 information gap, therefore,
In the embodiment of the present invention, HMD can be adjusted using the attitude information at newest moment to rendered image data, further matching
Head rotation and the view data of display.Therefore, using the embodiment of the present invention, it can eliminate because head rotation causes scene to become
Change the pattern drafting delay produced.In addition, the attitude information and the information gap of the attitude information at the first moment that passed through for the 3rd moment
And second the moment attitude information and the information gap of the attitude information at the first moment two are passed through to the frame image data rendered
Secondary different adjustment of displacement, realizes the interleave processing of view data, and reduce view data render frame per second, reduce figure
As the transmission bandwidth of data.
Referring to Fig. 7, Fig. 7 is a kind of structural representation of rendering apparatus 20 provided in an embodiment of the present invention.Such as Fig. 7 institutes
Show, the equipment includes wireless transmission unit 21 and rendering display unit 22, wherein, unit is described in detail as follows.
Wireless transmission unit 21, for receive from head-mounted display the first moment attitude information and with it is described
The first time information bound in the attitude information at the first moment, first time information is used to indicate for the first moment；
Display unit 22 is rendered, for the posture described in being received according to the wireless transmission unit 21 at the first moment
Information, renders and obtains view data；
The wireless transmission unit 21, be additionally operable to the head-mounted display send described in render what display unit 22 was obtained
The first time information bound with described image data that described image data and the wireless transmission unit 21 are received.
In embodiments of the present invention, it is concurrent in the attitude information at the first moment by head-mounted display collection head-mounted display
Rendering apparatus is delivered to, is rendered by rendering apparatus according to the attitude information and obtains video data/view data, then by video counts
Head-mounted display is back to according to/view data, head-mounted display gathers head-mounted display after the attitude information at the second moment, root
Video data/the view data received is entered with the information gap of the attitude information at the first moment according to the attitude information at the second moment
Row adjustment.Without the attitude information at the second moment is sent to rendering apparatus, the propagation delay time of attitude information is reduced so that adjust
View data after whole can match head rotation, on the one hand reduce because the picture that head rotation causes scene changes to produce is painted
System delay, on the other hand, shown in such as Fig. 2 (b), due to the picture number for generating secondary rendering in T2 moment rendering apparatus (such as AP)
According to A1 transmit to head-mounted display (such as) HMD shown when, the propagation delay time based on view data/video data and cause AP
Just received at the T3 moment and display image data A1, and A1 is generated according to MU2 and MU0 information gap, and be in fact
Elimination pattern drafting delay, the view data that now HMD is shown should be obtained according to MU3 and MU0 information gap, therefore,
In the embodiment of the present invention, HMD can be adjusted using the attitude information at newest moment to rendered image data, further matching
Head rotation and the view data of display.Therefore, using the embodiment of the present invention, it can eliminate because head rotation causes scene to become
Change the pattern drafting delay produced.
Head-mounted display in embodiment illustrated in fig. 8 can be realized with the head-mounted display shown in Fig. 3 and Fig. 6.Such as Fig. 8 institutes
Show, be that, the embodiments of the invention provide a kind of structural representation of head-mounted display, the head-mounted display 1000 shown in Fig. 8 is wrapped
Include：Processor 1001, transceiver 1004 and sensor 1005.Wherein, processor 1001 is connected with transceiver 1004, such as by total
Line 1002 is connected.Optionally, the data sending device 1000 can also include memory 1003.It should be noted that actual
Transceiver 1004 is not limited to one in, and the structure of the data sending device 1000 does not constitute the limit to the embodiment of the present invention
It is fixed.
Wherein, processor 1001 is applied in the embodiment of the present invention, for realizing the graphics processing unit 13 shown in Fig. 6
Function.Transceiver 1004 includes transmitter and receiver, and transceiver 1004 is applied in the embodiment of the present invention, for realizing Fig. 6
The function of shown wireless transmission unit 11.Sensor 1005 includes motion sensor, and it is real that sensor 1005 is applied to the present invention
Apply in example, the function for realizing the movement sensor unit 12 shown in Fig. 6.
Processor 1001 can be central processing unit (Central Processing Unit, CPU), general processor, number
Word signal transacting (Digital Signal Processing, DSP), integrated circuit (Application Specific
Integrated Circuit, ASIC), field programmable gate array (Field-Programmable Gate Array,
FPGA) or other PLDs, transistor logic, hardware component or its any combination.It can be realized
Or various exemplary logic blocks of the execution with reference to described by the disclosure of invention, module and circuit.Processor 1001
It can be the combination for realizing computing function, for example, be combined comprising one or more microprocessors, combination of DSP and microprocessor etc.
Deng.
Bus 1002 may include a path, and information is transmitted between said modules.Bus 1002 can be that external components are mutual
Connect standard (Peripheral Component Interconnect, PCI) bus or EISA (Extended
Industry Standard Architecture, EISA) bus etc..Bus 1002 can be divided into address bus, data/address bus,
Controlling bus etc..For ease of representing, only represented in Fig. 8 with a thick line, it is not intended that only one bus or a type
Bus.
Memory 1003 can be read-only storage (read-only memory, ROM) or can store static information and refer to
The other kinds of static storage device of order, random access memory (random access memory, RAM) or can be stored
Information and the other kinds of dynamic memory or EEPROM of instruction
(Electrically Erasable Programmable Read-Only Memory, EEPROM), read-only optical disc (Compact
Disc Read-Only Memory, CD-ROM) or other optical disc storages, laser disc storage (including compression laser disc, laser disc, light
Dish, Digital Versatile Disc, Blu-ray Disc etc.), magnetic disk storage medium or other magnetic storage apparatus or can be used in carrying or
Storage with instruction or data structure form desired program code and can by computer access any other medium, but
Not limited to this.
Optionally, memory 1003 is used to store the application code for performing the present invention program, and by processor 1001
To control to perform.Processor 1001 is used to perform the application code stored in memory 1003, is implemented with realizing shown in Fig. 6
The action for the head-mounted display that example is provided.
A kind of computer-readable storage medium is additionally provided in embodiments of the present invention, for saving as above-mentioned head-mounted display institute
Computer software instructions, it, which is included, is used to perform program of the above-mentioned aspect designed by head-mounted display.
Rendering apparatus in embodiment illustrated in fig. 9 can be realized with the rendering apparatus shown in Fig. 7.As shown in figure 9, being this hair
Bright embodiment provides a kind of structural representation of rendering apparatus, and the rendering apparatus 2000 shown in Fig. 9 includes：The He of processor 2001
Transceiver 2004.Wherein, processor 2001 is connected with transceiver 2004, is such as connected by bus 2002.Optionally, rendering apparatus
2000 can also include memory 2003.It should be noted that transceiver 2004 is not limited to one in practical application, the data connect
The structure of receiving apparatus 2000 does not constitute the restriction to the embodiment of the present invention.
Wherein, processor 2001 is applied in the embodiment of the present invention, and display unit 22 is rendered shown in Fig. 7 for realizing
Function.Transceiver 2004 includes Receiver And Transmitter, and transceiver 2004 is applied in the embodiment of the present invention, for realizing Fig. 7
The function of shown wireless transmission unit 21.
Processor 2001 can be CPU, general processor, DSP, ASIC, FPGA or other PLDs, crystalline substance
Body pipe logical device, hardware component or its any combination.It can realize or perform with reference to described by the disclosure of invention
Various exemplary logic blocks, module and circuit.Processor 2001 can also be the combination for realizing computing function, for example, wrap
Containing one or more microprocessors combination, the combination of DSP and microprocessor etc..
Bus 2002 may include a path, and information is transmitted between said modules.Bus 2002 can be pci bus or
Eisa bus etc..Bus 2002 can be divided into address bus, data/address bus, controlling bus etc..For ease of representing, only used in Fig. 9
One thick line is represented, it is not intended that only one bus or a type of bus.
Memory 2003 can be ROM or can store the other kinds of static storage device of static information and instruction, RAM
Or can storage information and the other kinds of dynamic memory or EEPROM, CD-ROM or other CDs of instruction
Storage, laser disc storage (including compressing laser disc, laser disc, laser disc, Digital Versatile Disc, Blu-ray Disc etc.), magnetic disk storage medium
Or other magnetic storage apparatus or can be used in carrying or store with instruction or data structure form desired program generation
Code and can by computer access any other medium, but not limited to this.
Optionally, memory 2003 is used to store the application code for performing the present invention program, and by processor 2001
To control to perform.Processor 2001 is used to perform the application code stored in memory 2003, is implemented with realizing shown in Fig. 7
The action for the rendering apparatus that example is provided.
A kind of computer-readable storage medium is additionally provided in embodiments of the present invention, for saving as used in above-mentioned rendering apparatus
Computer software instructions, it is included, and to be used to perform above-mentioned aspect be the program designed by rendering apparatus.
Although combining each embodiment herein, invention has been described, however, implementing the present invention for required protection
During, those skilled in the art are by checking the accompanying drawing, disclosure and appended claims, it will be appreciated that and it is real
Other changes of the existing open embodiment.In the claims, the word of " comprising " (comprising) one is not excluded for other compositions
Part or step, "a" or "an" are not excluded for multiple situations.Single processor or other units can realize claim
In some functions enumerating.Mutually different has been recited in mutually different dependent some measures, it is not intended that these are arranged
Apply to combine and produce good effect.
It will be understood by those skilled in the art that embodiments of the invention can be provided as method, device (equipment) or computer journey
Sequence product.Therefore, in terms of the present invention can be using complete hardware embodiment, complete software embodiment or combination software and hardware
The form of embodiment.Moreover, the present invention can be used in one or more calculating for wherein including computer usable program code
The computer program that machine usable storage medium is implemented on (including but is not limited to magnetic disk storage, CD-ROM, optical memory etc.)
The form of product.Computer program is stored/distributed in suitable medium, is provided together with other hardware or as the one of hardware
Part, it would however also be possible to employ other distribution forms, such as passes through the wired or wireless telecommunication systems of Internet or other.
The present invention be with reference to the method for the embodiment of the present invention, device (equipment) and computer program product flow chart with/
Or block diagram is described.It should be understood that can by each flow in computer program instructions implementation process figure and/or block diagram and/
Or square frame and the flow in flow chart and/or block diagram and/or the combination of square frame.These computer program instructions can be provided
To the processor of all-purpose computer, special-purpose computer, Embedded Processor or other programmable data processing devices to produce one
Individual machine so that produced by the instruction of computer or the computing device of other programmable data processing devices for realizing
The device for the function of being specified in one flow of flow chart or multiple flows and/or one square frame of block diagram or multiple square frames.
These computer program instructions, which may be alternatively stored in, can guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works so that the instruction being stored in the computer-readable memory, which is produced, to be included referring to
Make the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one square frame of block diagram or
The function of being specified in multiple square frames.
These computer program instructions can be also loaded into computer or other programmable data processing devices so that in meter
Series of operation steps is performed on calculation machine or other programmable devices to produce computer implemented processing, thus in computer or
The instruction performed on other programmable devices is provided for realizing in one flow of flow chart or multiple flows and/or block diagram one
The step of function of being specified in individual square frame or multiple square frames.
Although with reference to specific features and embodiment, invention has been described, it is clear that, do not departing from this hair
In the case of bright spirit and scope, various modifications and combinations can be carried out to it.Correspondingly, the specification and drawings are only institute
The exemplary illustration of the invention that attached claim is defined, and be considered as covered in the scope of the invention any and all and repair
Change, change, combining or equivalent.Obviously, those skilled in the art can to the present invention carry out it is various change and modification without
Depart from the spirit and scope of the present invention.So, if the present invention these modifications and variations belong to the claims in the present invention and its
Within the scope of equivalent technologies, then the present invention is also intended to comprising including these changes and modification.
Claims (12)
1. a kind of image processing method, methods described is performed by head-mounted display, it is characterised in that methods described includes：
Attitude information of the head-mounted display at the first moment is sent to rendering apparatus；
The video data from the rendering apparatus is received, the video data includes rendering according to the attitude information obtaining figure
As data；
Attitude information of the head-mounted display at the second moment is obtained, second moment is later than first moment；
According to the attitude information at second moment and the information gap of the attitude information at first moment, described image number is adjusted
According to the image of instruction, so as to obtain the image after the adjustment for display.
2. the method as described in claim 1, it is characterised in that methods described also includes：Being sent to the rendering apparatus is used for
First time information at first moment is indicated, first time information and the attitude information at first moment are bound；
The video data also includes first time information bound with the video data；
The information gap of the attitude information according to second moment and the attitude information at first moment, adjusts the figure
As the image that data are indicated, including：
The attitude information at the first moment bound with first time information is obtained, according to the appearance at second moment
State information and the information gap of the attitude information at first moment, the image that adjustment described image data are indicated.
3. method as claimed in claim 1 or 2, it is characterised in that the attitude information at first moment includes described first
The attitude angle at moment, the attitude information at second moment includes the attitude angle at second moment.
4. the method as described in any one of claims 1 to 3, it is characterised in that described to be believed according to the posture at second moment
The information gap of breath and the attitude information at first moment, the image that adjustment described image data are indicated, so that after being adjusted
Image, including：
The corresponding transformation matrix of information gap is multiplied with the image that described image information is indicated, so as to obtain the figure after the adjustment
Picture.
5. the method according to any one of Claims 1-4, it is characterised in that methods described also includes：
Attitude information of the head-mounted display at the 3rd moment is obtained, the 3rd moment is later than first moment, described
Second moment and the 3rd moment are not synchronizations；
According to the attitude information at the 3rd moment and the information gap of the attitude information at first moment, described image number is adjusted
According to the image of instruction, so as to obtain the image after another adjustment for display.
6. a kind of image processing method, methods described is performed by rendering apparatus, it is characterised in that methods described includes：
Receive being tied up in the attitude information at the first moment and with the attitude information at the first moment from head-mounted display
The first fixed time information, first time information is used to indicate for the first moment；
According to the attitude information at the first moment, render and obtain view data；
The first time information for sending described image data to the head-mounted display and being bound with described image data.
7. a kind of head-mounted display, it is characterised in that the head-mounted display includes：
Wireless transmission unit, for sending attitude information of the head-mounted display at the first moment to rendering apparatus；
The wireless transmission unit, is additionally operable to receive the video data from the rendering apparatus, the video data includes root
Rendered according to the attitude information and obtain view data；
Movement sensor unit, for obtaining attitude information of the head-mounted display at the second moment, the evening at the second moment
In first moment；
Graphics processing unit, for the attitude information at second moment that is obtained according to the movement sensor unit with it is described
The information gap of the attitude information at the first moment, adjusts the figure for the described image data instruction that the wireless transmission unit is received
Picture, so as to obtain the image after the adjustment for display.
8. head-mounted display as claimed in claim 7, it is characterised in that the head-mounted display also includes：
The wireless transmission unit, is additionally operable to believe to the first moment that the rendering apparatus is sent for indicating first moment
Breath, first time information and the attitude information at first moment are bound；
The video data also includes first time information bound with the video data；
The movement sensor unit, is additionally operable to obtain the posture at the first moment bound with first time information
Information,
Described image processing unit specifically for：The posture letter at second moment obtained according to the movement sensor unit
The information gap of breath and the attitude information at first moment, the described image data that the adjustment wireless transmission unit is received refer to
The image shown.
9. head-mounted display as claimed in claim 7 or 8, it is characterised in that the attitude information at first moment includes institute
The attitude angle at the first moment is stated, the attitude information at second moment includes the attitude angle at second moment.
10. the head-mounted display as described in any one of claim 7 to 9, it is characterised in that described image processing unit is specifically used
In：
The corresponding transformation matrix of information gap is multiplied with the image that described image information is indicated, so as to obtain the figure after the adjustment
Picture.
11. the head-mounted display as described in any one of claim 7 to 10, it is characterised in that the head-mounted display also includes：
The movement sensor unit, is additionally operable to obtain attitude information of the head-mounted display at the 3rd moment, the described 3rd
Moment is later than first moment, and second moment and the 3rd moment are not synchronizations；
Described image processing unit, is additionally operable to the attitude information at the 3rd moment obtained according to the movement sensor unit
With the information gap of the attitude information at first moment, the described image data instruction that the wireless transmission unit is received is adjusted
Image, so as to obtain the image after another adjustment for display.
12. a kind of rendering apparatus, it is characterised in that the rendering apparatus includes：
Wireless transmission unit, for receive from head-mounted display the attitude information at the first moment and with described first
First time information of the attitude information binding at moment, first time information is used to indicate for the first moment；
Display unit is rendered, for the attitude information described in being received according to the wireless transmission unit at the first moment, wash with watercolours
Dye obtains view data；
The wireless transmission unit, be additionally operable to the head-mounted display send described in render the described image that display unit is obtained
The first time information bound with described image data that data and the wireless transmission unit are received.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710169082.6A CN106998409B (en) | 2017-03-21 | 2017-03-21 | Image processing method, head-mounted display and rendering equipment |
| PCT/CN2018/078131 WO2018171421A1 (en) | 2017-03-21 | 2018-03-06 | Image processing method, head mounted display, and rendering device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710169082.6A CN106998409B (en) | 2017-03-21 | 2017-03-21 | Image processing method, head-mounted display and rendering equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106998409A true CN106998409A (en) | 2017-08-01 |
| CN106998409B CN106998409B (en) | 2020-11-27 |
Family
ID=59431689
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710169082.6A Active CN106998409B (en) | 2017-03-21 | 2017-03-21 | Image processing method, head-mounted display and rendering equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN106998409B (en) |
| WO (1) | WO2018171421A1 (en) |
Cited By (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107835404A (en) * | 2017-11-13 | 2018-03-23 | 歌尔科技有限公司 | Method for displaying image, equipment and system based on wear-type virtual reality device |
| WO2018171421A1 (en) * | 2017-03-21 | 2018-09-27 | 华为技术有限公司 | Image processing method, head mounted display, and rendering device |
| CN108829627A (en) * | 2018-05-30 | 2018-11-16 | 青岛小鸟看看科技有限公司 | Synchronisation control means and system between virtual reality device |
| CN109271022A (en) * | 2018-08-28 | 2019-01-25 | 北京七鑫易维信息技术有限公司 | A kind of display methods of VR equipment, device, VR equipment and storage medium |
| WO2019076356A1 (en) * | 2017-10-20 | 2019-04-25 | 腾讯科技（深圳）有限公司 | Video processing method based on virtual reality scene, server, and virtual reality device and system |
| CN109725730A (en) * | 2019-01-02 | 2019-05-07 | 京东方科技集团股份有限公司 | Wear display equipment and its driving method, display system and its driving method |
| CN109743626A (en) * | 2019-01-02 | 2019-05-10 | 京东方科技集团股份有限公司 | A kind of image display method, image processing method and relevant device |
| CN109753158A (en) * | 2019-01-11 | 2019-05-14 | 京东方科技集团股份有限公司 | A kind of the determination method and controlling terminal of the delay of VR equipment |
| CN109782912A (en) * | 2019-01-03 | 2019-05-21 | 京东方科技集团股份有限公司 | Method, apparatus, medium and electronic equipment for measuring device delay |
| WO2019183914A1 (en) * | 2018-03-30 | 2019-10-03 | Intel Corporation | Dynamic video encoding and view adaptation in wireless computing environments |
| CN110505518A (en) * | 2019-08-05 | 2019-11-26 | 青岛小鸟看看科技有限公司 | One kind wearing display equipment, data transmission method and system |
| WO2020043105A1 (en) * | 2018-08-28 | 2020-03-05 | 华为技术有限公司 | Image display method, device, and system |
| CN111064985A (en) * | 2018-10-16 | 2020-04-24 | 北京凌宇智控科技有限公司 | System, method and device for realizing video streaming |
| CN111065053A (en) * | 2018-10-16 | 2020-04-24 | 北京凌宇智控科技有限公司 | System and method for video streaming |
| CN111243027A (en) * | 2020-02-28 | 2020-06-05 | 京东方科技集团股份有限公司 | Delay measurement method, device and system |
| CN111338546A (en) * | 2020-02-28 | 2020-06-26 | 歌尔科技有限公司 | Method for controlling head-mounted display device, terminal and storage medium |
| CN111586391A (en) * | 2020-05-07 | 2020-08-25 | 中国联合网络通信集团有限公司 | Image processing method, device and system |
| CN111813230A (en) * | 2020-09-14 | 2020-10-23 | 芋头科技（杭州）有限公司 | Interaction method and device on AR glasses |
| WO2020238741A1 (en) * | 2019-05-24 | 2020-12-03 | 华为技术有限公司 | Image processing method, related device and computer storage medium |
| CN112073632A (en) * | 2020-08-11 | 2020-12-11 | 联想(北京)有限公司 | Image processing method, apparatus and storage medium |
| WO2020248442A1 (en) * | 2019-06-13 | 2020-12-17 | 歌尔股份有限公司 | Image processing method and apparatus for electronic device, and electronic device |
| CN112104855A (en) * | 2020-09-17 | 2020-12-18 | 联想(北京)有限公司 | Image processing method and device |
| WO2021218683A1 (en) * | 2020-04-30 | 2021-11-04 | 华为技术有限公司 | Image processing method and apparatus |
| CN113728615A (en) * | 2020-03-31 | 2021-11-30 | 深圳市大疆创新科技有限公司 | Image processing method, image processing device, user equipment, aircraft and system |
| CN114168096A (en) * | 2021-12-07 | 2022-03-11 | 深圳创维新世界科技有限公司 | Display method, system, mobile terminal and storage medium of output picture |
| US11500455B2 (en) | 2018-10-16 | 2022-11-15 | Nolo Co., Ltd. | Video streaming system, video streaming method and apparatus |
| WO2022242704A1 (en) * | 2021-05-19 | 2022-11-24 | 闪耀现实(无锡)科技有限公司 | Method for refreshing screen of head-mounted display device and head-mounted display device |
| WO2023050533A1 (en) * | 2021-09-29 | 2023-04-06 | 歌尔股份有限公司 | Control method for head-mounted device, and picture rendering method |
| WO2023217198A1 (en) * | 2022-05-11 | 2023-11-16 | 闪耀现实(无锡)科技有限公司 | Method and apparatus for displaying picture on head-mounted display device, and electronic device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105809144A (en) * | 2016-03-24 | 2016-07-27 | 重庆邮电大学 | Gesture recognition system and method adopting action segmentation |
| CN105807798A (en) * | 2014-12-31 | 2016-07-27 | 上海乐相科技有限公司 | Method and device for controlling jittering phenomenon of head-mounted intelligent glasses |
| CN105847785A (en) * | 2016-05-09 | 2016-08-10 | 上海乐相科技有限公司 | Image processing method, device and system |
| CN106354258A (en) * | 2016-08-30 | 2017-01-25 | 上海乐相科技有限公司 | Image display method and device of virtual reality equipment |
| CN106502427A (en) * | 2016-12-15 | 2017-03-15 | 北京国承万通信息科技有限公司 | Virtual reality system and its scene rendering method |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20160139461A (en) * | 2015-05-27 | 2016-12-07 | 엘지전자 주식회사 | Head mounted display and, the controlling method thereof |
| CN106385625A (en) * | 2016-09-29 | 2017-02-08 | 宇龙计算机通信科技(深圳)有限公司 | Image intermediate frame generation method and device |
| CN106998409B (en) * | 2017-03-21 | 2020-11-27 | 华为技术有限公司 | Image processing method, head-mounted display and rendering equipment |
-
2017
- 2017-03-21 CN CN201710169082.6A patent/CN106998409B/en active Active
-
2018
- 2018-03-06 WO PCT/CN2018/078131 patent/WO2018171421A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105807798A (en) * | 2014-12-31 | 2016-07-27 | 上海乐相科技有限公司 | Method and device for controlling jittering phenomenon of head-mounted intelligent glasses |
| CN105809144A (en) * | 2016-03-24 | 2016-07-27 | 重庆邮电大学 | Gesture recognition system and method adopting action segmentation |
| CN105847785A (en) * | 2016-05-09 | 2016-08-10 | 上海乐相科技有限公司 | Image processing method, device and system |
| CN106354258A (en) * | 2016-08-30 | 2017-01-25 | 上海乐相科技有限公司 | Image display method and device of virtual reality equipment |
| CN106502427A (en) * | 2016-12-15 | 2017-03-15 | 北京国承万通信息科技有限公司 | Virtual reality system and its scene rendering method |
Cited By (47)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018171421A1 (en) * | 2017-03-21 | 2018-09-27 | 华为技术有限公司 | Image processing method, head mounted display, and rendering device |
| CN109698949A (en) * | 2017-10-20 | 2019-04-30 | 腾讯科技（深圳）有限公司 | Method for processing video frequency, device and system based on virtual reality scenario |
| CN109698949B (en) * | 2017-10-20 | 2020-08-21 | 腾讯科技（深圳）有限公司 | Video processing method, device and system based on virtual reality scene |
| WO2019076356A1 (en) * | 2017-10-20 | 2019-04-25 | 腾讯科技（深圳）有限公司 | Video processing method based on virtual reality scene, server, and virtual reality device and system |
| CN107835404A (en) * | 2017-11-13 | 2018-03-23 | 歌尔科技有限公司 | Method for displaying image, equipment and system based on wear-type virtual reality device |
| US11375244B2 (en) | 2018-03-30 | 2022-06-28 | Intel Corporation | Dynamic video encoding and view adaptation in wireless computing environments |
| WO2019183914A1 (en) * | 2018-03-30 | 2019-10-03 | Intel Corporation | Dynamic video encoding and view adaptation in wireless computing environments |
| CN108829627B (en) * | 2018-05-30 | 2020-11-20 | 青岛小鸟看看科技有限公司 | Synchronous control method and system between virtual reality devices |
| CN108829627A (en) * | 2018-05-30 | 2018-11-16 | 青岛小鸟看看科技有限公司 | Synchronisation control means and system between virtual reality device |
| KR20210046759A (en) * | 2018-08-28 | 2021-04-28 | 후아웨이 테크놀러지 컴퍼니 리미티드 | Image display methods, devices and systems |
| US20210181506A1 (en) * | 2018-08-28 | 2021-06-17 | Huawei Technologies Co., Ltd. | Image Display Method, Apparatus, and System |
| KR102503337B1 (en) * | 2018-08-28 | 2023-02-23 | 후아웨이 테크놀러지 컴퍼니 리미티드 | Image display method, apparatus and system |
| CN109271022A (en) * | 2018-08-28 | 2019-01-25 | 北京七鑫易维信息技术有限公司 | A kind of display methods of VR equipment, device, VR equipment and storage medium |
| WO2020043105A1 (en) * | 2018-08-28 | 2020-03-05 | 华为技术有限公司 | Image display method, device, and system |
| CN110868581A (en) * | 2018-08-28 | 2020-03-06 | 华为技术有限公司 | Image display method, device and system |
| CN109271022B (en) * | 2018-08-28 | 2021-06-11 | 北京七鑫易维信息技术有限公司 | Display method and device of VR equipment, VR equipment and storage medium |
| US11662580B2 (en) * | 2018-08-28 | 2023-05-30 | Huawei Technologies Co., Ltd. | Image display method, apparatus, and system to reduce display latency |
| US11500455B2 (en) | 2018-10-16 | 2022-11-15 | Nolo Co., Ltd. | Video streaming system, video streaming method and apparatus |
| CN111065053A (en) * | 2018-10-16 | 2020-04-24 | 北京凌宇智控科技有限公司 | System and method for video streaming |
| CN111064985A (en) * | 2018-10-16 | 2020-04-24 | 北京凌宇智控科技有限公司 | System, method and device for realizing video streaming |
| WO2020140657A1 (en) * | 2019-01-02 | 2020-07-09 | Boe Technology Group Co., Ltd. | Head mounted display apparatus, virtual reality display system and driving method thereof |
| US11574613B2 (en) | 2019-01-02 | 2023-02-07 | Beijing Boe Optoelectronics Technology Co., Ltd. | Image display method, image processing method and relevant devices |
| US11275248B2 (en) | 2019-01-02 | 2022-03-15 | Beijing Boe Optoelectronics Technology Co., Ltd. | Head mounted display apparatus, virtual reality display system and driving method thereof |
| CN109743626A (en) * | 2019-01-02 | 2019-05-10 | 京东方科技集团股份有限公司 | A kind of image display method, image processing method and relevant device |
| CN109725730A (en) * | 2019-01-02 | 2019-05-07 | 京东方科技集团股份有限公司 | Wear display equipment and its driving method, display system and its driving method |
| CN109782912A (en) * | 2019-01-03 | 2019-05-21 | 京东方科技集团股份有限公司 | Method, apparatus, medium and electronic equipment for measuring device delay |
| CN109753158A (en) * | 2019-01-11 | 2019-05-14 | 京东方科技集团股份有限公司 | A kind of the determination method and controlling terminal of the delay of VR equipment |
| WO2020238741A1 (en) * | 2019-05-24 | 2020-12-03 | 华为技术有限公司 | Image processing method, related device and computer storage medium |
| WO2020248442A1 (en) * | 2019-06-13 | 2020-12-17 | 歌尔股份有限公司 | Image processing method and apparatus for electronic device, and electronic device |
| CN110505518A (en) * | 2019-08-05 | 2019-11-26 | 青岛小鸟看看科技有限公司 | One kind wearing display equipment, data transmission method and system |
| CN111243027B (en) * | 2020-02-28 | 2023-06-23 | 京东方科技集团股份有限公司 | Delay measurement method, device and system |
| CN111243027A (en) * | 2020-02-28 | 2020-06-05 | 京东方科技集团股份有限公司 | Delay measurement method, device and system |
| CN111338546A (en) * | 2020-02-28 | 2020-06-26 | 歌尔科技有限公司 | Method for controlling head-mounted display device, terminal and storage medium |
| CN113728615A (en) * | 2020-03-31 | 2021-11-30 | 深圳市大疆创新科技有限公司 | Image processing method, image processing device, user equipment, aircraft and system |
| WO2021218683A1 (en) * | 2020-04-30 | 2021-11-04 | 华为技术有限公司 | Image processing method and apparatus |
| CN111586391A (en) * | 2020-05-07 | 2020-08-25 | 中国联合网络通信集团有限公司 | Image processing method, device and system |
| CN111586391B (en) * | 2020-05-07 | 2022-07-08 | 中国联合网络通信集团有限公司 | Image processing method, device and system |
| CN112073632A (en) * | 2020-08-11 | 2020-12-11 | 联想(北京)有限公司 | Image processing method, apparatus and storage medium |
| CN111813230B (en) * | 2020-09-14 | 2021-03-19 | 芋头科技（杭州）有限公司 | Interaction method and device on AR glasses |
| CN111813230A (en) * | 2020-09-14 | 2020-10-23 | 芋头科技（杭州）有限公司 | Interaction method and device on AR glasses |
| CN112104855A (en) * | 2020-09-17 | 2020-12-18 | 联想(北京)有限公司 | Image processing method and device |
| CN112104855B (en) * | 2020-09-17 | 2022-05-31 | 联想(北京)有限公司 | Image processing method and device |
| WO2022242704A1 (en) * | 2021-05-19 | 2022-11-24 | 闪耀现实(无锡)科技有限公司 | Method for refreshing screen of head-mounted display device and head-mounted display device |
| WO2023050533A1 (en) * | 2021-09-29 | 2023-04-06 | 歌尔股份有限公司 | Control method for head-mounted device, and picture rendering method |
| CN114168096A (en) * | 2021-12-07 | 2022-03-11 | 深圳创维新世界科技有限公司 | Display method, system, mobile terminal and storage medium of output picture |
| CN114168096B (en) * | 2021-12-07 | 2023-07-25 | 深圳创维新世界科技有限公司 | Display method and system of output picture, mobile terminal and storage medium |
| WO2023217198A1 (en) * | 2022-05-11 | 2023-11-16 | 闪耀现实(无锡)科技有限公司 | Method and apparatus for displaying picture on head-mounted display device, and electronic device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106998409B (en) | 2020-11-27 |
| WO2018171421A1 (en) | 2018-09-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106998409A (en) | A kind of image processing method, head-mounted display and rendering apparatus | |
| US10453175B2 (en) | Separate time-warping for a scene and an object for display of virtual reality content | |
| CN109743626B (en) | Image display method, image processing method and related equipment | |
| CN106919248A (en) | It is applied to the content transmission method and equipment of virtual reality | |
| JP2019507509A (en) | Adjust video rendering speed of virtual reality content and process stereoscopic images | |
| US10871822B2 (en) | Image generation apparatus, image generation system, and image generation method | |
| CN110488977A (en) | Virtual reality display methods, device, system and storage medium | |
| US20200134923A1 (en) | Generating and Modifying Representations of Objects in an Augmented-Reality or Virtual-Reality Scene | |
| CN112198959A (en) | Virtual reality interaction method, device and system | |
| US20180011681A1 (en) | Information processing method and system for executing the information processing method | |
| RU2723920C1 (en) | Support of augmented reality software application | |
| CN113568170A (en) | Virtual image generation system and method of operating the same | |
| CN111988598B (en) | Visual image generation method based on far and near view layered rendering | |
| US10891890B1 (en) | Asymmetric pixel operation for compensating lens optics limitations | |
| CN110351480A (en) | Image processing method, device and electronic equipment for electronic equipment | |
| CN110537208A (en) | Head-mounted display and method | |
| CN106507128A (en) | Virtual reality imagery transmission method, player method and the program using the method | |
| CN104216533B (en) | A kind of wear-type virtual reality display based on DirectX9 | |
| CN109462745A (en) | A kind of white balancing treatment method and mobile terminal | |
| CN107835404A (en) | Method for displaying image, equipment and system based on wear-type virtual reality device | |
| CN105653035B (en) | virtual reality control method and system | |
| CN105721856A (en) | Remote image display method for near-to-eye display | |
| CN105260441A (en) | 3D human body data model scene introduction method and process | |
| CN109076101A (en) | Cloud platform control method, equipment and computer readable storage medium | |
| US11328487B2 (en) | Method and program for providing augmented reality image by using depth data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |