TWM548794U - Head mounted display and display system - Google Patents

Abstract

A head mounted display (HMD) is provided. The HMD includes: a display panel; a motion sensor; and a display controller. The motion sensor is configured to detect motion of the HMD to generate sensor data. The display controller is configured to retrieve an input image from a host, and obtain a target object from the input image. The display controller calculates a motion pattern of the target object in the input image according to the sensor data and predicts a motion trajectory of the target object. When the display controller determines that the trajectory is larger than a predetermined threshold, the display controller updates a sub-image associated with the target object in the input image, and displays the updated input image on the display panel.

Description

Head mounted display and display system

This creation relates to display devices, and more particularly to a head mounted display and display system.

With the development of technology, Head Mounted Display is also often used to view virtual reality images. However, a head-mounted display usually needs to receive a virtual reality image from a host, and the head-mounted display and the host are connected through a data transmission line (for example, an HDMI cable). In addition, when the user wears a head-mounted display to view the virtual reality picture, the user's head may not be completely stationary, and there may be displacement or rotation. At this time, the master needs to recalculate the virtual reality picture corresponding to the above movement according to the situation, and then transmit the picture to the head mounted display through the data transmission line. It takes a considerable amount of time to recalculate the picture and transfer the picture through the data transmission line, which means that there may be a picture delay.

Because the user's eyes are quite close to the display panel when wearing the head-mounted display, it is easier to check the picture delay.

Therefore, there is a need for a head mounted display and display system to solve the above problems.

The present invention provides a head mounted display comprising: a display panel; a motion sensor for detecting movement of the head mounted display to generate a sensor data; and a display controller for The master obtains an input image and obtains one of the target objects in the input image, wherein the display controller further calculates the target object in the input image according to the sensor data from the motion sensor. a moving pattern, and predicting a movement trajectory of the target object; wherein when the display controller determines that the movement trajectory exceeds a predetermined threshold, the display controller locally updates the sub-image of the target object in the input image, And the updated input image is played on the display panel.

The present invention further provides a display system comprising: a main control terminal; and a head mounted display, wherein the main control terminal is connected to the head mounted display through a data transmission line; wherein the head mounted display is from a The master obtains an input image and obtains one of the target objects in the input image, wherein the head mounted display is further calculated according to one sensor data generated by one of the mobile sensors in the head mounted display. Inputting a movement pattern of the target object in the image, and predicting a movement trajectory of the target object; wherein when the head mounted display determines that the movement trajectory exceeds a predetermined threshold, the head mounted display is locally updated Input a sub-image of the target object in the image, and play the updated input image on the head-mounted display.

100‧‧‧Display system

110‧‧‧Master

111‧‧‧ Processor

112‧‧‧ memory unit

112A‧‧‧ volatile memory

112B‧‧‧Non-volatile memory

113‧‧‧Data transmission interface

120‧‧‧ head mounted display

121‧‧‧Display controller

122‧‧‧ memory unit

123‧‧‧Data transmission interface

124‧‧‧ display panel

124L‧‧‧Left eye display panel

124R‧‧‧right eye display panel

125‧‧‧left eye lens

126‧‧‧right eye lens

127‧‧‧Mobile Sensor

130‧‧‧ data transmission line

131‧‧‧Shell

210-214‧‧‧ Target object

S310-S350‧‧‧Steps

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a block diagram showing a display system in accordance with an embodiment of the present invention.

Figure 2 is a diagram showing the movement of a target object in a picture in accordance with an embodiment of the present invention.

Figure 3 is a flow chart showing a method of data mapping for a storage device in accordance with an embodiment of the present invention.

In order to make the above objects, features and advantages of the present invention more comprehensible, the following detailed description of the preferred embodiments and the accompanying drawings are set forth below.

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a block diagram showing a display system in accordance with an embodiment of the present invention.

As shown in FIG. 1 , the display system 100 includes a main control terminal 110 and a head mounted display 120 . The main control terminal 110 and the head mounted display 120 are connected through a data transmission line 130 . For example, display system 100 can be used for visual display of virtual reality or simulated reality. For example, the host 110 can generate or capture an image to be played (for example, a stereoscopic image, a panoramic image, etc.) from an image source, and transmit the head-mounted display of the image to be played through the data transmission line 130 for playback.

In an embodiment, the host 110 includes a processor 111, a memory unit 112, and a data transmission interface 113. The memory unit 112 can store one or more image files, and the processor 111 can select one of the image files, and perform corresponding image/video processing on the image file to obtain a virtual reality video picture to be played. For example, the memory unit 112 can include a volatile memory 112A and a non-volatile memory 112B. Among them, volatile memory can be, for example For a static random access memory (SRAM) or a dynamic random access memory (DRAM), the non-volatile memory 112B can be, for example, a hard disk drive, a solid state hard disk, or a flash memory, but this creation Not limited to this.

The data transmission interface 113 is configured to receive image data (for example, a left eye image and a right eye image, for example, an RGB image) from the processor 111 of the host 110, and transmit the image data to the head mounted display through the display controller 121. 120 to play. For example, the data transmission interface 113 can be a High Definition Multimedia Interface (HDMI) or a DisplayPort (DP) interface, but the creation is not limited thereto.

The head mounted display 120 includes a display controller 121, a memory unit 122, a data transmission interface 123, a display panel 124, a left lens head 125, a right lens 126, a motion sensor 127, and a housing 131. . The display panel 124, the left lens head 125, and the right lens head 126 are disposed in the housing 131 in a predetermined optical arrangement for allowing the user of the head mounted display 120 to feel the virtual reality picture. The memory unit 122 can be, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM) for storing input images from the host and serving as an image buffer for outputting images.

The housing 131 can include a strap or other auxiliary device (not shown) for the user to wear on the head to view the screen through the head mounted display 120. In some embodiments, the display panel 124 further includes a left-eye display panel 124L and a right-eye display panel 124R for playing the left-eye image and the right-eye image, respectively.

The display controller 121 is for controlling the display screen on the display panel 124. For example, the display controller 121 receives image data from the host terminal 110 via the data transmission interface 123, for example, including a left eye image and a right eye image. general In this case, the frame rate of the screen played by the head mounted display 120 needs to reach 60 frames/second, so that the user does not obviously feel the jitter of the screen when viewing the virtual reality movie or playing the game using the head mounted display 120. delay.

The data transmission interface 123 is configured to receive image data (for example, a left eye image and a right eye image, for example, an RGB image) from the data transmission interface of the host 110, and transmit the image data to the head mounted display through the display controller 121. The display panel 124 of 120 plays. For example, the data transmission interface 113 can be a High Definition Multimedia Interface (HDMI) or a DisplayPort (DP) interface, but the creation is not limited thereto.

It should be noted that the data transmission line 130 is compliant with the data transmission protocol and multimedia transmission protocol supported by the data transmission interfaces 113 and 123, and the data transmission interfaces 113 and 123 can be the same data transmission protocol interface, for example, the same HDMI or DisplayPort interface. Or can be converted to a compatible interface through the data transmission line 130, for example, can be converted from HDMI interface to DisplayPort, or converted from DisplayPort to HDMI interface.

The mobile sensor 127 detects the movement of the head mounted display 120 and reports the detected movement information of the head mounted display 120 to the display controller 121 and the host terminal 110. For example, the motion sensor 127 can detect the movement of the head mounted display 120 on the 9-axis, that is, the displacement and rotation on the x-axis, the y-axis, and the z-axis (ie, roll/pitch/yaw rotation, respectively). ).

Figure 2 is a diagram showing the movement of a target object in a picture in accordance with an embodiment of the present invention. In an embodiment, if a target object 210 is included in the scene 200 to move from left to right in the scene 200, then the game is played differently. The image of the scene of time 200 will see objects 211, 212, 213, 214, etc. at different locations. In fact, these objects 210~214 may belong to the same target object.

Furthermore, the display controller 121 can recognize the target object 210 from the screen of the scene 200 and predict its movement trajectory. In some embodiments, the target object 210 can be one of a plurality of segments cut in the image of the scene 200.

For example, to predict the movement trajectory of each segment, based on the location and speed of the previous region. In other words, the trajectory T of each segment can be expressed as one of its previous position L and the previous velocity V: T(t)=L(t ₀ )+V(tt ₀ )

Where t ₀ is the tag of the previous time and t is the tag of the future time. It is possible to use the position and velocity of the target object at the current time t0 to predict the movement trajectory of the target object at a future time point t.

In general, because of the size, weight, heat dissipation, etc., the computing power and system resources of the head mounted display 120 tend to be less than that of the host terminal 110, and the display controller 121 of the head mounted display 120 can be used, for example, on site. A field-programmable gate array (FPGA) is implemented with a corresponding digital signal processor and associated circuitry. In some embodiments, the display controller 121 of the head mounted display 120 can also be implemented using a system-on-chip (SOC).

In some embodiments, the display controller 121 can analyze the input image from the host 110 and retrieve the target object from the input image, such as through foreground/background segmentation. Reason. In some embodiments, when the host 110 transmits the input image to the head mounted display 120 through the data transmission line 130, the corresponding object information in the image is also transmitted, for example, according to the corresponding depth image of the input image or A binary mask image is used to obtain at least one target object in the image. After the target object is obtained, the display controller 121 can calculate the motion pattern or the amount of movement of the target object based on the sensor data from the motion sensor 127. The movement pattern can be, for example, a motion vector of the target object or a relative movement speed, rotational speed, or displacement.

Next, the display controller 121 predicts a movement trajectory of one of the target objects. If the amount of movement/rotation of the target object is greater than a predetermined threshold within a unit time interval (eg, 16 ms, but not limited), the display controller 121 will control the image of the self-control terminal 110 and play it. On the display panel 124. If the amount of movement/rotation of the target object within a unit time interval (for example, 16 ms, but not limited) is not greater than a predetermined threshold, the display controller 121 performs a screen update of the local end.

For example, because the master 110 receives the sensor data from the motion sensor 127, it takes time to calculate a picture corresponding to the movement of the head mounted display 120. In addition, there is a slight delay in transmitting data through the data transmission line 130, depending on the length of the data transmission line 130. In addition, when the user views the image by using the head mounted display 120, since the distance between the image and the eye is relatively close, the user can easily detect that the picture is delayed. Therefore, the display controller 121 can perform the screen update of the local end when the target object moves less, move the trajectory prediction with the target object of a lower computation amount, and update only the image of the region of the target object in the screen, and Can not be noticed by the user The screen update is performed in the late case, that is, the updated image on the local end of the head mounted display 120 is played on the display panel 124 of the head mounted display 120.

Figure 3 is a flow chart showing a method of updating a picture for a head mounted display in accordance with an embodiment of the present invention.

At step S310, sensor data is taken from the motion sensor of the head mounted display.

At step S320, the amount of movement from one of the target objects in the master is calculated.

In step S330, one of the target objects is predicted to move the trajectory.

In step S335, it is determined whether the movement trajectory exceeds a predetermined threshold. If yes, go to step S350, if no, go to step S340. The display controller 121 can perform the local-end picture update when the target object moves less, move the trajectory prediction with the target object with a lower computational amount, and update the picture about the target object only in the picture, and can be used. The screen update can be performed without being aware of the screen delay. If the movement trajectory is too large (ie, exceeds the predetermined threshold), the range that can be calculated by the display controller 121 is exceeded, and the display controller 121 plays the input image from the main control directly on the head mounted display.

In step S340, the target object in the input image is updated by the head mounted display, and the updated image is played on the head mounted display.

In step S350, the input image from the host terminal is directly played on the head mounted display.

The present invention is disclosed in the above preferred embodiments, and is not intended to limit the scope of the present invention. Any person having ordinary skill in the art can make some changes without departing from the spirit and scope of the present invention. Retouching The scope of protection of this creation is subject to the definition of the scope of the patent application.

100‧‧‧Display system

110‧‧‧Master

111‧‧‧ Processor

112‧‧‧ memory unit

113‧‧‧Data transmission interface

112A‧‧‧ volatile memory

112B‧‧‧Non-volatile memory

120‧‧‧ head mounted display

121‧‧‧Display controller

122‧‧‧ memory unit

123‧‧‧Data transmission interface

124‧‧‧ display panel

124L‧‧‧Left eye display panel

124R‧‧‧right eye display panel

125‧‧‧left eye lens

126‧‧‧right eye lens

127‧‧‧Mobile Sensor

130‧‧‧ data transmission line

131‧‧‧Shell

Claims (10)

A head mounted display comprising: a display panel; a motion sensor for detecting movement of the head mounted display to generate a sensor data; and a display controller for obtaining from a master Inputting an image and obtaining a target object in the input image, wherein the display controller further calculates a movement pattern of the target object in the input image according to the sensor data from the motion sensor. And predicting a movement trajectory of the target object; wherein when the display controller determines that the movement trajectory exceeds a predetermined threshold, the display controller locally updates the sub-image of the target object in the input image, and displays the sub-image on the target object The panel plays the updated input image. The head mounted display of claim 1, wherein the display controller is configured to predict the target object after a predetermined time interval according to a position and a speed of the target object in the input image. Move the track. The head mounted display of claim 1, wherein the display controller performs a foreground/background separation process to retrieve the target object from the input image. The head mounted display of claim 1, wherein the display controller obtains the target object according to a corresponding depth map or a binary mask image of the input image from the main control end. The head mounted display of claim 1, wherein the display controller plays the input image directly on the display panel when the display controller further determines that the movement trajectory does not exceed a predetermined threshold. A display system includes: a master terminal; and a head mounted display, wherein the master terminal is connected to the head mounted display through a data transmission line; wherein the head mounted display is obtained from a master terminal Inputting an image and obtaining a target object in the input image, wherein the head mounted display is further calculated in the input image according to one sensor data generated by a mobile sensor in the head mounted display One of the target objects moves the pattern and predicts a movement trajectory of the target object; wherein when the head mounted display determines that the movement trajectory exceeds a predetermined threshold, the head mounted display is locally updated in the input image A sub-image of the target object, and the updated input image is played on the head mounted display. The display system of claim 6, wherein the head mounted display predicts the movement of the target object after a predetermined time interval according to a position and a speed of the target object in the input image. Track. The display system of claim 6, wherein the head mounted display performs a foreground/background separation process to retrieve the target object from the input image. The display system of claim 6, wherein the head mounted display obtains the target object according to a corresponding depth map or a binary mask image of the input image from the main control end. The display system of claim 6, wherein the head mounted display directly plays the input image on the display panel when the head mounted display determines that the movement trajectory does not exceed a predetermined threshold.