CN109445596B - Integrated mixed reality head-mounted display system - Google Patents
Integrated mixed reality head-mounted display system Download PDFInfo
- Publication number
- CN109445596B CN109445596B CN201811301980.3A CN201811301980A CN109445596B CN 109445596 B CN109445596 B CN 109445596B CN 201811301980 A CN201811301980 A CN 201811301980A CN 109445596 B CN109445596 B CN 109445596B
- Authority
- CN
- China
- Prior art keywords
- image
- information
- mounted display
- mixed reality
- head
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02B—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO BUILDINGS, e.g. HOUSING, HOUSE APPLIANCES OR RELATED END-USER APPLICATIONS
- Y02B20/00—Energy efficient lighting technologies, e.g. halogen lamps or gas discharge lamps
- Y02B20/40—Control techniques providing energy savings, e.g. smart controller or presence detection
Abstract
The embodiment of the invention provides an integrated mixed reality head-mounted display system, which comprises head-mounted display equipment, handle equipment and an upper computer, wherein an inertial measurement unit in the handle equipment measures handle posture information of the handle equipment and sends the information to the upper computer; the key unit acquires operation information of a user and sends the operation information to the upper computer; the lamp strip is used for displaying images displayed by the lamp strip; the head-mounted display device acquires a lamp band display image and transmits display data information of the head-mounted display device to the upper computer; the upper computer is arranged in the head-mounted display device, generates a mixed reality image according to the handle posture information, the operation information, the lamp band display image and the display data information, and sends the mixed reality image to the head-mounted display device for display. The method and the device realize the mixed display of the virtual image and the real image, thereby realizing the mixed interaction of the user to the virtual world and the real scene, improving the sense of reality of the user experience, and having small volume, convenient carrying and wide application range.
Description
Technical Field
The invention relates to the technical field of mixed reality, in particular to an integrated mixed reality head-mounted display system.
Background
The mixed reality, virtual reality and augmented reality technologies are high-new technologies which are developed in recent years, and a virtual world of a three-dimensional space is generated by means of computer graphics simulation, so that a user can interact with objects in the three-dimensional space in real time in visual, audible and other aspects without limitation. In the virtual reality, a user interacts with a pure virtual world, in the augmented reality, the user can utilize virtual content to superimpose and augment the real world, a mixed reality technology is further developed and fused with a virtual reality technology, and the technology builds an interactive feedback information loop among the virtual world, the real world and the user by introducing digital reality scene information into a virtual environment so as to enhance the sense of reality of user experience.
Most of the existing head-mounted display devices are virtual reality head-mounted display devices, users can only interact with the pure virtual world and cannot combine with the real scene, the reality of user experience is poor, and the integrated mixed reality head-mounted display system is used as important equipment for next-generation mixed reality interaction, has wide application prospects in the fields of visualization, virtual training, entertainment, art and the like of a data model, and is still in an early development stage at present, and no mature product exists at home and abroad.
Disclosure of Invention
The embodiment of the invention provides an integrated mixed reality head-mounted display system, which aims to solve the problems that most of head-mounted display devices in the prior art are virtual reality head-mounted display devices, users can only interact with a pure virtual world, cannot combine with a real scene and are poor in reality sense of user experience.
The embodiment of the invention provides an integrated mixed reality head-mounted display system, which comprises: wear display device, handle equipment and host computer, wherein, the handle equipment includes: the device comprises an inertia measurement unit, a key unit and a lamp strip; the inertial measurement unit measures handle posture information of the handle equipment and sends the handle posture information to the upper computer; the key unit acquires operation information of a user and sends the operation information to the upper computer; the lamp strip is used for displaying images displayed by the lamp strip; the head-mounted display device acquires the lamp band display image and sends the lamp band display image and display data information of the head-mounted display device to an upper computer; the upper computer is arranged in the head-mounted display device, generates a mixed reality image according to the handle posture information, the operation information, the lamp band display image and the display data information, and sends the mixed reality image to the head-mounted display device for display.
Optionally, the handle device further comprises: the data transmission unit acquires the handle posture information and the operation information and sends the handle posture information and the operation information to the upper computer.
Optionally, the handle device further comprises: and the lamp band adjusting unit acquires the operation information and adjusts the lamp band display image according to the operation information.
Optionally, the display data information includes: real scene image information and motion information; the head-mounted display device includes: the device comprises an image acquisition unit, a sensor measurement unit, a communication unit and a display unit, wherein the image acquisition unit acquires the image information of the real scene, acquires the display image of the lamp band, and sends the display image of the lamp band and the image information of the real scene to the upper computer through the communication unit; the sensor measuring unit measures the motion information of the head-mounted display device and sends the motion information to the upper computer through the communication unit; the display unit displays the mixed reality image sent by the upper computer through the communication unit.
Optionally, the head-mounted display device further comprises: and the head-mounted fixing device is used for fixing the head-mounted display equipment on the head of the user by the user.
Optionally, the display unit includes: and the first screen and the second screen are respectively arranged corresponding to the left eye and the right eye of the user, and synchronously display the mixed reality image.
Optionally, when the user operates the handle device with a left hand, the operation information is left-hand operation information, and the light band adjusting unit adjusts a light band display image of the light band to be a first preset image; when a user operates the handle equipment by right hand, the operation information is right hand operation information, and the lamp band adjusting unit adjusts a lamp band display image of the lamp band to be a second preset image; the upper computer identifies the handle equipment according to the first preset image and the second preset image.
Optionally, the light strip is composed of a plurality of LED lights.
Optionally, the image acquisition unit is at least one camera.
Optionally, the at least one camera is laid out according to a preset arrangement mode.
Optionally, the upper computer includes: the real-time map construction unit is used for establishing a current real-time map of the integrated mixed reality head-mounted display system by adopting a real-time positioning and map construction algorithm; the position and posture calculating unit is used for calculating position and posture information of the handle equipment and the head-mounted display equipment in the current instant map according to the handle posture information and the display data information; a virtual image generating unit for generating virtual image information according to the user operation information and the current instant map; and the mixed reality image generating unit acquires the real scene image information under the instant map from the display data information, and renders and superimposes the real scene image information and the virtual image information to generate a mixed reality image.
Optionally, the mixed reality image generating unit includes: the plane detection module is used for carrying out plane detection on the object in the real scene image information by adopting a plane detection algorithm to obtain the position relation information under the instant map coordinate system; and the image fusion module is used for rendering and superposing the position relation information and the virtual image information under the instant map coordinate system to generate a mixed reality image.
Optionally, the upper computer further includes: the safety monitoring unit is used for judging whether the head-mounted display equipment is in a preset safety operation area or not according to the real scene image information; and when the head-mounted display device is not in the preset safe operation area, the safety monitoring unit alarms to the user.
The technical scheme of the invention has the following advantages:
the integrated mixed reality head-mounted display system comprises head-mounted display equipment, handle equipment and an upper computer, wherein an inertia measurement unit in the handle equipment measures handle posture information of the handle equipment, a key unit obtains operation information of a user, the handle posture information and the operation information are sent to the upper computer, and a lamp strip is used for displaying a lamp strip display image; the head-mounted display equipment acquires a lamp band display image and sends the lamp band display image and display data information of the head-mounted display equipment to the upper computer; the upper computer is arranged in the head-mounted display device, generates a mixed reality image according to the handle posture information, the operation information, the lamp band display image and the display data information, and sends the mixed reality image to the head-mounted display device for display. The mixed display of the virtual image and the real image is realized, so that the mixed interaction of the user to the virtual world and the real scene is realized, the sense of reality of user experience is improved, in addition, the volume of the whole system is reduced by arranging the upper computer in the head-mounted display device, the carrying is convenient, and the interaction range of the user is enlarged.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic structural diagram of an integrated mixed reality head-mounted display system according to an embodiment of the invention;
fig. 2 is another schematic structural diagram of an integrated mixed reality head-mounted display system according to an embodiment of the invention;
FIG. 3 is a schematic diagram of a display unit according to an embodiment of the invention;
FIG. 4 is a schematic diagram of an upper computer in an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a mixed reality image generating unit according to an embodiment of the invention.
Detailed Description
The following description of the embodiments of the present invention will be made apparent and fully in view of the accompanying drawings, in which some, but not all embodiments of the invention are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the description of the present invention, it should be noted that the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, it should be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; the two components can be directly connected or indirectly connected through an intermediate medium, or can be communicated inside the two components, or can be connected wirelessly or in a wired way. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.
In addition, the technical features of the different embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
The embodiment of the invention provides an integrated mixed reality head-mounted display system, as shown in fig. 1, which comprises: head-mounted display device 1, handle equipment 2 and host computer 3, wherein, handle equipment 2 includes: an inertial measurement unit 21, a key unit 22, and a light belt 23; the inertia measurement unit 21 measures the handle posture information of the handle device 2 and transmits the handle posture information to the upper computer 3; the key unit 22 acquires operation information of a user and transmits the operation information to the upper computer 3; the light strip 23 is used for displaying a light strip display image; the head-mounted display device 1 acquires a lamp band display image and transmits the lamp band display image and display data information of the head-mounted display device 1 to the upper computer 3; the host computer 3 generates a mixed reality image from the handle posture information, the operation information, the light band display image, and the display data information, and transmits the mixed reality image to the head-mounted display device 1 for display.
The integrated mixed reality head-mounted display system provided by the embodiment of the invention realizes the mixed display of the virtual image and the real image, thereby realizing the mixed interaction of the user on the virtual world and the real scene, improving the sense of reality of the user experience.
The integrated mixed reality head mounted display system provided by the embodiment of the invention will be described in detail below with reference to specific examples.
In a preferred embodiment, as shown in fig. 2, the handle apparatus 2 further comprises: the data transmission unit 24, the data transmission unit 24 acquires the handle posture information and the operation information, and transmits the handle posture information and the operation information to the host computer 3. In practical applications, the data transmission unit 24 may transmit the above information to the host computer 3 in a wireless mode using bluetooth 4.0.
In a preferred embodiment, as shown in fig. 2, the handle apparatus 2 further comprises: and a lamp band adjusting unit 25, the lamp band adjusting unit 25 acquiring the operation information, and adjusting the lamp band display image according to the operation information. Specifically, when the user operates the handle apparatus 2 with the left hand, the operation information is left-hand operation information, and the lamp band adjusting unit 25 adjusts the lamp band display image of the lamp band 23 to be a first preset image; when the user right-hand operates the handle device 2, the operation information is right-hand operation information, and the lamp band adjusting unit 25 adjusts the lamp band display image of the lamp band 23 to a second preset image; the upper computer 3 recognizes the handle device 2 according to the first preset image and the second preset image. In practical applications, the light band adjusting unit 25 may also adjust different light band display images according to the operation information of the user when the multiple users interact together, so that the host computer 3 may identify the handle device 2 according to the obtained light band display images, and further identify the user.
Specifically, in practical application, the above-described inertial measurement unit 21 measures the handgrip posture information of the handgrip apparatus 2 including: acceleration information and angular velocity information of the handle device 2. In practical applications, the MPU9250 may be used as the inertial measurement unit 21, which includes a three-axis gyroscope and a three-axis accelerometer, to measure the acceleration and angular velocity of the handle device 2, respectively, and transmit them to the host computer 3 through the bluetooth 4.0 mode. The key unit 22 acquires operation information of the user, which is touch information of the manual key of the user, and the information is transmitted to the upper computer 3 through the data transmission unit 24 bluetooth 4.0. The above-mentioned light band adjusting unit 25 adjusts the light band display image of the above-mentioned light band 23 according to the user's operation information, so that the light band display images in the different user's handle devices 2 are different, so as to facilitate the recognition of the different handle devices 2 of multiple users. In practical application, the light band 23 is composed of a plurality of LED lamps, and the light band adjusting unit 25 adjusts the on/off states of the LED lamps according to the operation information, so as to display images of different light bands.
In a preferred embodiment, the display data information includes: real scene image information and motion information. As shown in fig. 2, the head-mounted display device 1 described above includes: the image acquisition unit 11, the sensor measurement unit 12, the communication unit 13 and the display unit 14, wherein the image acquisition unit 11 acquires the real scene image information, acquires the lamp band display image, and sends the lamp band display image and the real scene image information to the upper computer 3 through the communication unit 13; the sensor measurement unit 12 measures the motion information of the head-mounted display device 1 and transmits the motion information to the upper computer 3 through the communication unit 13; the display unit 14 displays the mixed reality image transmitted from the host computer 3 through the communication unit 13.
Specifically, in practical application, the image obtaining unit 11 is at least one camera, and in this embodiment of the present invention, the image obtaining unit 11 is a binocular camera, and the binocular camera is arranged according to a preset arrangement mode, for example, the preset arrangement mode may be that the binocular camera is placed in parallel, and the camera looks down at 15 ° horizontally, and the camera may adopt OV580 or the like. It should be noted that, in the embodiment of the present invention, the image obtaining unit 11 is taken as an example of a binocular camera, and other types of cameras may be used in practical applications, and the types, numbers and preset arrangement modes of the cameras may be set according to practical needs, which is not limited to this embodiment. The communication unit 13 is a USB3.0, and captures an image of a real scene where the user is located and a display image of a light band displayed by the light band in the handle device 2 through the camera, and transmits the captured image to the host computer 3 through a USB3.0 wired mode. The sensor measurement unit 12 may employ the MPU9250 including a three-axis gyroscope and a three-axis accelerometer, and the measured motion information thereof includes angular velocity information and acceleration information of the head-mounted display device 1 and is transmitted to the upper computer 3 through the USB3.0 wired mode.
In a preferred embodiment, as shown in fig. 3, the display unit 14 includes: the first screen 141 and the second screen 142 are respectively provided corresponding to the left eye and the right eye of the user, and the mixed reality image is synchronously displayed. In practical application, in order to enhance the experience of the user, ensure the consistency of the images seen by the eyes of the user after wearing the head-mounted display device 1, the first screen 141 and the second screen 142 need to be separately provided at the left eye and the right eye, and the mixed reality images sent by the upper computer 3 through the communication unit 13 need to be synchronously displayed.
In a preferred embodiment, as shown in fig. 2, the head-mounted display device 1 further includes: a head mount 15, by which a user secures the head mounted display device 1 to the user's head. In practical applications, the head-mounted fixing device 15 may be fixedly connected to the head-mounted display device 1 or detachably connected to the head-mounted display device 1, and the head-mounted fixing device 15 may fix the head-mounted display device 1 to the head of the user by using a structure such as a buckle or a strap.
In a preferred embodiment, as shown in fig. 4, the above-mentioned upper computer 3 includes: the instant map construction unit 31 adopts an instant positioning and map construction algorithm to establish a current instant map of the integrated mixed reality head mounted display system. In practical applications, the current instant map established by the instant map establishing unit 31 is to establish a map of the user wearing the head-mounted display device 1 and the surrounding environment thereof by using the user as a reference point, and provide location information for the subsequent user to perform interactive operation.
A position and orientation calculation unit 32 for calculating position and orientation information of the handle device 2 and the head-mounted display device 1 in the current instant map according to the handle orientation information and the display data information; in practical application, the position and posture calculating unit 32 identifies the handle device 2 according to the acceleration and angular velocity information of the handle measured by the inertial measuring unit 21 and the display image of the light band captured by the camera, and calculates the position and posture information of the handle device 2 in the current instant map.
A virtual image generation unit 33 that generates virtual image information from the user operation information and the current instant map; in practical applications, the virtual image information may be set according to the interactive scene of the user, for example, the user selects an experience scene of a game on the key unit 22, and the virtual image generating unit 33 generates the virtual scene image selected by the user according to the current instant map.
The mixed reality image generating unit 34 acquires real scene image information under the instantaneous map from the display data information, and renders and superimposes the real scene image information with the virtual image information to generate a mixed reality image. In practical applications, the above-mentioned real scene image information may be obtained by capturing with the camera in the above-mentioned head-mounted display device 1, and the mixed reality image generating unit 34 identifies the real scene image information by a deep learning algorithm, and renders and fuses the identification result with the above-mentioned virtual image to generate a mixed reality image. For example, if the real scene image is an image including a chair and the chair is not present in the virtual image, the mixed reality image generating unit 34 renders and superimposes the image of the chair under the current instant map and the virtual image by identifying the chair, so that the generated mixed reality image includes both the virtual image information and the information of the chair in the real scene, and the sense of reality of the user interaction experience is improved.
In a preferred embodiment, as shown in fig. 5, the mixed reality image generating unit 34 includes: the plane detection module 341 performs plane detection on the object in the image information of the real scene by adopting a plane detection algorithm to obtain the position relation information under the coordinate system of the instant map. In practical application, the specific process of the plane detection module 341 for performing plane detection on an object in the real scene image information by adopting a plane detection algorithm is that a dense point cloud of the real scene image information is obtained first, then the dense point cloud is subjected to position clustering, and finally the position relation information such as a plane center point coordinate, a plane direction, a plane size and the like is obtained from the dense point cloud of each type of clustering.
The image fusion module 342 renders and superimposes the position relation information and the virtual image information under the real-time map coordinate system, and generates a mixed reality image. In practical application, the image fusion module 342 is overlapped on the plane according to the above-mentioned position relationship information such as the plane center point coordinates, the plane direction, the plane dimensions, etc., and the position and the posture of the virtual object in the virtual image under the world coordinate system, because the real scene image shot by the camera is in a solid connection relationship with the above-mentioned current instant map, the coordinate system of the virtual object and the real object is fixedly connected, and the image fusion of the virtual image and the real object is formed.
In a preferred embodiment, as shown in fig. 4, the above-mentioned upper computer 3 further includes: a safety monitoring unit 35, configured to determine whether the head-mounted display device 1 is within a preset safety operation area according to the real scene image information; the safety monitoring unit 35 alerts the user when the head mounted display device 1 is not within the preset safe operating region.
In practical application, the safety monitoring unit 35 may control the virtual safety area through a virtual wall, wherein the virtual safety area is a closed-loop area, and when the user approaches the edge of the virtual safety area, the virtual wall is displayed to remind the user to pay attention to the operation area, and when the user leaves the preset safety operation area; in practical application, the real images shot by the camera can be projected to the display unit 14 of the head-mounted display device 1, so that a user can still see a real scene under the condition of wearing the head-mounted display device 1, and touch obstacles are avoided; depth information can be obtained by calculating the image shot by the camera, and whether an obstacle exists in the visible area of the camera is judged according to the depth information; it is also possible to judge whether there is an obstacle or the like in a 360 ° range around the user according to the positional relationship of the head-mounted display device 1, the positional relationship of the posture information, and the object in the scene. The invention is not limited thereto.
In practical application, the integrated mixed reality head-mounted display system provided by the embodiment of the invention realizes the work flow of rendering and superposing real object images and virtual three-dimensional object information as follows:
initializing an integrated mixed reality head-mounted display system; the OV580 camera shoots a real scene picture; after the upper computer 3 obtains the picture of the OV580 camera, the picture shot by the OV580 camera is overlapped with the virtual three-dimensional object by utilizing a virtual reality development engine Unity; the upper computer 3 uses the high-pass 835 on-chip display card to render the picture shot by the OV580 camera and the virtual three-dimensional object, displays the picture and the virtual three-dimensional object on the display unit 14 of the head-mounted display device 1, and returns to the step of shooting the real scene picture by the OV580 camera.
In practical application, the mixed reality interactive workflow of the integrated mixed reality head-mounted display system based on deep learning provided by the embodiment of the invention is as follows:
initializing an integrated mixed reality head-mounted display system; the OV580 camera shoots a real scene picture; distortion correction is carried out on a real scene picture shot by an OV580 camera, a distortion-free image is obtained, and the method is based on a deep learning algorithm, for example: the TensorFlow algorithm carries out object recognition on the undistorted image to obtain object semantic information, and based on the object semantic information, a current instant map of the mixed reality head-mounted display system is established by adopting an instant positioning and map construction algorithm to obtain position and posture information of a camera, and further information such as position, posture and size of a recognized object under a coordinate system of the current instant map is obtained; creating a corresponding virtual object in the virtual scene based on the identified object; judging whether to continue object recognition, if so, returning to the OV580 camera to shoot a real scene picture, otherwise, entering a virtual scene created in a corresponding virtual object created in the virtual scene based on the recognized object, wherein the objects in the scene are correspondingly created by the object obtained by object recognition, and the relative relation between the objects corresponds to the real scene.
In practical application, the integrated mixed reality head-mounted display system provided by the embodiment of the invention realizes the following interaction workflow of multiple persons in the same field:
initializing an integrated mixed reality head-mounted display system; carrying out current instant map creation by utilizing an integrated mixed reality head-mounted display system of one user to obtain a navigation map; distributing the created navigation map to other local integrated mixed reality head-mounted display systems; the integrated mixed reality head-mounted display system of other local users imports the distributed navigation map; the integrated mixed reality head-mounted display system of other local users carries out current instant map navigation according to the imported navigation map, so that a plurality of head-mounted display systems carry out navigation positioning under the same coordinate system, and multi-user same-field interaction is realized.
Through the cooperative cooperation of the components, the integrated mixed reality head-mounted display system provided by the embodiment of the invention realizes the mixed display of the virtual image and the real image, thereby realizing the mixed interaction of the user to the virtual world and the real scene, improving the sense of reality of user experience, reducing the volume of the whole system by arranging the upper computer in the head-mounted display equipment, being convenient to carry, expanding the interaction range of the user, realizing the adjustment of the display image of the lamp belt, enhancing the recognition capability of the hand-mounted equipment, providing a basis for the recognition of each user during the common interaction of multiple users and improving the user experience.
It is apparent that the above examples are given by way of illustration only and are not limiting of the embodiments. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. While still being apparent from variations or modifications that may be made by those skilled in the art are within the scope of the invention.
Claims (12)
1. An integrated mixed reality head mounted display system, comprising: a head-mounted display device (1), a handle device (2) and an upper computer (3), wherein,
the handle device (2) comprises: an inertia measurement unit (21), a key unit (22) and a lamp belt (23);
the inertial measurement unit (21) measures the handle posture information of the handle equipment (2) and sends the handle posture information to the upper computer (3);
the key unit (22) acquires operation information of a user and sends the operation information to the upper computer (3);
the lamp strip (23) is used for displaying a lamp strip display image;
the head-mounted display device (1) acquires the lamp band display image and sends the lamp band display image and display data information of the head-mounted display device (1) to an upper computer (3);
the upper computer (3) is arranged in the head-mounted display device (1), and the upper computer (3) generates a mixed reality image according to the handle posture information, the operation information, the lamp band display image and the display data information and sends the mixed reality image to the head-mounted display device (1) for display;
the upper computer (3) comprises:
an instant map construction unit (31) for establishing a current instant map of the integrated mixed reality head-mounted display system by adopting an instant positioning and map construction algorithm;
a position and posture calculation unit (32) for calculating position and posture information of the handle device (2) and the head-mounted display device (1) in the current instant map according to the handle posture information and the display data information;
a virtual image generation unit (33) that generates virtual image information on the basis of the user operation information and the current instantaneous map;
and a mixed reality image generating unit (34) for acquiring the real scene image information under the instant map from the display data information, and rendering and superposing the real scene image information and the virtual image information to generate a mixed reality image.
2. The integrated mixed reality head mounted display system of claim 1, characterized in that the handle device (2) further comprises:
the data transmission unit (24) acquires the handle posture information and the operation information, and sends the handle posture information and the operation information to the upper computer (3).
3. The integrated mixed reality head mounted display system of claim 1, characterized in that the handle device (2) further comprises:
and the lamp strip adjusting unit (25) is used for acquiring the operation information and adjusting the lamp strip display image according to the operation information.
4. The integrated mixed reality head mounted display system of claim 1, wherein the display data information comprises: real scene image information and motion information;
the head-mounted display device (1) comprises: an image acquisition unit (11), a sensor measurement unit (12), a communication unit (13) and a display unit (14), wherein,
the image acquisition unit (11) acquires the real scene image information, acquires the lamp band display image and sends the lamp band display image and the real scene image information to the upper computer (3) through the communication unit (13);
the sensor measuring unit (12) measures the motion information of the head-mounted display device (1) and sends the motion information to the upper computer (3) through the communication unit (13);
the display unit (14) displays the mixed reality image transmitted by the upper computer (3) through the communication unit (13).
5. The integrated mixed reality head mounted display system of claim 4, characterized in that the head mounted display device (1) further comprises: -a head mount fixture (15) by which a user secures the head mounted display device (1) to the user's head.
6. The integrated mixed reality head mounted display system of claim 4, wherein the display unit (14) comprises: a first screen (141) and a second screen (142), the first screen (141) and the second screen (142) being respectively arranged corresponding to the left eye and the right eye of the user, and the mixed reality image being synchronously displayed.
7. The integrated mixed reality head mounted display system of claim 3, wherein the integrated mixed reality head mounted display system comprises,
when a user operates the handle device (2) by left hand, the operation information is left hand operation information, and the lamp band adjusting unit (25) adjusts a lamp band display image of the lamp band (23) to be a first preset image;
when a user operates the handle device (2) by right hand, the operation information is right hand operation information, and the lamp band adjusting unit (25) adjusts a lamp band display image of the lamp band (23) to be a second preset image;
the upper computer (3) identifies the handle equipment (2) according to the first preset image and the second preset image.
8. The integrated mixed reality head mounted display system of claim 1, characterized in that the light strip (23) is comprised of a plurality of LED lights.
9. The integrated mixed reality head mounted display system of claim 4, wherein the image acquisition unit (11) is at least one camera.
10. The mixed reality head mounted display system of claim 9, wherein the at least one camera is arranged according to a predetermined arrangement.
11. The integrated mixed reality head mounted display system of claim 1, wherein the mixed reality image generating unit (34) comprises:
the plane detection module (341) is used for carrying out plane detection on the object in the real scene image information by adopting a plane detection algorithm to obtain the position relation information under the instant map coordinate system;
and the image fusion module (342) is used for rendering and superposing the position relation information and the virtual image information under the instant map coordinate system to generate a mixed reality image.
12. The integrated mixed reality head mounted display system of claim 1, wherein the host computer (3) further comprises:
the safety monitoring unit (35) is used for judging whether the head-mounted display device (1) is in a preset safety operation area or not according to the real scene image information;
the safety monitoring unit (35) alerts the user when the head mounted display device (1) is not within the preset safety operating region.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811301980.3A CN109445596B (en) | 2018-11-02 | 2018-11-02 | Integrated mixed reality head-mounted display system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811301980.3A CN109445596B (en) | 2018-11-02 | 2018-11-02 | Integrated mixed reality head-mounted display system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109445596A CN109445596A (en) | 2019-03-08 |
| CN109445596B true CN109445596B (en) | 2023-09-08 |
Family
ID=65549517
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811301980.3A Active CN109445596B (en) | 2018-11-02 | 2018-11-02 | Integrated mixed reality head-mounted display system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109445596B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112286343A (en) | 2020-09-16 | 2021-01-29 | 青岛小鸟看看科技有限公司 | Positioning tracking method, platform and head-mounted display system |
| CN114973042A (en) * | 2022-05-13 | 2022-08-30 | 歌尔股份有限公司 | VR handle position detection method, device, equipment and medium |
| CN115396656A (en) * | 2022-08-29 | 2022-11-25 | 歌尔科技有限公司 | AR SDK-based augmented reality method, system, device and medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202196425U (en) * | 2011-06-23 | 2012-04-18 | 北京北方卓立科技有限公司 | Device for enhancing reversed infrared action capture in reality |
| CN105549725A (en) * | 2016-02-03 | 2016-05-04 | 深圳市中视典数字科技有限公司 | Three-dimensional scene interaction display device and method |
| CN106843456A (en) * | 2016-08-16 | 2017-06-13 | 深圳超多维光电子有限公司 | A kind of display methods, device and virtual reality device followed the trail of based on attitude |
| CN106990851A (en) * | 2017-05-24 | 2017-07-28 | 北京无远弗届科技有限公司 | Virtual reality hand-held device and preparation method thereof, virtual reality interactive system |
| CN106990537A (en) * | 2017-05-25 | 2017-07-28 | 巢湖市金辉自控设备有限公司 | A kind of space walk device based on VR glasses and locating module |
| CN107168515A (en) * | 2017-03-31 | 2017-09-15 | 北京奇艺世纪科技有限公司 | The localization method and device of handle in a kind of VR all-in-ones |
| CN206604176U (en) * | 2017-02-16 | 2017-11-03 | 沈阳市创奇游乐设备有限公司 | A kind of dodgem of achievable virtual reality function |
| CN107533230A (en) * | 2015-03-06 | 2018-01-02 | 索尼互动娱乐股份有限公司 | Head mounted display tracing system |
| CN108124150A (en) * | 2017-12-26 | 2018-06-05 | 歌尔科技有限公司 | Virtual reality wears display device and observes the method for real scene by it |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9600067B2 (en) * | 2008-10-27 | 2017-03-21 | Sri International | System and method for generating a mixed reality environment |
-
2018
- 2018-11-02 CN CN201811301980.3A patent/CN109445596B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202196425U (en) * | 2011-06-23 | 2012-04-18 | 北京北方卓立科技有限公司 | Device for enhancing reversed infrared action capture in reality |
| CN107533230A (en) * | 2015-03-06 | 2018-01-02 | 索尼互动娱乐股份有限公司 | Head mounted display tracing system |
| CN105549725A (en) * | 2016-02-03 | 2016-05-04 | 深圳市中视典数字科技有限公司 | Three-dimensional scene interaction display device and method |
| CN106843456A (en) * | 2016-08-16 | 2017-06-13 | 深圳超多维光电子有限公司 | A kind of display methods, device and virtual reality device followed the trail of based on attitude |
| CN206604176U (en) * | 2017-02-16 | 2017-11-03 | 沈阳市创奇游乐设备有限公司 | A kind of dodgem of achievable virtual reality function |
| CN107168515A (en) * | 2017-03-31 | 2017-09-15 | 北京奇艺世纪科技有限公司 | The localization method and device of handle in a kind of VR all-in-ones |
| CN106990851A (en) * | 2017-05-24 | 2017-07-28 | 北京无远弗届科技有限公司 | Virtual reality hand-held device and preparation method thereof, virtual reality interactive system |
| CN106990537A (en) * | 2017-05-25 | 2017-07-28 | 巢湖市金辉自控设备有限公司 | A kind of space walk device based on VR glasses and locating module |
| CN108124150A (en) * | 2017-12-26 | 2018-06-05 | 歌尔科技有限公司 | Virtual reality wears display device and observes the method for real scene by it |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109445596A (en) | 2019-03-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109358754B (en) | Mixed reality head-mounted display system | |
| US11796309B2 (en) | Information processing apparatus, information processing method, and recording medium | |
| CN104380347B (en) | Video processing equipment, method for processing video frequency and processing system for video | |
| JP2022002144A (en) | System and method for augmented reality | |
| US9454006B2 (en) | Head mounted display and image display system | |
| CN105608746B (en) | A method of reality is subjected to Virtual Realization | |
| CN109445596B (en) | Integrated mixed reality head-mounted display system | |
| CN110969905A (en) | Remote teaching interaction and teaching aid interaction system for mixed reality and interaction method thereof | |
| CN204442580U (en) | A kind of wear-type virtual reality device and comprise the virtual reality system of this equipment | |
| EP3734423A1 (en) | Camera-based location of optically detectable markers | |
| JP2017120556A (en) | Head-mounted display for operation, control method of head-mounted display for operation, and program for head-mounted display for operation | |
| CN108132490A (en) | Detection system and detection method based on alignment system and AR/MR | |
| EP3552052A1 (en) | Head mounted display with user head rotation guidance | |
| CN108830944B (en) | Optical perspective three-dimensional near-to-eye display system and display method | |
| TW201341848A (en) | Telescopic observation for virtual reality system and method thereof using intelligent electronic device | |
| JP2021060627A (en) | Information processing apparatus, information processing method, and program | |
| CN111127822B (en) | Augmented reality-based fire-fighting auxiliary method and intelligent wearable equipment | |
| CN106325523A (en) | Man-machine interaction display device and system | |
| CN111491159A (en) | Augmented reality display system and method | |
| CN109445598A (en) | A kind of augmented reality system and device of view-based access control model | |
| EP3729235B1 (en) | Data processing | |
| WO2022176450A1 (en) | Information processing device, information processing method, and program | |
| CN212012916U (en) | Augmented reality's display device | |
| WO2021237952A1 (en) | Augmented reality display system and method | |
| US20220230357A1 (en) | Data processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |