CN117058504A - Method and system for fusing vehicle-mounted sensor and MR technology - Google Patents

Method and system for fusing vehicle-mounted sensor and MR technology Download PDF

Info

Publication number
CN117058504A
CN117058504A CN202310991500.5A CN202310991500A CN117058504A CN 117058504 A CN117058504 A CN 117058504A CN 202310991500 A CN202310991500 A CN 202310991500A CN 117058504 A CN117058504 A CN 117058504A
Authority
CN
China
Prior art keywords
video
vehicle
sensor
shake
user
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310991500.5A
Other languages
Chinese (zh)
Inventor
严哲峰
冯宪伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xiaoxiang Intelligent Shenzhen Co ltd
Original Assignee
Xiaoxiang Zero Run Suzhou Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xiaoxiang Zero Run Suzhou Technology Co ltd filed Critical Xiaoxiang Zero Run Suzhou Technology Co ltd
Priority to CN202310991500.5A priority Critical patent/CN117058504A/en
Publication of CN117058504A publication Critical patent/CN117058504A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01D—MEASURING NOT SPECIALLY ADAPTED FOR A SPECIFIC VARIABLE; ARRANGEMENTS FOR MEASURING TWO OR MORE VARIABLES NOT COVERED IN A SINGLE OTHER SUBCLASS; TARIFF METERING APPARATUS; MEASURING OR TESTING NOT OTHERWISE PROVIDED FOR
    • G01D21/00—Measuring or testing not otherwise provided for
    • G01D21/02—Measuring two or more variables by means not covered by a single other subclass
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/49—Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects

Abstract

The invention discloses a method and a system for fusing a vehicle-mounted sensor and an MR technology, wherein the method comprises the following steps: firstly, intercepting and processing related to the field of view of a user from a panoramic video generated by a vehicle-mounted sensor; step two, the video anti-shake is realized by reversely compensating a vehicle-mounted triaxial acceleration sensor; intercepting a user visual field, wherein the visual field 1 slightly larger than the user visual field is used for transmission and the visual field 2 which is focused by the user is used for actual processing; step four, the result after image recognition comprises the label of the object and the corresponding pixel point coordinates; step five, the labels in the step four are converted into corresponding digital images or other formatted information during virtual reality fusion, and the corresponding coordinate points are displayed in the fused video; the invention realizes better integration of the vehicle-mounted MR and the vehicle-mounted sensor, has no shake, small time delay and good user experience.

Description

Method and system for fusing vehicle-mounted sensor and MR technology
Technical Field
The invention belongs to the technical field of mixed reality, and particularly relates to a method and a system for fusing a vehicle-mounted sensor and an MR technology.
Background
Current developments in automotive intelligence and autopilot technology, vehicles tend to carry a large number of external sensors for sensing information in various dimensions of the surroundings. These sensors include, but are not limited to: visible light cameras (common, wide angle, depth of field), infrared cameras, laser radars, millimeter wave radars, ultrasonic radars, and the like, which can provide visible light, infrared images/videos, and signals returned by the radars can also be converted into images and videos for identifying reflectors by using a traditional method.
The mixed reality technology (MR) is a further development of AR/VR technology that enhances the realism of the user experience by presenting virtual scene information in a real scene, and by placing an interactive feedback information loop between the real world, the virtual world and the user.
In the vehicle-mounted MR device, if the digital virtual scene and the image/video output by the vehicle-mounted sensor are fused, a very unique user experience can be realized.
However, the original image/video is directly played in the MR, and the image/video is dithered due to jolt of the automobile, so that dizziness of a user is easily caused.
In addition, the sensor data is directly accessed, and because the data volume is large, particularly 360-degree peripheral data, more time and resources are needed for transmission and processing, the time delay can be increased, and at the moment, the user can see inconsistent with the actual sense of body, and dizziness is easy to cause.
Disclosure of Invention
The invention aims to overcome the existing defects, and provides a method and a system for fusing a vehicle-mounted sensor and an MR technology, so as to solve the problems of high image video jitter and transmission/processing data volume in the background technology and realize better fusion of sensor data and vehicle-mounted MR.
In order to achieve the above purpose, the present invention provides the following technical solutions: a method of vehicle-mounted sensor and MR technology fusion, comprising the steps of:
firstly, intercepting and processing related to the field of view of a user from a panoramic video generated by a vehicle-mounted sensor;
step two, the video anti-shake is realized by reversely compensating a vehicle-mounted triaxial acceleration sensor;
intercepting a user visual field, wherein the visual field 1 slightly larger than the user visual field is used for transmission and the visual field 2 which is focused by the user is used for actual processing;
step four, the result after image recognition comprises the label of the object and the corresponding pixel point coordinates;
and fifthly, converting the labels in the fourth step into corresponding digital images or other formatted information during virtual reality fusion, and displaying the digital images or other formatted information in the fused video according to the corresponding coordinate points.
Preferably, in the first step: the processing steps include video anti-shake and image recognition.
Preferably, in the first step: and fusing the processed anti-shake video with the result of image recognition to perform virtual reality.
A system for fusing a vehicle-mounted sensor and an MR technology comprises at least one video synthesis module, an object identification module, an image anti-shake module, an image splicing module and a triaxial acceleration sensor/vibration sensor connected with the anti-shake module.
Preferably, the video synthesis module mainly functions to process video signals from the vehicle-mounted sensor, and the non-video sensor is processed into video signals of a reflector with a certain azimuth by a traditional method, and is integrated into full-scale video data of 360 degrees.
Preferably, the video synthesis module further needs to cut the whole data according to the current user viewing angle, and only further processes part of the data, so as to save time and resources.
Preferably, the object recognition and the image/video anti-shake processing can be respectively performed on the processed data; the video anti-shake module is respectively responsible for carrying out anti-shake processing by combining a triaxial acceleration sensor and a vibration sensor; and the data processed by the two modules are summarized to an image/video splicing module, and are spliced to form a fused video/image finally displayed to a user.
Compared with the prior art, the invention provides a method and a system for fusing a vehicle-mounted sensor and an MR technology, which have the following beneficial effects: the invention realizes better integration of the vehicle-mounted MR and the vehicle-mounted sensor, has no shake, small time delay and good user experience.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate and together with the embodiments of the invention and do not constitute a limitation to the invention, and in which:
FIG. 1 is a diagram of a system architecture according to the present invention;
fig. 2 is a schematic diagram of one way of actually deploying the system according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1-2, the present invention provides a technical solution: a method of vehicle-mounted sensor and MR technology fusion, comprising the steps of:
firstly, intercepting and processing related to the field of view of a user from a panoramic video generated by a vehicle-mounted sensor;
step two, the video anti-shake is realized by reversely compensating a vehicle-mounted triaxial acceleration sensor;
intercepting a user visual field, wherein the visual field 1 slightly larger than the user visual field is used for transmission and the visual field 2 which is focused by the user is used for actual processing;
step four, the result after image recognition comprises the label of the object and the corresponding pixel point coordinates;
and fifthly, converting the labels in the fourth step into corresponding digital images or other formatted information during virtual reality fusion, and displaying the digital images or other formatted information in the fused video according to the corresponding coordinate points.
In the present invention, preferably, in the first step: the processing steps include video anti-shake and image recognition.
In the present invention, preferably, in the first step: and fusing the processed anti-shake video with the result of image recognition to perform virtual reality.
A system for fusing a vehicle-mounted sensor and an MR technology comprises at least one video synthesis module, an object identification module, an image anti-shake module, an image splicing module and a triaxial acceleration sensor/vibration sensor connected with the anti-shake module.
In the invention, preferably, the main function of the video synthesis module is to process the video signal from the vehicle-mounted sensor, and the non-video sensor is processed into the video signal of the reflector with a certain azimuth by the traditional method, and is integrated into the full-volume video data of 360 degrees.
In the invention, preferably, the video synthesis module further needs to cut the whole data according to the current user viewing angle, and only further processes part of the data so as to save time and resources.
In the present invention, preferably, the object recognition and the image/video anti-shake processing may be performed on the processed data; the video anti-shake module is respectively responsible for carrying out anti-shake processing by combining a triaxial acceleration sensor and a vibration sensor; and the data processed by the two modules are summarized to an image/video splicing module, and are spliced to form a fused video/image finally displayed to a user.
Example 1
Referring to fig. 1, the present invention provides a technical solution: a method of vehicle-mounted sensor and MR technology fusion, comprising the steps of:
firstly, intercepting and processing related to the field of view of a user from a panoramic video generated by a vehicle-mounted sensor;
in the first step: the processing steps include video anti-shake and image recognition.
Preferably, in the first step: the processed anti-shake video and the image recognition result are fused in a virtual reality mode;
step two, the video anti-shake is realized by reversely compensating a vehicle-mounted triaxial acceleration sensor;
intercepting a user visual field, wherein the visual field 1 slightly larger than the user visual field is used for transmission and the visual field 2 which is focused by the user is used for actual processing;
step four, the result after image recognition comprises the label of the object and the corresponding pixel point coordinates;
and fifthly, converting the labels in the fourth step into corresponding digital images or other formatted information during virtual reality fusion, and displaying the digital images or other formatted information in the fused video according to the corresponding coordinate points.
Example two
Referring to fig. 1, the present invention provides a technical solution: a system for fusing a vehicle-mounted sensor and an MR technology comprises at least one video synthesis module, an object identification module, an image anti-shake module, an image splicing module and a triaxial acceleration sensor/vibration sensor connected with the anti-shake module.
In the invention, preferably, the main function of the video synthesis module is to process the video signal from the vehicle-mounted sensor, and the non-video sensor is processed into the video signal of the reflector with a certain azimuth by the traditional method, and is integrated into the full-volume video data of 360 degrees.
In the invention, preferably, the video synthesis module further needs to cut the whole data according to the current user viewing angle, and only further processes part of the data so as to save time and resources.
In the present invention, preferably, the object recognition and the image/video anti-shake processing may be performed on the processed data; the video anti-shake module is respectively responsible for carrying out anti-shake processing by combining a triaxial acceleration sensor and a vibration sensor; and the data processed by the two modules are summarized to an image/video splicing module, and are spliced to form a fused video/image finally displayed to a user.
Example III
Referring to fig. 2, the present invention provides another technical solution: the system is characterized in that a vehicle-mounted sensor and an MR technology are integrated, an intelligent car cabin is generally provided with an intelligent car machine system at present, the intelligent car cabin is connected with a sensor at the front end, a video synthesis module and a video anti-shake module are required to be deployed on the car machine system, and time delay caused by signal transmission is reduced;
the rest modules can be deployed on the MR and can also be deployed on the vehicle-mounted system, and the rest modules depend on the computing power of the vehicle-mounted system and the MR;
the connection between the MR and the vehicle machine can be wired or wireless; the wireless protocol is generally WIFI, and can also be 4G/5G; the cable may be through an HDMI cable or an ethernet cable.
The processing flow of the system is as follows:
the method comprises the steps that firstly, a front-end video signal is input to a video synthesis module, the video synthesis module is spliced into a 360-degree annular video according to the coverage azimuth of the video signal, and the technology can adopt a video splicing algorithm commonly used in vehicle-mounted 360-degree images and digital shooting; for an image/video from a non-visible light signal, it may be the outline of a reflector, superimposed on the corresponding orientation of the annular video;
step two, the video synthesis module continuously inquires the view angle of the current MR user periodically (for example, taking 20 milliseconds as a period and not more than 100 milliseconds at maximum), and only takes video data with a slightly wider view range in the video stream to carry out the next processing according to the view angle direction, for example, 105 degrees around the view angle center line and recorded as a view range 1;
step three, the video data output in the step two are further reduced, only the part of the middle part of the video data, which is easy to be focused by a user, is marked as a visual field range 2, for example, the middle 150 degrees are processed in the following step four and step five, and the time and the resource required by the processing are further reduced; the fourth step and the fifth step can be parallel, and the parallel benefit is that the concurrency capability of the current computer system is fully utilized, so that the processing is quickened;
step four, video anti-shake, which combines the real-time acceleration signals in the horizontal and vertical directions acquired by the triaxial acceleration sensor/vibration sensor to reversely compensate the next frame of image; the compensation algorithm is as examples: reading acceleration data from a triaxial acceleration sensor at 10ms period, wherein accelerations ax and az in a left-right horizontal direction (X-axis) and a vertical direction (Z-axis) are used for compensating for shake, respectively; when displaying a frame of image, horizontally moving x pixels in the opposite direction according to ax, and vertically moving z pixels in the opposite direction according to az; the video jitter is best compensated when ax and x, az and z respectively accord with the linear relation, so that the approximate linear relation can be obtained through preliminary experiments, the linear relation is converted from ax and az to x and z, and the frame is reversely moved, so that better anti-jitter is realized;
fifthly, performing image recognition on the video based on a depth neural network, for example, adopting models such as RNN, DNN and the like, identifying objects (including outlines generated by non-visible light sensing data in the first step) appearing in the video, identifying (recording relative pixel point positions) and labeling;
step six, the data output from the step five can read the preset digital image matched with the label or other formatted data information according to the label, splice the anti-shake data output from the step four in a video splicing module according to the recorded corresponding pixel points to form a virtual+real stable (de-shake) video;
and step seven, when the user head rotates, the visual field range 1 and the visual field range 2 in the step one are correspondingly changed, and at the moment, the visual field range 1 is larger than the visual field range 2, so that the data can be directly supplemented to the visual field range 2 along with the rotation direction from the visual field range 1, the transmission delay of the supplement data is reduced, and the processing of the step four and the step five is not delayed.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims (7)

1. A method for fusing an on-board sensor and MR technology, comprising the steps of:
firstly, intercepting and processing related to the field of view of a user from a panoramic video generated by a vehicle-mounted sensor;
step two, the video anti-shake is realized by reversely compensating a vehicle-mounted triaxial acceleration sensor;
intercepting a user visual field, wherein the visual field 1 slightly larger than the user visual field is used for transmission and the visual field 2 which is focused by the user is used for actual processing;
step four, the result after image recognition comprises the label of the object and the corresponding pixel point coordinates;
and fifthly, converting the labels in the fourth step into corresponding digital images or other formatted information during virtual reality fusion, and displaying the digital images or other formatted information in the fused video according to the corresponding coordinate points.
2. The method of claim 1, wherein the method further comprises the steps of: in the first step: the processing steps include video anti-shake and image recognition.
3. The method of claim 1, wherein the method further comprises the steps of: in the first step: and fusing the processed anti-shake video with the result of image recognition to perform virtual reality.
4. A system for fusing an on-board sensor and MR technology, characterized in that: the system comprises at least one video synthesis module, an object identification module, an image anti-shake module, an image splicing module and a triaxial acceleration sensor/vibration sensor connected with the anti-shake module.
5. The system for vehicle sensor and MR technology integration according to claim 4, wherein: the main function of the video synthesis module is to process video signals from the vehicle-mounted sensor, and the non-video sensor is processed into video signals of a reflector with a certain azimuth by a traditional method and integrated into full-volume video data of 360 degrees.
6. The system for vehicle sensor and MR technology integration according to claim 5, wherein: the video synthesis module also needs to cut the whole data according to the current user view angle, and only further processes part of the data so as to save time and resources.
7. The system for vehicle sensor and MR technology integration according to claim 6, wherein: object recognition and image/video anti-shake processing can be respectively carried out on the processed data; the video anti-shake module is respectively responsible for carrying out anti-shake processing by combining a triaxial acceleration sensor and a vibration sensor; and the data processed by the two modules are summarized to an image/video splicing module, and are spliced to form a fused video/image finally displayed to a user.
CN202310991500.5A 2023-08-08 2023-08-08 Method and system for fusing vehicle-mounted sensor and MR technology Pending CN117058504A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310991500.5A CN117058504A (en) 2023-08-08 2023-08-08 Method and system for fusing vehicle-mounted sensor and MR technology

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310991500.5A CN117058504A (en) 2023-08-08 2023-08-08 Method and system for fusing vehicle-mounted sensor and MR technology

Publications (1)

Publication Number Publication Date
CN117058504A true CN117058504A (en) 2023-11-14

Family

ID=88658239

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310991500.5A Pending CN117058504A (en) 2023-08-08 2023-08-08 Method and system for fusing vehicle-mounted sensor and MR technology

Country Status (1)

Country Link
CN (1) CN117058504A (en)

Similar Documents

Publication Publication Date Title
US11935197B2 (en) Adaptive vehicle augmented reality display using stereographic imagery
US11303806B2 (en) Three dimensional rendering for surround view using predetermined viewpoint lookup tables
JP6977720B2 (en) Generation device and generation method, and playback device and playback method
US20200202498A1 (en) Computing system for rectifying ultra-wide fisheye lens images
JP7040466B2 (en) Image processing device and image processing method
KR20010112433A (en) Image processing device and monitoring system
CN101442618A (en) Method for synthesizing 360 DEG ring-shaped video of vehicle assistant drive
CN107235008B (en) Vehicle-mounted driving-assisting panoramic image system and panoramic image acquisition method
CN212305474U (en) Large vehicle panoramic image splicing system and automobile
CN105774657B (en) Single-camera panoramic reverse image system
CN105216715A (en) A kind of motorist vision assists enhancing system
CN113890977A (en) Airborne video processing device and unmanned aerial vehicle with same
CN117058504A (en) Method and system for fusing vehicle-mounted sensor and MR technology
KR102235951B1 (en) Imaging Apparatus and method for Automobile
EP4156127A1 (en) Image processing system, mobile object, image processing method, and storage medium
JP6977725B2 (en) Image processing device and image processing method
US20230113406A1 (en) Image processing system, mobile object, image processing method, and storage medium
CN219266904U (en) Vehicle-mounted mixed reality experience expansion system
CN219039781U (en) Vehicle-mounted panoramic double-spectrum vision enhancement system
CN211731213U (en) Vehicle-mounted information entertainment system and vehicle
CN111824019B (en) Vehicle-mounted 3D panoramic all-around viewing system
CN110979319A (en) Driving assistance method, device and system
CN111942288B (en) Vehicle image system and vehicle positioning method using vehicle image
US20230368484A1 (en) Dynamic 360-degree virtual sensor mapping
CN116149480A (en) Vehicle-mounted mixed reality experience expansion system and method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20240119

Address after: 518000 Research Building 107, Tsinghua Information Port, West Side of South Gate, No.13 Langshan Road, Songpingshan Community, Xili Street, Nanshan District, Shenzhen, Guangdong Province

Applicant after: Xiaoxiang intelligent (Shenzhen) Co.,Ltd.

Address before: Room 236, Building 1, No. 5 Wenzhi Road, Nanjiao, Chengxiang Town, Taicang City, Suzhou City, Jiangsu Province, 215400

Applicant before: Xiaoxiang Zero Run (Suzhou) Technology Co.,Ltd.