CN114115246A - Remote driving auxiliary system and method thereof - Google Patents
Remote driving auxiliary system and method thereof Download PDFInfo
- Publication number
- CN114115246A CN114115246A CN202111322167.6A CN202111322167A CN114115246A CN 114115246 A CN114115246 A CN 114115246A CN 202111322167 A CN202111322167 A CN 202111322167A CN 114115246 A CN114115246 A CN 114115246A
- Authority
- CN
- China
- Prior art keywords
- driving
- remote
- unit
- motion information
- video data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000033001 locomotion Effects 0.000 claims abstract description 43
- 230000006399 behavior Effects 0.000 claims abstract description 4
- 230000008569 process Effects 0.000 claims description 14
- 230000004886 head movement Effects 0.000 claims description 10
- 238000005259 measurement Methods 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 5
- 210000001508 eye Anatomy 0.000 description 19
- 238000004364 calculation method Methods 0.000 description 11
- 230000004424 eye movement Effects 0.000 description 9
- 210000003128 head Anatomy 0.000 description 9
- 210000005252 bulbus oculi Anatomy 0.000 description 8
- 230000007246 mechanism Effects 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 6
- 230000008901 benefit Effects 0.000 description 4
- 238000004088 simulation Methods 0.000 description 4
- 230000001133 acceleration Effects 0.000 description 3
- 241000282414 Homo sapiens Species 0.000 description 2
- 238000005034 decoration Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
- G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Electromagnetism (AREA)
- Aviation & Aerospace Engineering (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- General Physics & Mathematics (AREA)
- Automation & Control Theory (AREA)
- Traffic Control Systems (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
The invention provides a remote driving assistance system.A first assistance unit is used for acquiring motion information of a remote driver in remote driving, generating a posture control instruction according to the motion information and sending the posture control instruction to a second assistance unit, wherein the motion information is used for representing body motion information of the remote driver to a driving focus point in the remote driving, and the driving focus point is a key point of driving reaction of the remote driver to video data displayed by a display unit according to driving behaviors; the second auxiliary unit is used for adjusting the acquisition angle of the camera shooting unit according to the attitude control instruction to acquire new video data when receiving the attitude control instruction, and transmitting the new video data to the display unit so that a remote driver can make driving reaction based on the new video data displayed by the display unit. The invention also provides a remote driving assistance method. The invention can really realize the real-time control of the remote vehicle by observing the video under the current network bandwidth condition.
Description
Technical Field
The invention belongs to the field of automatic driving, and particularly relates to a remote driving and remote pipe taking system.
Background
With the popularization of 5G communication technology and the gradual maturity of automatic driving technology, more and more researches and attentions have been made on remote driving and remote takeover of vehicles.
Currently, a remote driving system/remote takeover system is basically implemented based on multiple high-definition videos of a remote vehicle, that is, a plurality of cameras are installed on the remote vehicle, data of the cameras are transmitted back to a remote console in real time through a 5G network, and a remote driver controls the remote vehicle according to a video picture. Although the scheme is simple and intuitive, and is matched with technologies such as a driving auxiliary indicating line and the like, a driving operator can remotely control the remote vehicle through proper training, for example, the remote driving method can be applied to remote driving scenes with single driving scenes such as mines, ports, docks and the like under the current technical conditions. However, the disadvantages of this solution are also apparent. This is because current 5G macronets typically have an upload speed of 100Mbit/s and are only capable of transmitting 1-way high definition video. And if the real-time transmission of multiple high-definition videos is realized, a 5G private network is required to be used, and the uploading speed of 5G needs to be optimized. Thus, not only the construction cost of the system is greatly increased, but also the use occasions are limited. Meanwhile, in order to adapt to the observation habit of an operator, multiple paths of videos need to be spliced into one picture at local or remote places, so that the real-time performance of the system is further reduced, and high-speed driving of the vehicle cannot be performed (for example, the vehicle speed is higher than 30 km/h).
In addition, a remote driving system/remote takeover system with a single driving scene has a technical scheme that a high-precision simulation scene is combined with a dynamic target object, for example, a technical scheme disclosed in patent document "a real-time remote driving system and method for monitoring the state of multiple vehicles" (with the publication number being CN113022540A) is that a high-precision map of a target area is acquired in advance, a target object list identified by a local automatic driving system of a remote vehicle is transmitted back to a remote driving end, and a simulation server of the remote driving end performs simulation reduction. Although the scheme can reduce the requirement on network bandwidth, and can be applied to a remote driving system/remote takeover system with a single driving scene and relatively stable targets, the scheme lacks real-time video, and the safety needs to be improved when the scheme is applied to a traffic scene with a large number of dynamic targets (pedestrians and other traffic vehicles). Meanwhile, the scheme has very high requirements on the computing capacity of the simulation server.
Based on the above, the present application provides a technical solution to solve the above technical problems.
Disclosure of Invention
The first purpose of the present invention is to obtain a remote driving assistance system with greatly improved safety, which can really realize real-time control of a remote vehicle by observing videos under the current network bandwidth condition.
The second objective of the present invention is to obtain a driving assistance method for remote driving with greatly improved safety, which can really realize real-time control of a remote vehicle by observing videos under the current network bandwidth condition.
The invention provides an auxiliary system for remote driving, which is applied to assist a remote driver to remotely drive a remote vehicle, and comprises a first auxiliary unit and a display unit which are arranged at the remote driving end, and a second auxiliary unit which is arranged at the remote vehicle end;
the first auxiliary unit is used for acquiring motion information of the remote driver in the remote driving, generating a posture control instruction according to the motion information and sending the posture control instruction to the second auxiliary unit, wherein the motion information is used for representing body motion information of the remote driver to a driving focus point in the remote driving, and the driving focus point is a key point of driving reaction of the remote driver to video data displayed by the display unit according to driving behaviors;
the second auxiliary unit is used for adjusting the acquisition angle of the camera unit according to the attitude control instruction to acquire new video data and transmitting the new video data to the display unit when receiving the attitude control instruction from the first auxiliary unit, so that the remote driver can make driving reaction based on the new video data displayed by the display unit.
In one embodiment of the invention, the remote driving assistance system is a remote driving take-over system.
Specifically, the first auxiliary unit comprises a remote driver motion information unit, a camera attitude calculation unit and a display unit.
Specifically, the second auxiliary unit comprises a servo control system and a camera unit. Specifically, the camera unit is a front camera unit. For example, the front camera unit is mounted on the single-axis servo control system whose position is adjustable. Preferably, the front camera unit is mounted on a single-shaft servo control system and is integrally mounted at a proper position in the front of the vehicle. In this position, images of the front, left front, and right front of the vehicle can be acquired by rotation of the camera.
In a preferred embodiment of the present invention, the number of cameras protected by the front camera unit is not higher than 2, and is preferably one.
In a preferred embodiment of the invention, the information collected by the remote driver movement information unit comprises eye movement information and/or head movement information.
In a preferred embodiment of the present invention, the information collected by the remote driver movement information unit is the information of the eye movement direction of the remote driver tracked and collected by the eye tracking system. More preferably, the acquired head movement information of the remote driver is tracked by an inertial measurement unit from the eye tracking system. In particular, the eye tracking system is a head-mounted device that is capable of tracking the direction of eye movement of a wearer (remote driver). At the same time, it also comprises an Inertial Measurement Unit (IMU) containing an acceleration sensor and a gyroscope, the inertial measurement unit being used to track the head movements of the wearer.
In one embodiment, the camera attitude calculation unit calculates the rotation direction of the camera (i.e. the rotation around the Z axis) on the horizontal plane in real time according to the output of the eye tracking system and the information such as the installation position of the camera on the vehicle created in advance, and finally converts the rotation direction into a camera attitude control command, and sends the command to the single-axis servo control system on the vehicle side through a 4G or 5G network.
In one embodiment of the present invention, the single-axis servo control system is a high-precision and high-dynamic micro-servo mechanism, which comprises a micro-motor, an embedded controller and a corresponding mechanical structure, and can control the camera connected with the micro-motor to rotate and position around a vertical axis according to instructions.
In a preferred embodiment of the invention, the attitude control command is sent to the second auxiliary unit via a 4G/5G network.
In a preferred embodiment of the invention, the new video data is transmitted to the display unit via a 4G/5G network.
In a second aspect of the present invention, there is provided a driving assistance method for remote driving, which is applied to the driving assistance system for remote driving of the present invention, the method including:
the method comprises the steps that through a first auxiliary unit, motion information of a remote driver in remote driving is obtained, a posture control instruction is generated according to the motion information, and the posture control instruction is sent to a second auxiliary unit;
and when the attitude control instruction is received, the second auxiliary unit adjusts the acquisition angle of the camera shooting unit according to the attitude control instruction to acquire new video data and transmits the new video data to the display unit, so that the remote driver can make a driving reaction based on the new video data displayed by the display unit.
In a preferred embodiment of the invention, the attitude control command is sent to the second auxiliary unit via a 4G/5G network.
In a preferred embodiment of the invention, the new video data is transmitted to the display unit via a 4G/5G network.
In a preferred embodiment of the present invention, the process of the driving assistance method is performed at a frequency of not less than 50 hz.
In a preferred embodiment of the invention, the command tracking response delay of the single-axis servo controller is not more than 5 ms.
The invention can bring at least one of the following beneficial effects:
(1) the invention provides a remote driving assistance method which can really realize real-time control of a remote vehicle by observing videos under the condition of the current network bandwidth and greatly improve the safety.
(2) In one embodiment of the present invention, the system provided by the present invention can effectively solve the disadvantages of the existing video-based remote driving/takeover system. The camera is used for follow-up control and eye tracking of a driver, only one-way high-definition video is transmitted between the vehicle and a remote driving system, and the requirement on computing capacity is greatly reduced; but can satisfy the remote driver to look over the condition around the vehicle in real time simultaneously, can really realize controlling in real time to remote vehicle through observing the video under the condition of current 5G network bandwidth, improved the security greatly.
Drawings
The foregoing features, technical features, advantages and embodiments are further described in the following detailed description of the preferred embodiments, which is to be read in connection with the accompanying drawings.
FIG. 1 is a block diagram of an exemplary system of the present invention;
fig. 2 is a schematic diagram of an exemplary system operation of the present invention.
Detailed Description
In the invention, the inventor finds out an operator auxiliary facility and a device for the remote takeover system of the automatic driving vehicle through extensive and intensive experiments, so that the video of the surrounding environment of the remote vehicle is transmitted according to the attention point of the operator, the effectiveness of video transmission under the condition of current lower network bandwidth can be realized, the driving habit of human is better met, and the invention aim of improving the safety is fulfilled.
More specifically, the inventive concept of the present invention combines remote control technology, high-speed servo technology, eye tracking technology and high-precision indoor positioning technology to obtain a preferred embodiment.
Unless explicitly stated or limited otherwise, the term "or" as used herein includes the relationship of "and". The "sum" is equivalent to the boolean logic operator "AND", the "OR" is equivalent to the boolean logic operator "OR", AND "is a subset of" OR ".
Various aspects of the invention are described in detail below:
auxiliary system
The invention provides an auxiliary system for remote driving, which is applied to assist a remote driver to remotely drive a remote vehicle and comprises a first auxiliary unit and a display unit which are arranged at a remote driving end, and a second auxiliary unit which is arranged at the remote vehicle end;
the first auxiliary unit is used for acquiring motion information of the remote driver in remote driving, generating a posture control instruction according to the motion information and sending the posture control instruction to the second auxiliary unit, wherein the motion information is used for representing body motion information of the remote driver to a driving focus point in remote driving, and the driving focus point is a key point of driving reaction of the remote driver to video data displayed by the display unit according to driving behaviors;
and the second auxiliary unit is used for adjusting the acquisition angle of the camera unit according to the attitude control instruction to acquire new video data and transmitting the new video data to the display unit when receiving the attitude control instruction, so that the remote driver can make a driving reaction based on the new video data displayed by the display unit.
The inventor finds that the video of the surrounding environment of the remote vehicle is transmitted according to the attention point of the driver, so that the effectiveness of video transmission under the condition of current lower network bandwidth can be realized, the driving habit of human beings is better met, the real-time control of the remote vehicle by observing the video can be really realized under the condition of current network bandwidth, and the safety is greatly improved.
In one embodiment of the invention, the attitude control command of the first auxiliary unit is sent to the second auxiliary unit via a 4G/5G network.
In one embodiment of the invention, the new video data is transmitted to the display unit over a 4G/5G network.
The "4G/5G network" of the present invention may be 4G, 5G or a combination thereof.
It should be understood that although the invention adopts a new technical concept, the invention achieves the purpose of the invention, namely, the invention can really realize the real-time control of the remote vehicle by observing the video under the current network bandwidth condition; however, the present invention may be used for instruction transmission or data transmission using various other networks in the art as long as the object of the present invention is not limited.
It should be understood that the sending of the attitude control command and the transmission of the video data may be in the same network or in different networks.
In one embodiment of the invention, the data of the camera unit is turned back to the video receiving and displaying equipment of the remote driving end in real time through the 4G/5G network.
In a preferred embodiment of the present invention, the number of cameras protected by the front camera unit is not higher than 2, and is preferably one. It will be appreciated that one of the advantages of the inventor's solution is to allow remote driving of the vehicle with the current 5G bandwidth. Even in the case where the bandwidth condition is satisfied, the present invention has another advantage that the demand for the number of cameras is reduced, and thus the setting of the number of cameras can be reduced, thereby obtaining technical advantages.
In one embodiment of the invention, the remote driving assistance system is a remote driving take-over system.
First auxiliary unit
Specifically, the first auxiliary unit comprises a remote driver motion information unit and a camera posture calculation unit. The remote driver movement information unit is configured to acquire movement information of the remote driver during remote driving. The camera attitude calculation unit is configured to receive the motion information acquired by the remote driver motion information unit, generate an attitude control instruction according to the motion information, and send the generated attitude control instruction to the second auxiliary unit.
In a preferred embodiment of the invention, the information collected by the remote driver movement information unit comprises eye movement information and/or head movement information.
In a preferred embodiment of the present invention, the information collected by the remote driver movement information unit is the information of the eye movement direction of the remote driver tracked and collected by the eye tracking system.
More preferably, the collected head movement information of the remote driver is tracked from the inertial measurement unit.
In one embodiment, the eye tracking system is a head-mounted device that is capable of tracking the direction of eye movement of the wearer (remote driver). At the same time, it also comprises an Inertial Measurement Unit (IMU) containing an acceleration sensor and a gyroscope, the inertial measurement unit being used to track the head movements of the wearer.
In one embodiment, the camera attitude calculation unit calculates the rotation direction of the camera (i.e. the rotation around the Z axis) on the horizontal plane in real time according to the output of the eye tracking system and the information such as the installation position of the camera on the vehicle created in advance, and finally converts the rotation direction into a camera attitude control command, and sends the command to the single-axis servo control system on the vehicle side through a 4G or 5G network.
In one embodiment of the present invention, the system provided by the present invention can effectively solve the disadvantages of the existing video-based remote driving/takeover system. The camera is used for follow-up control and eye tracking of a driver, only one-way high-definition video is transmitted between the vehicle and a remote driving system, and the requirement on computing capacity is greatly reduced; but can satisfy the remote driver to look over the condition around the vehicle in real time simultaneously, can really realize controlling in real time to remote vehicle through observing the video under the condition of current 5G network bandwidth, improved the security greatly.
The inventor finds that an eye tracker is used for acquiring the attention point of an operator (remote driver) to a vehicle-mounted video in the driving process, and a camera is adjusted to a desired position in real time; more preferably, the eye tracker not only tracks the eyeball movement of the operator, but also tracks the head movement of the operator through a built-in IMU, so that the region and the position of the operator, which are concerned with the vehicle-mounted video, can be ensured to be accurately acquired.
The inventor finds that in the technical scheme of the invention, only one servo control mechanism is needed to adjust the posture of the camera in real time according to the requirement of an operator (remote driver), so that only one camera and one high-definition video are needed to meet the requirement threshold of the operator on the vehicle-mounted video in the driving process, and the remote driving of the vehicle is realized under the condition of the current 5G bandwidth. The application range of 5G remote driving is expanded, so that the remote driving method can be applied to real traffic environments. It should be noted that although the invention can save resources to the maximum extent, only one camera and one path of high definition video are needed to meet the requirement threshold of the driver, thereby allowing the application under the current 5G bandwidth limitation; however, the present invention is not limited to a single servo control mechanism, a single camera, and a single channel of high definition video in order to achieve better regulation or with improved 5G bandwidth.
Second auxiliary unit
Specifically, the second auxiliary unit includes a servo control system and a camera unit.
Specifically, the servo control system is configured to adjust the acquisition angle of the camera unit according to the attitude control instruction to acquire new video data when receiving the attitude control instruction.
Preferably, the servo control system is a single axis servo control system.
The "camera unit" is configured to acquire new video data and transmit the new video data to the display unit so that the remote driver makes a driving reaction based on the new video data displayed by the display unit. Book (I)
It will be understood by those skilled in the art that the "camera unit" is used to collect the driving information related to the surrounding environment, and is generally installed at a suitable collecting position of the vehicle, as long as the object of the present invention is not limited thereto.
Specifically, the camera unit is a front camera unit. For example, the front camera unit is mounted on the single-axis servo control system whose position is adjustable.
In one embodiment of the invention, the front-end camera unit is mounted on a single-axis servo control system and is integrally mounted in a suitable position in the front of the vehicle. In this position, images of the front, left front, and right front of the vehicle can be acquired by rotation of the camera.
In one embodiment of the present invention, the single-axis servo control system is a high-precision and high-dynamic micro-servo mechanism, which comprises a micro-motor, an embedded controller and a corresponding mechanical structure, and can control the camera connected with the micro-motor to rotate and position around a vertical axis according to instructions.
The inventor finds that in the technical scheme of the invention, only one servo control mechanism is needed to adjust the posture of the camera in real time according to the requirement of an operator (remote driver), so that only one camera and one high-definition video are needed to meet the requirement threshold of the operator on the vehicle-mounted video in the driving process, and the remote driving of the vehicle is realized under the condition of the current 5G bandwidth. The application range of 5G remote driving is expanded, so that the remote driving method can be applied to real traffic environments. It should be noted that although the invention can save resources to the maximum extent, only one camera and one path of high definition video are needed to meet the requirement threshold of the driver, thereby allowing the application under the current 5G bandwidth limitation; however, the present invention is not limited to a single servo control mechanism, a single camera, and a single channel of high definition video in order to achieve better regulation or with improved 5G bandwidth.
Display unit
The "display unit" is arranged to receive the video data stream of the second auxiliary unit. Specifically, herein, the "display unit" is used to receive real-time video data from a camera on the vehicle-mounted side. It may be any device having a function of receiving video from a network and displaying it, as long as it does not limit the object of the present invention. Specifically, the display unit is a video receiving display device.
Driving method
The invention provides a driving assistance method for remote driving, which is applied to the driving assistance system for remote driving, and the method comprises the following steps:
the method comprises the steps that through a first auxiliary unit, motion information of a remote driver in remote driving is obtained, a posture control instruction is generated according to the motion information, and the posture control instruction is sent to a second auxiliary unit;
and when the attitude control instruction is received, the second auxiliary unit adjusts the acquisition angle of the camera shooting unit according to the attitude control instruction to acquire new video data and transmits the new video data to the display unit, so that the remote driver can make a driving reaction based on the new video data displayed by the display unit.
In a preferred embodiment of the invention, the attitude control command is sent to the second auxiliary unit via a 4G/5G network.
In a preferred embodiment of the invention, the new video data is transmitted to the display unit via a 4G/5G network.
In a preferred embodiment of the present invention, the process of the driving assistance method is performed at a frequency of not less than 50 hz.
In a preferred embodiment of the invention, the command tracking response delay of the single-axis servo controller is not more than 5 ms.
In one embodiment, when the remote driver intends to turn the vehicle to the left, naturally, the operator (i.e., the remote driver) will turn the head or eyeball to the left in an attempt to obtain video data of the left side of the vehicle. The eye tracking instrument acquires eyeball horizontal movement distance or head rotation angle data of an operator and sends the eyeball horizontal movement distance or head rotation angle data to the camera posture calculation unit. The camera attitude calculation unit converts the eye movement data and the head rotation data into attitude data of the camera and sends the attitude data to a single-axis servo controller of a vehicle-mounted end through a wireless network (4G/5G). The single-axis servo controller controls the motor to rotate according to expected camera attitude data, drives the camera to rotate to an expected position leftwards around a vertical axis, and in the rotating process, the camera acquires video data all the time and sends the video data back to video receiving and displaying equipment of a remote driving end through a network. The whole process is finished at a frequency of not less than 50hz, and the command tracking response delay of the single-shaft servo controller is not more than 5ms, so that the smooth picture and no blockage are ensured in the head or eyeball rotation process of an operator.
Based on the present application, one skilled in the art should appreciate that one aspect described herein may be implemented independently of any other aspects and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method practiced using any number and aspects set forth herein. Additionally, such an apparatus may be implemented and/or such a method may be practiced using other structure and/or functionality in addition to one or more of the aspects set forth herein.
Each of the technical means described in the present invention is commercially available or is clear to those skilled in the art, unless otherwise specifically explained.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description will be made with reference to the accompanying drawings. It is obvious that the drawings in the following description are only some examples of the invention, and that for a person skilled in the art, other drawings and embodiments can be derived from them without inventive effort.
It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present application, and the drawings only show the components related to the present application rather than the number, shape and size of the components in actual implementation, and the type, amount and ratio of the components in actual implementation may be changed arbitrarily, and the layout of the components may be more complicated.
In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects may be practiced without these specific details. The terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features described as being defined as "first," "second," etc., may explicitly or implicitly include one or more of the features. In the description of the present invention, "a plurality" means two or more unless otherwise specified.
As shown in fig. 1, the whole system is composed of two parts, one part is a remote driving end 100 which is installed on a remote driving platform; the other part is a vehicle-mounted end 200 which is arranged on a vehicle needing remote driving.
The remote control 100 includes an eye tracking system 101, a camera pose calculation unit 102, and a video receiving and display device 103.
The camera attitude calculation unit 102 calculates the rotation direction of the camera 202 in the horizontal plane (i.e., the rotation about the Z axis) in real time based on the output of the eye tracking system in combination with information such as the mounting position of the camera on the vehicle created in advance, and finally converts the rotation direction into a camera attitude control command, and sends the command to the single-axis servo control system 201 of the vehicle-mounted terminal 200 through a 4G or 5G network.
The video receiving and displaying device 103 is used for receiving real-time video data from the camera 202 on the vehicle-mounted side, and can be any device capable of receiving video from a network and displaying the video.
The vehicle-mounted end 200 includes a single-axis servo control system 201 and a camera 202 mounted at a suitable position on the front of the vehicle.
The single-axis servo control system 201 is a high-precision and high-dynamic micro servo mechanism, and comprises a micro motor, an embedded controller and a corresponding mechanical structure, and the micro motor can control a camera connected to the single-axis servo control system to rotate around a vertical axis and position according to instructions.
The camera 202 is mounted on the single-axis servo control system 201 and is integrally mounted at a suitable position in the front of the vehicle. In this position, images of the front, left front, and right front of the vehicle can be acquired by rotation of the camera.
The data of the camera 202 is turned back to the video receiving and displaying device 103 of the remote driving end 100 in real time through the 4G/5G network.
In summary, the embodiment of the present invention as shown in fig. 2 achieves the following effects:
using an eye tracker 101 to obtain the attention point of an operator to a vehicle-mounted video in the driving process, and adjusting a camera to a desired position in real time;
the eye tracker 101 not only tracks the eyeball movement of the operator, but also tracks the head movement of the operator through a built-in IMU, so that the attention area and position of the operator to the vehicle-mounted video can be ensured to be accurately acquired;
the posture of the camera 202 is adjusted in real time by using the servo control mechanism 201 according to the requirement of an operator, so that the requirement of the operator on vehicle-mounted video in the driving process can be met by only using one camera 202 and one-way high-definition video, and the remote driving of the vehicle is realized under the condition of the current 5G bandwidth. The application range of 5G remote driving is expanded, so that the remote driving method can be applied to real traffic environments.
When the remote driver intends to turn the vehicle to the left, naturally, the operator will turn the head or eyes to the left in order to obtain video data of the left side of the vehicle.
The eye-tracking instrument 101 acquires the horizontal movement distance of the eyeball of the operator or the rotational angle data of the head, and sends it to the camera posture calculation unit 102.
The camera attitude calculation unit 102 converts the eye movement data and the head rotation data into attitude data of the camera 202, and sends it to the single-axis servo controller 201 of the vehicle-mounted end through the wireless network (4G/5G).
The single-axis servo controller 201 controls the motor to rotate according to expected camera attitude data, drives the camera 202 to rotate to a desired position leftwards around a vertical axis, and in the rotating process, the camera 202 acquires video data all the time and sends the video data back to the video receiving and displaying device 103 of the remote driving end through a network.
The whole process is finished at a frequency not lower than 50hz, and the command tracking response delay of the single-shaft servo controller 201 is not more than 5ms, so that the smooth picture and no blockage are ensured in the head or eyeball rotation process of an operator.
It should be noted that the above embodiments can be freely combined as necessary. The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. The remote driving assistance system is applied to assisting a remote driver to remotely drive a remote vehicle, and comprises a first assistance unit and a display unit which are arranged at the remote driving end, and a second assistance unit which is arranged at the remote vehicle end;
the first auxiliary unit is used for acquiring motion information of the remote driver in the remote driving, generating a posture control instruction according to the motion information and sending the posture control instruction to the second auxiliary unit, wherein the motion information is used for representing body motion information of the remote driver to a driving focus point in the remote driving, and the driving focus point is a key point of driving reaction of the remote driver to video data displayed by the display unit according to driving behaviors;
the second auxiliary unit is used for adjusting the acquisition angle of the camera unit according to the attitude control instruction to acquire new video data and transmitting the new video data to the display unit when receiving the attitude control instruction from the first auxiliary unit, so that the remote driver can make driving reaction based on the new video data displayed by the display unit.
2. The assistance system according to claim 1, characterized in that the motion information acquired by the first assistance unit includes eye motion information and/or head motion information of the remote driver.
3. The assistance system according to claim 1 or 2, wherein the motion information acquired by the first assistance unit is from eye motion information of the remote driver acquired by tracking by an eye tracking system;
more preferably, the eye tracking system is provided with an inertial measurement unit for tracking and collecting the head movement information of the remote driver.
4. Auxiliary system according to any of claims 1 to 3,
the attitude control instruction generated by the first auxiliary unit is sent to the second auxiliary unit through a 4G/5G network; and/or
The new video data acquired by the second auxiliary unit is transmitted to the display unit through a 4G/5G network.
5. Assistance system according to claim 4, characterised in that said second assistance unit is provided with a number of cameras of said camera unit not higher than 2, preferably 1.
6. A driving assistance method for remote driving, which is applied to the driving assistance system for remote driving according to any one of claims 1 to 5, the method comprising:
the method comprises the steps that through a first auxiliary unit, motion information of a remote driver in remote driving is obtained, a posture control instruction is generated according to the motion information, and the posture control instruction is sent to a second auxiliary unit;
and when the attitude control instruction is received, the second auxiliary unit adjusts the acquisition angle of the camera shooting unit according to the attitude control instruction to acquire new video data and transmits the new video data to the display unit, so that the remote driver can make a driving reaction based on the new video data displayed by the display unit.
7. The driving assist method according to claim 6, wherein the attitude control command generated by the first assist unit is transmitted to the second assist unit through a 4G/5G network.
8. The driving assist method according to any one of claims 6 or 7, wherein the new video data obtained by the second assist unit is transmitted to the display unit via a 4G/5G network.
9. The driving assist method according to any one of claims 6 to 8, wherein the process of the driving assist method is performed at a frequency of not less than 50 hz.
10. The driving assist method according to any one of claims 6 to 9, wherein a command tracking response delay of the single-axis servo controller is not more than 5 ms.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111322167.6A CN114115246B (en) | 2021-11-09 | 2021-11-09 | Remote driving auxiliary system and method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111322167.6A CN114115246B (en) | 2021-11-09 | 2021-11-09 | Remote driving auxiliary system and method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114115246A true CN114115246A (en) | 2022-03-01 |
| CN114115246B CN114115246B (en) | 2024-01-23 |
Family
ID=80377668
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111322167.6A Active CN114115246B (en) | 2021-11-09 | 2021-11-09 | Remote driving auxiliary system and method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114115246B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20130058991A (en) * | 2011-11-28 | 2013-06-05 | 현대자동차주식회사 | Driving assistant system having a function of gesture recognition |
| EP3530521A1 (en) * | 2018-02-22 | 2019-08-28 | Continental Automotive GmbH | Driver assistance method and apparatus |
| CN110228484A (en) * | 2019-06-17 | 2019-09-13 | 福州视驰科技有限公司 | A kind of low time delay intelligent remote control loop driving function with auxiliary |
| CN110850711A (en) * | 2019-12-06 | 2020-02-28 | 中国科学院自动化研究所 | Auxiliary driving control system and method based on cloud |
| KR20200141428A (en) * | 2020-12-09 | 2020-12-18 | 현대모비스 주식회사 | Method for driver attention-based video output and vehicle control determine |
| US20210237744A1 (en) * | 2018-04-25 | 2021-08-05 | Hitachi Transport System, Ltd. | Management assistance system |
-
2021
- 2021-11-09 CN CN202111322167.6A patent/CN114115246B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20130058991A (en) * | 2011-11-28 | 2013-06-05 | 현대자동차주식회사 | Driving assistant system having a function of gesture recognition |
| EP3530521A1 (en) * | 2018-02-22 | 2019-08-28 | Continental Automotive GmbH | Driver assistance method and apparatus |
| US20210237744A1 (en) * | 2018-04-25 | 2021-08-05 | Hitachi Transport System, Ltd. | Management assistance system |
| CN110228484A (en) * | 2019-06-17 | 2019-09-13 | 福州视驰科技有限公司 | A kind of low time delay intelligent remote control loop driving function with auxiliary |
| CN110850711A (en) * | 2019-12-06 | 2020-02-28 | 中国科学院自动化研究所 | Auxiliary driving control system and method based on cloud |
| KR20200141428A (en) * | 2020-12-09 | 2020-12-18 | 현대모비스 주식회사 | Method for driver attention-based video output and vehicle control determine |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114115246B (en) | 2024-01-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109891875B (en) | Image transmission apparatus, method, system, and storage medium | |
| JP7200946B2 (en) | Information processing device, control system, information processing method and program | |
| EP3787257A1 (en) | Patrol robot and patrol robot management system | |
| KR101544524B1 (en) | Display system for augmented reality in vehicle, and method for the same | |
| JP7331696B2 (en) | Information processing device, information processing method, program, and mobile object | |
| CN106527426A (en) | Indoor multi-target track planning system and method | |
| US20110228047A1 (en) | Method and apparatus for displaying stereographic images | |
| CN108495048B (en) | Double-camera image acquisition equipment based on holder control | |
| WO2004039077A1 (en) | Bionic vision and sight automatic control system and method | |
| JP7248037B2 (en) | Image processing device, image processing method, and program | |
| CN104781873A (en) | Image display device and image display method, mobile body device, image display system, and computer program | |
| CA2642591A1 (en) | Method for teleoperating an unmanned ground vehicle with a pan camera and such a ground vehicle | |
| CN107404642A (en) | The dynamic vergence of binocular display apparatus | |
| EP3174290B1 (en) | Method for generating an enhanced video signal and providing the enhanced video signal to the driver of a vehicle that is moving at a certain speed, system for generating an enhanced video signal, program and computer program product | |
| CN111886854B (en) | Exposure control device, exposure control method, program, imaging device, and moving object | |
| CN113473020A (en) | Focus following device control method, device and system | |
| CN110803109A (en) | Vehicle-mounted camera system and image acquisition method thereof | |
| CN102591014B (en) | Panoramic vision observing system and work method thereof | |
| CN114115246A (en) | Remote driving auxiliary system and method thereof | |
| US8538600B2 (en) | Method for providing control commands for a vehicle steerable by means of an operating unit and control system for a steerable vehicle of this type | |
| CN1873439A (en) | Low-level automatic tracking system of ground motion meter gauge based on control of bionic human eye | |
| CN216927439U (en) | Crawler-type unmanned target vehicle remote control system | |
| CN205356558U (en) | System is watched in remote control and make a video recording and hold and watch end thereof | |
| US20220244726A1 (en) | Information processing apparatus, information processing method, and program | |
| Sanders-Reed et al. | Vision systems for manned and robotic ground vehicles |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |