WO2023039650A1

WO2023039650A1 - Land vehicle remote driving system and method

Info

Publication number: WO2023039650A1
Application number: PCT/BR2022/050263
Authority: WO
Inventors: Rafael Da Rolt D'AGOSTINI
Original assignee: Dagostini Rafael Da Rolt
Priority date: 2021-09-16
Filing date: 2022-07-18
Publication date: 2023-03-23
Also published as: BR102021018438A2

Abstract

The present invention relates to a system (1) and method for remote driving of vehicles (2), especially land vehicles that circulate on public or private roads, comprising a vehicle (2) and a remote driving station (9), with communication between the vehicle (2) and the remote driving station (9) over a 5G or higher communications network of commercial telephone operators by means of a 5G or higher communications module (18) mounted in the vehicle (2) and another 5G or higher communications module (18) mounted in the remote station (9), wherein the communications module (18) mounted in the vehicle (2) transmits in real time images and sounds captured in the vehicle (2) to the remote station (9) and receives from the latter commands that are remotely sent by the remote driver (6); and an autonomous driving system (19) to stop the vehicle (2) safely when the loss of the real-time characteristic is detected in the data traffic.

Description

SYSTEM AND METHOD FOR REMOTE DRIVING OF LAND VEHICLES

[0001] The present invention relates to a system and a method for remote driving of vehicles, especially for land vehicles traveling on roads open to the public or private roads. This system and method does not seek to replace fully autonomous vehicle driving systems, but seeks to meet the demand for which fully autonomous driving is not safe or functional, therefore being a hybrid system between the autonomous system and the remote system. .

STATE OF THE TECHNIQUE

[0002] There are several commercially available systems or experimental systems that perform autonomous vehicle driving. In some of these systems, the presence of a driver inside the vehicle is required to intervene in the system in case of malfunction. Other systems already support fully remote driving, especially in aircraft and vehicles operating on private roads. It is observed that such systems often fail when faced with unforeseen situations, for which they were not trained. Such events often result in accidents.

SUMMARY OF THE INVENTION

[0003] The present invention seeks to realize a driving system and a hybrid method, in which remote driving is possible, complemented by an autonomous driving system, which only operates in emergency mode, when there is a failure in communication between the vehicle and the remote drive station, driving the vehicle to a safe stop, or returning drive to the remote drive station if communication is re-established.

[0004] Many roads, especially in developing countries, have a single track, in which vehicles travel in opposite directions separated by a few centimeters, many of these roads have winding stretches of mountains, which often do not have a shoulder. To make matters worse, such roads are often potholed or have poor or non-existent visual signage, and many of these roads are not even paved. Such factors preclude the use of autonomous driving systems known in the state of the art. Of this In this way, the remote driving system proves to be much safer and more reliable, being the only one capable of operating in such conditions.

[0005] As it is a remote system, it allows a public transport company for people or cargo transport to have a group of drivers who can work at the company's headquarters, without the need for them to travel with such vehicles , reducing costs and improving quality of life for drivers and their families. This system also allows the different drivers to take turns in shorter intervals than if they were driving the vehicle, since it is very common for drivers to drive from dawn to dusk without being able to take turns driving. In non-commercial use, this system can be used by citizens in their daily activities, such as taking the child to school or the elderly to the doctor, even if they live in different cities, states or even countries.

[0006] This system operates using commercial communication networks, which are operated by third parties. In such communication systems, there are interruptions in service continuity, a fact observed by any user of real-time communication applications or video and audio streaming transmissions. Such interruptions can be caused by overload in the system of the communication operators or can be caused by shadows in the coverage areas, a fact especially observed in moving vehicles.

[0007] Current commercial communication systems, that is, fifth generation communication (5G) or higher, allow high data rate traffic, which allows the implementation of systems such as the one proposed herein.

DETAILED DESCRIPTION OF THE INVENTION

[0008] The present invention can be better understood considering the following figures:

Figure 1 : Remote driving system;

Figure 2: Real-time operation check.

[0009] In Figure 1, we have the system for remote driving (1) of a land vehicle, comprising a vehicle (2) equipped with high-resolution cameras (3) for capturing images in real time, containing at least four cameras (3) : one facing the front of the vehicle (2), one facing the rear, one facing the left and one facing the right. It also comprises microphones (4) installed in the vehicle (2), containing at least two microphones, each facing one side, installed in the virtual driver's head position (5), which allow the remote driver (6) to see and hear just as if you were sitting in the virtual driver's seat (7). Such system (1) also comprises actuators (8) installed in the vehicle with remote actuation, such actuators being a brake pedal actuator, accelerator pedal actuator, clutch pedal actuator in manual transmission cars, gear selector lever actuator gearbox for both manual and automatic vehicles, steering wheel actuator, arrow key actuator, horn actuator, windshield wiper actuator, headlight actuator and other systems essential for driving the vehicle. It should be noted that the vehicle (2) also generates data related to its operation, such as speed, engine speed, engine temperature, fuel line pressure, among others. This data can be obtained from the OBDII port available in vehicles in general. Such data are transmitted from time to time from the vehicle (2) to the remote station (9), since they do not interfere with the real-time driving of the vehicle, however, they are of great importance for remote diagnosis and to avoid traffic violations, such as in the case of the vehicle speed value (2). All devices installed in the vehicle (2) related to the remote driving system (1) are commanded by a microprocessed central A (22) installed in the vehicle (2).

[0010] Such a system also comprises a remote station (9), fixedly installed and remote from the vehicle (2), in which the remote driver (6) has access to all available controls for driving the vehicle (2 ). This station (9) comprises a remote driver's seat (10) and sensors (11) for all commands available for driving the vehicle (2), such commands being a brake pedal, an accelerator pedal, a clutch pedal , a gearshift lever, a steering wheel, an arrow key, a horn button, a windshield wiper switch, a headlight switch and other essential systems for driving the vehicle. All these remote station commands (9) have sensors (11) suitable for reading the applied commands, and such readings simply comprise an on/off sensor, for on/off switches and force/position sensors, for representative commands. of the force required by the driver, as in the case of acceleration and braking. All these sensors (11) are already known by a person skilled in the art.

[0011] The remote driver (6) is provided with augmented reality glasses (12), that is, a kind of glasses equipped with a high-definition color screen and speakers (14) that reproduce the captured images and sounds from the point of view of the virtual driver's seat (7) installed in the vehicle (2). In such glasses (12) an acceleration sensor with three x,y,z axes (15) is installed, which captures the movements of the head of the remote driver (16) and transmits on the screen in front of him the image corresponding to the angle sought through the head of the remote conductor (16). Alternatively, a system equipped with high definition monitors (17) can also be built, involving the remote station (9) and eliminating the use of augmented reality glasses (12). All devices installed in the remote driving station (9) related to the remote driving system (1) are commanded by a microprocessed central B (23) installed in the station (9).

[0012] The communication between the vehicle (2) and the remote station (9) is carried out by the 5G or higher communication network of commercial telephone operators. For this, a 5G or higher communication module (18) is installed in the vehicle (2) and another 5G or higher communication module (18) is installed in the remote station (9). The communication module (18) installed in the vehicle (2) transmits in real time the images and sounds captured in the vehicle (2) to the remote station (9) and from this receives the commands sent remotely by the remote driver (6). The communication module (18) installed in the remote station (9) transmits in real time the commands generated by the remote driver (6) and receives the images and sounds captured in the vehicle (2). The cited communication modules (18) comprise commercial modules available on the market, known by the person skilled in the art.

[0013] The fundamental issue for the safe operation of the system for remote driving (1 ) is the operation in real time, that is, that the image and audio data collected in the vehicle (2) from the point of view of the virtual driver's head (5) reach the remote conductor (6) in real time or in the shortest possible time. It is also essential that the commands given by the remote driver (6) reach the actuators (8) installed in the vehicle (2) in real time or in the shortest possible time.

[0014] Considering that the traffic of such data occurs through 5G or higher communication networks of commercial telephone operators, it must be considered that the system of such operators can often be overloaded and not be able to carry out the data traffic in time real. Another factor is that, due to the fact that the vehicle (2) is in motion, it may occupy, for a few moments, places where the signal is shadowed by such telephone companies, which could result in serious accidents, as the image, sounds and commands to the vehicle (2) would be delayed. Such a delay could make the vehicle (2) make a curve outside the exact geographic coordinates of the same, continue in a straight line when in a curve, or invade the opposite lane, colliding with other vehicles that eventually travel in the opposite direction.

[0015] In this way, an autonomous driving system (19) is indispensable to make a safe stop of the vehicle (2), when detecting the loss of the real-time characteristic in the data traffic.

[0016] To verify the synchronism in the real-time operation, the vehicle (2) does not need a verification in which it guarantees that the commands received from the remote station (9) are adequate for that traffic condition. [0017] GPS satellite positioning systems emit time signals, which could be used to verify whether the command received from the remote station (9) is current or out of date for the operation of the vehicle (2). However, it is preferable that such GPS signals are used for purposes other than synchronism verification for real-time operation of the proposed system.

[0018] The commercially available GPS modules transfer the data calculated by triangulating the satellites in Earth orbit through a communication port using communication protocols already known in the state of the art. Such protocols, such as NMEA 0183, update their data through the communication port every second, and this time base is inadequate for the real-time operation of the proposed system.

[0019] For synchronization purposes, in the proposed system, in addition to sending images and audio from the vehicle (2) to the remote station (9) in real time, a sequential code is also sent, all via the communication module (18 ). Such sequential code is incremented at each transmission, and when such code reaches its maximum value, which depends on the number of bits in the register, it is reset to zero and the count starts again from zero.

[0020] In figure 2 we have the verification of operation in real time. When sending the first sequential code (step A), the part of the system (1) installed in the vehicle (2) starts a timer. The remote station (9), upon receiving this first sequential code, via the communication module (18), will send it back to the vehicle (2) without performing any operation with such sequential code.

[0021] The vehicle time counter (2) keeps counting until it receives and identifies the sequential code sent to the remote station (9) or until it reaches a maximum predefined time value as a limit for real-time operation (step B ).

[0022] If the time counter receives and identifies the sequential code sent to the remote station (9), the system (1) understands that real-time communication (RT) is taking place and the vehicle remains with the remote driver ( 6), returning the system to step A. Thus, the sequential code is incremented, sent to the remote station (9) and the time counter installed in the vehicle (2) is reset and starts counting again.

[0023] If the time counter reaches the maximum predefined time value as a limit for real-time operation, without having received or identified the sequential code sent to the remote station (9), the system (1) understands that the communication in real time is lost (OL) and transfers the driving of the vehicle (2) to the autonomous driving system (19), seeking a safe stop for the vehicle (2) (step C). Systems of autonomous driving in general are already known in the state of the art and are not object of the present invention. The autonomous driving system (19), when looking for a safe stop for the vehicle (2), performs the visual signal, for example, by the brake lights and hazard warning lights and also performs the sound signal, issuing an audio message or activating any vehicle siren or horn (2).

[0024] The maximum predefined time value means a time value in which the vehicle moves laterally on the road for a distance of less than 0.25 meters, which could lead such vehicle (2) to leave the lane or invade the oncoming lane. This lateral distance value is merely illustrative and may obviously vary depending on the width of the lane in which the vehicle (2) is traveling. Consequently, the predefined maximum time value also depends on the speed at which the vehicle (2) is traveling, and the higher the speed, the lower this value will be. As a merely illustrative and non-limiting value of the present invention, a maximum time value of 0.1 seconds can be cited.

[0025] In order for the driving of the vehicle (2) to be returned to the remote driver (6) in the event of an emergency stop, the system (1) must verify the resumption or not of the operating condition in real time (step D) For such verification, the system uses the same method that it uses for the verification of the operation in real time with some modifications. The part of the system (1 ) installed in the vehicle sends a first sequential code again and starts the time counter. The remote station (9), upon receiving this first sequential code via the communication module (18), will send it back to the vehicle (2) without performing any operation with such sequential code. The vehicle's time counter (2) follows its count until it receives and identifies the sequential code sent to the remote station (9). If the time counter has not reached the maximum predefined time value as a limit for real-time operation when identifying the sequential code, the part of the system (1 ) installed in the vehicle (2) understands that real-time communication has been reestablished, however, before returning the driving of the vehicle (2) to the remote driver (6), the system (1) remains testing the connection for a predetermined time (CT), as an example we can cite a test for a period of 30 seconds. If, within this predetermined time, the connection does not lose its real-time operation feature, driving the vehicle (2) is returned to the remote driver (6) and the system returns to operating in step A.

[0026] In order to enhance safety in the operation of the system (1), a driver can be allocated in the virtual driver's seat (7) installed in the vehicle (2), and this driver can share driving the vehicle (2) with the remote driver (6) and act so that the vehicle (2) can be driven when real-time operation is not possible.

[0027] While the remote conduction system (1) is operating in real time (step A), that is, it is receiving the codes sequential codes in a time shorter than the predefined maximum time value, the system (1) keeps incrementing and sending sequential codes to the remote station (9) whenever it receives and identifies the sequential code sent in the immediately previous iteration.

Claims

8

1 System for remote driving (1) of a land vehicle, characterized in that it comprises a vehicle (2) and a remote driving station (9), whereby communication between the vehicle (2) and the remote driving station (9) is performed by the 5G or higher communication network of commercial telephone operators through a 5G or higher communication module (18) installed in the vehicle (2) and another 5G or higher communication module (18) installed in the remote station (9), being that the communication module (18) installed in the vehicle (2) transmits in real time the images and sounds captured in the vehicle (2) to the remote station (9) and from this receives the commands sent remotely by the remote driver (6); and, an autonomous driving system (19) to make a safe stop of the vehicle (2), when detecting the loss of the real-time characteristic in the data traffic.

2 System according to claim 1, characterized in that the vehicle (2) comprises high resolution cameras (3) for capturing images in real time, containing at least four cameras (3), one facing the front of the vehicle (2), one facing the rear, one facing the left and one facing the right; microphones (4) installed in the vehicle (2), containing at least two microphones, each facing one side, installed in the virtual driver's head position (5), actuators (8) installed in the vehicle with remote activation, such actuators , a brake pedal actuator, accelerator pedal actuator, clutch pedal actuator on manual transmission cars, shift selector lever actuator for both manual and automatic vehicles, steering wheel actuator, arrow switch actuator, horn actuator, actuator for activating the windshield wiper, actuator for activating the headlights and other systems essential for driving the vehicle, all devices installed in the vehicle (2) related to the system for remote driving (1) controlled by a central microprocessor A (22) installed in the vehicle (2).

3 System according to claim 2, characterized in that the vehicle (2) generates data such as speed, engine speed, engine temperature, fuel line pressure, among others, and communicates such data via OBDII port to the central A (22), such data being transmitted from time to time from the vehicle (2) to the remote station (9).

4 System according to claim 1, characterized in that the remote station (9) is installed in a fixed and remote way in relation to the vehicle (2), in which the remote driver (6) has access to all available commands for driving the vehicle (2) and comprises a remote driver's seat (10) and sensors (11) for all controls available for driving the vehicle (2), such controls being a brake pedal, an accelerator pedal, a clutch pedal, a gear lever, a steering wheel 9 steering, an arrow key, a horn button, a windshield wiper activation key, a headlight activation key and other essential systems for driving the vehicle, all devices installed in the remote driving station ( 9) related to the remote driving system (1) commanded by a microprocessed central B (23) installed in the station (9).

5 System according to claim 4, characterized in that the remote station (9) has sensors (11) suitable for reading the commands applied, and such readings comprise an on/off sensor, for on/off switches and force/position sensors, for commands representative of the force required by the driver, as in the case of acceleration and braking.

6 System according to claim 4, characterized in that the remote station (9) comprises augmented reality glasses (12) made available to the remote driver (6), said glasses (12) comprising a high definition color screen and speakers (14) that reproduce the images and sounds captured from the point of view of the virtual driver's seat (7) installed in the vehicle (2), and a three-axis x,y,z acceleration sensor (15) , which captures the movements of the remote conductor's head (16) and transmits on the screen in front of it the image corresponding to the angle sought by the remote conductor's head (16).

7 System according to claim 4, characterized in that the remote station (9) comprises, alternatively, high definition monitors (17), which surround the remote station (9) and dispense with the use of augmented reality glasses (12).

8 Method for remotely driving a land vehicle characterized by capturing and transmitting images and audio in real time in the vehicle (2) through high-resolution cameras (3) and microphones (4) installed in the vehicle (2), acting on actuators ( 8) installed in the vehicle (2) through the commands read by the sensors (11) installed and sent in real time by the remote station (9), communicate the vehicle (2) and the remote station (9) through the communication network 5G or higher from commercial telephone operators through a 5G or higher communication module (18) installed in the vehicle (2) and another 5G or higher communication module (18) installed in the remote station (9), check the communication synchronism between the vehicle (2) and the remote station (9) by sending a sequential code via the communication module (18); and, transferring the driving of the vehicle (2) to the autonomous driving system (19) for a safe stop of the vehicle when verifying that the real-time communication between the vehicle (2) and the remote station (9) has been lost. 10

Method according to claim 8, characterized by checking the synchronism by sending a sequential code from the vehicle (2) via the communication module (18), incrementing the sequential code at each transmission, starting a timer in the vehicle (2), receive the first sequential code at the remote station (9) via the communication module (18), send back to the vehicle (2) the sequential code without performing any operation, increment the vehicle's time contactor (2) until receipt and identification of the sequential code sent to the remote station (9) or until reaching a predefined maximum time value as a limit for real-time operation, if the time counter receives and identifies the sequential code sent to the remote station (9 ), the system (1) understands that real-time communication (RT) is taking place and the vehicle remains driving with the remote driver (6), if the time counter reaches the maximum predefined time value as a limit for operation inreal time, without having received or identified the sequential code sent to the remote station (9), the system (1) understands that the real time communication has been lost (OL) and transfers the driving of the vehicle (2) to the autonomous driving (19), seeking a safe stop for the vehicle (2), and the autonomous driving system (19), when seeking a safe stop for the vehicle (2), performs the visual signaling, through the lights of brake and hazard warning lights and also performs sound signaling, issuing an audio message or activating a siren or horn on the vehicle (2).

10 Method according to claim 9, characterized by returning or not driving the vehicle (2) to the remote driver (6) in the event of an emergency stop when checking the resumption or not of the operating condition in real time by means of after sending a sequential code from the vehicle (2) via the communication module (18), increment the sequential code with each transmission, start a time counter in the vehicle (2), receive the first sequential code at the remote station (9) via the communication module (18), send the sequential code back to the vehicle (2) without performing any operation, increase the vehicle's time contactor (2) until the sequential code sent to the remote station (9) is received and identified, or until reaching a maximum predefined time value as a limit for real-time operation, in case the time counter receives and identifies the sequential code sent to the remote station (9) and has not yet reached the maximum predefined time value as a limit for real-time operation when identifying the sequential code, the part of the system (1) installed in the vehicle (2) understands that the real-time communication has been reestablished, however, before returning the vehicle (2) to the remote driver ( 6), the system (1) remains testing the connection for a predetermined time (CT), and if in this predetermined time the connection does not lose its real-time operation characteristic, driving the vehicle (2) is returned to the driver remote (6).