CN107505944B

CN107505944B - Method and device for remotely assisting vehicle

Info

Publication number: CN107505944B
Application number: CN201710891487.0A
Authority: CN
Inventors: 吴甘沙; 张玉新; 周鑫
Original assignee: Uisee Technologies Beijing Co Ltd
Current assignee: Uisee Technologies Beijing Co Ltd
Priority date: 2017-09-27
Filing date: 2017-09-27
Publication date: 2021-04-16
Anticipated expiration: 2037-09-27
Also published as: CN107505944A

Abstract

The method comprises the steps that a vehicle determines remote assistance requirement degree information of the vehicle based on vehicle sensing information of the vehicle, and the remote assistance requirement degree information is sent to a remote device; the remote equipment receives remote assistance demand degree information sent by a plurality of vehicles, and one or more monitored vehicles are optimized from the plurality of vehicles according to the remote assistance demand degree information; the remote device performs a remote assistance operation with respect to at least one monitored vehicle of the one or more monitored vehicles. The remote control vehicle assisting system can greatly improve the operating efficiency of a remote operator for remotely assisting the vehicle.

Description

Method and device for remotely assisting vehicle

Technical Field

The present application relates to the field of autopilot, and more particularly to a technique for remotely assisting a vehicle.

Background

The unmanned automobile or the automobile with the driving assisting function can sense the ambient environment information and the self state information of the automobile by utilizing a rich sensing system, the control system makes a decision, and the execution system drives the automobile to perform self running actions (such as steering, acceleration/deceleration and the like), so that the operation interface of a human driver on the automobile is reduced or eliminated. During driving, the unmanned vehicle may encounter various unexpected situations. For example, the environment of the vehicle is extremely complex, and the vehicle cannot make decisions by a perception system and a control system of the vehicle; some systems of the vehicle are in failure, so that the vehicle loses the control capability; and the like.

In the prior art, there is a remote control method for an unmanned vehicle, in which a remote operator remotely takes care of an unmanned vehicle, and once the unmanned vehicle needs remote assistance, the remote operator remotely controls the unmanned vehicle according to the current state of the vehicle and the environmental information of the vehicle. However, as the number of unmanned vehicles increases, accidents that need to be dealt with also increases, and the human input of the remote operator becomes enormous.

Disclosure of Invention

It is an object of the present application to provide a method for remote assistance with a vehicle.

According to one aspect of the present application, there is provided a method at a vehicle end for remote assistance with a vehicle, the method comprising:

determining remote assistance desirability information of a vehicle based on vehicle sensing information of the vehicle;

and sending the remote assistance demand degree information to remote equipment.

According to another aspect of the present application, there is provided a method for remotely assisting a vehicle at a remote device side, the method comprising:

receiving remote assistance demand degree information sent by a plurality of vehicles;

preferentially selecting one or more monitored vehicles from the plurality of vehicles according to the remote assistance demand degree information;

performing a remote assistance operation with respect to at least one of the one or more monitored vehicles.

According to one aspect of the present application, there is provided an apparatus at a vehicle end for remote assistance with a vehicle, the apparatus comprising:

the system comprises a calculation module, a display module and a display module, wherein the calculation module is used for determining the remote assistance demand degree information of a vehicle based on the vehicle sensing information of the vehicle;

and the sending module is used for sending the remote assistance demand degree information to remote equipment.

According to another aspect of the present application, there is provided an apparatus for remotely assisting a vehicle at a remote device side, wherein the apparatus includes:

the receiving module is used for receiving remote assistance demand degree information sent by a plurality of vehicles;

the selection module is used for preferably selecting one or more monitored vehicles from the plurality of vehicles according to the remote assistance demand degree information;

an execution module to execute a remote assistance operation with respect to at least one monitored vehicle of the one or more monitored vehicles.

According to one aspect of the present application, there is provided a method for remotely assisting a vehicle, the method comprising:

the method comprises the steps that a vehicle determines remote assistance requirement degree information of the vehicle based on vehicle sensing information of the vehicle, and the remote assistance requirement degree information is sent to remote equipment;

the remote equipment receives remote assistance demand degree information sent by a plurality of vehicles, and one or more monitored vehicles are optimized from the plurality of vehicles according to the remote assistance demand degree information;

the remote device performs a remote assistance operation with respect to at least one monitored vehicle of the one or more monitored vehicles.

According to one aspect of the present application, there is provided a vehicle for remote assistance with the vehicle, the vehicle comprising:

a processor; and

a memory arranged to store computer executable instructions that, when executed, cause the processor to:

According to another aspect of the present application, there is provided a remote device for remote assistance with a vehicle, the remote device comprising:

a processor; and

According to one aspect of the present application, there is provided a computer-readable medium comprising instructions that, when executed, cause a system to:

According to another aspect of the application, there is provided a computer-readable medium comprising instructions that, when executed, cause a system to:

Compared with the prior art, the remote assistance demand degree information sent by the plurality of vehicles is received, the vehicles needing remote assistance are preferably selected based on the running credibility information of the vehicles, fewer remote operators are used for monitoring and remotely controlling more vehicles, the operation efficiency of the remote operators for remotely assisting the vehicles is greatly improved, and manpower is saved.

Drawings

Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:

FIG. 1 illustrates a flow diagram of a method for remotely assisting a vehicle according to one embodiment of the present application;

FIG. 2 illustrates a schematic diagram of a system for remotely assisting a vehicle according to another embodiment of the present application;

FIG. 3 shows a flow chart of a method for remotely assisting a vehicle at a vehicle end according to another embodiment of the present application;

FIG. 4 illustrates a flow chart of a method for determining a degree of need for remote assistance based on vehicle sensory information according to another embodiment of the present application;

FIG. 5 shows a flow chart of a method for remote assistance with a vehicle at a remote device end according to another embodiment of the present application;

FIGS. 6-8 each illustrate a flow diagram for remotely assisting a vehicle according to one embodiment of the present application;

fig. 9 is a schematic structural view of a vehicle-side remote assistance apparatus for remotely assisting a vehicle at a vehicle side according to another embodiment of the present application;

FIG. 10 is a block diagram of a computing module for determining a degree of need for remote assistance based on vehicle sensory information according to another embodiment of the present application;

fig. 11 is a schematic structural diagram of a remote equipment-side remote assistance apparatus for remotely assisting a vehicle at a remote equipment side according to another embodiment of the present application;

FIG. 12 illustrates a schematic structural diagram of an on-board system of a vehicle according to another embodiment of the present application;

FIG. 13 shows a schematic diagram of a remote device according to another embodiment of the present application;

the same or similar reference numbers in the drawings identify the same or similar elements.

Detailed Description

The present application is described in further detail below with reference to the attached figures.

In a typical configuration of the present application, the terminal, the device serving the network, and the trusted party each include one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

The device referred to in this application includes, but is not limited to, a user device, a network device, or a device formed by integrating a user device and a network device through a network. The user equipment includes, but is not limited to, any mobile electronic product, such as a smart phone, a tablet computer, etc., capable of performing human-computer interaction with a user (e.g., human-computer interaction through a touch panel), and the mobile electronic product may employ any operating system, such as an android operating system, an iOS operating system, etc. The network device includes an electronic device capable of automatically performing numerical calculation and information processing according to a preset or stored instruction, and hardware thereof includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like. The network device includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud of a plurality of servers; here, the Cloud is composed of a large number of computers or web servers based on Cloud Computing (Cloud Computing), which is a kind of distributed Computing, one virtual supercomputer consisting of a collection of loosely coupled computers. Including, but not limited to, the internet, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless Ad Hoc network (Ad Hoc network), etc. Preferably, the device may also be a program running on the user device, the network device, or a device formed by integrating the user device and the network device, the touch terminal, or the network device and the touch terminal through a network.

Of course, those skilled in the art will appreciate that the foregoing is by way of example only, and that other existing or future devices, which may be suitable for use in the present application, are also encompassed within the scope of the present application and are hereby incorporated by reference.

In the description of the present application, "a plurality" means two or more unless specifically limited otherwise.

Fig. 1 shows a flowchart of a method for remotely assisting a vehicle, in which the vehicle determines remote assistance desirability information of the vehicle based on vehicle sensing information of the vehicle in step S11 and transmits the remote assistance desirability information to a remote device in step S12; the remote device receives the remote assistance required degree information transmitted from the vehicle in step S21, prefers one or more monitored vehicles from the plurality of vehicles according to the remote assistance required degree information vehicle traveling reliability information in step S22, and performs a remote assistance operation with respect to at least one monitored vehicle of the one or more monitored vehicles in step S23.

The vehicle sensing information is from a sensing system of the vehicle, and the remote assistance demand degree information of the vehicle is obtained by vehicle-mounted or cloud computing based on the vehicle sensing information. In some embodiments, the vehicle sensing information is used to determine the vehicle state and the vehicle surrounding environment condition (e.g., surrounding environment complexity), and the remote assistance desirability information calculated according to the vehicle state and the vehicle surrounding environment condition is used to evaluate whether the vehicle needs remote assistance from the remote device. After receiving the remote assistance requirement degree information from the plurality of vehicles, the remote equipment preferably selects one or more of the vehicles as monitored vehicles according to the remote assistance requirement degree information, so that a remote operator can monitor the monitored vehicles and remotely assist the monitored vehicles when necessary; or preferably one or more of, as a monitored vehicle and provide remote assistance to the monitored vehicle when necessary, including but not limited to allocating cloud computing resources to the vehicle, providing a high precision environmental map of where the vehicle is located, providing environmental awareness resources of V2I devices or remote drones or the like to the vehicle, and the like.

Fig. 2 is a schematic structural diagram of a possible system for remotely assisting a vehicle, where a plurality of vehicles send remote assistance requirement information to a remote device, and after the remote device receives the remote assistance requirement information sent by the vehicle, the vehicles are sorted and presented based on the remote assistance requirement information, for example, after the vehicles are arranged from high to low according to the remote assistance requirement information, inherent information of the corresponding vehicles, vehicle sensing information (including but not limited to image sensing information, position sensing information, speed sensing information, fault sensing information, etc.) and the like are presented on a plurality of displays or in a plurality of areas of one display.

In various embodiments, the remote assistance desirability information may exist in various forms including, but not limited to, vehicle autonomous driving reliability information, vehicle accident rate information, vehicle serious failure rate information, and the like. Accordingly, the manner in which the remote device prefers the monitored vehicle from among the plurality of vehicles that transmit the remote assistance desirability information may also differ corresponding to different forms of the remote assistance desirability information. For example, in some embodiments, the remote assistance demand degree information includes vehicle accident rate information or vehicle serious failure rate information, and when the remote assistance demand degree information of the vehicle is higher, the vehicle is more likely to fail to complete autonomous driving and need remote assistance, so the remote device preferably selects the vehicle with higher remote assistance demand degree information as the monitored vehicle, for example, preferably selects the vehicle with higher remote assistance demand degree information than a preset demand degree threshold value as the monitored vehicle. In other embodiments, the remote assistance demand degree information is in the form of vehicle autonomous driving reliability information, and since the lower the vehicle autonomous driving reliability is, the more likely the vehicle is unable to complete autonomous driving and needs remote assistance, the remote device preferably selects a vehicle with lower vehicle autonomous driving reliability information as the monitored vehicle, for example, a vehicle with a remote assistance demand degree higher than a preset demand degree threshold value is selected as the monitored vehicle, where the remote assistance demand degree information is related to the vehicle autonomous driving reliability, and the lower the vehicle autonomous driving reliability is, the higher the remote assistance demand degree information is.

Of course, it should be understood by those skilled in the art that the above-described preferred monitored vehicle is by way of example only, and that other existing or future preferred monitored vehicle configurations, as may be suitable for use in the present application, are also within the scope of the present application and are hereby incorporated by reference. In particular, the vehicle can also determine the vehicle running credibility information thereof based on the vehicle sensing information thereof and transmit the vehicle running credibility information to the remote device; the remote device receives the vehicle running reliability information sent by the plurality of vehicles, and prefers one or more monitored vehicles from the plurality of vehicles according to the vehicle running reliability information, for example, prefers a fixed number of vehicles with low vehicle running reliability information as the monitored vehicles, or prefers vehicles with vehicle running reliability information lower than a preset reliability threshold value as the monitored vehicles. At this time, the vehicle running reliability information reflects the degree of the vehicle needing remote assistance from another angle, that is, the remote assistance requirement degree of the vehicle is reflected from another angle, and both the vehicle running reliability information and the remote assistance requirement degree can substantially realize the measurement of the degree of the vehicle remote assistance requirement. In some embodiments, the vehicle travel reliability information and the remote assistance request degree are traded off, for example, they are inversely proportional, or added together to a fixed value.

According to one aspect of the present application, there is provided a method for remotely assisting a vehicle at a vehicle end, the method including steps S11 and S12. Referring to fig. 3, in step S11, the vehicle determines remote assistance necessity degree information of the vehicle based on vehicle sensing information of the vehicle; in step S12, the vehicle transmits the remote assistance required degree information to the remote device.

Specifically, in step S11, the vehicle determines the remote assistance necessity degree information of the vehicle based on the vehicle sensing information of the vehicle. The vehicle is provided with a rich perception system, for example, the vehicle is provided with a sensing module consisting of a plurality of sensors, or is further provided with a database corresponding to sensing information of the sensing module; in some embodiments, the vehicle may be unmanned or have a drive-assist function. The sensors in the sensing module include, but are not limited to, a sound sensor for sensing environmental or vehicle interior sound, a pressure sensor for sensing pressure of various pipelines or squeezing pressure between components of the vehicle, a vehicle speed sensor for sensing vehicle running speed, a steering angle sensor for sensing vehicle steering angle or a sensor for sensing other vehicle state signals, and may further include a laser radar, a camera, a millimeter wave radar or a sensor for sensing other vehicle peripheral signals.

In some embodiments, the vehicle may calculate the remote assistance desirability information of the vehicle based on the sensing data collected by the sensors. In other embodiments, after the vehicle collects the sensing data of the sensors in real time, the vehicle performs multi-sensing information fusion based on the corresponding database, and calculates the remote assistance demand degree information of the vehicle. For example, the database includes a fault alarm database and a situation alarm database, wherein the alarm information in the fault alarm database includes, but is not limited to, actuator fault alarm information (e.g., brake and steering system fault alarm information, throttle and suspension fault alarm information), sensor fault alarm information (e.g., environment sensing and navigation positioning related sensor fault alarm information, vehicle entertainment and interaction system related sensor fault alarm information), computing module fault alarm information, and the like, and the alarm information in the situation alarm database includes, but is not limited to, weak communication signal strength, frequent switching of steering/brake/throttle, and the like, uncertain types of obstacles, uncertain positions, being trapped in a certain posture, uncertain types of environments/objects, being trapped overtime, and the like. The alarm information can determine a weight according to the safety relevance between the alarm information and the safety of the passengers and the pedestrians, and participate in the calculation of the information of the remote assistance demand degree, for example, the higher the safety relevance between the passengers and the pedestrians is, the larger the weight participating in the calculation is, and the larger the influence on the remote assistance demand degree is. Taking the alarm information in the fault alarm database as an example, the fault of the braking and steering system has larger influence on the safety of passengers and pedestrians than the fault of the accelerator and the suspension, so that the weight value of the fault alarm information of the braking and steering system is higher than that of the fault alarm information of the accelerator and the suspension; the related sensor faults of the environment sensing and navigation positioning have larger influence on the safety of passengers and pedestrians than the related sensor faults of the vehicle-mounted entertainment and interaction system, so the weight value of the related sensor fault alarm information of the environment sensing and navigation positioning is higher than that of the related sensor fault alarm information of the vehicle-mounted entertainment and interaction system.

After the vehicle determines the remote assistance requirement degree information according to the sensing information calculation, in step S12, the remote assistance requirement degree information is sent to the remote device. The remote device is used for a remote operator to monitor the relevant vehicle and send a remote assistance operation instruction to the relevant vehicle, or is used for allocating cloud computing resources for the vehicle, providing a high-precision environment map of the position of the vehicle, providing environment perception resources of V2I equipment or a remote unmanned aerial vehicle and the like for the vehicle and the like.

Here, the vehicle may obtain the vehicle driving reliability information of the autonomous driving of the vehicle by a method similar to the method for obtaining the remote assistance demand information, and send the vehicle driving reliability information to the remote device, which is not described herein again and is included herein by reference.

In some embodiments, the method further comprises step S13. In step S13, the vehicle receives operation result information that the remote apparatus performs the remote assistance operation on the vehicle. The operation result information of the remote assistance operation performed on the vehicle includes, but is not limited to, a remote operation instruction of a remote operator, cloud computing resources allocated to the vehicle, a high-precision environment map of a location where the vehicle is located provided for the vehicle, environment sensing resources of a V2I device or a remote drone and the like provided for the vehicle, and the like. For example, after the vehicle sends the remote assistance requirement information to the remote device, the remote operator sends a remote operation instruction to the vehicle according to the remote assistance requirement information, and the remote operation instruction is an operation result of performing remote assistance operation on the vehicle.

In some embodiments, the operation result information includes a current travel task set for the vehicle; the method further comprises step S14 (not shown). In step S14, the vehicle executes the current travel task. For example, the remote operator or the remote device sends a remote operation instruction to the vehicle according to the remote assistance requirement information sent by the vehicle, wherein the operation instruction comprises a running task set for the vehicle, and the running task comprises but is not limited to parking at the side, emergency braking, driving to the nearest maintenance site and the like; after receiving the remote operation command, the vehicle executes a travel task set by a remote operator or a remote device.

In some embodiments, the method further comprises step S15 (not shown). In step S15, the vehicle resumes its autonomous driving mode according to the instructions of the remote device. For example, the remote device continuously receives the remote assistance demand level information of the vehicle, after the remote device sends a remote assistance operation instruction to the vehicle, the environment where the vehicle is located or the state of the vehicle itself changes, for example, fault information such as weak communication signal strength and uncertain type of obstacles disappears, and after the vehicle recovers to a state sufficient for reliable autonomous driving, the remote device sends an instruction to the vehicle, and the vehicle recovers to an autonomous driving mode according to the instruction.

Referring to fig. 4, in some embodiments, step S11 includes sub-step S11a and sub-step S11 b. In sub-step S11a, the vehicle identifies the corresponding safety event information for the vehicle based on its vehicle sensing information. For example, after the vehicle collects the sensing data of its sensors in real time, the safety events of the vehicle are identified according to the sensing data, which may be related to the safety of its passengers or pedestrians outside the vehicle, including but not limited to actuator failure alarm information (e.g., brake and steering system failure alarm information, throttle and suspension failure alarm information), sensor failure alarm information (e.g., environment sensing and navigation positioning related sensor failure alarm information, vehicle entertainment and interaction system related sensor failure alarm information), computing module failure alarm information, weak communication signal strength, frequent switching of steering/brake/throttle, etc., uncertain type of obstacles, uncertain position, being trapped in a certain posture, uncertain type of environment/object, being trapped overtime, etc. In sub-step S11b, the vehicle determines remote assistance desirability information for the vehicle based on the safety event information. For example, the remote assistance desirability information of the vehicle is calculated based on the weight of each safety event.

Preferably, in sub-step S11a, the vehicle matches the query in a malfunction alerting database or a context database based on its vehicle sensing information to obtain corresponding safety event information for the vehicle. The fault alarm database is related to the state of the vehicle, and can be used for inquiring vehicle fault alarm information, such as fault alarm information of an actuator, fault alarm information of a sensor, fault alarm information of a calculation module and the like; the alarm information in the situation database is related to the environment where the vehicle is located, and can be used for inquiring alarm information such as weak communication signal strength, frequent switching of steering/braking/accelerator and the like, uncertain type of obstacles, uncertain positions, trapped in a certain posture, uncertain type of environment/object, trapped overtime and the like. The vehicle can obtain corresponding alarm information by matching and inquiring in a fault alarm database or a situation database based on the sensor data so as to determine corresponding safety event information.

According to another aspect of the present application, there is provided a method for remotely assisting a vehicle at a remote device side, the method including step S21, step S22, and step S23. Referring to fig. 5, in step S21, the remote device receives remote assistance request level information transmitted by a plurality of vehicles; in step S22, the remote device prefers one or more monitored vehicles from the plurality of vehicles according to the remote assistance requirement degree information; in step S23, the remote device performs a remote assistance operation with respect to at least one monitored vehicle of the one or more monitored vehicles.

In step S21, the remote device receives the remote assistance request level information transmitted by the plurality of vehicles. The remote assistance demand degree information is obtained by vehicle-mounted or cloud computing based on sensing data acquired by each sensor in a sensing system of the vehicle, and the sensing data is related to the state and the environment complexity of the vehicle and can reflect the demand degree of the vehicle needing external remote assistance or further reflect the credibility of the vehicle capable of completing an autonomous driving task.

In step S22, the remote device prefers one or more monitored vehicles from the plurality of vehicles according to the remote assistance necessity degree information. For example, the remote device sorts a plurality of vehicles according to the remote assistance demand degree information, and selects a fixed number of vehicles to monitor, for example, selects 5 vehicles with the highest remote assistance demand degree to monitor; for another example, the remote device compares the remote assistance requirement information with a preset requirement threshold, and monitors the vehicle once the remote assistance requirement of the vehicle is higher than the requirement threshold. The monitoring of the vehicle may be performed automatically by the remote device or by an operator of the remote device, for example, the remote device presents the monitored vehicle on a screen for monitoring of its condition by a remote operator. Fig. 6 shows a possible scenario in which the cloud continuously receives and sequences the remote assistance desirability information of the vehicles, and presents the sensed information of the vehicles (e.g., a look-around video capable of reflecting the surrounding conditions of the vehicles) to the operator of the remote device.

In step S13, the remote device performs a remote assistance operation with respect to at least one monitored vehicle of the one or more monitored vehicles. In one embodiment, the monitoring of the vehicle is automatically performed by the remote device, and when the remote assistance requirement information of the vehicle is higher than a preset requirement threshold, the remote device performs remote assistance operation and sends a remote assistance operation instruction to the vehicle, such as emergency braking; or based on the sensing information or alarm information of the vehicle, selecting an appropriate remote assistance operation instruction, such as parking at the side when a fault occurs. In another embodiment, the monitoring of the vehicle is performed by an operator of the remote device, and the operator operates the remote device to cause the remote device to perform the remote assistance operation, and the remote assistance operation instruction is sent to the vehicle.

In some embodiments, there may be two situations when the remote assistance operation instruction sent by the remote device to the vehicle: 1) the environment of the vehicle is complex, such as front construction, excessive pedestrians around, broken trees and the like which exceed the processing capacity of the vehicle; 2) the state of the vehicle is abnormal, for example, a certain sensing device has serious failure, a decision control program has errors, and the like. For case 1), the process flow refers to fig. 7, the remote device extracts the vehicle surrounding environment information from the sensing information sent by the vehicle (for example, by performing machine vision recognition on the video or picture data sent by the vehicle), and performs manual or non-manual remote assistance according to the extracted environment information, during which the current remote assistance desirability information recalculated by the vehicle is continuously received until the vehicle can complete the automatic driving task. For case 2), the processing flow refers to fig. 8, the remote device extracts the environment and the fault information of the vehicle, and takes corresponding remote assistance measures according to the fault information of the vehicle, or sends a remote assistance instruction of a remote operator to the vehicle, so as to remotely assist the vehicle to safely travel to a maintenance point or other safe areas.

In conjunction with the schematic configuration of the system for remotely assisting a vehicle shown in fig. 2, in some embodiments, the method further includes step S24 (not shown). In step S24, the remote device presents the one or more monitored vehicles. The monitored vehicles presented by the remote equipment can be used for a remote operator to monitor the vehicles needing remote assistance so as to react in time and realize real-time monitoring on a plurality of vehicles by fewer remote operators. The presentation of the vehicle may be based on a plurality of display devices, such as a plurality of display screens; it is also possible to base multiple display areas of a single display device, for example multiple divided areas of a display screen.

Wherein, preferably, in step S24, the remote device presents the one or more monitored vehicles according to the remote assistance requirement degree information of the monitored vehicles. In some embodiments, the monitored vehicles are presented after being ranked based on the remote assistance desirability information of each vehicle, for example, the monitored vehicles are ranked in descending order according to the remote assistance desirability information, so that the vehicle with the highest remote assistance desirability is presented first; in other embodiments, the monitored vehicle is presented based on a relationship between the remote assistance desirability information of the vehicle and a preset desirability threshold, for example, when the remote assistance desirability information is higher than the preset desirability threshold, the corresponding vehicle is presented. The monitored vehicle is presented according to the remote assistance demand information, the utilization rate of remote equipment display resources can be improved, the operation efficiency of a remote operator is improved, the misoperation rate is reduced, and therefore the vehicle remote assistance efficiency and the vehicle operation safety are improved.

In some embodiments, the method further comprises step S25 (not shown). In step S25, the remote device acquires vehicle sensing information of the monitored vehicle; in step S24, vehicle sensory information for the one or more monitored vehicles and each monitored vehicle is presented. For example, the remote device receives the remote assistance desirability information and the vehicle sensing information of the monitored vehicle at the same time, and presents the vehicle sensing information of the monitored vehicle and each monitored vehicle on the display device respectively. The vehicle sensing information can help a remote operator to monitor the vehicle state in real time, and further improves the vehicle remote assistance efficiency and the vehicle running safety.

In some embodiments, in step S23, the remote device performs the remote assistance operation with respect to at least one monitored vehicle of the one or more monitored vehicles according to the monitoring operation of the monitoring user on the at least one monitored vehicle. For example, a remote operator finds that the state of a monitored vehicle is abnormal, and performs monitoring operation on the monitored vehicle; the remote device detects a monitoring operation by a remote operator and performs a manual or non-manual remote assistance operation on the vehicle, wherein the manual remote assistance operation is based on an input of the remote operator on the remote device, and the non-manual remote assistance operation can be automatically completed by the remote device.

In some embodiments, the method further comprises step S26 (not shown) and step S27 (not shown). In step S26, the remote device detects whether the remote assistance requirement information of the monitored vehicle is greater than or equal to a predetermined requirement threshold, and if so, sends a monitoring prompt message about the monitored vehicle to the monitoring user; in step S27, the remote device acquires a monitoring operation of the monitoring user on at least one monitored vehicle of the one or more monitored vehicles. For example, the remote device continuously receives the remote assistance desirability information of the monitored vehicle and compares the information with a preset desirability threshold, and once the remote assistance desirability of the vehicle is higher than the preset desirability threshold, the remote operator is prompted to pay attention to the vehicle. The manner in which the remote device prompts the remote operator includes, but is not limited to, sound, flashing lights, color changes of the content displayed by the display device, vibrations of the vibration device, and the like. Of course, if the remote device receives the vehicle running reliability information sent by the vehicle, it detects whether the vehicle running reliability information is less than or equal to the predetermined reliability threshold, and prompts when the vehicle running reliability information is less than or equal to the reliability threshold, which is not described herein again.

In some embodiments, in step S23, the remote device performs the remote assistance operation with respect to at least one monitored vehicle of the one or more monitored vehicles when the remote assistance requirement information of the at least one monitored vehicle is greater than or equal to a predetermined requirement threshold. The remote assistance operation of the monitored vehicle can be automatically completed by the remote equipment, and can also be completed by the remote equipment after determining a corresponding remote assistance instruction according to the input of a remote operator.

In some embodiments, the remote device prefers one or more monitored vehicles from the plurality of vehicles based on the remote assistance desirability information and a threshold number of vehicles that can be monitored. For example, when the display resources of the remote device are limited and only the monitoring information of 6 monitored vehicles at most (for example, videos or pictures taken by a camera of the vehicle or other sensor information of the vehicle) can be displayed, the monitored vehicles are arranged in a descending order according to the remote assistance requirement information, and the monitoring information of the 6 monitored vehicles at most with the highest remote assistance requirement is presented. For another example, the number of monitored vehicles that can be monitored by each remote operator at the same time is limited, and the remote device only displays monitoring information of no more than the number of vehicles, so that the situation that the remote operator is stressed too much due to the fact that too many monitoring information of the monitored vehicles are displayed is avoided, and misoperation is reduced.

Wherein preferably the threshold number of monitorable vehicles is determined on the basis of the number of available monitoring personnel or available monitoring screens. For example, a monitorable vehicle quantity threshold is determined based on the quantity of vehicle monitoring information that each remote operator can monitor simultaneously and the number of remote operators working simultaneously, the quantity of monitored vehicles displayed by the remote device not exceeding the monitorable vehicle quantity threshold; as another example, the number of displays or display areas of the remote device is limited, which determines a threshold number of vehicles that can be monitored. Vehicles exceeding the threshold number of monitorable vehicles may be assigned to other remote devices if remote assistance is required.

In some embodiments, the remote assistance operations include, but are not limited to:

1) allocating task computing resources to the at least one monitored vehicle, for example, providing stronger artificial intelligence processing capacity for the unmanned vehicle with a fault or accident at the cloud end by an unmanned vehicle operation management system, or putting more cloud computing power into the unmanned vehicle to assist the unmanned vehicle in completing decision control in a complex scene;

2) allocating environment sensing resources to the at least one monitored Vehicle, for example, providing a higher-precision environment map where the Vehicle is located for the unmanned Vehicle at the cloud end by an unmanned Vehicle operation management system, or invoking more V2I (Vehicle to Infrastructure communication) devices or remote unmanned aerial vehicles, and the like, so as to enhance the sensing capability of the Vehicle;

3) allocating an external rescue resource for the at least one monitored vehicle, such as calling a trailer, clearing an obstacle vehicle, etc.;

4) and setting the current running task of the at least one monitored vehicle, such as a vehicle fault signal to determine the type of vehicle fault, and determining a remote intervention mode to be taken according to the environment of the vehicle to remotely assist the vehicle to run to a maintenance point or other safe areas.

In some embodiments, the method further comprises step S28 (not shown) and step S29 (not shown). In step S28, the remote device updates the remote assistance desirability information of the at least one monitored vehicle; in step S29, when the updated remote assistance desirability information of the at least one monitored vehicle is less than a predetermined desirability threshold, the remote device resumes the autonomous driving mode of the at least one monitored vehicle. For example, the remote device continuously receives the remote assistance desirability information of the monitored vehicle, compares the remote assistance desirability information with a preset desirability threshold, and when the environment of the vehicle changes or the vehicle is remotely assisted, reduces the remote assistance desirability information of the vehicle to be less than the preset desirability threshold, sends an autonomous driving instruction to the vehicle, ends the remote assistance, and restores the autonomous driving mode of the vehicle. Of course, if the remote device receives the vehicle running reliability information sent by the vehicle, it detects whether the vehicle running reliability information is greater than or equal to the predetermined reliability threshold, and recovers the autonomous driving mode of the vehicle when the vehicle running reliability information is greater than or equal to the reliability threshold, which is not described herein again.

In some embodiments, the remote device further performs a ranking operation on the plurality of vehicles according to the remote assistance desirability information; in step S22, the remote device prefers one or more monitored vehicles from the plurality of vehicles according to the ranking result of the ranking operation. In some embodiments, the number of vehicles that the remote device is capable of remotely assisting at the same time is limited, or the number of display screens or divided display areas on the display screen used by the remote device to present monitored vehicle information is limited, or when the number of remote operators is limited, and thus the number of vehicles that the remote operator can monitor and remotely assist at one time is limited. In order to allocate limited remote assistance resources to the most needed vehicles as much as possible, the remote device may, after receiving the remote assistance desirability information sent by the plurality of vehicles, sort the vehicles according to the remote assistance desirability information, for example, sort the vehicles in a descending order according to the remote assistance desirability information, and select one or more controlled vehicles with the highest remote assistance desirability as monitored vehicles. In some embodiments, after the vehicle with the highest remote assistance demand degree information is remotely assisted, the vehicles can be sorted again according to the remote assistance demand degree information received in real time, and the monitored vehicle list is updated.

According to an aspect of the present application, there is provided a vehicle-side remote assistance apparatus 1 for remotely assisting a vehicle at a vehicle side, the apparatus including a calculation module 11 and a transmission module 12. Referring to fig. 9, the calculation module 11 determines remote assistance desirability information of a vehicle based on vehicle sensing information of the vehicle; the sending module 12 sends the remote assistance desirability information to a remote device.

Specifically, the calculation module 11 determines the remote assistance necessity degree information of the vehicle based on the vehicle sensing information of the vehicle. The vehicle is provided with a rich perception system, for example, the vehicle is provided with a sensing module consisting of a plurality of sensors, or is further provided with a database corresponding to sensing information of the sensing module; in some embodiments, the vehicle may be unmanned or have a drive-assist function. The sensors in the sensing module include, but are not limited to, a sound sensor for sensing environmental or vehicle interior sound, a pressure sensor for sensing pressure of various pipelines or squeezing pressure between components of the vehicle, a vehicle speed sensor for sensing vehicle running speed, a steering angle sensor for sensing vehicle steering angle or a sensor for sensing other vehicle state signals, and may further include a laser radar, a camera, a millimeter wave radar or a sensor for sensing other vehicle peripheral signals.

After the vehicle calculates and determines the remote assistance requirement degree information according to the sensing information, the sending module 12 sends the remote assistance requirement degree information to the remote device. The remote device is used for a remote operator to monitor the relevant vehicle and send a remote assistance operation instruction to the relevant vehicle, or is used for allocating cloud computing resources for the vehicle, providing a high-precision environment map of the position of the vehicle, providing environment perception resources of V2I equipment or a remote unmanned aerial vehicle and the like for the vehicle and the like.

In some embodiments, the apparatus further comprises a receiving module 13. The receiving module 13 receives operation result information of the remote device performing the remote assistance operation on the vehicle. The operation result information of the remote assistance operation performed on the vehicle includes, but is not limited to, a remote operation instruction of a remote operator, cloud computing resources allocated to the vehicle, a high-precision environment map of a location where the vehicle is located provided for the vehicle, environment sensing resources of a V2I device or a remote drone and the like provided for the vehicle, and the like. For example, after the vehicle sends the remote assistance requirement information to the remote device, the remote operator sends a remote operation instruction to the vehicle according to the remote assistance requirement information, and the remote operation instruction is an operation result of performing remote assistance operation on the vehicle.

In some embodiments, the operation result information includes a current travel task set for the vehicle; the apparatus also includes an execution module 14 (not shown). The execution module 14 executes the current travel task. For example, the remote operator or the remote device sends a remote operation instruction to the vehicle according to the remote assistance requirement information sent by the vehicle, wherein the operation instruction comprises a running task set for the vehicle, and the running task comprises but is not limited to parking at the side, emergency braking, driving to the nearest maintenance site and the like; after receiving the remote operation command, the vehicle executes a travel task set by a remote operator or a remote device.

In some embodiments, the apparatus further comprises a recovery module 15 (not shown). The recovery module 15 recovers its autonomous driving mode according to the instructions of the remote device. For example, the remote device continuously receives the remote assistance demand level information of the vehicle, after the remote device sends a remote assistance operation instruction to the vehicle, the environment where the vehicle is located or the state of the vehicle itself changes, for example, fault information such as weak communication signal strength and uncertain type of obstacles disappears, and after the vehicle recovers to a state sufficient for reliable autonomous driving, the remote device sends an instruction to the vehicle, and the vehicle recovers to an autonomous driving mode according to the instruction.

Referring to fig. 10, in some embodiments, the calculation module 11 includes a recognition unit 11a and a calculation unit 11 b. The identification unit 11a identifies the safety event information corresponding to the vehicle based on the vehicle sensing information thereof. For example, after the vehicle collects the sensing data of its sensors in real time, the safety events of the vehicle are identified according to the sensing data, which may be related to the safety of its passengers or pedestrians outside the vehicle, including but not limited to actuator failure alarm information (e.g., brake and steering system failure alarm information, throttle and suspension failure alarm information), sensor failure alarm information (e.g., environment sensing and navigation positioning related sensor failure alarm information, vehicle entertainment and interaction system related sensor failure alarm information), computing module failure alarm information, weak communication signal strength, frequent switching of steering/brake/throttle, etc., uncertain type of obstacles, uncertain position, being trapped in a certain posture, uncertain type of environment/object, being trapped overtime, etc. The calculation unit 11b determines the remote assistance necessity degree information of the vehicle based on the safety event information. For example, the remote assistance desirability information of the vehicle is calculated based on the weight of each safety event.

Preferably, in the identification unit 11a, the vehicle matches the query in the malfunction alerting database or the context database based on its vehicle sensing information to obtain the corresponding safety event information of the vehicle. The fault alarm database is related to the state of the vehicle, and can be used for inquiring vehicle fault alarm information, such as fault alarm information of an actuator, fault alarm information of a sensor, fault alarm information of a calculation module and the like; the alarm information in the situation database is related to the environment where the vehicle is located, and can be used for inquiring alarm information such as weak communication signal strength, frequent switching of steering/braking/accelerator and the like, uncertain type of obstacles, uncertain positions, trapped in a certain posture, uncertain type of environment/object, trapped overtime and the like. The vehicle can obtain corresponding alarm information by matching and inquiring in a fault alarm database or a situation database based on the sensor data so as to determine corresponding safety event information.

According to another aspect of the present application, there is provided a remote equipment-side remote assistance apparatus 2 for remotely assisting a vehicle at a remote equipment side, the apparatus including a receiving module 21, a selecting module 22, and an executing module 23. Referring to fig. 11, the receiving module 21 receives remote assistance desirability information transmitted by a plurality of vehicles; the selection module 22 prefers one or more monitored vehicles from the plurality of vehicles according to the remote assistance desirability information; the execution module 23 performs a remote assistance operation with respect to at least one monitored vehicle of the one or more monitored vehicles.

The receiving module 21 receives the remote assistance demand degree information sent by the plurality of vehicles. The remote assistance demand degree information is obtained by vehicle-mounted or cloud computing based on sensing data acquired by each sensor in a sensing system of the vehicle, and the sensing data is related to the state and the environment complexity of the vehicle and can reflect the demand degree of the vehicle needing external remote assistance or further reflect the credibility of the vehicle capable of completing an autonomous driving task.

The selection module 22 prefers one or more monitored vehicles from the plurality of vehicles according to the remote assistance desirability information. For example, the remote device sorts a plurality of vehicles according to the remote assistance demand degree information, and selects a fixed number of vehicles to monitor, for example, selects 5 vehicles with the highest remote assistance demand degree to monitor; for another example, the remote device compares the remote assistance requirement information with a preset requirement threshold, and monitors the vehicle once the remote assistance requirement of the vehicle is higher than the requirement threshold. The monitoring of the vehicle may be performed automatically by the remote device or by an operator of the remote device, for example, the remote device presents the monitored vehicle on a screen for monitoring of its condition by a remote operator. Fig. 6 shows a possible scenario in which the cloud continuously receives and sequences the remote assistance desirability information of the vehicles, and presents the sensed information of the vehicles (e.g., a look-around video capable of reflecting the surrounding conditions of the vehicles) to the operator of the remote device.

The execution module 23 performs a remote assistance operation with respect to at least one monitored vehicle of the one or more monitored vehicles. In one embodiment, the monitoring of the vehicle is automatically performed by the remote device, and when the remote assistance requirement information of the vehicle is higher than a preset requirement threshold, the remote device performs remote assistance operation and sends a remote assistance operation instruction to the vehicle, such as emergency braking; or based on the sensing information or alarm information of the vehicle, selecting an appropriate remote assistance operation instruction, such as parking at the side when a fault occurs. In another embodiment, the monitoring of the vehicle is performed by an operator of the remote device, and the operator operates the remote device to cause the remote device to perform the remote assistance operation, and the remote assistance operation instruction is sent to the vehicle.

In conjunction with the schematic configuration of the system for remotely assisting a vehicle shown in fig. 2, in some embodiments, the apparatus further includes a presentation module 24 (not shown). The presentation module 24 presents the one or more monitored vehicles. The monitored vehicles presented by the remote equipment can be used for a remote operator to monitor the vehicles needing remote assistance so as to react in time and realize real-time monitoring on a plurality of vehicles by fewer remote operators. The presentation of the vehicle may be based on a plurality of display devices, such as a plurality of display screens; it is also possible to base multiple display areas of a single display device, for example multiple divided areas of a display screen.

Wherein, preferably, the presenting module 24 presents the one or more monitored vehicles according to the remote assistance desirability information of the monitored vehicles. In some embodiments, the monitored vehicles are presented after being ranked based on the remote assistance desirability information of each vehicle, for example, the monitored vehicles are ranked in descending order according to the remote assistance desirability information, so that the vehicle with the highest remote assistance desirability is presented first; in other embodiments, the monitored vehicle is presented based on a relationship between the remote assistance desirability information of the vehicle and a preset desirability threshold, for example, when the remote assistance desirability information is higher than the preset desirability threshold, the corresponding vehicle is presented. The monitored vehicle is presented according to the remote assistance demand information, the utilization rate of remote equipment display resources can be improved, the operation efficiency of a remote operator is improved, the misoperation rate is reduced, and therefore the vehicle remote assistance efficiency and the vehicle operation safety are improved.

In some embodiments, the apparatus further comprises an acquisition module 25 (not shown). The obtaining module 25 obtains vehicle sensing information of the monitored vehicle; the presentation module 24 presents the one or more monitored vehicles and the vehicle sensory information for each monitored vehicle. For example, the remote device receives the remote assistance desirability information and the vehicle sensing information of the monitored vehicle at the same time, and presents the vehicle sensing information of the monitored vehicle and each monitored vehicle on the display device respectively. The vehicle sensing information can help a remote operator to monitor the vehicle state in real time, and further improves the vehicle remote assistance efficiency and the vehicle running safety.

In some embodiments, the execution module 23 executes the remote assistance operation with respect to at least one monitored vehicle of the one or more monitored vehicles according to the monitoring operation of the monitoring user on the at least one monitored vehicle. For example, a remote operator finds that the state of a monitored vehicle is abnormal, and performs monitoring operation on the monitored vehicle; the remote device detects a monitoring operation by a remote operator and performs a manual or non-manual remote assistance operation on the vehicle, wherein the manual remote assistance operation is based on an input of the remote operator on the remote device, and the non-manual remote assistance operation can be automatically completed by the remote device.

In some embodiments, the apparatus further comprises a detection module 26 (not shown) and a monitoring module 27 (not shown). The detection module 26 detects whether the remote assistance requirement information of the monitored vehicle is greater than or equal to a preset requirement threshold, and if so, sends monitoring prompt information about the monitored vehicle to the monitoring user; the monitoring module 27 obtains the monitoring operation of the monitoring user on at least one monitored vehicle of the one or more monitored vehicles. For example, the remote device continuously receives the remote assistance desirability information of the monitored vehicle and compares the information with a preset desirability threshold, and once the remote assistance desirability of the vehicle is higher than the preset desirability threshold, the remote operator is prompted to pay attention to the vehicle. The manner in which the remote device prompts the remote operator includes, but is not limited to, sound, flashing lights, color changes of the content displayed by the display device, vibrations of the vibration device, and the like. Of course, if the remote device receives the vehicle running reliability information sent by the vehicle, it detects whether the vehicle running reliability information is less than or equal to the predetermined reliability threshold, and prompts when the vehicle running reliability information is less than or equal to the reliability threshold, which is not described herein again.

In some embodiments, the execution module 23 executes the remote assistance operation with respect to at least one monitored vehicle of the one or more monitored vehicles when the remote assistance requirement information of the at least one monitored vehicle is greater than or equal to a predetermined requirement threshold. The remote assistance operation of the monitored vehicle can be automatically completed by the remote equipment, and can also be completed by the remote equipment after determining a corresponding remote assistance instruction according to the input of a remote operator.

In some embodiments, the apparatus further comprises an update module 28 (not shown) and a recovery module 29 (not shown). The updating module 28 updates the remote assistance desirability information of the at least one monitored vehicle; the restoring module 29 restores the autonomous driving mode of the at least one monitored vehicle when the updated remote assistance desirability information of the at least one monitored vehicle is less than a predetermined desirability threshold. For example, the remote device continuously receives the remote assistance desirability information of the monitored vehicle, compares the remote assistance desirability information with a preset desirability threshold, and when the environment of the vehicle changes or the vehicle is remotely assisted, reduces the remote assistance desirability information of the vehicle to be less than the preset desirability threshold, sends an autonomous driving instruction to the vehicle, ends the remote assistance, and restores the autonomous driving mode of the vehicle. Of course, if the remote device receives the vehicle running reliability information sent by the vehicle, it detects whether the vehicle running reliability information is greater than or equal to the predetermined reliability threshold, and recovers the autonomous driving mode of the vehicle when the vehicle running reliability information is greater than or equal to the reliability threshold, which is not described herein again.

In some embodiments, the remote device further comprises a ranking module. The sequencing module is used for sequencing the vehicles according to the remote assistance demand degree information; the selection module 22 prefers one or more monitored vehicles from the plurality of vehicles according to the ranking result of the ranking operation. In some embodiments, the number of vehicles that the remote device is capable of remotely assisting at the same time is limited, or the number of display screens or divided display areas on the display screen used by the remote device to present monitored vehicle information is limited, or when the number of remote operators is limited, and thus the number of vehicles that the remote operator can monitor and remotely assist at one time is limited. In order to allocate limited remote assistance resources to the most needed vehicles as much as possible, the remote device may, after receiving the remote assistance desirability information sent by the plurality of vehicles, sort the vehicles according to the remote assistance desirability information, for example, sort the vehicles in a descending order according to the remote assistance desirability information, and select one or more controlled vehicles with the highest remote assistance desirability as monitored vehicles. In some embodiments, after the vehicle with the highest remote assistance demand degree information is remotely assisted, the vehicles can be sorted again according to the remote assistance demand degree information received in real time, and the monitored vehicle list is updated.

According to one aspect of the present application, there is provided a vehicle for remote assistance with a vehicle, wherein the vehicle comprises:

a processor; and

Fig. 12 illustrates a block diagram of an exemplary vehicle-side remote assistance device for remotely assisting a vehicle at a vehicle side according to some embodiments of the present application. Wherein the above-described method for remotely assisting a vehicle may be performed by the apparatus, and wherein some operations are optionally combined and/or the order of some operations is optionally changed in the above-described method for remotely assisting a vehicle.

In some embodiments, the apparatus includes a sensing module and a control module. The sensing module includes one or more sensors including, but not limited to, a sound sensor for sensing ambient sound, a pressure sensor for sensing a squeezing pressure between components or a fluid pressure in a pipe, a vehicle speed sensor for sensing a running speed of the vehicle, a steering angle sensor for sensing a steering angle of the vehicle, an acceleration sensor for sensing an acceleration of the vehicle, other vehicle state signal sensors for sensing other states of the vehicle, and a sensor for sensing an environment where the vehicle is located, such as a laser radar, a camera, a millimeter wave radar, and other vehicle peripheral signal sensors. The sensors are directly or indirectly connected to a processor in the control module so that the processor can acquire the sensing information collected by the sensors.

The control module also comprises a wireless transceiver module connected with the processor and used for communicating with remote equipment, a CAN (Controller Area Network) bus used for carrying out communication between vehicle digital components, a memory, a human-computer interface used for carrying out human-computer interaction and other modules. The memory includes high speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid state storage devices. The man-machine interaction comprises that the machine provides a large amount of related information and prompt requests for people through an output or display device, and the people input the related information into the machine through an input device, answer questions and prompt requests and the like.

The control module also comprises a control unit which is connected with the processor and the execution module and is used for driving the corresponding unit in the execution module to carry out control action according to the instruction of the processor. The execution module comprises a braking unit (sometimes called a brake) for braking the vehicle, a steering unit for steering the vehicle, a warning unit for providing warning information or external warning light, warning sound and the like, and other actuators for executing other actions.

Of course, those skilled in the art will appreciate that the vehicle-end remote assistance device described above is merely exemplary, optionally having more or fewer components than shown in fig. 12, optionally combining two or more components, or optionally having a different configuration or arrangement of these components; also, other existing or future devices that may be suitable for use with the present application are intended to be within the scope of the present application and are hereby incorporated by reference.

According to another aspect of the present application, there is provided a remote device for remote assistance with a vehicle, wherein the remote device comprises:

a processor; and

Fig. 13 illustrates a block diagram of an exemplary remote assistance device at a remote device end for remotely assisting a vehicle according to some embodiments of the present application. Wherein the above-described method for remotely assisting a vehicle may be performed by the apparatus, and wherein some operations are optionally combined and/or the order of some operations is optionally changed in the above-described method for remotely assisting a vehicle.

The device comprises a remote assistance control module, optionally a signal transceiver module, for receiving remote data and transmitting control signals. The remote assistance control module also typically includes one or more processing units, memory, and one or more communication buses for interconnecting these components. The communication bus optionally includes circuitry (sometimes called a chipset) that interconnects and controls communication between system components.

The device also comprises a display device connected with the remote assistance control module, wherein the display device comprises a plurality of display areas, and the plurality of display areas can be respectively provided by independent display devices or a single display device. Wherein the display device optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, AR/VR/MR technology for presenting information about the monitored vehicle in a look-around/VR/AR/MR manner.

The device also comprises a remote control device which is connected with the remote assistance control module and is used for a remote operator to carry out remote assistance operation, and the remote control device optionally comprises a physical key, a virtual key, a track ball or a somatosensory input unit.

Of course, those skilled in the art will appreciate that the above-described remote-device-side remote assistance apparatus is merely exemplary, and that the remote-device-side remote assistance apparatus may alternatively have more or fewer components than shown in fig. 13, may alternatively combine two or more components, or may alternatively have a different configuration or arrangement of such components; also, other existing or future devices that may be suitable for use with the present application are intended to be within the scope of the present application and are hereby incorporated by reference.

It should be noted that the present application may be implemented in software and/or a combination of software and hardware, for example, implemented using Application Specific Integrated Circuits (ASICs), general purpose computers or any other similar hardware devices. In one embodiment, the software programs of the present application may be executed by a processor to implement the steps or functions described above. Likewise, the software programs (including associated data structures) of the present application may be stored in a computer readable recording medium, such as RAM memory, magnetic or optical drive or diskette and the like. Additionally, some of the steps or functions of the present application may be implemented in hardware, for example, as circuitry that cooperates with the processor to perform various steps or functions.

In addition, some of the present application may be implemented as a computer program product, such as computer program instructions, which when executed by a computer, may invoke or provide methods and/or techniques in accordance with the present application through the operation of the computer. Those skilled in the art will appreciate that the form in which the computer program instructions reside on a computer-readable medium includes, but is not limited to, source files, executable files, installation package files, and the like, and that the manner in which the computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Computer-readable media herein can be any available computer-readable storage media or communication media that can be accessed by a computer.

Communication media includes media by which communication signals, including, for example, computer readable instructions, data structures, program modules, or other data, are transmitted from one system to another. Communication media may include conductive transmission media such as cables and wires (e.g., fiber optics, coaxial, etc.) and wireless (non-conductive transmission) media capable of propagating energy waves such as acoustic, electromagnetic, RF, microwave, and infrared. Computer readable instructions, data structures, program modules, or other data may be embodied in a modulated data signal, for example, in a wireless medium such as a carrier wave or similar mechanism such as is embodied as part of spread spectrum techniques. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The modulation may be analog, digital or hybrid modulation techniques.

By way of example, and not limitation, computer-readable storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memory such as random access memory (RAM, DRAM, SRAM); and non-volatile memory such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic/ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disk, tape, CD, DVD); or other now known media or later developed that can store computer-readable information/data for use by a computer system.

An embodiment according to the present application comprises an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein the computer program instructions, when executed by the processor, trigger the apparatus to perform a method and/or a solution according to the aforementioned embodiments of the present application.

It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the apparatus claims may also be implemented by one unit or means in software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.

Various aspects of various embodiments are defined in the claims. These and other aspects of the various embodiments are specified in the following numbered clauses:

1. a method at a vehicle end for remote assistance with a vehicle, wherein the method comprises:

2. The method of clause 1, wherein the method further comprises:

and receiving operation result information of the remote assistance operation performed on the vehicle by the remote equipment.

3. The method according to clause 2, wherein the operation result information includes a current travel task set for the vehicle;

wherein the method further comprises:

and executing the current running task.

4. The method of any of clauses 1-3, wherein the method further comprises:

and recovering the autonomous driving mode of the vehicle according to the instruction of the remote equipment.

5. The method of any of clauses 1-3, wherein the determining remote assistance desirability information for the vehicle based on vehicle sensory information for the vehicle comprises:

identifying safety event information corresponding to a vehicle based on vehicle sensing information of the vehicle;

determining remote assistance desirability information for the vehicle based on the safety event information.

6. The method of clause 5, wherein the identifying the vehicle-based vehicle sensing information corresponding to the vehicle comprises:

matching and inquiring in a fault alarm database or a situation database based on vehicle sensing information of the vehicle so as to obtain safety event information corresponding to the vehicle.

7. A method at a remote device end for remote assistance with a vehicle, wherein the method comprises:

8. The method of clause 7, wherein the method further comprises:

presenting the one or more monitored vehicles.

9. The method of clause 8, wherein the presenting the one or more monitored vehicles comprises:

and presenting the one or more monitored vehicles according to the remote assistance demand degree information of the monitored vehicles.

10. The method of clause 8, wherein the method further comprises:

acquiring vehicle sensing information of the monitored vehicle;

wherein said presenting the one or more monitored vehicles comprises:

vehicle sensory information is presented for the one or more monitored vehicles and for each monitored vehicle.

11. The method of clause 8, wherein the performing the remote assistance operation with respect to at least one of the one or more monitored vehicles comprises:

and according to the monitoring operation of the monitoring user on at least one monitored vehicle in the one or more monitored vehicles, executing remote assistance operation on the at least one monitored vehicle.

12. The method of clause 11, wherein the method further comprises:

detecting whether the remote assistance desirability information of the monitored vehicle is greater than or equal to a predetermined desirability threshold;

if so, sending monitoring prompt information about the monitored vehicle to the monitoring user;

and acquiring the monitoring operation of the monitoring user on at least one monitored vehicle in the one or more monitored vehicles.

13. The method of clause 7, wherein the performing the remote assistance operation with respect to at least one of the one or more monitored vehicles comprises:

and when the remote assistance requirement degree information of at least one monitored vehicle in the one or more monitored vehicles is larger than or equal to a preset requirement degree threshold value, executing remote assistance operation related to the at least one monitored vehicle.

14. The method of clause 7, wherein the preferring one or more monitored vehicles from the plurality of vehicles according to the remote assistance desirability information comprises:

and preferably selecting one or more monitored vehicles from the plurality of vehicles according to the remote assistance demand degree information and the threshold value of the number of the monitored vehicles.

15. The method of clause 14, wherein the threshold number of monitorable vehicles is determined based on the number of available monitoring personnel or available monitoring screens.

16. The method of clause 7, wherein the remote assistance operation includes at least any one of:

allocating task computing resources to the at least one monitored vehicle;

allocating a context aware resource to the at least one monitored vehicle;

allocating an external rescue resource to the at least one monitored vehicle;

setting a current driving task of the at least one monitored vehicle;

17. the method of clause 7, wherein the method further comprises:

updating remote assistance desirability information of the at least one monitored vehicle;

and when the updated remote assistance demand degree information of the at least one monitored vehicle is smaller than a preset demand degree threshold value, recovering the autonomous driving mode of the at least one monitored vehicle.

18. The method of clause 7, wherein the method further comprises:

sequencing the plurality of vehicles according to the remote assistance demand degree information;

the preferring one or more monitored vehicles from the plurality of vehicles according to the remote assistance desirability information includes:

and optimizing one or more monitored vehicles from the plurality of vehicles according to the sequencing result of the sequencing operation.

19. An apparatus at a vehicle end for remote assistance with a vehicle, wherein the apparatus comprises:

20. The apparatus of clause 19, wherein the apparatus further comprises:

the receiving module is used for receiving operation result information of the remote assistance operation executed by the remote equipment on the vehicle.

21. The apparatus according to clause 20, wherein the operation result information includes a current travel task set for the vehicle;

wherein the apparatus further comprises:

and the execution module is used for executing the current running task.

22. The apparatus of any of clauses 19 to 21, wherein the apparatus further comprises:

and the recovery module is used for recovering the autonomous driving mode of the vehicle according to the instruction of the remote equipment.

23. The apparatus of any of clauses 19 to 21, wherein the computing module comprises:

the safety event information processing device comprises an identification unit, a safety event processing unit and a safety event processing unit, wherein the identification unit is used for identifying safety event information corresponding to a vehicle based on vehicle sensing information of the vehicle;

a computing unit to determine remote assistance desirability information of the vehicle based on the safety event information.

24. The method of clause 23, wherein the identification unit is configured to:

25. An apparatus for remotely assisting a vehicle at a remote device side, wherein the apparatus comprises:

26. The apparatus of clause 25, wherein the apparatus further comprises:

a presentation module to present the one or more monitored vehicles.

27. The apparatus of clause 26, wherein the presentation module is to:

28. The apparatus of clause 26, wherein the apparatus further comprises:

the acquisition module is used for acquiring vehicle sensing information of the monitored vehicle;

wherein the presentation module is to:

29. The apparatus of clause 26, wherein the execution module is to:

30. The apparatus of clause 29, wherein the apparatus further comprises:

the detection module is used for detecting whether the remote assistance demand degree information of the monitored vehicle is greater than or equal to a preset demand degree threshold value or not, and if yes, sending monitoring prompt information about the monitored vehicle to the monitoring user;

and the monitoring module is used for acquiring the monitoring operation of the monitoring user on at least one monitored vehicle in the one or more monitored vehicles.

31. The apparatus of clause 25, wherein the execution module is to:

32. The apparatus of clause 25, wherein the selection module is to:

33. The apparatus of clause 32, wherein the threshold number of monitorable vehicles is determined based on the number of available monitoring personnel or available monitoring screens.

34. The apparatus of clause 25, wherein the remote assistance operation includes at least any one of:

allocating task computing resources to the at least one monitored vehicle;

allocating a context aware resource to the at least one monitored vehicle;

allocating an external rescue resource to the at least one monitored vehicle;

setting a current driving task of the at least one monitored vehicle;

35. the apparatus of clause 25, wherein the apparatus further comprises:

the updating module is used for updating the remote assistance demand degree information of the at least one monitored vehicle;

and the recovery module is used for recovering the autonomous driving mode of the at least one monitored vehicle when the updated remote assistance demand degree information of the at least one monitored vehicle is smaller than a preset demand degree threshold value.

36. The apparatus of clause 25, wherein the apparatus further comprises:

the sequencing module is used for sequencing the vehicles according to the remote assistance demand degree information;

the selection module is configured to:

37. A method for remotely assisting a vehicle, wherein the method comprises:

38. A vehicle for remotely assisting a vehicle, wherein the vehicle comprises:

a processor; and

a memory arranged to store computer executable instructions that, when executed, cause the processor to perform operations of the method according to any of clauses 1 to 6.

39. A remote device for remote assistance with a vehicle, wherein the remote device comprises:

a processor; and

a memory arranged to store computer executable instructions that, when executed, cause the processor to perform operations of the method according to any of clauses 7 to 18.

40. A computer-readable medium comprising instructions that, when executed, cause a system to perform operations of the method of any of clauses 1 to 18.

Claims

1. A method at a remote device end for remote assistance with a vehicle, wherein the method comprises:

receiving remote assistance demand degree information sent by a plurality of vehicles; the remote assistance demand information comprises vehicle accident rate information and/or vehicle autonomous driving credibility information;

sequencing the plurality of vehicles according to the remote assistance demand degree information; one or more monitored vehicles are preferably selected from the plurality of vehicles according to the remote assistance desirability information, and the method comprises the following steps:

one or more monitored vehicles are optimized from the plurality of vehicles according to the sequencing result of the sequencing operation;

performing a remote assistance operation with respect to at least one of the one or more monitored vehicles, comprising:

when the remote assistance requirement degree information of at least one monitored vehicle in the one or more monitored vehicles is larger than or equal to a preset requirement degree threshold value, executing remote assistance operation related to the at least one monitored vehicle;

the remote assistance demand degree is determined based on the alarm information and the weight value corresponding to the alarm information, and the weight value corresponding to the alarm information is positively correlated with the safety correlation of the alarm information to passengers and/or pedestrians.

2. The method of claim 1, wherein the method further comprises:

presenting the one or more monitored vehicles.

3. The method of claim 2, wherein the presenting the one or more monitored vehicles comprises:

4. The method of claim 2, wherein the method further comprises:

acquiring vehicle sensing information of the monitored vehicle;

wherein said presenting the one or more monitored vehicles comprises:

5. The method of claim 2, wherein the performing the remote assistance operation with respect to at least one of the one or more monitored vehicles comprises:

6. The method of claim 5, wherein the method further comprises:

sending monitoring prompt information about the monitored vehicle to the monitoring user under the condition that the remote assistance demand degree information of the monitored vehicle is detected to be greater than or equal to a preset demand degree threshold value;

7. The method of claim 1, wherein the remote assistance operation comprises at least any one of:

allocating task computing resources to the at least one monitored vehicle;

allocating a context aware resource to the at least one monitored vehicle;

allocating an external rescue resource to the at least one monitored vehicle;

setting a current driving task of the at least one monitored vehicle.

8. The method of claim 1, wherein the method further comprises:

9. An apparatus for remotely assisting a vehicle at a remote device side, wherein the apparatus comprises:

the receiving module is used for receiving remote assistance demand degree information sent by a plurality of vehicles; the remote assistance demand information comprises vehicle accident rate information and/or vehicle autonomous driving credibility information;

the sequencing module is used for sequencing the vehicles according to the remote assistance demand degree information; a selection module configured to prefer one or more monitored vehicles from the plurality of vehicles according to the remote assistance desirability information, the selection module including: one or more monitored vehicles are optimized from the plurality of vehicles according to the sequencing result of the sequencing operation;

an execution module to execute a remote assistance operation with respect to at least one monitored vehicle of the one or more monitored vehicles; the execution module is used for executing remote assistance operation related to at least one monitored vehicle when the remote assistance demand degree information of at least one monitored vehicle in the one or more monitored vehicles is larger than or equal to a preset demand degree threshold;

10. The apparatus of claim 9, wherein the apparatus further comprises:

a presentation module to present the one or more monitored vehicles.

11. The apparatus of claim 10, wherein the presentation module is to:

12. The apparatus of claim 10, wherein the apparatus further comprises:

wherein the presentation module is to:

13. The apparatus of claim 10, wherein the execution module is to:

14. The apparatus of claim 13, wherein the apparatus further comprises:

the monitoring module is used for sending monitoring prompt information about the monitored vehicle to the monitoring user when the remote assistance demand degree information of the monitored vehicle is detected to be greater than or equal to a preset demand degree threshold;

15. The apparatus of claim 9, wherein the remote assistance operation comprises at least any one of:

allocating task computing resources to the at least one monitored vehicle;

allocating a context aware resource to the at least one monitored vehicle;

allocating an external rescue resource to the at least one monitored vehicle;

setting a current driving task of the at least one monitored vehicle.

16. The apparatus of claim 9, wherein the apparatus further comprises:

17. A method for remotely assisting a vehicle, wherein the method comprises:

the method comprises the steps that a vehicle determines remote assistance requirement degree information of the vehicle based on vehicle sensing information of the vehicle, and the remote assistance requirement degree information is sent to remote equipment; the remote assistance demand information comprises vehicle accident rate information and/or vehicle autonomous driving credibility information;

the remote equipment receives remote assistance demand degree information sent by a plurality of vehicles and carries out sequencing operation on the vehicles according to the remote assistance demand degree information; and preferably one or more monitored vehicles from the plurality of vehicles, comprising: one or more monitored vehicles are optimized from the plurality of vehicles according to the sequencing result of the sequencing operation;

the remote device performs a remote assistance operation with respect to at least one of the one or more monitored vehicles, including:

18. A remote device for remote assistance with a vehicle, wherein the remote device comprises:

a processor; and

a memory arranged to store computer-executable instructions that, when executed, cause the processor to perform operations according to the method of any one of claims 1 to 8.

19. A computer-readable medium comprising instructions that, when executed, cause a system to perform operations according to any one of claims 1 to 8.