WO2019047576A1

WO2019047576A1 - Method and system for automatically controlling camera of self-driving vehicle

Info

Publication number: WO2019047576A1
Application number: PCT/CN2018/089981
Authority: WO
Inventors: 张云飞; 郁浩; 闫泳杉; 郑超; 唐坤; 姜雨
Original assignee: 百度在线网络技术（北京）有限公司
Priority date: 2017-09-05
Filing date: 2018-06-05
Publication date: 2019-03-14
Also published as: CN107728646A; CN107728646B

Abstract

The present invention provides a system for automatically controlling a camera of a self-driving vehicle, comprising: a camera installed on a vehicle and used for capturing video; a processor configured to process said video so as to determine the error of the position and/or angle of said camera; a drive device which is connected to the camera and which drives the camera to change position and/or angle according to said error. Thus the present invention solves the problem of movement of a vehicle for a long period of time causing image and initial calibration inconsistency. The video captured by the camera is used for determining error, and is ultimately used for adjusting the position and/or angle of the camera, thus achieving true closed-loop control.

Description

Method and system for automatically controlling a camera of an autonomous vehicle

Cross-reference to related applications

This application claims the Chinese patent application number "201710792884.2" submitted by Baidu Online Network Technology (Beijing) Co., Ltd. on September 5, 2017, and the invention titled "Method and System for Automatic Control of Cameras of Self-driving Vehicles" priority.

Technical field

The present disclosure relates to automatic driving of a vehicle, and more particularly to control of a camera of an autonomous vehicle.

Background technique

Autonomous driving is a smart car that is driven by a computer system. Automated driving vehicles rely on artificial intelligence, visual computing, radar, surveillance devices and global positioning systems to work together to allow the computer to operate the vehicle automatically and safely without any human active operation.

The camera is the main sensor for autonomous driving, and its attitude is especially important for the entire system. During the automatic driving of the vehicle, there is an error in the position and angle of the camera, which is disadvantageous for autonomous driving.

Despite the rapid development of the automatic driving field, the automatic control of the camera's pose is still a problem that we need to study and solve.

Summary of the invention

In accordance with an embodiment of the present invention, it is desirable to provide a method and system capable of automatically controlling a camera of an autonomous vehicle, by which it is desirable to have the camera substantially always in a position and posture that facilitates automatic driving to ensure The stability and safety of autonomous driving.

According to an embodiment of an aspect of the present invention, a system for implementing automatic control of a camera of an autonomous vehicle is provided, including:

a camera mounted on the vehicle to capture video;

a processor configured to process the video to determine an error in position and/or angle of the camera;

The drive unit is connected to the camera and drives the camera to change position and/or angle based on the error.

Further, the driving device includes:

a connection portion configured to be connected to the camera;

a communication interface communicatively coupled to the processor, configured to receive a signal from the processor indicative of an error in the position and/or angle of the camera;

A movable portion that is connected to the connection portion and adjusts the position and/or angle of the camera through the connection portion according to a signal from the communication interface.

Further, the movable portion has a first rotating shaft and a second rotating shaft, and can drive the camera to rotate along at least one of the rotating shafts.

Further, the movable portion further has a third rotating shaft and can drive the camera to rotate along the third rotating shaft.

Further, the processor processes the video using a deep learning network.

Further, the processor includes: a first unit configured to extract an image of the at least one moment from the video; and a second unit configured to compare the image with an existing image in the deep learning network to obtain a position of the camera and / or the angle of the error with the global optimal value.

Further, the processor further includes: a third unit configured to determine vehicle control information based on the video.

Further, the third unit provides vehicle control information to the automated driving system of the vehicle for controlling the automatic driving process.

According to an embodiment of another aspect of the present invention, a driving device for a camera of an autonomous vehicle is provided, including:

a connection portion configured to be connected to the camera;

a communication interface configured to receive a signal indicative of an error in the position and/or angle of the camera;

A movable portion that is connected to the connecting portion and adjusts the position and/or angle of the camera through the connecting portion according to the signal.

According to still another aspect of the present invention, a method for automatically controlling a camera of an autonomous vehicle is provided, including:

a. capture video through the camera of the vehicle;

b. processing the video with a computer to determine an error in the position and/or angle of the camera;

c. The drive device changes the position and/or angle based on the error by the drive.

Further, the step c includes: the driving device drives the camera to rotate along at least one rotating shaft of the driving device.

Further, step b includes processing the video using a deep learning network.

Further, wherein the processing the video by using the deep learning network comprises: extracting an image of the at least one moment from the video; and

The extracted image is compared with an existing image in the deep learning network to obtain an error with a global optimum value of the position and/or angle of the camera.

Further, the method further includes determining vehicle control information based on the video.

Further, the method further includes providing vehicle control information to the automated driving system of the vehicle for controlling the automatic driving process.

According to an embodiment of still another aspect of the present invention, there is provided an automatic driving system for a vehicle including the aforementioned system for realizing automatic control of a camera of an automatically driven vehicle.

Compared with the prior art, the embodiment of the present invention has the following advantages: 1. By automatically controlling the position and/or angle of the camera, the problem that the image and the initial calibration are inconsistent due to long-time movement of the vehicle is solved; 2. The camera The video captured by itself is used for the determination of the error, and finally used for the adjustment of the position and/or angle of the camera, realizing the closed-loop control; 3. The above control can be continuously updated according to the latest video data to always adapt to the complexity. The automatic driving environment; 4. The proposed method and system can be applied to the environment where the artificially driven vehicle and the self-driving vehicle are mixed, and the environment in which only the self-driving vehicle exists is highly adaptable.

DRAWINGS

Other features, objects, and advantages of the present invention will become apparent from the Detailed Description of Description

1 shows a schematic diagram of a driving environment to which the method and system are applicable in accordance with an embodiment of the present invention;

2 is a schematic diagram of a system for implementing automatic control of a camera of an autonomously driven vehicle, in accordance with an embodiment of the present invention;

3 is a schematic diagram of a driving device of a camera of an autonomous vehicle according to an embodiment of the present invention;

4 is a schematic flow chart of a method of automatically controlling a camera of an autonomous vehicle in accordance with an embodiment of the present invention.

The same or similar reference numerals in the drawings denote the same or similar components.

Detailed ways

Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as a process or method depicted as a flowchart. Although the flowcharts describe various operations as a sequential process, many of the operations can be implemented in parallel, concurrently or concurrently. In addition, the order of operations can be rearranged. The process may be terminated when its operation is completed, but may also have additional steps not included in the figures. The processing may correspond to methods, functions, procedures, subroutines, subroutines, and the like.

By "computer device", also referred to as "computer" in the context, is meant an intelligent electronic device that can perform predetermined processing, such as numerical calculations and/or logical calculations, by running a predetermined program or instruction, which can include a processor and The memory is executed by the processor to execute a predetermined process pre-stored in the memory to execute a predetermined process, or is executed by hardware such as an ASIC, an FPGA, a DSP, or the like, or a combination of the two. Computer devices include, but are not limited to, servers, personal computers, laptops, tablets, smart phones, and the like.

The computer device includes a user device and a network device. The user equipment includes, but is not limited to, a computer, a smart phone, a PDA, etc.; the network device includes but is not limited to a single network server, a server group composed of multiple network servers, or a cloud computing based computer Or a cloud composed of a network server, wherein cloud computing is a type of distributed computing, a super virtual computer composed of a group of loosely coupled computers. Wherein, the computer device can be operated separately to implement the present invention, and can also access the network and implement the present invention by interacting with other computer devices in the network. The network in which the computer device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, and the like.

It should be noted that the user equipment, the network equipment, the network, and the like are merely examples, and other existing or future possible computer equipment or networks, such as those applicable to the present invention, are also included in the scope of the present invention. It is included here by reference.

The methods discussed below, some of which are illustrated by flowcharts, can be implemented in hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to carry out the necessary tasks can be stored in a machine or computer readable medium, such as a storage medium. The processor(s) can perform the necessary tasks.

The specific structural and functional details disclosed are merely representative and are for the purpose of describing exemplary embodiments of the invention. The present invention may, however, be embodied in many alternative forms and should not be construed as being limited only to the embodiments set forth herein.

It will be understood that when a unit is referred to as "connected" or "coupled" to another unit, it can be directly connected or coupled to the other unit, or an intermediate unit can be present. In contrast, when a unit is referred to as being "directly connected" or "directly coupled" to another unit, there is no intermediate unit. Other words used to describe the relationship between the units should be interpreted in a similar manner (eg "between" and "directly between" and "adjacent to" Than "directly adjacent to", etc.).

It should be understood that although the terms "first," "second," etc. may be used herein to describe the various elements, these elements should not be limited by these terms. These terms are used only to distinguish one unit from another. For example, a first unit could be termed a second unit, and similarly a second unit could be termed a first unit, without departing from the scope of the exemplary embodiments. The term "and/or" used herein includes any combination of one or more of the associated listed items.

The terminology used herein is for the purpose of describing the particular embodiments, The singular forms "a", "an", It is also to be understood that the terms "comprising" and """ Other features, integers, steps, operations, units, components, and/or combinations thereof.

It should also be noted that in some alternative implementations, the functions/acts noted may occur in a different order than that illustrated in the drawings. For example, two figures shown in succession may in fact be executed substantially concurrently or sometimes in the reverse order, depending on the function/acts involved.

The invention is further described in detail below with reference to the accompanying drawings.

1 shows a schematic diagram of a driving environment to which the method and system are applicable in accordance with an embodiment of the present invention. The scenario is an environment 1 at an intersection in which there are three

cars

11, 12 and 13 that are driving or waiting, and a number of pedestrians 14 that are crossing the crosswalk. Among them, each car is equipped with a camera 15 for automatic driving. The camera 15 provides road information to the on-vehicle automatic driving system by taking a video, and the automatic driving system generates control information based on current road conditions and the like, and the control information acts on each device and component of the automobile to start, continue or stop the automatic driving. . The Chinese invention patent application No. 201610027742.2 proposes an automatic driving device that selects a path with a low computational amount associated with automatic driving in the case where a plurality of paths are searched. The automatic driving device includes: a position acquisition unit that acquires position information of another vehicle that automatically drives and manually drives another vehicle; a route search unit that searches for a route; and a calculation unit that searches for a plurality of routes by the route search unit, a ratio of the number of other vehicles that are automatically driven on the path of each path to the total of all other vehicles on the path based on the position information of the other vehicle and the other vehicle that is manually driven; the selection unit is selected by the operation unit The calculated maximum path for automatically driving other vehicles is a route for the own vehicle to travel; and the control unit performs automatic driving of the own vehicle to travel along the selected route. Autopilot system devices such as these can be used for the autonomous driving mentioned herein.

Although each vehicle in the environment 1 has a camera 15, it should be noted that the self-driving vehicle using the method and system of the present invention can also be in a driving environment with a manually driven vehicle without causing a driving environment. Adverse effects, on the contrary, are often beneficial to the driving environment as the behavior of autonomous vehicles is protected and improved.

Four directions, D1-D4, are shown in FIG. Referring to Figure 1, the car 11 is ready to pass through the intersection in the direction D2. The car 13 has basically passed the intersection in the direction D2 and continues to travel. The driving direction of the car 12 is D4, and is waiting for the green light. The plurality of pedestrians 14 are in the direction of D1 or D2. Through the crosswalk.

The long-term movement of the car may cause the image captured by the camera to be inconsistent with the initial calibration. In order to protect the self-driving car itself and other cars and pedestrians in the environment 1, the inventors of the present invention first realized that the posture of the camera 15 is required. (ie position and / or angle (angle can also be called attitude angle)) make the necessary adjustments. The specific manner will be further explained below in conjunction with the drawings.

2 is a schematic diagram of a system for implementing automatic control of a camera of an autonomous vehicle. System 2 primarily includes a camera 25, primarily for capturing video, a drive 26 coupled to camera 25, and a processor 27 communicatively coupled to camera 25 and drive unit 26. Processor 28 processes the video and obtains an error in the position and/or angle of camera 25, which is provided to drive unit 26, which adjusts the position and/or angle of camera 25 based on the error. The processor 27 is operatively in communication with the memory 28 for performing the data access required to perform its functions, and the processor 27 and memory 28 may be in-vehicle, such that the cloud processor and memory may be used to provide transmission delays, etc. The problems, which are the auto-driving environments that are required for safety, are avoided as much as possible.

The number of cameras 25 may be one or more. Its position can be located at the front of the car so that it is advantageously able to look substantially straight ahead of the direction of travel of the car. Of course, it is also feasible to perform the teachings of the present invention using a video captured by a camera located on the side of the vehicle body or even behind. In order to facilitate the provision of vehicle control information to the automatic driving system, the video captured by the camera 25 should be used to identify the following: lane lines in front of the vehicle, traffic signs, traffic signs and obstacles (pedestrians, other vehicles, etc.), and then A camera (not shown) can identify lane lines and obstacles behind the vehicle, which can share the same processor for video processing and determination of respective pose errors.

Alternatively, the calibration method mentioned in the Chinese Patent Application Publication No. CN103077518A can also be used to generate the aforementioned error value. Although the present disclosure tends to implement the above-described deep learning network-based implementation, it is not limited thereto.

3 is a schematic diagram of a driving device of a camera of an autonomous vehicle according to an embodiment of the present invention. It can be seen that the camera 35 captures a video, and the captured video is transmitted to the processor 37 through a communication interface with the processor 37 and processed. The result of the processing is an error value meaningful to the pose of the camera 35. The value is communicated to the drive device via communication interface 362 between processor 37 and drive unit 36. The drive unit 36 is mechanically coupled to the camera 35 via a connection portion 361, and the position and/or angle of the camera 35 is adjusted accordingly according to a signal (error value) from the communication interface 362 by the movable portion 363.

Referring to FIG. 2, the movable portion may have a first rotation axis a and a second rotation axis b, and may drive the camera to rotate along at least one of the rotation axes. Further, the movable portion may further have a third rotation axis c, and can drive the camera to rotate along the third rotation axis. It should be understood that the above-described arrangement of the rotating shaft may have other variations, and is not limited to the above examples.

Referring to FIG. 3, processor 37 processes the video captured by camera 35 using a deep learning network, in cooperation with (if desired) memory, in accordance with one embodiment. Specifically, the processor 37 can be implemented to include a first unit and a second unit. The first unit is configured to extract an image of at least one time from the video, and the second unit is configured to compare the image with an existing image in the deep learning network to obtain a position and/or an angle of the camera 35. The error of the global optimal value.

Further, the processor 35 may further include a third unit configured to determine vehicle control information based on the video and provide the vehicle control information to the automatic driving system of the vehicle for controlling the automatic driving process, including but not limited to Steal or brake while avoiding obstacles, continue driving, turning, etc.

Preferably, after one adjustment, the camera 35 will capture a new video and continue to provide to the processor 37. After processing the updated video, the processor 37 will obtain new error value information, which should generally be Better than before. In one example, the processor 37 maintains a threshold. When the adjusted error is less than the threshold, it may be considered that the pose of the camera 35 need not be adjusted again until the next time it is triggered, for example, after the new video processing, the error is again high. At this threshold.

4 is a schematic flow chart of a method of automatically controlling a camera of an autonomous vehicle in accordance with an embodiment of the present invention. The method can include:

Step 41: capturing a video through a camera of the vehicle;

Step 42: processing the video with a computer to determine an error in position and/or angle of the camera;

Step 43: The camera changes the position and/or angle according to the error by the driving device.

In a preferred embodiment, in step 43, the drive unit drives the camera to rotate along at least one axis of rotation of the drive unit.

In a preferred embodiment, in step 42, a computer (e.g., a processor) processes the video using a deep learning network.

The various step features of the method correspond to the features of the system, drive device described above in connection with Figures 1-3, which can be understood and implemented with reference to the related description of the above-described embodiments.

In one non-limiting embodiment, the step of processing the video using the deep learning network includes extracting an image of the at least one time from the video; and comparing the image to an existing image in the deep learning network to derive the camera The error of the position and/or angle with a global optimum.

Optionally, the method further comprises determining vehicle control information based on the video, and providing vehicle control information to an automated driving system of the vehicle for controlling an automated driving process.

Preferably, after an adjustment, in step 41, the camera will capture a new video and continue to provide to the processor. In step 42, after processing the updated video, the processor will obtain new error value information. Value information should generally be improved over the previous period. In one example, the processor maintains a threshold. When the adjusted error is less than the threshold, it may be considered that the step of the camera is not required to be performed again to adjust the pose of the camera until the next time it is triggered, for example, the error is found after the new video processing. Above this threshold again.

The automatic driving system mentioned in the embodiments of the present invention, the system for implementing automatic camera control, and the processor include, but are not limited to, a computer device and an automotive electronic device. A computer device refers to an intelligent electronic device that can perform a predetermined process such as numerical calculation and/or logic calculation by running a predetermined program or instruction, which can include a processor and a memory, which are executed by the processor to execute a pre-stored survival instruction in the memory. The predetermined processing is performed by a hardware such as an ASIC, an FPGA, a DSP, or the like, or a combination of the two. Computer devices include, but are not limited to, servers, personal computers, notebook computers, tablets, smart phones, and the like. The server includes, but is not limited to, a single server, a plurality of servers, or a cloud-based cloud composed of a large number of computers or servers, wherein the cloud computing is a type of distributed computing, a super virtual composed of a group of loosely coupled computers. computer. Automotive electronics are electronic devices used in or related to automobiles.

It is noted that at least a portion of the present invention can be implemented in software and/or a combination of software and hardware. For example, the various devices of the present invention can be implemented using an application specific integrated circuit (ASIC) or any other similar hardware device. In one embodiment, the software program of the present invention may be executed by a processor to implement the steps or functions described above. As such, the software programs (including related data structures) of the present invention can be stored in a computer readable medium. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive lists) of computer readable storage media include: electrical connections having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), Erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium can be any tangible medium that can contain or store a program, which can be used by or in connection with an instruction execution system, apparatus or device. A computer readable signal medium may include a data signal that is propagated in the baseband or as part of a carrier, carrying computer readable program code. Such propagated data signals can take a variety of forms including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer readable signal medium can also be any computer readable medium other than a computer readable storage medium, which can transmit, propagate, or transport a program for use by or in connection with the instruction execution system, apparatus, or device. . Program code embodied on a computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing. Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including an object oriented programming language such as Java, Smalltalk, C++, and conventional Procedural programming language—such as C language or a similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer, partly on the remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (eg, using an Internet service provider) Internet connection).

Additionally, some of the steps or functions of the present invention may be implemented in hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.

It is apparent to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, and the present invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the present embodiments are to be considered as illustrative and not restrictive, and the scope of the invention is defined by the appended claims instead All changes in the meaning and scope of equivalent elements are included in the present invention. Any reference signs in the claims should not be construed as limiting the claim. In addition, it is to be understood that the word "comprising" does not exclude other elements or steps. A plurality of units or devices recited in the system claims can also be implemented by a unit or device by software or hardware. The first, second, etc. words are used to denote names and do not denote any particular order.

Claims

A system for automatically controlling a camera of an autonomous vehicle, comprising:

a camera mounted to the vehicle for capturing video;

a processor configured to process the video to determine an error in position and/or angle of the camera;

A drive device coupled to the camera and driving the camera to change position and/or angle in accordance with the error.
The system of claim 1 wherein said driving means comprises:

a connection portion configured to be coupled to the camera;

a communication interface communicatively coupled to the processor, configured to receive a signal from the processor indicative of an error in position and/or angle of the camera;

a movable portion connected to the connecting portion and adjusting a position and/or an angle of the camera through the connecting portion according to the signal.
The system of claim 2, wherein the movable portion has a first axis of rotation, a second axis of rotation, and the camera is rotatable along at least one of the axes of rotation.
The system of claim 3, wherein the movable portion further has a third axis of rotation and can drive the camera to rotate along the third axis of rotation.
The system of any of claims 1 to 4, wherein the processor processes the video using a deep learning network.
The system of claim 5 wherein said processor comprises:

a first unit configured to extract an image of the at least one moment from the video;

And a second unit configured to compare the image with an existing image in the deep learning network to derive an error having a global optimal value for the position and/or angle of the camera.
The system of claim 5 wherein said processor further comprises:

A third unit configured to determine vehicle control information based on the video.
The system of claim 7 wherein said third unit provides said vehicle control information to an automated driving system of said vehicle for controlling an automated driving process.
A driving device for a camera of an autonomous vehicle, comprising:

a connection portion configured to be coupled to the camera;

a communication interface configured to receive a signal indicative of an error in position and/or angle of the camera;

a movable portion connected to the connecting portion and adjusting a position and/or an angle of the camera through the connecting portion according to the signal.
The driving apparatus according to claim 9, wherein said movable portion has a first rotating shaft, a second rotating shaft, and is rotatable by said camera along at least one of the rotating shafts.
The driving device according to claim 10, wherein said movable portion further has a third rotation shaft and is rotatable by said camera along said third rotation axis.
A method for automatically controlling a camera of an autonomous vehicle includes:

a. capturing video through the camera of the vehicle;

b. processing the video with a computer to determine an error in position and/or angle of the camera;

c. The camera drives the camera to change position and/or angle based on the error.
The method of claim 12 wherein said step c comprises:

The driving device drives the camera to rotate along at least one rotating shaft of the driving device.
The method of claim 12 or 13, wherein said step b comprises:

The video is processed using a deep learning network.
The method of claim 14 wherein said processing said video using a deep learning network comprises:

Extracting an image of at least one moment from the video; and

The image is compared to an existing image in the deep learning network to derive an error with a global optimum of the position and/or angle of the camera.
The method of claim 15 further comprising:

Vehicle control information is determined based on the video.
The method of claim 16 further comprising:

The vehicle control information is provided to an autopilot system of the vehicle for controlling an autopilot process.
An automatic driving system for a vehicle, comprising the system for automatically controlling a camera of an autonomously driven vehicle according to any one of claims 1 to 8.