WO2021253741A1

WO2021253741A1 - Scenario identification-based vehicle adaptive sensor system

Info

Publication number: WO2021253741A1
Application number: PCT/CN2020/133599
Authority: WO
Inventors: 田爽; 高享久; 张伟方
Original assignee: 重庆长安汽车股份有限公司
Priority date: 2020-06-15
Filing date: 2020-12-03
Publication date: 2021-12-23
Also published as: CN111767910B; CN111767910A

Abstract

A scenario identification-based vehicle adaptive sensor system, comprising: an AVM around view sensor (1), an AVM around view controller (3), and an ADAS controller (5); the AVM around view sensor (1) includes a camera, a position adjustment mechanism (11), and a position sensor (12); the camera transmits collected image information to the AVM around view controller (3), the position adjustment mechanism (11) receives a position adjustment instruction of the AVM around view controller (3) and adjusts a position of the camera, and the position sensor provides feedback on the position of the camera to the AVM around view controller (3); and the AVM around view controller (3) transmits identified target information to the ADAS controller (5). The described system obtains an ROI region of interest adapted for a current vehicle operating scenario by means of scenario identification optimization, improving image processing performance of the system; within the AVM around view sensor (1), target identification is performed on an image, and output target and scenario information may be used by the ADAS controller (5), which can reduce the needed image processing chip capability of the ADAS controller (5), and can also reduce power consumption of an ADAS system, effectively reducing configuration costs.

Description

Vehicle adaptive sensor system based on scene recognition

Technical field

The invention relates to the field of sensor performance optimization under an automatic driving situation, and in particular to an adaptive sensor system.

Background technique

For complex driving scenarios, the existing sensor hardware systems for autonomous driving and parking cannot recognize multiple scenarios and make corresponding hardware and software adjustments. AVM panoramic camera (AVM, Around View Monitor) FOV field of view (FOV, Field of View) and ROI region of interest (ROI, Region of Interest) are fixed, the maximum detection range is also fixed, so the system is In high-speed scenes, a wider area cannot be detected, which may result in the omission of information. The effective detection areas of parking scenes and high-speed driving scenes are different, and the existing panoramic surround view camera sensor cannot automatically adapt to different scenes, changing its performance, mainly involving the field of view and the region of interest.

In the future, hardware processing capabilities will increase rapidly, target detection and classification will become more refined, and sensor fusion requirements will become higher and higher, but the current collaborative processing efficiency between sensors is low, and a single control system often requires a separate sensor. Therefore, for future autonomous driving systems, it is very necessary to improve the performance of the AVM controller and realize the multi-function fusion of sensors.

However, the existing solutions only rely on hardware upgrades or software optimizations. In the future, we need a comprehensive upgrade of these two parts. More cameras require higher costs, and more sensors will also make vehicle design more complicated. Since the optimal installation position of the sensor and the aesthetic appearance are difficult to compromise, if more sensors are used, it will bring greater difficulty to the vehicle design. At the same time, every OEM wants to make a beautiful and unobtrusive sensor design, which makes it difficult to implement the idea of installing more sensors on the vehicle.

Summary of the invention

At least some of the embodiments of the present invention provide a vehicle adaptive sensor system based on scene recognition, which realizes the desired position adjustment of the camera according to different scenes, optimizes the region of interest, realizes target detection in the sensor stage and transmits the result directly to ADAS control The driver (Advanced Driver Assistance System) optimizes the scene recognition accuracy and system computing burden.

In one of the embodiments of the present invention, a vehicle adaptive sensor system based on scene recognition is provided, which includes an AVM surround view sensor, an AVM surround view controller, and an ADAS controller;

The AVM surround view sensor (1) includes a camera, a position adjustment mechanism and a position sensor; the camera transmits the collected image information to the AVM surround view controller, and the position adjustment mechanism receives the position adjustment instruction of the AVM surround view controller and adjusts the position of the camera , The position sensor feeds back the position information of the camera to the AVM surround view controller;

The AVM surround view controller is connected with the ADAS controller, and transmits the identified target information to the ADAS controller;

During work, the AVM surround view controller determines the current scene information according to the image information input by the camera, calculates the desired position of the camera according to different scene information, and generates a position adjustment instruction based on the desired position and outputs it to the position adjustment mechanism;

After the position adjustment mechanism adjusts the position of the camera, the AVM surround view controller receives the updated position information and the updated image information of the camera, optimizes the region of interest for the image information input after the position adjustment, and performs the target after obtaining the current region of interest Identify, output the identified target information to the ADAS controller.

In an optional embodiment, when the vehicle is in low light conditions, the speed and gear of the vehicle are low, there are many target obstacles, or there are traffic lights and pedestrian obstacles, the position adjustment mechanism adjusts the horizontal and vertical angles of the camera to lower the camera. And reduce the scope of the current area of interest. When the vehicle is in high light conditions, the speed and gear of the vehicle are high, and the number of target obstacles is small, the position adjustment mechanism should increase the vertical position of the camera to increase the camera's field of view and increase the current area of interest.

In an optional embodiment, the AVM surround view controller includes: MCU and SOC; the camera of the AVM surround view sensor is connected to the SOC through LVDS and transmits image information to the SOC; the MCU is connected to the position adjustment mechanism and the position sensor through a hard wire; MCU Connect with ADAS controller via CAN or Ethernet.

In an optional embodiment, the camera of the AVM surround view sensor transmits image information to the SOC10, and the SOC10 determines the current driving scene and outputs the scene information to the MCU. The MCU calculates the desired position of the camera according to different scene information, generates a position adjustment instruction based on the desired position of the camera, and outputs the position adjustment instruction to the position adjustment mechanism to adjust the position of the camera. After the position of the camera is adjusted, the position sensor will adjust the position of the camera. The position information is fed back to the MCU, and the MCU is fed back to the SOC; the camera inputs the adjusted image information to the SOC, and the SOC optimizes the region of interest based on the updated position information of the camera and the updated image information. After the current region of interest is obtained Perform target recognition and transmit target information to the MCU. The MCU transmits the identified target information to the ADAS controller.

In an optional embodiment, the factors that affect the judgment of the scene include: light, the number of obstacles, the speed of the vehicle, the gear position of the vehicle, and the obstacles.

In an optional embodiment, the scene classification includes:

Parking lot entrance: the scene of the parking lot entrance is dimly lit, and the driving space is narrow, there are many target obstacles, the speed and gear of the vehicle are low, and there are fence restrictions;

Expressway: The speed and gear of the vehicle are high, the light is sufficient, the visibility is high, the driving space is wide, there are few obstacles, and there is a situation where the vehicle quickly cuts in;

Toll station: When the vehicle passes through the toll station, the passage is narrow, and there are checkpoint restrictions, the speed and gear of the vehicle are low, and there are many target obstacles;

Urban roads: In urban road scenes, the speed and gears of vehicles are low, the surrounding environment is complex, traffic lights, vehicles, and pedestrians increase, and there are many obstacles and types.

In an optional embodiment, the optimization of the region of interest includes: after the position of the camera is adjusted, the AVM surround view controller receives the updated position information and the updated image information of the camera, and recognizes the current driving scene according to different scenes. The information gets the corresponding region of interest.

In an optional embodiment, the target recognition includes: after the ROI region of interest is divided, performing target recognition on the image in the divided ROI region of interest again through deep learning, and outputting the identified target information to ADAS controller.

In an optional embodiment, the AVM surround view controller performs image processing and stitching according to the image information input by the camera, and combines the deep learning training model to determine the current scene;

In an optional embodiment, the adaptive sensor system based on scene recognition includes four AVM surround view sensors, and the four AVM surround view sensors are respectively installed at the front bumper, the rear door, and under the left and right rearview mirrors.

The beneficial technical effects of at least some of the embodiments of the present invention are:

1) Through scene recognition and optimization, an ROI region of interest suitable for the current driving scene is obtained, which improves the image processing performance of the system;

2) Perform target recognition on the image in the AVM surround view sensor. The output target and scene information can also be used in the ADAS controller, which can reduce the image processing chip capability required by the ADAS controller, and at the same time can reduce the power consumption of the ADAS system, effectively Reduce configuration costs.

3) The system can realize the optimization of sensor performance and power consumption in the field of autonomous driving and parking perception, and improve the recognition accuracy.

Description of the drawings

Fig. 1 is a schematic structural diagram of a vehicle adaptive sensor system based on scene recognition according to one of the embodiments of the present invention.

Fig. 2 is a schematic diagram of the control principle of an AVM surround view controller according to one of the alternative embodiments of the present invention.

Fig. 3 is a schematic diagram of an SOC image processing process according to an alternative embodiment of the present invention.

Fig. 4 is a schematic diagram of a region of interest in a high-speed scene before adjustment according to one of the optional embodiments of the present invention.

Fig. 5 is a schematic diagram of the adjusted high-speed scene interest area according to one of the optional embodiments of the present invention.

Fig. 6 is a schematic diagram of a region of interest in a low-speed scene before adjustment according to an alternative embodiment of the present invention.

Fig. 7 is a schematic diagram of the adjusted low-speed scene interest area according to one of the optional embodiments of the present invention.

Among them, 1-AVM surround view sensor, 2-LVDS, 13-AVM surround view controller, 4-CAN or Ethernet, 5-ADAS controller, 6-MCU, 7-SOC, 11-position adjustment mechanism, 12-position sensor .

detailed description

In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is a part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

It should be noted that the terms “first” and “second” in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and not necessarily used to describe a specific sequence or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those clearly listed. Those steps or units may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or equipment.

As shown in FIG. 1, in one of the embodiments of the present invention, an adaptive sensor system based on scene recognition is provided, including: an AVM surround view sensor 1, an AVM surround view controller 3, and an ADAS controller 5.

The AVM surround view sensor 1 includes a camera, a position adjustment mechanism 11 and a position sensor 12. The AVM surround view controller 3 is composed of MCU6 (Micro Control Unit) and SOC7 (System-on-a-Chip).

The camera of the AVM surround view sensor is connected to the SOC7 through LVDS2 (Low-Voltage Differential Signaling), and transmits image information to the SOC7. SOC7 pre-processes the input image information to obtain stitched images, and judges the current driving scene through deep learning; SOC7 outputs the scene information of the current driving scene to MCU6.

The MCU6 is connected to the position adjustment mechanism 11 and the position sensor 12 through a hard wire. The MCU6 calculates the desired position of the camera according to different scenarios, generates a position adjustment instruction (such as angle adjustment PWM signal) based on the desired position of the camera, and outputs the position adjustment instruction to the camera position adjustment mechanism 11 to control the position sensor to adjust the position of the camera. After the position of the camera is adjusted, the position sensor feeds back the position information of the camera to the MCU6, and the MCU6 feeds back the SOC7. At the same time, the camera inputs the adjusted image to the SOC7, and the SOC7 optimizes the region of interest based on the updated position information and image information of the camera. That is, when the vehicle is in a scene with good road conditions and a wide field of view, such as a highway, the area of interest is expanded; when the vehicle is in a scene with a poor environment such as congested, dim, or narrow, the area of interest is reduced. After the current region of interest is obtained, target recognition is performed, and the target information is transmitted to MCU6.

The MCU6 is connected to an ADAS controller 5 (Advanced Driver Assistance System) via CAN or Ethernet 4, and transmits the identified target information to the ADAS controller 5.

In an optional embodiment, the ROI optimization function principle includes: after the camera position is adjusted, the SOC7 in the AVM surround view controller 3 receives the updated position information and the updated image information of the camera, and recognizes the current driving scene , Obtain the corresponding ROI area of interest according to different scenes, that is, when the car linkage is in a scene with good road conditions and a wide field of view, such as a highway, expand the area of interest; when the vehicle is in a congested, dim, or narrow scene with a poor environment, Reduce the area of interest.

In an optional embodiment, the functional principle of target recognition includes: after the ROI region of interest is divided, target recognition is performed on the image in the segmented ROI region of interest again through deep learning, and the identified target information is output To ADAS controller 5.

In an optional embodiment, the factors that affect the judgment of the scene include: light, the number of obstacles, the speed of the own vehicle, the gear position of the own vehicle, and special obstacles.

In an optional embodiment, the scene classification includes:

Parking lot entrance: In the scene of the parking lot entrance, the light is dim, the driving space is narrow, there are many target obstacles, the speed and gear of the vehicle are low, and there are fence restrictions.

Expressway: The vehicle has high speed and gear, sufficient light, high visibility, wide driving space, few obstacles, and the situation where the vehicle cuts in quickly.

Toll station: When a vehicle passes through a toll station, the passage is narrow, and there are checkpoint restrictions, the speed and gear of the vehicle are low, and there are many target obstacles.

In an optional embodiment, the factors that affect the system scene recognition mainly include factors such as light, the number of obstacles, the speed and gear of the vehicle, and special obstacles. For example, when the vehicle is in low light conditions, the speed and gear of the vehicle are low, there are many target obstacles, and there are special obstacles (traffic lights, pedestrians), it can be considered that the vehicle is driving in an urban road scene. At this time, the position adjustment mechanism 11 should adjust the horizontal and vertical angles of the camera to reduce the field of view. For another example, when the vehicle is in high light conditions, the number of obstacles is small, the speed and gear of the vehicle are high, and there are no special obstacles, it can be considered that the vehicle is in a highway scene, and the camera should be raised at this time. The vertical position of the sensor to increase the field of view of the sensor.

In an optional embodiment, the adaptive sensor system based on scene recognition includes four AVM surround view sensors 1, and the four AVM surround view sensors 1 are respectively installed at the front bumper, the rear door, and under the left and right rearview mirrors.

As shown in Figure 4 to Figure 7, in a high-speed scene, a wider effective field of view can be obtained after adjusting the camera. The white shaded part shown in Figure 4 is the region of interest before optimization, and the white shaded part shown in Figure 5 is based on the optimized region of interest in the high-speed scene. It can be seen that through the optimization of the region of interest, you can get More image information. The white shaded part shown in Figure 6 is the region of interest before optimization, and the white shaded part shown in Figure 7 is the optimized region of interest. It can be seen that in a narrow and low-speed scene, the interest is increased. The area of the area also removes the invalid area of interest where the vehicle itself is located, so more image information can be obtained. The target information obtained by the SOC7 processing can be used in the ADAS controller 5, which can replace the role of the image chip in the ADAS control system, effectively reducing the hardware cost and the power consumption of the ADAS system.

The sequence numbers of the foregoing embodiments of the present invention are only for description, and do not represent the superiority or inferiority of the embodiments.

In the above-mentioned embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in an embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units may be a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or may be Integrate into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be indirect couplings or communication connections through some interfaces, units or modules, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present invention essentially or the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium. , Including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage media include: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes. .

The above are only the preferred embodiments of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications are also It should be regarded as the protection scope of the present invention.

Claims

A vehicle adaptive sensor system based on scene recognition, including: AVM surround view sensor (1), AVM surround view controller (3) and ADAS controller (5);

The AVM surround view sensor (1) includes a camera, a position adjustment mechanism (11) and a position sensor (12); the camera transmits the collected image information to the AVM surround view controller (3), and the position adjustment mechanism ( 11) Receive the position adjustment instruction of the AVM surround view controller (3) and adjust the position of the camera, and the position sensor feeds back the position information of the camera to the AVM surround view controller (3);

The AVM surround view controller (3) is connected to the ADAS controller (5), and transmits the identified target information to the ADAS controller (5);

During operation, the AVM surround view controller (3) determines the current scene information according to the image information input by the camera, calculates the desired position of the camera according to different scene information, and generates the position adjustment based on the desired position The instruction is output to the position adjustment mechanism (11);

After the position adjustment mechanism (11) adjusts the position of the camera, the AVM surround view controller (3) receives the updated position information and the updated image information of the camera, and performs processing on the image information input after the position adjustment The region of interest is optimized. After the current region of interest is obtained, target recognition is performed, and the recognized target information is output to the ADAS controller (5).
The vehicle adaptive sensor system based on scene recognition of claim 1, wherein:

When the vehicle is in low light conditions, the speed and gear of the vehicle are low, the number of target obstacles is large, or there are traffic lights and pedestrian obstacles, the position adjustment mechanism adjusts the horizontal and vertical angles of the camera to reduce the field of view of the camera Range and reduce the range of the current region of interest;

When the vehicle is in high light conditions, the speed and gear of the vehicle are high, and the number of target obstacles is small, the position adjustment mechanism raises the vertical position of the camera to increase the field of view of the camera and increase the The extent of the current area of interest.
The vehicle adaptive sensor system based on scene recognition according to claim 2, wherein the AVM surround view controller includes: MCU (6) and SOC (7); the camera of the AVM surround view sensor is connected with the LVDS (2) through the LVDS (2) The SOC (7) is connected, and the image information is transmitted to the SOC (7); the MCU (6) is connected with the position adjustment mechanism and the position sensor through a hard wire; the MCU (6) It is connected with the ADAS controller (5) through CAN or Ethernet (4).
The vehicle adaptive sensor system based on scene recognition of claim 3, wherein:

The camera of the AVM surround view sensor (1) transmits the image information to the SOC (7), and the SOC (7) determines the current driving scene and outputs the scene information to the MCU (6);

The MCU (6) calculates the desired position of the camera according to different scene information, generates the position adjustment instruction based on the desired position of the camera, and outputs the position adjustment instruction to the position adjustment mechanism to adjust the The position of the camera, after the position of the camera is adjusted, the position sensor (12) feeds back the position information of the camera to the MCU (6), and the MCU (6) feeds back to the SOC (7); The camera inputs the adjusted image information to the SOC (7), and the SOC (7) optimizes the region of interest based on the updated position information and updated image information of the camera, and then Perform target recognition after the current region of interest, and transmit the target information to the MCU (6);

The MCU (6) transmits the identified target information to the ADAS controller (5).
The vehicle adaptive sensor system based on scene recognition of claim 4, wherein the factors that affect the scene judgment include: light, the number of obstacles, the speed of the vehicle, the gear position of the vehicle, and the obstacles.
The vehicle adaptive sensor system based on scene recognition of claim 5, wherein: scene classification includes:

Parking lot entrance: the scene of the parking lot entrance is dimly lit, and the driving space is narrow, there are many target obstacles, the speed and gear of the vehicle are low, and there are fence restrictions;

Expressway: The speed and gear of the vehicle are high, the light is sufficient, the visibility is high, the driving space is wide, there are few obstacles, and there is a situation where the vehicle quickly cuts in;

Toll station: When the vehicle passes through the toll station, the passage is narrow, and there are checkpoint restrictions, the speed and gear of the vehicle are low, and there are many target obstacles;

Urban roads: In urban road scenes, the speed and gears of vehicles are low, the surrounding environment is complex, traffic lights, vehicles, and pedestrians increase, and there are many obstacles and types.
The vehicle adaptive sensor system based on scene recognition according to claim 6, wherein the optimization of the region of interest comprises: after the position of the camera is adjusted, the AVM surround view controller (3) receives the camera The updated position information and the updated image information of the image information, identify the current driving scene, and obtain the corresponding region of interest according to different scene information.
The vehicle adaptive sensor system based on scene recognition according to claim 7, wherein the target recognition comprises: after the region of interest is divided, performing target recognition on the images in the divided region of interest through deep learning, The identified target information is output to the ADAS controller (5).
The vehicle adaptive sensor system based on scene recognition according to claim 8, wherein the AVM surround view controller (5) performs image processing and stitching according to the image information input by the camera, combined with a deep learning training model, Determine the current scene.
The vehicle adaptive sensor system based on scene recognition of claim 9, wherein the adaptive sensor system based on scene recognition includes four AVM surround view sensors (1), and the four AVM surround view sensors are respectively installed on the front bumper At the back door and under the left and right rearview mirrors.