WO2022166308A1

WO2022166308A1 - Control instruction generation method and device, and control method and device for visual sensor

Info

Publication number: WO2022166308A1
Application number: PCT/CN2021/131695
Authority: WO
Inventors: 李文斌
Original assignee: 华为技术有限公司
Priority date: 2021-02-07
Filing date: 2021-11-19
Publication date: 2022-08-11
Also published as: CN114911219A

Abstract

A control instruction generation method for a visual sensor (10), wherein the visual sensor (10) acquires image data by scanning, and the method comprises: acquiring image data; determining, according to the image data, an object about to collide with a vehicle; generating an image acquisition range adjustment instruction according to the object; sending the image acquisition range adjustment instruction to the visual sensor (10), the image acquisition range adjustment instruction being used for instructing the visual sensor (10) that originally acquires images according to a preset range to acquire images according to a temporary range, and the temporary range being smaller than the preset range and comprising an area where the object is located. Therefore, in an emergency, the vehicle can acquire, in time, image data of the object identified as being about to collide with the vehicle, thereby reducing the occurrence of traffic accidents.

Description

A method and device for generating a control instruction for a vision sensor, and a control method and device

technical field

The present application relates to the field of vehicle automatic driving, and in particular, to a control instruction generation method and device for a visual sensor, and a control method and device.

Background technique

In autonomous driving platforms, sensors play an extremely important role. A number of different sensors can be installed on the vehicle, for example, vision sensors that can provide reversing images, front-view images, rear-view images, top-view images, and panoramic parking images for outside the vehicle; Fatigue driving sensors, sensors that can monitor the status of the instrument panel; and sensors that can provide forward collision warnings for Advanced Driving Assistance Systems (ADAS), sensors that can provide lane departure warnings, can Sensors that automatically control high beams, sensors that can recognize traffic signals, sensors that can detect pedestrians, sensors that can perform adaptive cruise control, sensors that can perform blind-spot detection, and sensors that have night vision capabilities.

The autonomous driving control platform can be connected with a variety of sensors. After the data received by the sensor is transmitted to the automatic driving control platform, the automatic driving control platform performs a series of processing on the data, and finally outputs the control instructions for the vehicle. Under this premise, the end-to-end low latency from the sensor receiving the data to the vehicle executing the autonomous driving control command has become the goal that the industry is constantly pursuing.

SUMMARY OF THE INVENTION

In view of the above problems in the prior art, the purpose of the present application is to provide a control instruction generation method and device for a visual sensor, and a control method and device. The time from the visual sensor receiving the data to the vehicle executing the automatic driving control instruction can be reduced, and the occurrence of traffic accidents caused by the untimely acquisition of the data of the object determined to collide with the vehicle can be reduced.

A first aspect of the embodiments of the present application provides a method for generating a control instruction for a visual sensor, where the visual sensor collects image data by scanning, and the method includes: acquiring image data; The object with which the vehicle collides; an image acquisition range adjustment instruction is generated according to the object; the image acquisition range adjustment instruction is sent to the vision sensor, and the image acquisition range adjustment instruction is used to instruct the image acquisition range adjustment instruction to be originally collected according to the preset range. The visual sensor collects images according to a temporary range, and the temporary range is smaller than the preset range and includes the area where the object is located.

Through the above settings, the vision sensor can immediately scan the area where the object identified as about to collide with the vehicle is located in an emergency, without waiting for the vision sensor to scan the preset range in the current cycle and then rescan the preset range to reduce the image. The size of the collection range reduces the time to acquire the image data of the object that will collide with the vehicle, thereby reducing the risk of traffic accidents due to the untimely collection of the image data of the object that will collide with the vehicle in an emergency.

In a possible implementation manner, the temporary area is a rectangle.

Through the above settings, the image acquisition range can be narrowed down to the rectangular area where the object identified as about to collide with the vehicle is located, thereby reducing the image acquisition time and preventing the untimely avoidance of danger due to the excessively long image acquisition time in an emergency. occur.

In a possible implementation, the visual sensor includes a camera.

A second aspect of the embodiments of the present application provides a control method for a visual sensor, where the visual sensor acquires image data by scanning, the method includes: acquiring an image acquisition range adjustment instruction; adjusting according to the image acquisition range The instruction controls the visual sensor to adjust the image acquisition range, and the image acquisition range adjustment instruction is used to instruct the vision sensor that originally acquired the image according to the preset range to acquire the image according to the temporary range, the temporary range is smaller than the preset range, and includes The area where the object identified as about to collide with the vehicle is located.

Through the above settings, the vision sensor can immediately obtain the image data of the object recognized as the object that will collide with the vehicle, without waiting for the vision sensor to scan the preset range in the current cycle and then re-pre-predict all the preset ranges. Set the range to scan, reduce the size of the image acquisition range, reduce the time to acquire the image data of the object that is identified as about to collide with the vehicle, and thus reduce the risk of being identified as about to collide with the vehicle in an emergency. Risk of traffic accidents due to untimely acquisition of image data of the colliding objects.

In a possible implementation manner, the temporary area is a rectangle.

Through the above arrangement, the image acquisition range can be narrowed down to the rectangular area where the object identified as about to collide with the vehicle is located, thereby reducing the image acquisition time and preventing untimely avoidance due to excessive image acquisition time in emergency situations situation occurs.

In a possible implementation manner, the temporary range of the current image acquisition cycle is adjusted, and when the acquired image range includes the temporary range, image acquisition is performed on the union of the acquired image range and the temporary range .

Through the above setting, repeated image acquisition for the temporary range caused by the complete or partial duplication of the acquired image range and the temporary range is avoided, and the time spent on image acquisition is further reduced.

In a possible implementation, the visual sensor includes a camera.

A third aspect of the embodiments of the present application provides a control instruction generation device for a visual sensor, the visual sensor collects image data by scanning, and the device includes: an image data acquisition module for acquiring image data; a module, which is used to determine the object that will collide with the vehicle according to the image data; a control instruction generation module, which is used to generate an image acquisition range adjustment instruction according to the object; a control instruction sending module, which is used to send the visual The sensor sends the image acquisition range adjustment instruction, and the image acquisition range adjustment instruction is used to instruct the visual sensor that originally acquired the image according to the preset range to acquire the image according to the temporary range, the temporary range is smaller than the preset range, and Contains the area in which the object is located.

In a possible implementation manner, the temporary area is a rectangle.

In a possible implementation, the visual sensor includes a camera.

A fourth aspect of the embodiments of the present application provides a control device for a visual sensor, the visual sensor collects image data by scanning, and the device includes: a control instruction receiving module, which is configured to acquire an image acquisition range adjustment instruction a control module, which is used to control the visual sensor to adjust the image acquisition range according to the image acquisition range adjustment instruction, and the image acquisition range adjustment instruction is used to instruct the visual sensor that originally acquired the image according to the preset range to acquire the image according to the temporary range , the temporary range is smaller than the preset range and includes an area where an object identified as about to collide with the vehicle is located.

In a possible implementation manner, the temporary area is a rectangle.

In a possible implementation manner, the method further includes adjusting the temporary range of the current image acquisition cycle, and when the acquired image range includes the temporary range, performing the process on the union of the acquired image range and the temporary range. Image Acquisition.

In a possible implementation, the visual sensor includes a camera.

A fifth aspect of the embodiments of the present application provides a driving risk prediction method, including:

acquiring image data, the image data is obtained by scanning with a vision sensor;

determining an object that will collide with the vehicle according to the image data;

generating an image acquisition range adjustment instruction according to the object;

sending the image acquisition range adjustment instruction to the vision sensor,

Acquire an image acquisition range adjustment instruction, and control the visual sensor to adjust the image acquisition range according to the image acquisition range adjustment instruction, where the image acquisition range adjustment instruction is used to instruct the visual sensor that originally acquired the image according to the preset range to acquire the image according to the temporary range , the temporary range is smaller than the preset range and includes the area where the object is located.

In a possible implementation manner, the temporary range is a rectangle.

In a possible implementation manner, when receiving the image acquisition range adjustment instruction, the temporary range of the current image acquisition cycle is adjusted, and when the acquired image range includes the temporary range, the acquired image The union of the range and the temporary range is imaged.

In a possible implementation, the visual sensor includes a camera.

A sixth aspect of the present application provides a computer-readable storage medium on which program instructions are stored, and when executed by a computer, the program instructions cause the computer to execute the above-mentioned first, second and fifth aspects and possible implementations thereof. any of the methods provided.

A seventh aspect of the present application provides a computer program, by running the program, a computer can execute any one of the methods provided in the first, second, and fifth aspects and their possible implementations, or as the second aspect and its possible implementations. Any of the means provided by the implementation of the .

These and other aspects of the present application will be more clearly understood in the following description of the embodiment(s).

Description of drawings

The various features of the present application and the connections between the various features are further explained below with reference to the accompanying drawings. The drawings are exemplary, some features are not shown to scale, and some of the drawings may omit features that are customary in the field to which the application relates and not essential to the application, or additionally show The non-essential features of the present application, and the combination of individual features shown in the drawings are not intended to limit the present application. In addition, the same reference numerals refer to the same contents throughout the present specification. The specific drawings are described as follows:

1a is a schematic structural diagram of a driving risk prediction system provided by an embodiment of the present application;

FIG. 1b is a schematic structural diagram of a control device for a visual sensor provided by an embodiment of the present application;

2a is a flowchart of a visual sensor control method provided by an embodiment of the present application;

Fig. 2b is a sub-flow chart of the visual sensor control method provided by the embodiment of the present application;

3a-3f are exemplary images of a traffic scene captured by a visual sensor provided in an embodiment of the present application;

4a-4d are schematic diagrams of the vision sensor scanning a scene according to different image acquisition range adjustment instructions provided by an embodiment of the present application;

5a-5d are images of the scene captured by the vision sensor provided by the embodiment of the present application at times t ₁ -t ₄ .

Description of reference numerals

Vision sensor 10; vision sensor control device 11; ADAS calculation and control device 20; perception module 21; home control module 22; hazardous area identification module 23; control instruction generation module 24; ISP30; ADAS platform 100; vision sensor scanning control module 110 ; comparison module 111 ; driving risk prediction system 200 ; ECU 210 ; image 300 ; intersection 301 ; road 302 ; traffic sign 303 ;

Detailed ways

The technical solutions involved in the specific embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments. Before describing the specific content of the technical solution, the terms used in this application are briefly explained.

The words "first, second, third, etc." in the description and claims, or similar terms such as module A, module B, module C, etc., are only used to distinguish similar objects, and do not represent a specific ordering of objects, which can be understood Indeed, where permitted, the specific order or sequence may be interchanged to enable the embodiments of the application described herein to be practiced in sequences other than those illustrated or described herein.

The term "comprising" used in the description and claims should not be interpreted as being limited to what is listed thereafter; it does not exclude other elements or steps. Accordingly, it should be interpreted as specifying the presence of said features, integers, steps or components mentioned, but not excluding the presence or addition of one or more other features, integers, steps or components and groups thereof. Therefore, the expression "apparatus comprising means A and B" should not be limited to apparatuses consisting of parts A and B only.

Reference in this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the terms "in one embodiment" or "in an embodiment" in various places in this specification are not necessarily all referring to the same embodiment, but can refer to the same embodiment. Furthermore, the particular features, structures or characteristics can be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more embodiments.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. If there is any inconsistency, the meaning described in this specification or the meaning derived from the content described in this specification shall prevail. In addition, the terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

Advanced Driving Assistance System (ADAS) includes multiple sensors and data processing platforms. Its working principle is to collect the data of the moving body and its surrounding environment through a variety of sensors installed on the moving body. After the data is processed and analyzed by the data processing platform, the traveling path of the moving body is planned, and control commands are sent to The control module performs the relevant operations.

Image Signal Processor (ISP) is a device used to process the image output by the front-end image sensor.

Electronic control unit (Electronic Control Unit, ECU for short) is used to calculate various input data and process various input instructions according to a pre-designed program, and further control each actuator to perform various predetermined control functions.

First, the deficiencies in the prior art discovered by the inventors are explained:

In current ADAS systems, the vision sensor scans the image multiple times, frame by frame. After each scan, the vision sensor is able to obtain information about a line of images from the real world and feed it to the ISP. After the ISP receives a complete frame of images composed of multiple lines of images, image processing is performed on the frame of images. The processed images are transmitted to the data processing module in the ADAS data processing platform for algorithm processing, and finally generate control instructions.

When the frame rate of the vision sensor is 30fps (30 frames of image exposure per second), the exposure interval of two adjacent frames of images is about 33ms. As a result, within a time interval of 33ms, the ADAS data processing platform cannot obtain image information in the real world. If the data processing platform recognizes a danger signal and needs to immediately obtain the image of the specified area in the current lens screen, it needs to wait for a maximum of 33ms. In the case of emergency risk avoidance, emergency decision-making and other scenarios, there is a problem of large delay.

In view of such problems in the prior art, the embodiments of the present application provide a method and device for generating a control instruction for a visual sensor, a control method and device, and a driving risk prediction method and system. It can obtain the image of the specified position in the current lens screen in the emergency avoidance and emergency decision-making scenarios, and suppress the traffic accident caused by the delayed acquisition of the image.

An outline of one embodiment of the present application is described in the following embodiments.

The first embodiment: a driving risk prediction system.

FIG. 1 a shows the module structure of a driving risk prediction system 200 with a vision sensor control device 11 and an ADAS calculation and control device 20 .

The driving risk prediction system 200 of the present embodiment is located on the vehicle. The vehicle is, for example, a car. The driving risk prediction system 200 may include a plurality of sensors, a plurality of sensor control devices, an ADAS platform 100 and a vehicle ECU 210 . The visual sensor control device 11 is connected to the visual sensor 10 for controlling the acquisition of image data by the visual sensor 10 .

The ADAS platform 100 can be connected with the vision sensor 10 and the vision sensor control device 11 through lines, for sending instructions to the vision sensor control device 11 and receiving and processing data collected by the vision sensor and other sensors. The vehicle ECU 210 can receive control instructions from the ADAS platform 100, and further control the vehicle to perform corresponding driving operations.

In some embodiments of the present application, the sensor may be a visual sensor 10, such as a camera, or a sensor that can acquire data by scanning, such as a lidar sensor, a millimeter-wave sensor, or the like. Correspondingly, the sensor control device may be the visual sensor control device 11 . The ADAS platform 100 may include a driving hazard prediction device, an ISP 30 configured as the ADAS calculation and control device 20 . The ADAS calculation and control device 20 may include: a perception module 21 , a control module 22 , a dangerous area identification module 23 , and a control instruction generation module 24 . The vehicle ECU 210 can be connected to the ADAS computing and control device 20 in a wired or wireless manner to further control the vehicle to perform corresponding operations according to the instructions generated by the ADAS computing and control device 20 .

The vision sensor 10 is used to acquire image data of a traffic scene, and it can be installed in different positions of the vehicle. The visual sensor 10 has a pixel array composed of a plurality of unit pixels arranged two-dimensionally. When acquiring the image data of the scene, the vision sensor can scan the preset range line by line in chronological order, and scan the image of one line area, that is, it can acquire the image data of one line area, and output the data from the line image through the line to ISP30. When the last line of images is scanned, the vision sensor adds an end mark to the scanned image data of the last line, indicating that one cycle of image acquisition is completed. The ISP 30 starts to process the received image data of the preset range according to the end marker. The vision sensor 10 may also scan the temporary range instructed to perform image capture according to the image capture range adjustment instruction. After the vision sensor 10 has finished scanning the last line of images in the temporary range, the vision sensor 10 adds an end mark to the last line of image data, indicating that the image acquisition for the current cycle is completed. The ISP 30 starts processing the received temporary image according to the end marker.

The ISP 30 is a processor that performs image processing on image data output from the vision sensor 10 . The ISP 30 receives the line image data output from the vision sensor 10 in chronological order, and after receiving the line image data with the end mark, performs, for example, gamma correction, color interpolation, and Processing of automatic white balance, etc. The ISP 30 may be a chip integrated in the ADAS platform 100 or a chip integrated with the visual sensor 10 . In this embodiment, the ISP 30 is integrated in the ADAS platform 100, and is provided with an image interface to receive image data sent by the visual sensor 10 through a data line.

The ADAS calculation and control device 20 is used for processing image data obtained from a plurality of sensors, and generating control instructions to control the vehicle ECU to perform corresponding operations. The ADAS calculation and control device 20 has a perception module 21 , a control module 22 , a dangerous area identification module 23 and a control instruction generation module 24 .

The perception module 21 is a device capable of performing algorithmic processing on the image data from the ISP 30 . It is used to perform image detection on the acquired image, so as to identify the object in the image and obtain the information of the object. The objects may be traffic participants in the surroundings of the vehicle. The traffic participants may include pedestrians, surrounding vehicles, traffic signs, obstacles, and the like. The information of the object may include: the position of the object in the world coordinate system, and the size of the object. The perception module can perform image detection using neural network models, object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other computer vision techniques.

The perception module 21 determines the location and size of the traffic participant according to the identified pixel coordinates of the traffic participant and the calibration parameters of the visual sensor 10 . The calibration parameters may be internal parameters, external parameters, and position information of the vision sensor lens. The perception module 21 obtains any one of the traffic participants according to the pixel coordinates of the same traffic participant in the first image and the second image obtained by the visual sensors located at different positions of the vehicle at the same time, the internal reference and the external reference corresponding to the visual sensor. The position information corresponding to the pixel coordinate point in the world coordinate system, and then determine the position of the traffic participant; according to the image area composed of multiple pixel coordinates and the zoom factor of the visual sensor, determine the traffic participant in the world coordinate system. size. During the operation of the vehicle, the perception module 21 can obtain the positioning information of the vehicle in real time through the inertial navigation device/lidar, or can obtain the positioning information of the vehicle in real time by using satellite positioning technology (eg, GPS technology), and can also use other existing Any positioning technology acquires the positioning information of the vehicle in real time, which is not limited in this embodiment of the present application. The positioning information of the vehicle may include longitude and latitude, altitude, and attitude information of the vehicle (such as the heading of the vehicle). The latitude, longitude and altitude in the positioning information of the vehicle are data in a world coordinate system (also referred to as a geographic coordinate system). The distance of the traffic participant relative to the vehicle is determined according to the positioning information of the vehicle and the location of the traffic participant. The perception module 21 may also receive the current planning of the vehicle's travel trajectory by the control module 22 . The sensing module 21 includes: an input interface, an output interface, a program memory, a working memory and a microcontroller. The input interface is used to receive image data output from the ISP; the output interface is used to output the information of traffic participants to the control module 22 and the danger area identification module 23; the microcontroller can read commands from the program memory and execute each process in sequence. The microcontroller temporarily expands the program stored in the program memory in advance into the working memory, and performs various actions according to the command group. Algorithms used to obtain information about traffic participants in scene data are implemented through a combination of microcontrollers and software. The software may be a module constituting a computer program for executing specific processing corresponding to each functional block. Such computer programs may be stored in program memory.

The dangerous area identification module 23 is an algorithm processing device different from the perception module 21 , and can determine whether the traffic participant is about to collide with the vehicle according to the information of the traffic participant obtained from the perception module 21 . The conditions for determining whether the traffic participant is about to collide with the vehicle may include: at the current moment, the position of the same traffic participant intersects the current driving trajectory of the vehicle, the distance between the same traffic participant and the vehicle is less than the first distance, and The size of the same traffic participant exceeds the preset value. When the traffic participant meets the above judgment criteria, it indicates that the traffic participant is about to collide with the vehicle, and the image of the traffic participant needs to be obtained immediately to further plan the vehicle's driving trajectory, and cannot wait for the visual sensor to scan the current traffic in chronological order. The preset range of the scene is then scanned again according to the preset range.

The control instruction generation module 24 is used to generate an image acquisition range adjustment instruction when the traffic participant is judged to be about to collide with the vehicle, and controls the vision sensor that originally acquired image data according to the preset range to perform image acquisition according to the temporary range. Areas smaller than the preset range and containing traffic participants identified as about to collide with the vehicle. The image acquisition range adjustment instruction is used to instruct the vision sensor to perform image acquisition on the area where the traffic participant identified as about to collide with the vehicle is located. According to the different scanning methods of the visual sensor, the temporary range can be the row area or column area where the traffic participants are located for image acquisition, and the temporary range can also be the rectangular, triangular and circular areas where the traffic participants are located. For example, when the traffic participant is a pedestrian, the temporary extent is a rectangular area containing the area where the pedestrian is located. The control instruction sending module is configured to send the image acquisition range adjustment instruction to the visual sensor 10 . The control command sending module can be a data line, one end of which is connected to the control command generating module, and the other end is connected to the vision sensor control device through the ISP30. The image acquisition range adjustment instruction can also be sent to the vision sensor control device 11 in the form of signal transmission.

The home control module 22 is a computing device different from the perception module 21 and the dangerous area identification module 23 , and is used to plan the driving path of the vehicle and generate vehicle driving control instructions according to the information of the traffic participants acquired by the perception module 21 .

The vehicle ECU 210 includes a microprocessor (CPU), a memory (ROM, RAM), an input/output interface (I/O), an analog-to-digital converter (A/D), and an integrated circuit. The vehicle ECU is connected to the home control module 22 through the input interface through the connecting line, and is used for receiving the vehicle travel control command generated by the home control module, and further controls each actuator to perform the corresponding travel operation according to the vehicle travel control command.

The vision sensor control device 11 can control and control the vision sensor 10 to scan images according to the temporary range based on the image acquisition range adjustment instruction output from the control instruction generation module 24 . The temporary range is obtained by reducing the preset range, and the temporary range includes the traffic participants identified by the danger area identification module 23 as being about to collide with the vehicle.

FIG. 1 b shows a schematic block diagram of the vision sensor control device 11 . The visual sensor control device 11 may include a visual sensor scanning control module 110 and a comparison module 111 .

The vision sensor scanning control module 110 may be a control circuit that controls the vision sensor 10, and is integrated on the vision sensor 10 to control the scanning range of the vision sensor according to the image acquisition range adjustment instruction. For example, when the vision sensor scanning control module 110 receives the image acquisition range adjustment instruction for data acquisition for the Nth row to the Mth row, it controls the vision sensor 10 to complete the row area currently scanned, and starts to scan from the Nth row, Until the M-th line is scanned.

The comparison module 111 may adjust the temporary range of the current image capture cycle according to the image capture range adjustment instruction of the control command generation module 24 . The comparison module 111 respectively compares the temporary range with the uncollected range in which image capture has not been performed and the captured range in which image capture has been carried out. When the acquired range includes the temporary range, the comparison module 111 generates a final adjustment instruction for performing image acquisition on the union of the acquired range and the temporary range.

When the acquired range does not include the temporary range, that is, the temporary range is within the uncollected range, the comparison module 111 generates a final adjustment instruction for image acquisition of the temporary range, and sends the final adjustment instruction to the vision sensor scanning control module 110 . The vision sensor completes the scanning of the current line area, adds an end mark to the image data corresponding to the current line area, and then starts image acquisition for the temporary range. For example, when the visual sensor scans the A-th row, the comparison module 111 receives an image acquisition range adjustment instruction for image acquisition for the N-th row to the M-th row (M>N>A). The vision sensor scan control module 110 controls the vision sensor 10 to stop the current scan, and starts to scan from the Nth row until the Mth row is scanned.

When the acquired range all includes the temporary range, there is no need to perform image acquisition on the temporary range again, just send the image data of the acquired range to the ISP. The comparison module 111 generates a final adjustment instruction for the image acquisition at the end of the current acquired range, and sends the final adjustment instruction to the vision sensor scanning control module 110 . The vision sensor scanning control module 110 controls the vision sensor to perform image acquisition again according to the preset range. For example, when the visual sensor captures the image of the A-th row, when the comparison module 111 receives the image capture range adjustment instruction for the image capture of the N-th row to the M-th row (A>M>N), the captured image The range already includes the temporary range, just send the currently captured image range to the ISP for processing. Therefore, the visual sensor scanning control module 110 controls the visual sensor 10 to complete the scanning of the A-th row, and adds an end mark to the A-th row to end the image acquisition of the currently acquired range.

When the acquired range includes part of the temporary range, it is not necessary to perform image acquisition on the temporary range again, and it is only necessary to continue image acquisition on the basis of the currently acquired image range. The comparison module 111 generates a final adjustment instruction for continuing image acquisition, and sends the final adjustment instruction to the vision sensor scanning control module 110 . The visual sensor scanning control module 110 receives the image acquisition range adjustment instruction, and controls the visual sensor 10 to continue image acquisition. For example, when the visual sensor has scanned from the 1st row to the Pth row, the comparison module 111 receives the image acquisition range adjustment instruction (N<P<M) for image acquisition for the Nth row to the Mth row (N<P<M). The range already partially contains the temporary range, just let the vision sensor continue to scan until the temporary range is scanned. Therefore, the vision sensor scanning control module 110 controls the vision sensor 10 to continue to perform the current scanning sequence until the scanning of the M-th row area is completed.

The operation of each component of the driving risk prediction system will be described below with reference to the accompanying drawings.

The scanning method of the vision sensor.

Figures 3a-3f show exemplary images 300 of a traffic scene captured by a vision sensor with a frame resolution of 1920x1080. The image 300 is a color image presented with black and white lines, in order to comply with the provisions of the Implementing Regulations of the Patent Law. Image 300 includes road 302 with intersection 301 , traffic signs 303 , traffic lights 304 ,

other vehicles

305 , 306 , 307 , 308 , and pedestrians 309 .

When the vision sensor control device 11 does not receive the image acquisition range adjustment instruction from the control instruction generation module 24, the vision sensor 10 scans the scene shown in FIG. 3a according to the preset range. That is, scanning is performed line by line from the L1 th line in the image range shown in FIG. 3 until the scanning reaches the L1080 th line to complete the scanning of one frame of image. In order to clearly illustrate the vehicles and pedestrians in the figure, only the L1th row, the L2th row and the L1080th row are schematically drawn with dotted lines.

When the visual sensor control device 11 receives the image acquisition range adjustment instruction from the control instruction generation module 24, it scans the scene according to the image acquisition range adjustment instruction. The image acquisition range adjustment instructions are different, and the scanning methods of the vision sensor are also different. The following describes the scanning method of the vision sensor according to three different image acquisition range adjustment instructions.

The first scanning mode: the image acquisition range adjustment instruction instructs to scan the rectangular area.

As shown in Fig. 3b, when the vision sensor control device 11 is placed on the line L22 when the vision sensor scans the line L22 and receives the image acquisition range adjustment instruction for scanning the lines L33-L44, the comparison module 111 adjusts the image acquisition range adjustment instruction The indicated line L33-L44 region is aligned with the scanned line L1-L22 region and the unscanned line L23-L1080 region. The scanned image range does not include the scanned area indicated by the image acquisition range adjustment instruction, and the comparison module 111 generates the final adjustment instruction for scanning the L33-L44, and sends it to the vision sensor scanning control module 110 . The vision sensor scanning control module 110 controls the vision sensor to stop scanning the line L22, and start scanning line by line from the L33 line until scanning to the L44 line. When the scanning of the L44th line is completed, the scanning starts from the L1th line again.

As shown in Fig. 3c, when the vision sensor control device receives the image acquisition range adjustment instruction for scanning the lines L14-L44 when the vision sensor scans the line L100, the comparison module 111 compares the scanned lines L1-L100. The line area is compared with the L14-L44 line area indicated by the image acquisition range adjustment instruction. The scanned image range part all includes the range to be scanned by the image acquisition range adjustment instruction. The vision sensor scanning control module 110 controls the vision sensor 10 to add an end mark to the image data of the L100th line, and then starts scanning from the L1th line.

As shown in Fig. 3d, when the vision sensor scanning control module receives the image acquisition range adjustment instruction for scanning the lines L14-L500 when the vision sensor scans the line L200, the comparison module 111 compares the scanned lines L1-L200. The line area is compared with the area for scanning in the L14-L500 lines indicated by the image acquisition range adjustment instruction. The scanned image range part includes the scanned area indicated by the image acquisition range adjustment instruction, and the comparison module 111 generates a final adjustment instruction for continuing to scan until all the scanned areas indicated by the image acquisition range adjustment instruction are scanned, and sends it to the vision sensor for scanning. Control module 110 . The vision sensor scanning control module 110 controls the vision sensor 10 to continue scanning until the scanning reaches the L500 line. When the scanning of the L500th line is completed, the scanning starts from the L1th line again.

After scanning the area specified by the image acquisition range adjustment instruction, the vision sensor 10 is not limited to start scanning again from the L1th line, but can also continue to scan the area specified by the image acquisition range adjustment instruction line by line, and can also scan from any line as required. Start scanning.

The second scanning mode: the image acquisition range adjustment instruction specifies to scan the oval area where the traffic participants are located.

As shown in FIG. 3e, the image acquisition range adjustment instruction instructs to scan the elliptical area X in the image range of the vision sensor. In the following description, for the same processing in the second scanning method as the first scanning method, the content of the first embodiment will be cited for description, or only a brief description will be given.

When the vision sensor scanning control module 110 does not receive the image acquisition range adjustment instruction from the control instruction generation module 24, the vision sensor 10 scans the scene shown according to the preset range. It is the same as that in the first scanning mode, and will not be repeated here.

As shown in FIG. 3e, when the vision sensor scanning control module 110 receives the scan of the area X when the vision sensor 10 scans the L22th row, it controls the vision sensor to complete the scan of the L22th row, and then starts to scan the area X. Scan until the area X is scanned, and then start scanning again from line L1.

The third scanning method: the image acquisition range adjustment instruction specifies to scan the rectangular area where the traffic participants are located.

As shown in Fig. 3f, the image acquisition range adjustment instruction is to scan the area Y. In the following description, for the same processing in the third scanning method as the first scanning method, the content of the first embodiment will be cited for description, or only a brief description will be given.

When the vision sensor scanning control module does not receive the image acquisition range adjustment instruction from the control instruction generation module, the vision sensor scans the scene shown according to the preset range. It is the same as that in the first scanning mode, and will not be repeated here.

When the vision sensor scanning control module 110 receives the image acquisition range adjustment instruction from the sensor control instruction generation module 24 when the vision sensor 10 scans to the L22 line, it controls the vision sensor to adjust the scanned area. When the image acquisition range adjustment instruction is to scan the area Y, control the visual sensor 10 to scan the area Y after completing the scanning of the L22 row, until the scanning of the area Y is completed, start from the L1 row again. Start scanning.

Alternatively, the image acquisition range adjustment instruction may specify to scan the circular, square and other shaped areas where the traffic participants are located. It is also possible to designate to scan the pixel area constituting the traffic participant, as long as the area can all contain the area where the traffic participant identified as about to collide with the vehicle is located.

The working modes of the perception module 21 , the dangerous area identification module 23 , the control instruction generation module 24 and the control module 22 will be described below with reference to the scene shown in FIG. 3 a .

Refer again to the image of the scene shown in Figure 3a. After the perception module 21 receives the first image and the second image obtained by the visual sensors located at different positions of the vehicle at the same time, it uses the neural network model, object recognition algorithm, structure restoration algorithm in motion, video tracking and other computer vision technologies Detect the image, extract the features in the image, and match with the preset features to determine the traffic signs 303, traffic lights 304, pedestrians 309 and surrounding

vehicles

305, 306, 307, 308 and other traffic participants in the image; The pixel coordinates of each traffic participant and the calibration parameters of the visual sensor 10 are obtained to determine the location of the traffic participant and the size of the traffic participant; according to the obtained vehicle positioning information, the distance of the traffic participant relative to the vehicle is determined.

After the perception module 21 obtains the positions of the plurality of traffic participants, the size of the traffic participants, the distances of the traffic participants relative to the vehicle, and the driving trajectory of the vehicle currently planned by the control module 22, the information of these traffic participants is converted into Sent to the hazardous area identification module 23 . The dangerous area identification module 23 determines whether the traffic participant is about to collide with the vehicle according to the traffic participant information. The criterion for determining whether the traffic participant is about to collide with the vehicle may include: at the current moment, the position of the same traffic participant intersects the current driving trajectory of the vehicle, and the distance of the same traffic participant relative to the vehicle is less than The first distance and the size of the same traffic participant exceeds a preset value. When the above adjustment is satisfied, it indicates that the image of the traffic participant needs to be acquired immediately, and the vision sensor cannot wait for the visual sensor to scan the preset range in time sequence.

The control instruction generation module 24 generates an image acquisition range adjustment instruction for scanning the area where the one or more traffic participants are located according to the traffic participant who is about to collide with the vehicle identified by the danger area identification module 23, and converts the image to the image acquisition range. The acquisition range adjustment instruction is sent to the sensor scanning control module 11 through the ISP30.

The working mode of the dangerous area identification module 23 and the control instruction generation module 24 will be described below with reference to the accompanying drawings in conjunction with different scenarios:

Scenario 1: The vehicle ahead approaches the vehicle

Figures 4a-4d show 4 frames of images captured by the vision sensor at times t ₁ -t ₄ , respectively. The change of the position of the preceding vehicle 401 in the image range can be seen from Figs. 4a-4d. The proportion of the pixel connection area of the preceding vehicle 401 in the image range of the entire vision sensor gradually increases, and the distance between the preceding vehicle and the host vehicle decreases. According to the time t4, the dangerous area identification module 23 _finds that the position of the preceding vehicle 401 intersects the current driving trajectory of the vehicle, the distance of the vehicle ahead relative to the vehicle is less than the first distance, and the size of the vehicle ahead exceeds a preset value. It is determined that the preceding vehicle 401 is a traffic participant about to collide with the own vehicle. The control instruction generation module 24 generates, according to the identification result of the dangerous area identification module 23, an image acquisition range adjustment instruction for image acquisition of the area where the preceding vehicle 401 is located. As shown in Fig. 4d, the area where the front vehicle 401 is located corresponds to the area in the L500th to L900th line in the image range of the vision sensor, and the control instruction generation module generates an image acquisition range adjustment instruction for image acquisition of the L500th to L900th line area , and send the image acquisition range adjustment instruction to the sensor scanning control module through the ISP.

Scenario 2: The pedestrian in front approaches the vehicle

Figures 5a-5d show 4 frames of images captured by the vision sensor at times t ₅ -t ₈ , respectively. As can be seen from Figures 5a-5d, the position of the pedestrian 501 in the image range of the vision sensor varies. As can be seen in Figures 5a-5d, the pedestrian 501 in front moves from the right side of the image range to the center of the image range. The dangerous area identification module 23 determines that the pedestrian in front intersects the current driving trajectory of the vehicle at the position of the pedestrian 501 in front at time _t8 , the distance between the pedestrian 501 in front and the vehicle is less than the first distance, and the size of the pedestrian in front exceeds a preset value, and determines the pedestrian in front. 501 is the traffic participant who is about to collide with the vehicle. The control instruction generation module 24 generates an image acquisition range adjustment instruction for imaging the area where the pedestrian 501 in front is located. As shown in FIG. 5d , the area where the pedestrian 501 is located corresponds to the L460th to L980th lines in the image range of the vision sensor. Therefore, the control instruction generation module 24 generates an image acquisition range for the image acquisition of the area of the L460th to L980th lines. An adjustment instruction is sent, and the image acquisition range adjustment instruction is sent to the sensor scanning control module through the ISP.

Referring to FIG. 3a again, after the perception module 21 obtains the traffic participant information about the traffic signs 303 , traffic lights 304 , pedestrians 309 and surrounding

vehicles

305 , 306 , 307 , 308 and other traffic participants in the multi-frame scene images, The traffic participant information of these traffic participants is also sent to the control module 24 . The home control module plans the driving path of the vehicle and generates control instructions according to the traffic participant information obtained by the perception module, and the vehicle ECU further controls the various components of the vehicle to perform corresponding operations according to the control instructions.

Second Embodiment: Driving Risk Prediction Method

Figures 2a-2b show a flow chart of a driving risk prediction method.

The driving hazard prediction method includes the following steps:

Step S1: the vision sensor scans the scene.

The vision sensor scans the preset image range of the vision sensor line by line in chronological order, and after scanning a line area, the image data of the line area is transmitted to the ISP.

Step S2: The ISP processes the received image.

Wherein, the ISP is configured to process the previously received images of all the line areas after receiving the image of the line area with the end mark.

The processing of the image by the ISP may include adjustment of parameters such as image-to-white balance, scanning, and optimization of image noise.

Step S3: using the perception module to perform algorithmic processing on the image data processed by the ISP, to obtain the information of the traffic participants in the image.

For the description of the algorithm processing, reference may be made to the description of the perception module in the first embodiment of the present application.

Step S41: The danger area identification module determines an object that will collide with the vehicle.

The dangerous area identification module determines whether the traffic participant is about to collide with the vehicle according to the acquired traffic participant information. When the following conditions are met, it is determined that the traffic participant may collide with the vehicle, and the traffic participant is the object that will collide with the vehicle: at the current moment, the position of the same traffic participant The current driving trajectory of the vehicle The intersection, the distance of the same traffic participant relative to the own vehicle is less than the first distance, and the size of the same traffic participant exceeds a preset value.

Step S42: when the traffic participant is determined to collide with the vehicle, the control instruction generation module generates an image acquisition range adjustment instruction, and performs image acquisition on a temporary range including the traffic participant.

Wherein, the temporary area may be a rectangular area including the traffic participant.

Step S43: The visual sensor control device receives and executes the image acquisition range adjustment instruction generated in step S42.

Wherein, step S43 may also include the following sub-steps:

Step S431: The comparison module receives the image acquisition range adjustment instruction generated by the control instruction generation module, and compares the temporary range with the scanned image range and the unscanned image range respectively.

The comparison module adjusts the temporary range of the current image acquisition cycle according to the comparison result, and when the acquired image range includes the temporary range, performs image acquisition on the union of the acquired image range and the temporary range.

Step S4321: When the captured image range includes all the temporary ranges, the comparison module generates a final adjustment instruction for ending the current image capture.

Step S4331: The visual sensor scanning control module 110 controls the visual sensor to add an end mark to the current row area after completing the image acquisition of the current row area to complete the image acquisition of the currently acquired image range.

Step S4322: When the captured image range includes part of the temporary range, the comparison module generates a final adjustment instruction for continuing image capture.

Step S4332: The vision sensor scanning control module 110 controls the vision sensor to complete the currently collected line area, and continues to perform image collection on the remaining uncollected temporary areas.

Step S4323 : when the captured image range does not include the temporary range, the comparison module generates a final adjustment instruction for image capturing for the temporary range.

Step S4333: The vision sensor scanning control module 110 controls the vision sensor to complete the currently collected line area, and starts image collection for the temporary area.

For the description of step S43 and its sub-steps performed by the visual sensor control apparatus, reference may be made to the description of the visual sensor control apparatus in the first embodiment of the present application, and for the sake of brevity, only a brief description is given here.

Wherein, the generated final adjustment instruction or image acquisition range adjustment instruction can be transmitted to the vision sensor scanning control module through the ISP. The vision sensor scanning control module controls the vision sensor to scan according to the control instruction.

While executing steps S41-S43, step S5 is also executed: the control module plans the travel path of the vehicle according to the acquired traffic participant information and generates a travel control instruction.

Step S6: The vehicle ECU controls the vehicle to perform corresponding operations according to the driving control instruction.

After step S43 is executed, steps S2, S3, S5 and S6 may be executed in sequence, so as to avoid collision between the vehicle and the traffic participants.

A specific embodiment of the driving risk prediction method will be described below with reference to the scenarios shown in FIGS. 4 a to 4 d .

Step S100: the visual sensor scans the image range shown in Fig. 4a-Fig. 4d in time sequence.

The vision sensor transmits the image data to the ISP after scanning a line of area. When the last line of area is scanned, the vision sensor adds an end mark to the last line of area.

Step S200: After the ISP receives the line area with the end mark. Processes the previously received images of all the row regions, and sends the processed images to the perception module.

Step S300: The perception module performs algorithm processing on the image processed by the ISP, and obtains the information of the vehicle 401 in the image range shown in FIG. 4a-FIG. 4d.

Step S410 : the dangerous area identification module determines whether the vehicle 401 is about to collide with the own vehicle according to the acquired information of the vehicle 401 .

It can be seen from Fig. 4a to Fig. _4d that during the period from t ₁ to t ₄ , the proportion of the vehicle 401 in the image range of the entire vision sensor gradually increases. The current travel trajectories intersect, the distance of the vehicle 401 relative to the own vehicle is less than the first distance, and the size of the vehicle 401 exceeds a preset value. The vehicle 401 is about to collide with the own vehicle, and the location information of the vehicle 401 needs to be obtained immediately.

Step S420: Generate an image acquisition range adjustment instruction for scanning the vehicle 401.

The image acquisition range adjustment instruction is to scan the L508-L806 line area where the vehicle 401 is located.

Step S430: The visual sensor control device receives and executes the image acquisition range adjustment instruction.

Step S110 : the vision sensor scans the L508-L806 row area where the vehicle 401 is located.

Step S210: The ISP processes the received images in the L508-L806 line area, and sends the processed images to the perception module.

Step S310: The perception module performs algorithmic processing on the image of the L508-L806 row area, and obtains information of the vehicle 401 in the image of the L508-L806 row area, the information may include the position of the vehicle 401, the distance of the vehicle 401 relative to the vehicle, and The size of the vehicle 401 .

Step S500: The control module plans the driving path of the vehicle according to the information of the vehicle 401 and generates a control instruction.

Step S600: The vehicle ECU controls the vehicle to perform corresponding operations according to the control instruction.

Third Embodiment: Computer-readable storage medium

Embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processing device to generate a control instruction and a sensor control method and calculation, and the method includes the solutions described in the foregoing embodiments at least one of the.

The computer storage medium of the embodiments of the present application may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples (non-exhaustive list) of computer readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM) ), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or apparatus.

A computer-readable signal medium may include a propagated data signal in baseband or as part of a carrier wave with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium, other than a computer-readable storage medium, that can transmit, propagate, or transmit data for use by or in connection with the instruction execution system, apparatus, or apparatus. program.

Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for performing the operations of the present application may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, but also conventional Procedural programming language - such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or service. Where a remote computer is involved, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, through the Internet using an Internet service provider) connect).

Fourth Embodiment: Computer Program

The fifth embodiment of the present application provides a computer program, and the computer can execute the control method provided by the embodiment of the present application by running the program, or function as the above-mentioned control device.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a service device, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only storage device (Read-Only Memory, ROM), random access storage device (Random Access Memory, RAM), magnetic disk or optical disk and other various programs that can store program codes medium.

Note that the above are only the preferred embodiments of the present application and the applied technical principles. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present application, all of which belong to The scope of protection of this application.

Claims

A control instruction generation method for a vision sensor, characterized in that the vision sensor collects image data by scanning, and the method includes:

get image data;

determining an object about to collide with the vehicle according to the image data;

generating an image acquisition range adjustment instruction according to the object;

sending the image acquisition range adjustment instruction to the vision sensor,

The image acquisition range adjustment instruction is used to instruct the vision sensor that originally acquired the image according to the preset range to acquire the image according to the temporary range, and the temporary range is smaller than the preset range and includes the area where the object is located.
The method of claim 1, wherein the temporary area is a rectangle.
The method of claim 1 or 2, wherein the visual sensor comprises a camera.
A control method for a vision sensor, characterized in that the vision sensor collects image data by scanning, the method comprising:

Obtain the image acquisition range adjustment instruction;

Control the vision sensor to adjust the image acquisition range according to the image acquisition range adjustment instruction,

The image acquisition range adjustment instruction is used to instruct the vision sensor that originally acquired the image according to the preset range to collect the image according to the temporary range, the temporary range is smaller than the preset range, and includes the image that is identified as about to collide with the vehicle. The area where the object is located.
The method of claim 4, wherein the temporary area is a rectangle.
The method according to claim 4, further comprising: adjusting the temporary range of the current image acquisition cycle, and when the acquired image range includes the temporary range, adjusting the acquired image range and the temporary range The union of images is collected.
The control method for a visual sensor according to any one of claims 4-6, wherein the visual sensor comprises a camera.
A control instruction generation device for a vision sensor, characterized in that the vision sensor collects image data by scanning, and the device includes:

an image data acquisition module for acquiring image data;

an identification module for determining an object that will collide with the vehicle according to the image data;

a control instruction generation module, which is used for generating an image acquisition range adjustment instruction according to the object;

a control instruction sending module, which is used for sending the image acquisition range adjustment instruction to the visual sensor,

The image acquisition range adjustment instruction is used to instruct the vision sensor that originally acquired the image according to the preset range to acquire the image according to the temporary range, and the temporary range is smaller than the preset range and includes the area where the object is located.
The apparatus of claim 8, wherein the temporary area is rectangular.
The apparatus of claim 8 or 9, wherein the visual sensor comprises a camera.
A control device for a vision sensor, characterized in that the vision sensor collects image data by scanning, and the device comprises:

a control instruction receiving module, which is used to obtain an image acquisition range adjustment instruction;

a control module, which is used to control the vision sensor to adjust the image acquisition range according to the image acquisition range adjustment instruction,

The image acquisition range adjustment instruction is used to instruct the vision sensor that originally acquired the image according to the preset range to acquire the image according to the temporary range, the temporary range is smaller than the preset range, and includes the objects identified as about to collide with the vehicle. The area where the object is located.
The apparatus of claim 11, wherein the temporary area is rectangular.
The device according to claim 11, further comprising: adjusting the temporary range of the current image acquisition cycle, and when the acquired image range includes the temporary range, adjusting the acquired image range and the temporary range The union of images is collected.
The apparatus of any one of claims 11-13, wherein the vision sensor comprises a camera.