JP2023069374A - Information processing device, information processing method, and program - Google Patents

Abstract

To suggest an information processing device, an information processing method, and a program which can raise accuracy of self-position estimation by appropriately coordinating a camera and a radar.SOLUTION: An information processing device comprises: a visibility range calculation unit; and a radar parameter determination unit. The visibility range calculation unit extracts visibility range information from image data of a camera. The radar parameter determination unit determines an operation parameter for controlling an effective positioning range of a radar on the basis of the visibility range information.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device, an information processing method, and a program.

Self-contained navigation using an IMU (Inertial Measurement Unit) is used complementarily in tunnels where it is difficult for GNSS (Global Navigation Satellite System) radio waves to reach. However, in self-contained navigation using an IMU, there is a concern that position estimation errors by the IMU may accumulate when the GNSS malfunctions for a long period of time. Therefore, additional external sensing means are desired in addition to the assistance provided by the IMU.

JP 2010-091356 A JP 2014-075026 A

Cameras and radar can be considered as strong candidates for auxiliary sensing means. However, cameras and radars each have advantages and disadvantages, and depending on the situation, the sensing effect may be reduced. For example, optical means such as cameras are concerned about performance degradation in extreme environmental conditions such as rain, fog, snow, and strong light. With radar, false images are likely to occur due to multipath, and it is difficult to accurately detect the azimuth of an object. Patent Document 2 proposes a self-position estimation method that combines a camera and a radar. It's not something you raise.

Therefore, the present disclosure proposes an information processing device, an information processing method, and a program capable of appropriately linking a camera and a radar to improve the accuracy of self-position estimation.

According to the present disclosure, information including a visibility calculation unit that extracts visibility information from camera image data, and a radar parameter determination unit that determines operation parameters for controlling the effective positioning range of the radar based on the visibility information A processing device is provided. Further, according to the present disclosure, there are provided an information processing method in which the information processing of the information processing device is executed by a computer, and a program for causing the computer to implement the information processing of the information processing device.

It is a figure explaining the outline|summary of an auxiliary sensing mechanism. 1 is a block diagram showing a configuration example of a vehicle control system; FIG. FIG. 4 is a diagram showing an example of a sensing area by an external recognition sensor; 1 is a schematic diagram of a conventional vehicle control system; FIG. 1 is a schematic diagram of a vehicle control system of the present disclosure that is an improvement over conventional systems; FIG. It is a figure which shows the structural example of an image signal processing part. It is a figure which shows the structural example of a radar signal processing part. 4 is a flow chart showing an example of information processing for controlling radar operation based on visibility information. FIG. 5 is a schematic diagram of a vehicle control system according to a first modified example; FIG. 11 is a schematic diagram of a vehicle control system according to a second modified example;

Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

The description will be given in the following order.
[1. Outline of auxiliary sensing mechanism]
[2. Configuration example of vehicle control system]
[3. Sensing area of external recognition sensor]
[4. Auxiliary Sensing Method]
[4-1. Conventional vehicle control system]
[4-2. Improved vehicle control system]
[4-2-1. Image signal processing unit]
[4-2-2. Radar signal processor]
[4-3. Information processing method]
[4-4. Modification 1]
[4-5. Modification 2]
[5. effect]

[1. Outline of auxiliary sensing mechanism]
FIG. 1 is a diagram for explaining the outline of the auxiliary sensing mechanism.

The vehicle 1 has a vehicle control system 11 that assists driving based on sensing information such as GNSS 91 and IMU 92 . The vehicle control system 11 is an information processing device that processes various types of information. Vehicle 1 has an auxiliary sensing mechanism that combines camera 51 and radar 52 . Auxiliary sensing mechanisms are used to aid position detection by GNSS 91 and IMU 92 .

In recent years, GNSS91 has become an indispensable technology for realizing driving support and automatic driving. The GNSS 91 generally receives radio waves from a plurality of satellites, calculates the distance between each satellite and the vehicle 1, and calculates the three-dimensional position of the vehicle on the ground. However, since the GNSS 91 uses radio waves from satellites in the sky, there is a problem that radio waves are attenuated by shielding objects and signal quality is degraded due to multiple reflections. As a result, performance is severely compromised in certain situations, and in some cases, the vehicle cannot be located. For example, it is known that tunnels and streets surrounded by tall buildings often prevent the GNSS 91 from performing certain operations.

Conventionally, self-contained navigation using an IMU 92 has been widely used to deal with such problems of the GNSS 91 . According to the self-contained navigation, when the position cannot be detected by the GNSS 91, the amount of movement of the vehicle is calculated using the acceleration and rotation amount of the vehicle body obtained by the IMU 92. By carrying out like this, it becomes possible to compensate for the inoperable area of GNSS91, and the problem that GNSS91 cannot be received is mitigated to some extent.

However, in self-contained navigation by the IMU 92, there is a concern that position estimation errors by the IMU 92 may be accumulated when the GNSS 91 malfunctions for a long period of time. For this reason, in order to always maintain the detection accuracy of the vehicle position at a certain level or higher, an additional external sensing means is desired in addition to the assistance provided by the IMU 92 .

A vehicle-mounted camera is a strong candidate for such additional auxiliary sensing means. The image information obtained by the camera 51 has extremely high resolution, making it possible to acquire an extremely large amount of road information. Therefore, it is thought that it is a very effective means as an additional sensing means with respect to self-contained navigation by GNSS91 and IMU92. For example, from the surrounding image information obtained by the camera 51, by estimating the distance to a specific known location and incorporating the results of estimating the position of the vehicle 1 into the position estimation results of the GNSS 91 and the IMU 92, the further It is possible to improve the position accuracy.

Patent Literature 1 discloses a method in which a server identifies a position from a feature amount of an image using a camera image of the surroundings when GNSS 91 cannot be used.

However, it is generally known that the camera 51 suffers from severe deterioration in the quality of the resulting image under certain extreme environments such as rain, fog, snow and strong light. Also, in general, the camera 51 is an excellent means with extremely high spatial resolution performance, but it is difficult to measure distance and velocity with high accuracy even at long distances. Thus, in the specific cases in which the camera 51 is weak in assisted sensing by the camera 51 for self-contained navigation by the GNSS 91 and the IMU 92, further improvements are still desired.

On the other hand, a radar 52 is known as a sensing means having a high resistance to such environmental conditions that the camera 51 is not good at. However, using the radar 52 as an auxiliary sensing means for the camera 51 presents new problems unique to radar. For example, the radar 52 is prone to artifacts due to multipath. Alternatively, the radar 52 is suitable for measuring the distance and speed to a distant object, but it is difficult to accurately detect the azimuth of the object.

Cooperative sensing by the camera 51 and the radar 52 is disclosed in Patent Document 2. In the method of Patent Document 2, when the vehicle is stopped or at a speed below a predetermined speed, the illuminance of the surrounding area is detected, and when the surrounding illuminance is above a predetermined amount, the reliability of the camera is determined to be high, and the operation of the radar is stopped. Also, when the ambient illumination is less than the predetermined level, the reliability of the camera is determined to be low, and the operation of the camera is stopped. This reduces power consumption in sensing. However, Patent Literature 2 does not describe a method of assisting the GNSS 91 by adaptively varying the operating conditions of the radar based on the image information of the camera.

The present invention has been made in view of the above circumstances. The vehicle control system 11 of the present disclosure calculates the effective positioning range of the camera 51 from the image captured by the camera 51 . The effective positioning range means a range in which positioning accuracy can be guaranteed by the camera. The effective positioning range is defined based on visibility, for example.

Visibility refers to the maximum distance at which an object can be clearly seen with the naked eye. As a visibility measurement method, a known method defined by the WMO (World Meteorological Organization) or the like can be used. Visibility information may be presented as distance, or may be presented as level information obtained by scaling the visibility. For example, it is conceivable to set the visibility level to 5 when the field of view of the camera 51 is not obstructed, and to set the visibility level to 1 when there is thick fog. The auxiliary sensing mechanism changes the optimum operating parameters to be given to the radar 52 based on visibility information (visibility information) extracted from the image data of the camera 51 .

Under default (normal) environmental conditions, the visibility value obtained from the image data of the camera 51 is set as a value that enables accurate sensing of the operating parameters of the radar 52 at a long distance. On the other hand, when the environment is "foggy" or "backlit", the reliable distance measurement range obtained from the image data of the camera 51 is greatly reduced. Therefore, the visibility value also varies accordingly. The radar 52 is then given operating parameters that correspond to the reduced visibility value and allow it to operate accurately at close range.

Specifically, the operating parameters of radar 52 are adjusted by changing parameters such as chirp bandwidth, chirp duration, and chirp transmission interval in the transmitted FMCW signal used by radar 52 . In this disclosure, for example, chirp signal frequency bandwidth as an operating parameter is determined based on visibility information. If the visibility value is small, the operating parameters of radar 52 are tuned to optimum parameters at close range. If the visibility value is large, the operating parameters of radar 52 are tuned to optimum parameters at long range.

In the present disclosure, the operating parameters of radar 52 are variably controlled based on visibility information. Therefore, there is a need for one or more radars 52 whose operating parameters can be adaptively controlled. Such a radar 52 may be installed in the vehicle as a retrofit, or the radar 52 already installed in the vehicle 1 may be modified in design and used as the radar 52 for the present disclosure. In the example of FIG. 1 , the radar 52 for use in the present disclosure is retrofitted to the front of the vehicle 1 .

The above describes the operation in the case where there is one camera 51 and one radar 52 , but similar processing is possible even when there are a plurality of radars 52 for one camera 51 . When a plurality of radars 52 are present, optimum operating parameters are set for each radar 52 based on visibility information obtained from image data of the camera 51 .

In addition to such visibility information, the vehicle control system 11 of the present disclosure can determine the direction in which positioning should be performed by the radar 52 from image information obtained from the camera 51 and transmit the direction to the radar 52 . By doing so, the radar 52 can adjust the transmission/reception beam BM so as to be optimal in the direction indicated by the system, and perform sensing.

At this time, the image of the object detected by the radar 52 is fed back to the camera 51 again, compared with the image information acquired by the camera 51, and after eliminating the image that may be a false image of the radar 52, the image is obtained from the camera 51. Distances to feature points can be mapped using distances obtained from radar 52 . This improves the reliability of operation in environments in which the camera 51 is not good.

[2. Configuration example of vehicle control system]
FIG. 2 is a block diagram showing a configuration example of the vehicle control system 11. As shown in FIG.

The vehicle control system 11 is provided in the vehicle 1 and performs processing related to driving support and automatic driving of the vehicle 1 .

The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a position information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a storage unit 28, a driving A support/automatic driving control unit 29 , a DMS (Driver Monitoring System) 30 , an HMI (Human Machine Interface) 31 , and a vehicle control unit 32 are provided.

Vehicle control ECU 21, communication unit 22, map information storage unit 23, position information acquisition unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, storage unit 28, driving support/automatic driving control unit 29, driver monitoring system ( DMS) 30 , human machine interface (HMI) 31 , and vehicle control unit 32 are connected via a communication network 41 so as to be able to communicate with each other.

The communication network 41 is, for example, a CAN (Controller Area Network), a LIN (Local Interconnect Network), a LAN (Local Area Network), a FlexRay (registered trademark), an Ethernet (registered trademark), and other digital two-way communication standards. It is composed of a communication network, a bus, and the like. The communication network 41 may be used properly depending on the type of data to be transmitted. For example, CAN may be applied to data related to vehicle control, and Ethernet may be applied to large-capacity data.

It should be noted that each unit of the vehicle control system 11 performs wireless communication assuming relatively short-range communication, such as near field communication (NFC) or Bluetooth (registered trademark), without going through the communication network 41. may be connected directly using

In addition, hereinafter, when each part of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 shall be omitted. For example, when the vehicle control ECU 21 and the communication unit 22 communicate via the communication network 41, it is simply described that the vehicle control ECU 21 and the communication unit 22 communicate.

The vehicle control ECU 21 is composed of various processors such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The vehicle control ECU 21 controls the functions of the entire vehicle control system 11 or a part thereof.

The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. At this time, the communication unit 22 can perform communication using a plurality of communication methods.

Communication with the outside of the vehicle that can be performed by the communication unit 22 will be described schematically. The communication unit 22 is, for example, 5G (fifth generation mobile communication system), LTE (Long Term Evolution), DSRC (Dedicated Short Range Communications), etc., via a base station or an access point, by a wireless communication method such as on an external network. communicates with a server (hereinafter referred to as an external server) located in the The external network with which the communication unit 22 communicates is, for example, the Internet, a cloud network, or a provider's own network. The communication method that the communication unit 22 performs with the external network is not particularly limited as long as it is a wireless communication method that enables digital two-way communication at a communication speed of a predetermined value or more and a distance of a predetermined value or more.

Further, for example, the communication unit 22 can communicate with a terminal existing in the vicinity of the vehicle using P2P (Peer To Peer) technology. The terminals existing near the own vehicle are, for example, terminals worn by mobile bodies such as pedestrians and bicycles that move at a relatively low speed, terminals installed at fixed locations such as shops, or MTC (Machine Type Communication) terminal. Furthermore, the communication unit 22 can also perform V2X communication. V2X communication includes, for example, vehicle-to-vehicle communication with other vehicles, vehicle-to-infrastructure communication with roadside equipment, etc., and vehicle-to-home communication. communication, and communication between a vehicle and others, such as vehicle-to-pedestrian communication with a terminal or the like possessed by a pedestrian.

The communication unit 22 can receive, for example, a program for updating software that controls the operation of the vehicle control system 11 from the outside (Over The Air). The communication unit 22 can also receive map information, traffic information, information around the vehicle 1, and the like from the outside. Further, for example, the communication unit 22 can transmit information about the vehicle 1, information about the surroundings of the vehicle 1, and the like to the outside. The information about the vehicle 1 that the communication unit 22 transmits to the outside includes, for example, data indicating the state of the vehicle 1, recognition results by the recognition unit 73, and the like. Furthermore, for example, the communication unit 22 performs communication corresponding to a vehicle emergency call system such as e-call.

For example, the communication unit 22 receives electromagnetic waves transmitted by a vehicle information and communication system (VICS (registered trademark)) such as radio wave beacons, optical beacons, and FM multiplex broadcasting.

Communication with the inside of the vehicle that can be performed by the communication unit 22 will be described schematically. The communication unit 22 can communicate with each device in the vehicle using, for example, wireless communication. The communication unit 22 performs wireless communication with devices in the vehicle using a communication method such as wireless LAN, Bluetooth, NFC, and WUSB (Wireless USB) that enables digital two-way communication at a communication speed higher than a predetermined value. can be done. Not limited to this, the communication unit 22 can also communicate with each device in the vehicle using wired communication. For example, the communication unit 22 can communicate with each device in the vehicle by wired communication via a cable connected to a connection terminal (not shown). The communication unit 22 performs digital two-way communication at a communication speed equal to or higher than a predetermined speed by wired communication such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), and MHL (Mobile High-definition Link). can communicate with each device in the vehicle.

Here, the in-vehicle device refers to, for example, a device in the vehicle that is not connected to the communication network 41 . Examples of in-vehicle devices include mobile devices and wearable devices possessed by passengers PA such as drivers, information devices brought into the vehicle and temporarily installed, and the like.

The map information accumulation unit 23 accumulates one or both of an externally acquired map and a map created by the vehicle 1 . For example, the map information accumulation unit 23 accumulates a three-dimensional high-precision map, a global map covering a wide area, and the like, which is lower in accuracy than the high-precision map.

High-precision maps are, for example, dynamic maps, point cloud maps, vector maps, and the like. The dynamic map is, for example, a map consisting of four layers of dynamic information, quasi-dynamic information, quasi-static information, and static information, and is provided to the vehicle 1 from an external server or the like. A point cloud map is a map composed of a point cloud (point cloud data). A vector map is, for example, a map adapted to ADAS (Advanced Driver Assistance System) and AD (Autonomous Driving) by associating traffic information such as lane and traffic signal positions with a point cloud map.

The point cloud map and the vector map, for example, may be provided from an external server or the like, and based on the sensing results of the camera 51, radar 52, LiDAR 53, etc., as a map for matching with a local map described later. It may be created by the vehicle 1 and stored in the map information storage unit 23 . Further, when a high-precision map is provided from an external server or the like, in order to reduce the communication capacity, map data of, for example, several hundred meters square, regarding the planned route that the vehicle 1 will travel from now on, is acquired from the external server or the like. .

The position information acquisition unit 24 receives GNSS signals from GNSS91 (Global Navigation Satellite System) satellites and acquires position information of the vehicle 1 . The acquired position information is supplied to the driving support/automatic driving control unit 29 . Note that the location information acquisition unit 24 is not limited to the method using GNSS signals, and may acquire location information using beacons, for example.

The external recognition sensor 25 includes various sensors used for recognizing external conditions of the vehicle 1 and supplies sensor data from each sensor to each part of the vehicle control system 11 . The type and number of sensors included in the external recognition sensor 25 are arbitrary.

For example, the external recognition sensor 25 includes a camera 51 , a radar 52 , a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53 , and an ultrasonic sensor 54 . The configuration is not limited to this, and the external recognition sensor 25 may be configured to include one or more types of sensors among the camera 51 , radar 52 , LiDAR 53 , and ultrasonic sensor 54 . The numbers of cameras 51 , radars 52 , LiDARs 53 , and ultrasonic sensors 54 are not particularly limited as long as they are realistically installable in the vehicle 1 . Moreover, the type of sensor provided in the external recognition sensor 25 is not limited to this example, and the external recognition sensor 25 may be provided with other types of sensors. An example of the sensing area of each sensor included in the external recognition sensor 25 will be described later.

Note that the imaging method of the camera 51 is not particularly limited. For example, cameras of various types such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, and an infrared camera, which are capable of distance measurement, can be applied to the camera 51 as necessary. The camera 51 is not limited to this, and may simply acquire a photographed image regardless of distance measurement.

Also, for example, the external recognition sensor 25 can include an environment sensor for detecting the environment with respect to the vehicle 1 . The environment sensor is a sensor for detecting the environment such as weather, climate, brightness, etc., and can include various sensors such as raindrop sensors, fog sensors, sunshine sensors, snow sensors, and illuminance sensors.

Furthermore, for example, the external recognition sensor 25 includes a microphone used for detecting sounds around the vehicle 1 and the position of a sound source.

The in-vehicle sensor 26 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each section of the vehicle control system 11 . The types and number of various sensors included in the in-vehicle sensor 26 are not particularly limited as long as they are the types and number that can be realistically installed in the vehicle 1 .

For example, in-vehicle sensors 26 may comprise one or more of cameras, radar, seat sensors, steering wheel sensors, microphones, and biometric sensors. As the camera provided in the in-vehicle sensor 26, for example, cameras of various shooting methods capable of distance measurement, such as a ToF camera, a stereo camera, a monocular camera, and an infrared camera, can be used. The camera included in the in-vehicle sensor 26 is not limited to this, and may simply acquire a photographed image regardless of distance measurement. The biosensors included in the in-vehicle sensor 26 are provided, for example, on a seat, a steering wheel, or the like, and detect various biometric information of a passenger such as a driver.

The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1 and supplies sensor data from each sensor to each section of the vehicle control system 11 . The types and number of various sensors included in the vehicle sensor 27 are not particularly limited as long as the types and number are practically installable in the vehicle 1 .

For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) integrating them. For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of operation of the accelerator pedal, and a brake sensor that detects the amount of operation of the brake pedal. For example, the vehicle sensor 27 includes a rotation sensor that detects the number of rotations of an engine or a motor, an air pressure sensor that detects tire air pressure, a slip rate sensor that detects a tire slip rate, and a wheel speed sensor that detects the rotational speed of a wheel. A sensor is provided. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining battery level and temperature, and an impact sensor that detects external impact. Vehicle sensor 27 includes a sensor that detects driving status DS.

The storage unit 28 includes at least one of a nonvolatile storage medium and a volatile storage medium, and stores data and programs. The storage unit 28 is used, for example, as an EEPROM (Electrically Erasable Programmable Read Only Memory) and a RAM (Random Access Memory). And a magneto-optical storage device can be applied. The storage unit 28 stores various programs and data used by each unit of the vehicle control system 11 . For example, the storage unit 28 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving), and stores information on the vehicle 1 before and after an event such as an accident and information acquired by the in-vehicle sensor 26 .

The driving support/automatic driving control unit 29 controls driving support and automatic driving of the vehicle 1 . For example, the driving support/automatic driving control unit 29 includes an analysis unit 61 , an action planning unit 62 and an operation control unit 63 .

The analysis unit 61 analyzes the vehicle 1 and its surroundings. The analysis unit 61 includes a self-position estimation unit 71 , a sensor fusion unit 72 and a recognition unit 73 .

The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data from the external recognition sensor 25 and the high-precision map accumulated in the map information accumulation unit 23 . For example, the self-position estimation unit 71 generates a local map based on sensor data from the external recognition sensor 25, and estimates the self-position of the vehicle 1 by matching the local map and the high-precision map. The position of the vehicle 1 is based on, for example, the center of the rear wheel versus axle.

The local map is, for example, a three-dimensional high-precision map created using a technique such as SLAM (Simultaneous Localization and Mapping), an occupancy grid map, or the like. The three-dimensional high-precision map is, for example, the point cloud map described above. The occupancy grid map is a map that divides the three-dimensional or two-dimensional space around the vehicle 1 into grids (lattice) of a predetermined size and shows the occupancy state of objects in grid units. The occupancy state of an object is indicated, for example, by the presence or absence of the object and the existence probability. The local map is also used, for example, by the recognizing unit 73 for detection processing and recognition processing of the situation outside the vehicle 1 .

The self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the position information acquired by the position information acquisition unit 24 and sensor data from the vehicle sensor 27 .

The sensor fusion unit 72 combines a plurality of different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52) to perform sensor fusion processing to obtain new information. . Methods for combining different types of sensor data include integration, fusion, federation, and the like.

The recognition unit 73 executes a detection process for detecting the situation outside the vehicle 1 and a recognition process for recognizing the situation outside the vehicle 1 .

For example, the recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, and the like. .

Specifically, for example, the recognition unit 73 performs detection processing, recognition processing, and the like of objects around the vehicle 1 . Object detection processing is, for example, processing for detecting the presence or absence, size, shape, position, movement, and the like of an object. Object recognition processing is, for example, processing for recognizing an attribute such as the type of an object or identifying a specific object. However, detection processing and recognition processing are not always clearly separated, and may overlap.

For example, the recognition unit 73 detects objects around the vehicle 1 by clustering the point cloud based on sensor data obtained by the radar 52 or the LiDAR 53 or the like for each cluster of point groups. As a result, presence/absence, size, shape, and position of objects around the vehicle 1 are detected.

For example, the recognition unit 73 detects the motion of objects around the vehicle 1 by performing tracking that follows the motion of the cluster of points classified by clustering. As a result, the speed and traveling direction (movement vector) of the object around the vehicle 1 are detected.

For example, the recognition unit 73 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. based on the image data supplied from the camera 51 . Further, the recognition unit 73 may recognize types of objects around the vehicle 1 by performing recognition processing such as semantic segmentation.

For example, the recognition unit 73, based on the map accumulated in the map information accumulation unit 23, the estimation result of the self-position by the self-position estimation unit 71, and the recognition result of the object around the vehicle 1 by the recognition unit 73, Recognition processing of traffic rules around the vehicle 1 can be performed. Through this processing, the recognition unit 73 can recognize the position and state of traffic lights, the content of traffic signs and road markings, the content of traffic restrictions, the lanes in which the vehicle can travel, and the like.

For example, the recognition unit 73 can perform recognition processing of the environment around the vehicle 1 . Surrounding environments to be recognized by the recognizing unit 73 include presence/absence of pedestrians and surrounding vehicles, weather, temperature, humidity, brightness, road surface conditions, and the like.

The action plan section 62 creates an action plan for the vehicle 1 . For example, the action planning unit 62 creates an action plan by performing route planning and route following processing.

Note that global path planning is a process of planning a rough route from the start to the goal. This route planning is called trajectory planning, and includes trajectory generation (local path planning) that allows safe and smooth progress in the vicinity of the vehicle 1 in the planned route, taking into consideration the motion characteristics of the vehicle 1. It also includes the processing to be performed.

Route following is a process of planning actions for safely and accurately traveling a route planned by route planning within a planned time. The action planning unit 62 can, for example, calculate the target speed and target angular speed of the vehicle 1 based on the result of this route following processing.

The motion control section 63 controls the motion of the vehicle 1 in order to implement the action plan created by the action planning section 62 .

For example, the operation control unit 63 controls a steering control unit 81, a brake control unit 82, and a drive control unit 83 included in the vehicle control unit 32, which will be described later, so that the vehicle 1 can control the trajectory calculated by the trajectory plan. Acceleration/deceleration control and direction control are performed so as to advance. For example, the operation control unit 63 performs coordinated control aimed at realizing ADAS functions such as collision avoidance or shock mitigation, follow-up driving, vehicle speed maintenance driving, vehicle collision warning, and vehicle lane departure warning. For example, the operation control unit 63 performs cooperative control aimed at automatic driving in which the vehicle autonomously travels without depending on the operation of the driver.

The DMS 30 performs driver authentication processing, driver state recognition processing, and the like based on sensor data from the in-vehicle sensor 26 and input data input to the HMI 31, which will be described later. As the state of the driver to be recognized, for example, physical condition, wakefulness, concentration, fatigue, gaze direction, drunkenness, driving operation, posture, etc. are assumed.

It should be noted that the DMS 30 may perform authentication processing for passengers other than the driver and processing for recognizing the state of the passenger. Further, for example, the DMS 30 may perform recognition processing of the situation inside the vehicle based on the sensor data from the sensor 26 inside the vehicle. Conditions inside the vehicle to be recognized include temperature, humidity, brightness, smell, and the like, for example.

The HMI 31 inputs various data and instructions, and presents various data to passengers including the driver.

Data input by the HMI 31 will be schematically described. The HMI 31 comprises an input device for human input of data. The HMI 31 generates an input signal based on data, instructions, etc. input from an input device, and supplies the input signal to each section of the vehicle control system 11 . The HMI 31 includes operators such as a touch panel, buttons, switches, and levers as input devices. The HMI 31 is not limited to this, and may further include an input device capable of inputting information by a method other than manual operation using voice, gestures, or the like. Furthermore, the HMI 31 may use, as an input device, a remote control device using infrared rays or radio waves, or an external connection device such as a mobile device or wearable device corresponding to the operation of the vehicle control system 11 .

The presentation of data by HMI 31 will be described schematically. The HMI 31 generates visual information, auditory information, and tactile information for the passenger or outside the vehicle. In addition, the HMI 31 performs output control for controlling the output, output content, output timing, output method, and the like of each generated information. The HMI 31 generates and outputs visual information such as an operation screen, a status display of the vehicle 1, a warning display, and information (contents) indicated by an image or light such as a monitor image showing the situation around the vehicle 1. The HMI 31 also generates and outputs information indicated by sounds such as voice guidance, warning sounds, warning messages, etc., as auditory information. Furthermore, the HMI 31 generates and outputs, as tactile information, information given to the passenger's tactile sense by force, vibration, movement, or the like.

As an output device from which the HMI 31 outputs visual information, for example, a display device that presents visual information by displaying an image by itself or a projector device that presents visual information by projecting an image can be applied. . In addition to the display device having a normal display, the display device is, for example, a head-up display, a transmissive display, a wearable device with an AR (Augmented Reality) function, etc., which displays visual information within the field of view of the passenger. It may be a device. The HMI 31 can also use a display device provided in the vehicle 1 such as a navigation device, an instrument panel, a CMS (Camera Monitoring System), an electronic mirror, a lamp, etc., as an output device for outputting visual information.

As an output device from which the HMI 31 outputs auditory information, for example, audio speakers, headphones, and earphones can be applied.

As an output device for the HMI 31 to output tactile information, for example, a haptic element using haptic technology can be applied. A haptic element is provided at a portion of the vehicle 1 that is in contact with a passenger, such as a steering wheel or a seat.

The vehicle control section 32 controls each section of the vehicle 1 . The vehicle control section 32 includes a steering control section 81 , a brake control section 82 , a drive control section 83 , a body system control section 84 , a light control section 85 and a horn control section 86 .

The steering control unit 81 detects and controls the state of the steering system of the vehicle 1 . The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, and the like. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, and the like.

The brake control unit 82 detects and controls the state of the brake system of the vehicle 1 . The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative brake mechanism, and the like. The brake control unit 82 includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, and the like.

The drive control unit 83 detects and controls the state of the drive system of the vehicle 1 . The drive system includes, for example, an accelerator pedal, a driving force generator for generating driving force such as an internal combustion engine or a driving motor, and a driving force transmission mechanism for transmitting the driving force to the wheels. The drive control unit 83 includes, for example, a drive ECU that controls the drive system, an actuator that drives the drive system, and the like.

The body system control unit 84 detects and controls the state of the body system of the vehicle 1 . The body system includes, for example, a keyless entry system, smart key system, power window device, power seat, air conditioner, air bag, seat belt, shift lever, and the like. The body system control unit 84 includes, for example, a body system ECU that controls the body system, an actuator that drives the body system, and the like.

The light control unit 85 detects and controls the states of various lights of the vehicle 1 . Lights to be controlled include, for example, headlights, backlights, fog lights, turn signals, brake lights, projections, bumper displays, and the like. The light control unit 85 includes a light ECU that controls the light, an actuator that drives the light, and the like.

The horn control unit 86 detects and controls the state of the car horn of the vehicle 1 . The horn control unit 86 includes, for example, a horn ECU for controlling the car horn, an actuator for driving the car horn, and the like.

[3. Sensing area of external recognition sensor]
FIG. 3 is a diagram showing an example of sensing areas by the camera 51, the radar 52, the LiDAR 53, the ultrasonic sensor 54, etc. of the external recognition sensor 25. As shown in FIG. Note that FIG. 3 schematically shows a state in which the vehicle 1 is viewed from above. 3, the left side is the front end (front) of the vehicle 1, and the right end is the rear end (rear) of the vehicle 1, when viewing FIG.

A sensing region 101F and a sensing region 101B are examples of sensing regions of the ultrasonic sensor 54 . The sensing area 101</b>F covers the periphery of the front end of the vehicle 1 with a plurality of ultrasonic sensors 54 . The sensing area 101B covers the periphery of the rear end of the vehicle 1 with a plurality of ultrasonic sensors 54 .

The sensing results in the sensing area 101F and the sensing area 101B are used for parking assistance of the vehicle 1, for example.

Sensing areas 102F to 102B are examples of sensing areas of the radar 52 for short or medium range. The sensing area 102F covers the front of the vehicle 1 to a position farther than the sensing area 101F. The sensing area 102B covers the rear of the vehicle 1 to a position farther than the sensing area 101B. The sensing area 102L covers the rear periphery of the left side surface of the vehicle 1 . The sensing area 102R covers the rear periphery of the right side surface of the vehicle 1 .

The sensing result in the sensing area 102F is used, for example, to detect a vehicle, a pedestrian, or the like existing in front of the vehicle 1, or the like. The sensing result in the sensing area 102B is used for the rear collision prevention function of the vehicle 1, for example. The sensing results in the sensing area 102L and the sensing area 102R are used, for example, to detect an object in a blind spot on the side of the vehicle 1, or the like.

Sensing areas 103</b>F to 103</b>B are examples of sensing areas by the camera 51 . The sensing area 103F covers the front of the vehicle 1 to a position farther than the sensing area 102F. The sensing area 103B covers the rear of the vehicle 1 to a position farther than the sensing area 102B. The sensing area 103L covers the periphery of the left side surface of the vehicle 1 . The sensing area 103R covers the periphery of the right side surface of the vehicle 1 .

The sensing results in the sensing area 103F can be used, for example, in recognition of traffic lights and traffic signs, lane departure prevention support systems, and automatic headlight control systems. A sensing result in the sensing area 103B can be used for parking assistance and a surround view system, for example. Sensing results in the sensing area 103L and the sensing area 103R can be used, for example, in a surround view system.

A sensing area 104 shows an example of the sensing area of the LiDAR 53 . The sensing area 104 covers the front of the vehicle 1 to a position farther than the sensing area 103F. On the other hand, the sensing area 104 has a narrower lateral range than the sensing area 103F.

Sensing results in the sensing area 104 are used, for example, to detect objects such as surrounding vehicles.

A sensing area 105 is an example of a sensing area of the long-range radar 52 .
The sensing area 105 covers the front of the vehicle 1 to a position farther than the sensing area 104 . On the other hand, the sensing area 105 has a narrower lateral range than the sensing area 104 .

The sensing result in the sensing area 105 is used, for example, for ACC (Adaptive Cruise Control), emergency braking, collision avoidance, and the like.

The sensing regions of the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54 included in the external recognition sensor 25 may have various configurations other than those shown in FIG. Specifically, the ultrasonic sensor 54 may also sense the sides of the vehicle 1 , and the LiDAR 53 may sense the rear of the vehicle 1 . Moreover, the installation position of each sensor is not limited to each example mentioned above. Also, the number of each sensor may be one or plural.

[4. Auxiliary Sensing Method]
[4-1. Conventional vehicle control system]
The auxiliary sensing method of the present disclosure will be described below based on a comparison with the conventional method. FIG. 4 is a schematic diagram of a conventional vehicle control system 19. As shown in FIG.

A conventional vehicle control system 19 processes camera 51 and radar 52 data in parallel. For example, the data processing unit 200 uses the image signal processing unit 201 and the radar signal processing unit 202 to extract feature points of the object from the data of the camera 51 and the radar 52, respectively. The data processing unit 200 collates the feature points extracted from the data of the camera 51 and the radar 52 with the image feature point database 203 and the radar feature point database 204 .

The image feature point database 203 stores feature point data extracted from past image data. In the radar feature point database 204, data of feature points extracted from past measurement data of the radar 52 are accumulated. The data processing unit 200 estimates the position of the camera 51 by matching the feature point group extracted from the image data of the camera 51 with the image feature point database 203 . The data processing unit 200 estimates the position of the radar 52 by collating the feature point group extracted from the measurement data of the radar 52 with the radar feature point database 204 . A known method is adopted as a method of estimating the self-position using the feature point group. For example, Patent Literature 1 describes a method of extracting feature points from image data and estimating their positions.

The data processing unit 200 outputs the camera position estimation information and the radar position estimation information to the data integration unit 210 . The data integration unit 210 integrates position information data of the camera 51 and the radar 52 with self-contained navigation data derived from the GNSS 91 and the IMU 92 . At this time, the data integration unit 210 may refer to the map database 95 as appropriate to further confirm the validity of the position information. The data integration unit 210 integrates signals by, for example, Kalman filter processing or arbitrary signal processing, and outputs position information.

[4-2. Improved vehicle control system]
FIG. 5 is a schematic diagram of the vehicle control system 11 of the present disclosure, which is an improvement over the conventional system.

The vehicle control system 11 of the present disclosure integrates information on the feature point group extracted from the camera 51 and information on the feature point group extracted from the data of the radar 52 . The vehicle control system 11 performs self-position estimation using highly accurate feature point information obtained by integration. The process of generating feature point information is performed by the data processing unit 300 . The data integration unit 310 performs self-location estimation processing based on feature point information. For example, data processing section 300 includes sensor fusion section 72 . Data integrating section 310 includes self-position estimating section 71 .

The data processing section 300 has an image signal processing section 301 , a radar signal processing section 302 and an auxiliary sensor position information generating section 303 . The image signal processing unit 301 extracts feature point information from the image data of the camera 51 . The feature point information includes information on the position of each feature point included in the captured image. The image signal processing unit 301 also extracts visibility information from the image data, and determines operating parameters of the radar 52 based on the visibility information. The image signal processing unit 301 controls the positioning direction (feature point orientation) and the effective positioning range (direction and shape of the beam BM) of the radar 52 by adjusting the operating parameters of the radar 52 . Adjustment of the operating parameters of the radar 52 is performed by the drive control section 83 .

The effective positioning range of the radar 52 is defined by the maximum positioning distance of the radar 52 . The maximum positioning distance means the maximum distance for which positioning accuracy is guaranteed by the radar 52 . The maximum positioning distance of the radar 52 is set to a value slightly larger than the visibility calculated from the image data of the camera 51 . If the visibility is short, the maximum positioning distance of the radar 52 is also set short. If the visibility is long, the maximum positioning distance of the radar 52 is also set long. For example, the maximum positioning distance is set to a value that is 5 meters or more and 10 meters or less than the visibility obtained from the image data.

The radar signal processing unit 302 extracts the distance to each feature point (feature point distance) from the sensor data measured by the radar 52 . The image signal processing unit 301 integrates information regarding the position of each feature point extracted from the image data of the camera 51 and information regarding the distance to each feature point obtained from the radar signal processing unit 302 . Thereby, the image signal processing unit 301 increases the accuracy of the feature point information generated based on the image data.

The auxiliary sensor position information generation unit 303 performs self-position estimation based on the feature point information whose accuracy has been improved by the integration process. For example, the vehicle control system 11 (storage unit 28) stores an image feature point database CD in which data of feature points extracted from past image data are accumulated. The auxiliary sensor position information generator 303 estimates the position of the camera 51 by collating the feature point information after the integration processing with the image feature point database CD. The auxiliary sensor position information generation unit 303 generates auxiliary sensor position information indicating the position of the camera 51 and supplies it to the data integration unit 310 . The data integration unit 310 integrates the auxiliary sensor position information with self-contained navigation data obtained from the GNSS 91 and the IMU 92 to generate final position information of the vehicle 1 .

The method of the present disclosure differs from the conventional method in that the radar processing is a processing structure that accompanies the camera processing. Therefore, in the vehicle control system 11 of the present disclosure, information processed by the radar signal processing section 302 is provided indirectly via the image signal processing section 301 .

[4-2-1. Image signal processing unit]
FIG. 6 is a diagram showing a configuration example of the image signal processing unit 301. As shown in FIG.

The image signal processing unit 301 has a visibility calculation unit 311 , a feature point extraction unit 312 and a feature point information update unit 313 . The input is image data supplied from the camera 51 .

The visibility calculator 311 extracts visibility information from the image data of the camera 51 . Visibility information includes information about the maximum distance (visibility) at which an object can be clearly recognized by the naked eye. If the visibility value is uniform regardless of direction, such as fog, rain, and snow, the visibility information includes average visibility information that is independent of direction. If the visibility value varies depending on the azimuth due to the influence of the late afternoon sun, backlight, or the like, the visibility information includes information about the visibility value (visibility distribution) for each azimuth. Whether or not the visibility distribution is biased is determined based on a preset allowable criterion.

It is assumed that the visibility calculator 311 is configured using a multilayer or deep neural network. The visibility calculation unit 311 extracts visibility information using a multi-layer or deep neural network that has learned the relationship between the shooting scene and visibility. By having the neural network learn image scenes such as rain, fog, and backlight in advance, it is possible to obtain visibility information for the input image with high response.

A large number of images are used for learning, and the visibility value is determined with high accuracy for specific environmental conditions. For example, a visibility of 1 is assigned to a situation in which the fog is so thick that one cannot see directly ahead. On the other hand, visibility=5 is assigned to normally clear weather where the field of view of the camera 51 is not impaired. By scaling the field of view obtained by the camera 51 in this way, the parameter setting of the radar signal processing unit 302 can be changed stepwise.

A feature point extraction unit 312 extracts a plurality of feature points from image data. The feature point extraction unit 312 generates feature point information based on the position of each feature point extracted from the image data, and supplies the feature point information update unit 313 with the feature point information. Extraction of feature points is performed using, for example, algorithms such as SfM (Structure from Motion) and SuperPoint, which analyze image feature quantities, and algorithms developed from these algorithms.

In SfM, self-position estimation is performed through steps such as image acquisition, feature point extraction/association, camera position/orientation estimation, and 3D map generation. For example, Non-Patent Document A below describes an example of SfM.

[Non-Patent Document A] Photogrammetry and Remote Sensing, Vol. 55, No. 3

SuperPoint uses deep learning as means for estimating self-location using a camera. In SuperPoint, deep learning is used to learn in advance the relationship between the absolute orientation of the image sensor and the image captured in that orientation. When an image for orientation estimation is input to the trained model, the absolute position and orientation of the image sensor are estimated. For example, Non-Patent Document B below describes an example of SuperPoint.

[Non-Patent Literature B] Workshop on Deep Learning for Visual SLAM, 2018, "SuperPoint: Self-supervised interest point detection and description"

The feature point extracting unit 312 detects, as the feature point azimuth, the direction to be sensed in detail by the radar 52 based on the visibility information. The feature point azimuth is forward in the traveling direction of the vehicle 1 by default. If the visibility distribution satisfies the acceptance criteria, the feature point extraction unit 312 detects the default orientation as the feature point orientation. If the visibility distribution is biased beyond the permissible standard, the feature point extraction unit 312 extracts the orientation with the shortest visibility from the visibility distribution as the feature point orientation.

The feature point information updating unit 313 determines the reliability (accuracy) of the position of each feature point based on the visibility information. For example, when the position of the feature point obtained from the image data of the camera 51 exists at a position that exceeds the effective positioning range of the camera 51 calculated from the visibility information, the feature point information updating unit 313 determines whether the position is accurate. presumed to be low. In this case, the feature point information updating unit 313 corrects the position of the feature point estimated to be inaccurate based on the distance to the feature point obtained from the radar signal processing unit 302 .

For example, the feature point information updating unit 313 decomposes the position of the feature point estimated to be inaccurate into the orientation of the feature point and the distance to the feature point. The feature point information updating unit 313 replaces the distance to the feature point with the distance to the feature point obtained from the radar signal processing unit 302 without changing the direction of the feature point. As a result, the feature point information updating unit 313 updates the feature point information obtained from the image data of the camera 51 with high accuracy. The auxiliary sensor position information generation unit 303 collates the updated feature point information with the image feature point database CD to generate auxiliary sensor position information indicating the position of the camera 51 .

[4-2-2. Radar signal processor]
FIG. 7 is a diagram showing a configuration example of the radar signal processing unit 302. As shown in FIG.

The radar signal processing unit 302 has a radar parameter determination unit 321 and a feature point distance calculation unit 322 . The radar parameter determination unit 321 determines operation parameters for controlling the positioning direction and effective positioning range of the radar 52 based on the visibility information. The radar 52 performs distance measurements for feature points in the positioning direction (feature point azimuth) defined by the operating parameters and the effective positioning range. The operating parameters include parameters for controlling the range resolution of the radar 52 and the like. The feature point distance calculator 322 analyzes the measurement data of the radar 52 based on the operating parameters.

The operating parameters of the radar 52 are tuned to optimum parameters at close range if the visibility value is small. If the visibility value is large, the operating parameters of radar 52 are tuned for greater range. For example, the maximum positioning distance R _max of FMCW radar is generally given by the following formula (1).

In equation (1), "c" indicates the speed of light. “T _chirp ” indicates the chirp signal time width. "BW" indicates the chirp signal frequency bandwidth. “f _S ” indicates the sampling frequency of the IF signal.

In the FMCW radar, the range resolution ΔR is generally given by Equation (2) below.

Therefore, in order to increase the maximum positioning distance _Rmax , it is necessary to increase the chirp signal time width _Tchirp or the sampling frequency _fS , or decrease the chirp signal frequency bandwidth BW.

Here, if the chirp signal time width T _chirp is too large, there is a problem that the maximum speed that can be detected becomes small. Therefore, the chirp signal time width T _chirp cannot be made very large. Also, the sampling frequency _fS corresponds to the operating frequency of the A/D converter that samples the beat signal. Therefore, if the sampling frequency _fS is increased, there is a concern that the circuit cost will increase. Therefore, in order to increase the maximum positioning distance _Rmax , it is desirable to set the chirp signal frequency bandwidth BW to a small value. However, in this case, there is a demerit that the range resolution ΔR increases at the same time.

Thus, in choosing the operating parameters of radar 52, there is a trade-off between the maximum positioning range _Rmax of radar 52 and the range resolution .DELTA.R. However, in the method of the present disclosure, auxiliary positioning is performed by the radar 52 in an area equal to or greater than the effective limit distance for positioning by the camera 51 . The limit distance is determined with reference to visibility information determined by the image signal processing unit 301 . Therefore, the radar signal processing unit 302 can obtain the best range resolution ΔR by setting the maximum chirp signal frequency bandwidth BW that covers the range exceeding the limit distance.

The feature point distance calculator 322 calculates the distance to each feature point based on the measurement result of the radar 52 and supplies the feature point information updater 313 with the calculated distance. A feature point information updating unit 313 integrates the feature point information extracted from the image data and the distance information to each feature point to update the feature point information. The feature point distance calculator 322 A/D-converts the beat signal input from the radar 52, and then calculates the distance to each feature point. For example, the feature point distance calculator 322 performs FFT on the input signal and estimates the distance from the peak value of the spectrum to the feature point.

As described above, the radar signal processing unit 302 refers to the feature point azimuth given from the image signal processing unit 301 and feeds back distance information to the feature point to the image signal processing unit 301 . Such coordinated operation of the camera 51 and the radar 52 enables measurement with higher precision than in the past. For example, when sufficient measurement accuracy cannot be expected from sensing by the camera 51, supplementary measurement is performed by the radar 52, and the result is fused with information on the feature points of the camera 51, thereby obtaining high-precision feature points of the camera 51. Positioning data is obtained.

[4-3. Information processing method]
FIG. 8 is a flowchart showing an example of information processing for controlling the operation of the radar 52 based on visibility information.

The visibility calculator 311 extracts visibility information from the image data of the camera 51 (step S1). The feature point extraction unit 312 determines whether or not the visibility distribution extracted from the visibility information has a bias exceeding the allowable standard (step S2). For example, an area in which the visibility value is equal to or less than a predetermined value is defined as a low visibility area, and if the magnitude of variation in visibility in the low visibility area falls within a preset range, the visibility is uniform, that is, the visibility distribution. is determined to meet the acceptance criteria. If the visibility variation in the low visibility region does not fall within a preset range, it is determined that the visibility is uneven, that is, the visibility distribution is biased beyond the allowable standard.

If the visibility distribution extracted from the visibility information satisfies the permissible criteria (step S2: Yes), the radar parameter determining unit 321 determines the operating parameters of the radar 52 based on the average visibility calculated from the visibility distribution. (step S3). The average visibility means, for example, an average visibility value in a low visibility region. The auxiliary sensor position information generation unit 303 generates auxiliary sensor position information using feature point information of all feature points included in the image data of the camera 51 .

If the visibility distribution extracted from the visibility information is biased beyond the permissible standard (step S2: No), the feature point extraction unit 312 extracts the orientation with the shortest visibility from the visibility distribution as the feature point orientation (step S4). The radar parameter determination unit 321 sets the detection range of the radar 52 to a range corresponding to the feature point azimuth (step S5), and determines the operation parameter of the radar 52 based on the visibility of the feature point azimuth (step S6). The auxiliary sensor position information generation unit 303 selectively uses the feature point information of one or more feature points existing in the feature point azimuth among the plurality of feature points included in the image data of the camera 51 to generate the auxiliary sensor position information. Generate.

[4-4. Modification 1]
FIG. 9 is a schematic diagram of a vehicle control system 11A according to a first modified example.

The difference between this modified example and the configuration example shown in FIG. 5 is that the auxiliary sensor position information generator 340 and the image feature point database CD are provided in an external server. The data processing section 330 has a communication device 331 that communicates with the auxiliary sensor position information generating section 340 .

The image signal processing unit 301 supplies feature point information to the auxiliary sensor position information generating unit 340 via the communication device 331 . The auxiliary sensor position information generation unit 340 generates auxiliary sensor position information by comparing the feature point information with the image feature point database CD, and supplies the auxiliary sensor position information to the data processing unit 330 via the communication device 331 . In this configuration, heavy-load processing and large-capacity data are transferred to a high-performance external server. Therefore, the configuration of the vehicle 1 is simplified.

[4-5. Modification 2]
FIG. 10 is a schematic diagram of a vehicle control system 11B according to a second modification.

The difference between this modification and the configuration example shown in FIG. 5 is that the operating parameters of the plurality of radars 52 with different sensing regions are independently controlled based on the visibility information. The drive control section 83 has a plurality of radar control sections 97 corresponding to each radar 52 . The radar signal processing unit 302 has a radar selection unit 323 .

The radar selection unit 323 acquires information about operation parameters from the radar parameter determination unit 321 . Also, the radar selection unit 323 acquires information about the feature point orientation from the feature point extraction unit 312 . The radar selection unit 323 selectively gives the acquired operation parameter to the radar 52 having a sensing region corresponding to the feature point orientation. The radar control unit 97 drives the radar 52 using the given operation parameters to detect an object. The auxiliary sensor position information generation unit 303 selectively uses feature point information of one or more feature points existing in the feature point orientation to generate auxiliary sensor position information.

[5. effect]
The vehicle control system 11 has a visibility calculator 311 and a radar parameter determiner 321 . The visibility calculator 311 extracts visibility information from the image data of the camera 51 . The radar parameter determination unit 321 determines operation parameters for controlling the effective positioning range of the radar 52 based on the visibility information. In the information processing method of the present disclosure, the processing of the vehicle control system 11 is executed by a computer. The program of the present disclosure causes the computer to implement the processing of the vehicle control system 11 .

According to this configuration, the effective positioning range of the radar 52 is adjusted based on the effective positioning range of the camera 51 . Therefore, the camera 51 and the radar 52 can be appropriately linked to improve the accuracy of self-position estimation.

The vehicle control system 11 has a feature point extractor 312 , a feature point distance calculator 322 and a feature point information updater 313 . A feature point extraction unit 312 extracts a plurality of feature points from image data. The feature point distance calculator 322 calculates the distance to each feature point based on the measurement result of the radar 52 . The feature point information update unit 313 integrates the feature point information regarding the position of each feature point extracted from the image data and the distance information to each feature point to update the feature point information.

According to this configuration, highly accurate feature point information can be obtained by sensor fusion.

The vehicle control system 11 has an auxiliary sensor position information generator 303 . The auxiliary sensor position information generation unit 303 collates the updated feature point information with the image feature point database CD to generate auxiliary sensor position information indicating the position of the camera 51 .

According to this configuration, highly accurate auxiliary sensor position information can be obtained based on the updated feature point information. Compared to the case where the measurement result of the radar 52 and the measurement result of the camera 51 are individually compared with the feature point database, the number of times of comparison is only one. Therefore, the computational load is reduced.

The vehicle control system 11 has a data integration section 310 . The data integration unit 310 integrates the auxiliary sensor position information with the self-contained navigation data to generate position information.

According to this configuration, highly accurate position information is generated by sensor fusion.

If the visibility distribution extracted from the visibility information is biased beyond the permissible standard, the feature point extraction unit 312 extracts the orientation with the shortest visibility from the visibility distribution as the feature point orientation. The radar parameter determination unit 321 determines operation parameters based on the visibility of the feature point azimuth. The auxiliary sensor position information generation unit 303 selectively uses the feature point information of one or more feature points existing in the feature point azimuth among the plurality of feature points included in the image data to generate the auxiliary sensor position information.

According to this configuration, highly accurate auxiliary sensor position information can be generated using only the measurement result of the specific radar 52 optimized based on the visibility distribution.

The vehicle control system 11 has a radar selector 323 . The radar selection unit 323 selectively provides operation parameters to the radar 52 having a sensing region corresponding to the feature point orientation.

This configuration allows selective adjustment of only the necessary operating parameters of the radar 52 .

If the visibility distribution extracted from the visibility information satisfies the permissible criteria, the radar parameter determination unit 321 determines the operating parameters based on the average visibility calculated from the visibility distribution.

According to this configuration, an appropriate operating parameter can be obtained easily without performing complicated calculations.

The radar parameter determination unit 321 determines the chirp signal frequency bandwidth BW as an operating parameter based on the visibility information.

According to this configuration, the radar 52 performs measurement with an appropriate range resolution ΔR according to the effective positioning range of the radar 52 .

The visibility calculation unit 311 extracts visibility information using a multi-layer or deep neural network that has learned the relationship between the shooting scene and visibility.

According to this configuration, visibility information for the input image can be obtained with high response.

Note that the effects described in this specification are merely examples and are not limited, and other effects may be provided.

[Appendix]
Note that the present technology can also adopt the following configuration.
(1)
a visibility calculation unit that extracts visibility information from the image data of the camera;
a radar parameter determination unit that determines an operation parameter for controlling the effective positioning range of the radar based on the visibility information;
Information processing device having
(2)
a feature point extraction unit that extracts a plurality of feature points from the image data;
a feature point distance calculation unit that calculates the distance to each feature point based on the measurement result of the radar;
a feature point information updating unit that updates the feature point information by integrating feature point information regarding the position of each feature point extracted from the image data and information on the distance to each feature point;
The information processing apparatus according to (1) above.
(3)
an auxiliary sensor position information generation unit that generates auxiliary sensor position information indicating the position of the camera by comparing the updated feature point information with an image feature point database;
The information processing apparatus according to (2) above.
(4)
a data integration unit that integrates the auxiliary sensor position information with self-contained navigation data to generate position information;
The information processing apparatus according to (3) above.
(5)
When the visibility distribution extracted from the visibility information has a deviation exceeding an allowable standard, the feature point extracting unit extracts an orientation with the shortest visibility from the visibility distribution as a feature point orientation,
The radar parameter determination unit determines the operating parameter based on the visibility of the feature point orientation,
The auxiliary sensor position information generation unit selectively generates the auxiliary sensor position information by selectively using feature point information of one or more feature points existing in the feature point orientation among the plurality of feature points included in the image data. to generate
The information processing apparatus according to (4) above.
(6)
a radar selection unit that selectively applies the operation parameter to a radar having a sensing area corresponding to the feature point orientation;
The information processing apparatus according to (5) above.
(7)
When the visibility distribution extracted from the visibility information satisfies an allowable criterion, the radar parameter determination unit determines the operating parameter based on an average visibility calculated from the visibility distribution.
The information processing apparatus according to any one of (1) to (6) above.
(8)
The radar parameter determination unit determines a chirp signal frequency bandwidth as the operating parameter based on the visibility information.
The information processing apparatus according to any one of (1) to (7) above.
(9)
The visibility calculation unit extracts the visibility information using a multilayer or deep neural network that has learned the relationship between the shooting scene and the visibility.
The information processing apparatus according to any one of (1) to (8) above.
(10)
Extract visibility information from camera image data,
Determining operating parameters that control the effective positioning range of the radar based on the visibility information;
A computer-implemented information processing method, comprising:
(11)
Extract visibility information from camera image data,
Determining operating parameters that control the effective positioning range of the radar based on the visibility information;
A program that makes a computer do something.

11 vehicle control system (information processing device)
51 Camera 52 Radar 303, 340 Auxiliary sensor position information generation unit 310 Data integration unit 311 Visibility calculation unit 312 Feature point extraction unit 313 Feature point information update unit 321 Radar parameter determination unit 322 Feature point distance calculation unit 323 Radar selection unit CD Image features point database

Claims (11)

a visibility calculation unit that extracts visibility information from the image data of the camera;
a radar parameter determination unit that determines an operation parameter for controlling the effective positioning range of the radar based on the visibility information;
Information processing device having a feature point extraction unit that extracts a plurality of feature points from the image data;
a feature point distance calculation unit that calculates the distance to each feature point based on the measurement result of the radar;
a feature point information updating unit that updates the feature point information by integrating feature point information regarding the position of each feature point extracted from the image data and information on the distance to each feature point;
The information processing apparatus according to claim 1, comprising: an auxiliary sensor position information generating unit that compares the updated feature point information with an image feature point database to generate auxiliary sensor position information indicating the position of the camera;
The information processing apparatus according to claim 2. a data integration unit that integrates the auxiliary sensor position information with self-contained navigation data to generate position information;
The information processing apparatus according to claim 3. When the visibility distribution extracted from the visibility information has a deviation exceeding an allowable standard, the feature point extracting unit extracts an orientation with the shortest visibility from the visibility distribution as a feature point orientation,
The radar parameter determination unit determines the operating parameter based on the visibility of the feature point orientation,
The auxiliary sensor position information generation unit selectively generates the auxiliary sensor position information by selectively using feature point information of one or more feature points existing in the feature point orientation among the plurality of feature points included in the image data. to generate
The information processing apparatus according to claim 4. a radar selection unit that selectively applies the operation parameter to a radar having a sensing area corresponding to the feature point orientation;
The information processing device according to claim 5 . When the visibility distribution extracted from the visibility information satisfies an allowable criterion, the radar parameter determination unit determines the operating parameter based on an average visibility calculated from the visibility distribution.
The information processing device according to claim 1 . The radar parameter determination unit determines a chirp signal frequency bandwidth as the operating parameter based on the visibility information.
The information processing device according to claim 1 . The visibility calculation unit extracts the visibility information using a multilayer or deep neural network that has learned the relationship between the shooting scene and the visibility.
The information processing device according to claim 1 . Extract visibility information from camera image data,
Determining operating parameters that control the effective positioning range of the radar based on the visibility information;
A computer-implemented information processing method, comprising: Extract visibility information from camera image data,
Determining operating parameters that control the effective positioning range of the radar based on the visibility information;
A program that makes a computer do something.

Images