WO2022075039A1 - Information processing device, information processing system, and information processing method - Google Patents

Abstract

The present invention suppresses transfer delay. An information processing device according to an embodiment of the present invention comprises: a sensor which acquires environment information; and a control unit which adds, to the environment information, correction information for correcting shake that occurs in the environment information acquired by the sensor and that is based on an external impact.

Description

Information processing equipment, information processing system and information processing method

This disclosure relates to an information processing device, an information processing system, and an information processing method.

In recent years, with the autonomy of moving objects such as automobiles and robots and the spread of IoT (Internet of Things), there is a strong demand for higher speed and higher accuracy of image recognition.

Japanese Unexamined Patent Publication No. 2018-75923

In image recognition, recognition processing is generally performed on the image acquired by the image sensor, but the image acquired by the image sensor mounted on the moving object is subject to shaking or vibration of the image sensor itself. Distortion can occur due to this. The distortion generated in this way becomes a factor that lowers the recognition accuracy.

Therefore, the present disclosure proposes an information processing device, an information processing system, and an information processing method capable of suppressing a decrease in recognition accuracy.

In order to solve the above-mentioned problems, the information processing apparatus according to the present disclosure has a sensor for acquiring environmental information and for correcting shaking caused by an external impact in the environmental information acquired by the sensor. It is provided with a control unit for adding the correction information of the above to the environment information.

It is a block diagram which shows the configuration example of a vehicle control system. It is a figure which shows the example of the sensing area. It is a figure which shows an example of the image taken by the front camera of a vehicle at the time of normal running. It is a figure which shows an example of the image taken by the front camera of a vehicle at the time of a sudden stop. It is a block diagram which shows the outline of the recognition system which concerns on embodiment. It is a block diagram which shows the schematic structure example of the image pickup apparatus which concerns on embodiment. It is a schematic diagram for demonstrating the reading operation of a rolling shutter system. It is a figure for demonstrating an example of the distortion of frame data which may occur when an image sensor suddenly points downward during reading by a rolling shutter system. It is a flowchart for demonstrating an example of the reading operation which concerns on 1st method of Embodiment. It is a figure for supplementing the flowchart shown in FIG. It is a flowchart for demonstrating an example of the reading operation which concerns on the 2nd method of Embodiment. It is a figure which illustrates the case where two ROIs which partially overlap in a row direction are set in a pixel array part. It is a figure for demonstrating the reading of the image data (hereinafter referred to as ROI data) from each ROI shown in FIG. It is a figure which shows an example of the reading start timing of each row at the time of reading ROI data from two ROIs which partially overlap in a row direction in the reading operation of a rolling shutter system. It is a flowchart which shows an example of the reading operation which concerns on embodiment. It is a figure for supplementing the flowchart shown in FIG. It is a figure for demonstrating the difference of distortion when the image sensor and EVS are combined. It is a flowchart which shows an example of the operation which concerns on embodiment. It is a figure for demonstrating an example of the distortion correction shown in step S164 of FIG. It is a figure for demonstrating the ROI determination method which concerns on embodiment (low speed, straight-ahead). It is a figure for demonstrating the ROI determination method which concerns on embodiment (low speed, straight-ahead). It is a figure for demonstrating the ROI determination method which concerns on embodiment (high speed, straight-ahead). It is a figure for demonstrating the ROI determination method which concerns on embodiment (high speed, straight-ahead). It is a figure for demonstrating the ROI determination method which concerns on embodiment (turning). It is a figure for demonstrating the ROI determination method which concerns on embodiment (turning). It is a figure for demonstrating the fluctuation which occurred in the frame data. It is a figure for demonstrating super-resolution of frame data which concerns on embodiment (overall). It is a figure for demonstrating the super-resolution of the frame data which concerns on embodiment (ROI). It is a system diagram which shows an example of the vehicle control system which concerns on embodiment. It is a hardware block diagram which shows an example of the computer which realizes the function of the information processing apparatus which concerns on this disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same parts are designated by the same reference numerals, so that overlapping description will be omitted.

In addition, the present disclosure will be described according to the order of items shown below.
1. 1. Configuration example of vehicle control system 2. (1) Embodiment 2.1 Schematic configuration example of recognition system 2.2 Schematic configuration example of image pickup device 2.3 Correction of distortion caused by environmental information 2.3.1 Distortion caused by sudden change in direction 2. 3.2 Distortion due to overlapping ROIs in the row direction 3.3.3 Distortion correction 2.3.4 Operation example 2.4 ROI setting 2.5 Super-resolution image data 2.6 Action ・Effect 3. About vehicle control system 4. Hardware configuration

1. 1. Configuration Example of Vehicle Control System FIG. 1 is a block diagram showing a configuration example of a vehicle control system 11 which is an example of a mobile device control system to which the present technology is applied.

The vehicle control system 11 is provided in the vehicle 1 and performs processing related to driving support and automatic driving of the vehicle 1.

The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a GNSS (Global Navigation Satellite System) receiving unit 24, an external recognition sensor 25, an in-vehicle sensor 26, and a vehicle sensor 27. It includes a recording unit 28, a driving support / automatic driving control unit 29, a driver monitoring system (DMS) 30, a human machine interface (HMI) 31, and a vehicle control unit 32.

Vehicle control ECU 21, communication unit 22, map information storage unit 23, GNSS receiving unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, recording unit 28, driving support / automatic driving control unit 29, DMS30, HMI31, and , The vehicle control unit 32 is connected to each other so as to be able to communicate with each other via the communication network 41. The communication network 41 is in-vehicle compliant with digital bidirectional communication standards such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), and Ethernet (registered trademark). It consists of a communication network and a bus. The communication network 41 may be used properly depending on the type of data to be communicated. For example, CAN is applied for data related to vehicle control, and Ethernet is applied for large-capacity data. In addition, each part of the vehicle control system 11 does not go through the communication network 41, but wireless communication assuming relatively short-distance communication such as short-range wireless communication (NFC (Near Field Communication)) and Bluetooth (registered trademark). In some cases, it is directly connected using.

Hereinafter, when each part of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 shall be omitted. For example, when the vehicle control ECU 21 and the communication unit 22 communicate with each other via the communication network 41, it is described that the processor 21 and the communication unit 22 simply communicate with each other.

The vehicle control ECU 21 is composed of various processors such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit), for example. The vehicle control ECU 21 controls the functions of the entire vehicle control system 11 or a part of the vehicle control system 11.

The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. At this time, the communication unit 22 can perform communication using a plurality of communication methods.

The communication unit 22 will roughly explain the feasible communication with the outside of the vehicle. The communication unit 22 is on an external network via a base station or an access point by a wireless communication method such as 5G (5th generation mobile communication system), LTE (Long Term Evolution), DSRC (Dedicated Short Range Communications), etc. Communicates with a server (hereinafter referred to as an external server) that exists in. The external network with which the communication unit 22 communicates is, for example, the Internet, a cloud network, a network peculiar to a business operator, or the like. The communication method for communicating with the external network by the communication unit 22 is not particularly limited as long as it is a wireless communication method capable of digital bidirectional communication at a communication speed of a predetermined value or higher and a distance of a predetermined distance or more.

Further, for example, the communication unit 22 can communicate with a terminal existing in the vicinity of the own vehicle by using P2P (Peer To Peer) technology. Terminals that exist near the vehicle are, for example, terminals worn by moving objects that move at relatively low speeds such as pedestrians and bicycles, terminals that are fixedly installed in stores, or MTC (Machine Type Communication). ) It is a terminal. Further, the communication unit 22 can also perform V2X communication. V2X communication is, for example, vehicle-to-vehicle (Vehicle to Vehicle) communication with other vehicles, road-to-vehicle (Vehicle to Infrastructure) communication with roadside devices, etc., and vehicle-to-home (Vehicle to Home) communication. , And communication between the vehicle and others, such as vehicle-to-Pedestrian communication with terminals owned by pedestrians.

The communication unit 22 can receive, for example, a program for updating the software that controls the operation of the vehicle control system 11 from the outside (Over The Air). The communication unit 22 can further receive map information, traffic information, information around the vehicle 1, and the like from the outside. Further, for example, the communication unit 22 can transmit information about the vehicle 1, information around the vehicle 1, and the like to the outside. Information about the vehicle 1 transmitted by the communication unit 22 to the outside includes, for example, data indicating the state of the vehicle 1, recognition result by the recognition unit 73, and the like. Further, for example, the communication unit 22 performs communication corresponding to a vehicle emergency call system such as eCall.

The communication unit 22 will roughly explain the feasible communication with the inside of the vehicle. The communication unit 22 can communicate with each device in the vehicle by using, for example, wireless communication. The communication unit 22 performs wireless communication with devices in the vehicle by a communication method such as wireless LAN, Bluetooth, NFC, WUSB (Wireless USB), which enables digital bidirectional communication at a communication speed higher than a predetermined value by wireless communication. Can be done. Not limited to this, the communication unit 22 can also communicate with each device in the vehicle by using wired communication. For example, the communication unit 22 can communicate with each device in the vehicle by wired communication via a cable connected to a connection terminal (not shown). The communication unit 22 is digital bidirectional communication at a communication speed higher than a predetermined speed by wired communication such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), and MHL (Mobile High-definition Link). It is possible to communicate with each device in the car by the communication method capable of.

Here, the device in the vehicle refers to, for example, a device that is not connected to the communication network 41 in the vehicle. As the equipment in the vehicle, for example, mobile equipment and wearable equipment possessed by passengers such as drivers, information equipment brought into the vehicle and temporarily installed, and the like are assumed.

For example, the communication unit 22 receives an electromagnetic wave transmitted by a vehicle information and communication system (VICS (Vehicle Information and Communication System) (registered trademark)) such as a radio wave beacon, an optical beacon, and FM multiplex broadcasting.

The map information storage unit 23 stores one or both of the map acquired from the outside and the map created by the vehicle 1. For example, the map information storage unit 23 stores a three-dimensional high-precision map, a global map that is less accurate than the high-precision map and covers a wide area, and the like.

High-precision maps are, for example, dynamic maps, point cloud maps, vector maps, etc. The dynamic map is, for example, a map composed of four layers of dynamic information, quasi-dynamic information, quasi-static information, and static information, and is provided to the vehicle 1 from an external server or the like. The point cloud map is a map composed of point clouds (point cloud data). Here, the vector map refers to a map conforming to ADAS (Advanced Driver Assistance System) in which traffic information such as lanes and signal positions are associated with a point cloud map.

The point cloud map and the vector map may be provided from, for example, an external server or the like, and the vehicle 1 is used as a map for matching with a local map described later based on the sensing result by the radar 52, LiDAR 53, or the like. It may be created and stored in the map information storage unit 23. Further, when a high-precision map is provided from an external server or the like, in order to reduce the communication capacity, map data of, for example, several hundred meters square, related to the planned route on which the vehicle 1 will travel from now on is acquired from the external server or the like. ..

The GNSS receiving unit 24 receives the GNSS signal from the GNSS satellite and acquires the position information of the vehicle 1. The received GNSS signal is supplied to the driving support / automatic driving control unit 29. The GNSS receiving unit 24 is not limited to the method using the GNSS signal, and may acquire the position information by using, for example, a beacon.

The external recognition sensor 25 includes various sensors used for recognizing the external situation of the vehicle 1, and supplies sensor data from each sensor to each part of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.

For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ringing, Laser Imaging Detection and Ringing) 53, and an ultrasonic sensor 54. Not limited to this, the external recognition sensor 25 may be configured to include one or more of the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54. The number of cameras 51, radar 52, LiDAR 53, and ultrasonic sensors 54 is not particularly limited as long as they can be practically installed in the vehicle 1. Further, the type of sensor included in the external recognition sensor 25 is not limited to this example, and the external recognition sensor 25 may include other types of sensors. An example of the sensing area of each sensor included in the external recognition sensor 25 will be described later.

The shooting method of the camera 51 is not particularly limited as long as it is a shooting method capable of distance measurement. For example, as the camera 51, cameras of various shooting methods such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, and an infrared camera can be applied as needed. Not limited to this, the camera 51 may be simply for acquiring a captured image regardless of the distance measurement.

Further, for example, the external recognition sensor 25 can be provided with an environment sensor for detecting the environment for the vehicle 1. The environment sensor is a sensor for detecting the environment such as weather, weather, and brightness, and may include various sensors such as a raindrop sensor, a fog sensor, a sunshine sensor, a snow sensor, and an illuminance sensor.

Further, for example, the external recognition sensor 25 includes a microphone used for detecting the sound around the vehicle 1 and the position of the sound source.

The in-vehicle sensor 26 includes various sensors for detecting information in the vehicle, and supplies sensor data from each sensor to each part of the vehicle control system 11. The type and number of various sensors included in the in-vehicle sensor 26 are not particularly limited as long as they can be practically installed in the vehicle 1.

For example, the in-vehicle sensor 26 can include one or more of a camera, a radar, a seating sensor, a steering wheel sensor, a microphone, and a biosensor. As the camera included in the in-vehicle sensor 26, for example, a camera of various shooting methods capable of measuring a distance, such as a ToF camera, a stereo camera, a monocular camera, and an infrared camera, can be used. Not limited to this, the camera included in the in-vehicle sensor 26 may be simply for acquiring a captured image regardless of the distance measurement. The biosensor included in the in-vehicle sensor 26 is provided on, for example, a seat, a stelling wheel, or the like, and detects various biometric information of a passenger such as a driver.

The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from each sensor to each part of the vehicle control system 11. The type and number of various sensors included in the vehicle sensor 27 are not particularly limited as long as they can be practically installed in the vehicle 1.

For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU (Inertial Measurement Unit)) that integrates them. For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the operation amount of the accelerator pedal, and a brake sensor that detects the operation amount of the brake pedal. For example, the vehicle sensor 27 includes a rotation sensor that detects the rotation speed of an engine or a motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and a wheel speed that detects wheel rotation speed. Equipped with a sensor. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining amount and temperature of the battery, and an impact sensor that detects an impact from the outside.

The recording unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The recording unit 28 is used as, for example, an EEPROM (Electrically Erasable Programmable Read Only Memory) and a RAM (Random Access Memory), and as a storage medium, a magnetic storage device such as an HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, and the like. And a photomagnetic storage device can be applied. The recording unit 28 records various programs and data used by each unit of the vehicle control system 11. For example, the recording unit 28 is equipped with EDR (Event Data Recorder) and DSSAD (Data Storage System for Automated Driving), and records information on the vehicle 1 before and after an event such as an accident and biometric information acquired by the in-vehicle sensor 26. ..

The driving support / automatic driving control unit 29 controls the driving support and automatic driving of the vehicle 1. For example, the driving support / automatic driving control unit 29 includes an analysis unit 61, an action planning unit 62, and an motion control unit 63.

The analysis unit 61 analyzes the vehicle 1 and the surrounding conditions. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data from the external recognition sensor 25 and the high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on the sensor data from the external recognition sensor 25, and estimates the self-position of the vehicle 1 by matching the local map with the high-precision map. The position of the vehicle 1 is based on, for example, the center of the rear wheel-to-axle.

The local map is, for example, a three-dimensional high-precision map created by using a technology such as SLAM (Simultaneous Localization and Mapping), an occupied grid map (Occupancy Grid Map), or the like. The three-dimensional high-precision map is, for example, the point cloud map described above. The occupied grid map is a map that divides a three-dimensional or two-dimensional space around the vehicle 1 into a grid (grid) of a predetermined size and shows the occupied state of an object in grid units. The occupied state of an object is indicated by, for example, the presence or absence of an object and the probability of existence. The local map is also used, for example, in the detection process and the recognition process of the external situation of the vehicle 1 by the recognition unit 73.

The self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the GNSS signal and the sensor data from the vehicle sensor 27.

The sensor fusion unit 72 performs a sensor fusion process for obtaining new information by combining a plurality of different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). .. Methods for combining different types of sensor data include integration, fusion, and association.

The recognition unit 73 executes a detection process for detecting the external situation of the vehicle 1 and a recognition process for recognizing the external situation of the vehicle 1.

For example, the recognition unit 73 performs detection processing and recognition processing of the external situation of the vehicle 1 based on the information from the external recognition sensor 25, the information from the self-position estimation unit 71, the information from the sensor fusion unit 72, and the like. ..

Specifically, for example, the recognition unit 73 performs detection processing, recognition processing, and the like of objects around the vehicle 1. The object detection process is, for example, a process of detecting the presence / absence, size, shape, position, movement, etc. of an object. The object recognition process is, for example, a process of recognizing an attribute such as an object type or identifying a specific object. However, the detection process and the recognition process are not always clearly separated and may overlap.

For example, the recognition unit 73 detects an object around the vehicle 1 by performing clustering that classifies the point cloud based on the sensor data by the LiDAR 53, the radar 52, or the like into each block of the point cloud. As a result, the presence / absence, size, shape, and position of an object around the vehicle 1 are detected.

For example, the recognition unit 73 detects the movement of an object around the vehicle 1 by performing tracking that follows the movement of a mass of point clouds classified by clustering. As a result, the velocity and the traveling direction (movement vector) of the object around the vehicle 1 are detected.

For example, the recognition unit 73 detects or recognizes a vehicle, a person, a bicycle, an obstacle, a structure, a road, a traffic light, a traffic sign, a road sign, or the like with respect to the image data supplied from the camera 51. Further, the type of the object around the vehicle 1 may be recognized by performing the recognition process such as semantic segmentation.

For example, the recognition unit 73 is based on the map stored in the map information storage unit 23, the self-position estimation result by the self-position estimation unit 71, and the recognition result of the object around the vehicle 1 by the recognition unit 73. It is possible to perform recognition processing of traffic rules around the vehicle 1. By this processing, the recognition unit 73 can recognize the position and state of the signal, the content of the traffic sign and the road marking, the content of the traffic regulation, the lane in which the vehicle can travel, and the like.

For example, the recognition unit 73 can perform recognition processing of the environment around the vehicle 1. As the surrounding environment to be recognized by the recognition unit 73, weather, temperature, humidity, brightness, road surface condition, and the like are assumed.

The action planning unit 62 creates an action plan for the vehicle 1. For example, the action planning unit 62 creates an action plan by performing route planning and route tracking processing.

Note that route planning (Global path planning) is a process of planning a rough route from the start to the goal. This route plan is called a track plan, and in the route planned by the route plan, the track generation (Local) capable of safely and smoothly traveling in the vicinity of the vehicle 1 in consideration of the motion characteristics of the vehicle 1 is taken into consideration. The processing of path planning) is also included. The route plan may be distinguished from the long-term route plan and the activation generation from the short-term route plan or the local route plan. The safety priority route represents a concept similar to activation generation, short-term route planning, or local route planning.

Route tracking is a process of planning an operation for safely and accurately traveling on a route planned by route planning within a planned time. The action planning unit 62 can calculate, for example, the target speed and the target angular velocity of the vehicle 1 based on the result of this route tracking process.

The motion control unit 63 controls the motion of the vehicle 1 in order to realize the action plan created by the action plan unit 62.

For example, the motion control unit 63 controls the steering control unit 81, the brake control unit 82, and the drive control unit 83, which are included in the vehicle control unit 32 described later, and the vehicle 1 controls the track calculated by the track plan. Acceleration / deceleration control and direction control are performed so as to proceed. For example, the motion control unit 63 performs coordinated control for the purpose of realizing ADAS functions such as collision avoidance or impact mitigation, follow-up travel, vehicle speed maintenance travel, collision warning of own vehicle, and lane deviation warning of own vehicle. For example, the motion control unit 63 performs coordinated control for the purpose of automatic driving or the like that autonomously travels without being operated by the driver.

The DMS 30 performs driver authentication processing, driver status recognition processing, and the like based on sensor data from the in-vehicle sensor 26 and input data input to HMI 31 described later. In this case, as the state of the driver to be recognized by the DMS 30, for example, physical condition, arousal degree, concentration degree, fatigue degree, line-of-sight direction, drunkenness, driving operation, posture and the like are assumed.

Note that the DMS 30 may perform authentication processing for passengers other than the driver and recognition processing for the status of the passenger. Further, for example, the DMS 30 may perform the recognition processing of the situation inside the vehicle based on the sensor data from the sensor 26 in the vehicle. As the situation inside the vehicle to be recognized, for example, temperature, humidity, brightness, odor, etc. are assumed.

HMI31 inputs various data and instructions, and presents various data to the driver and the like.

The data input by HMI31 will be outlined. The HMI 31 includes an input device for a person to input data. The HMI 31 generates an input signal based on data, instructions, and the like input by the input device, and supplies the input signal to each part of the vehicle control system 11. The HMI 31 includes an operator such as a touch panel, a button, a switch, and a lever as an input device. Not limited to this, the HMI 31 may further include an input device capable of inputting information by a method other than manual operation by voice, gesture, or the like. Further, the HMI 31 may use, for example, a remote control device using infrared rays or radio waves, or an externally connected device such as a mobile device or a wearable device corresponding to the operation of the vehicle control system 11 as an input device.

The presentation of data by HMI31 will be outlined. The HMI 31 generates visual information, auditory information, and tactile information for the passenger or the outside of the vehicle. Further, the HMI 31 performs output control for controlling the output, output content, output timing, output method, etc. of each of the generated information. As visual information, the HMI 31 generates and outputs, for example, an image such as an operation screen, a status display of the vehicle 1, a warning display, a monitor image showing the situation around the vehicle 1, or information indicated by light. Further, the HMI 31 generates and outputs as auditory information, for example, information indicated by sounds such as voice guidance, warning sounds, and warning messages. Further, the HMI 31 generates and outputs tactile information that is given to the tactile sensation of the occupant by, for example, force, vibration, movement, or the like.

As an output device for which the HMI 31 outputs visual information, for example, a display device that presents visual information by displaying an image by itself or a projector device that presents visual information by projecting an image can be applied. .. In addition to the display device having a normal display, the display device displays visual information in the passenger's field of view, such as a head-up display, a transmissive display, and a wearable device having an AR (Augmented Reality) function. It may be a device. Further, the HMI 31 can also use a display device of a navigation device, an instrument panel, a CMS (Camera Monitoring System), an electronic mirror, a lamp, etc. provided in the vehicle 1 as an output device for outputting visual information.

As an output device for which the HMI 31 outputs auditory information, for example, an audio speaker, headphones, or earphones can be applied.

As an output device for which the HMI 31 outputs tactile information, for example, a haptics element using haptics technology can be applied. The haptic element is provided in a portion of the vehicle 1 in contact with the occupant, such as a steering wheel or a seat.

The vehicle control unit 32 controls each part of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, electric power steering, and the like. The steering control unit 81 includes, for example, a control unit such as an ECU that controls the steering system, an actuator that drives the steering system, and the like.

The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, ABS (Antilock Brake System), a regenerative brake mechanism, and the like. The brake control unit 82 includes, for example, a control unit such as an ECU that controls the brake system.

The drive control unit 83 detects and controls the state of the drive system of the vehicle 1. The drive system includes, for example, a drive force generator for generating a drive force of an accelerator pedal, an internal combustion engine, a drive motor, or the like, a drive force transmission mechanism for transmitting the drive force to the wheels, and the like. The drive control unit 83 includes, for example, a control unit such as an ECU that controls the drive system.

The body system control unit 84 detects and controls the state of the body system of the vehicle 1. The body system includes, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioner, an airbag, a seat belt, a shift lever, and the like. The body system control unit 84 includes, for example, a control unit such as an ECU that controls the body system.

The light control unit 85 detects and controls various light states of the vehicle 1. As the light to be controlled, for example, a headlight, a backlight, a fog light, a turn signal, a brake light, a projection, a bumper display, or the like is assumed. The light control unit 85 includes a control unit such as an ECU that controls the light.

The horn control unit 86 detects and controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a control unit such as an ECU that controls the car horn.

FIG. 2 is a diagram showing an example of a sensing region of the external recognition sensor 25 of FIG. 1 by a camera 51, a radar 52, a LiDAR 53, an ultrasonic sensor 54, and the like. Note that FIG. 2 schematically shows a view of the vehicle 1 from above, with the left end side being the front end (front) side of the vehicle 1 and the right end side being the rear end (rear) side of the vehicle 1.

The sensing area 91F and the sensing area 91B show an example of the sensing area of the ultrasonic sensor 54. The sensing region 91F covers the vicinity of the front end of the vehicle 1 by a plurality of ultrasonic sensors 54. The sensing region 91B covers the periphery of the rear end of the vehicle 1 by a plurality of ultrasonic sensors 54.

The sensing results in the sensing area 91F and the sensing area 91B are used, for example, for parking support of the vehicle 1.

The sensing area 92F to the sensing area 92B show an example of the sensing area of the radar 52 for a short distance or a medium distance. The sensing area 92F covers a position farther than the sensing area 91F in front of the vehicle 1. The sensing region 92B covers the rear of the vehicle 1 to a position farther than the sensing region 91B. The sensing region 92L covers the rear periphery of the left side surface of the vehicle 1. The sensing region 92R covers the rear periphery of the right side surface of the vehicle 1.

The sensing result in the sensing area 92F is used, for example, for detecting a vehicle, a pedestrian, or the like existing in front of the vehicle 1. The sensing result in the sensing region 92B is used, for example, for a collision prevention function behind the vehicle 1. The sensing results in the sensing region 92L and the sensing region 92R are used, for example, for detecting an object in a blind spot on the side of the vehicle 1.

The sensing area 93F to the sensing area 93B show an example of the sensing area by the camera 51. The sensing region 93F covers a position farther than the sensing region 92F in front of the vehicle 1. The sensing region 93B covers the rear of the vehicle 1 to a position farther than the sensing region 92B. The sensing region 93L covers the periphery of the left side surface of the vehicle 1. The sensing region 93R covers the periphery of the right side surface of the vehicle 1.

The sensing result in the sensing area 93F can be used, for example, for recognition of traffic lights and traffic signs, a lane departure prevention support system, and an automatic headlight control system. The sensing result in the sensing region 93B can be used, for example, for parking assistance and a surround view system. The sensing results in the sensing region 93L and the sensing region 93R can be used, for example, in a surround view system.

The sensing area 94 shows an example of the sensing area of LiDAR53. The sensing region 94 covers a position far from the sensing region 93F in front of the vehicle 1. On the other hand, the sensing area 94 has a narrower range in the left-right direction than the sensing area 93F.

The sensing result in the sensing area 94 is used for detecting an object such as a peripheral vehicle, for example.

The sensing area 95 shows an example of the sensing area of the radar 52 for a long distance. The sensing region 95 covers a position far from the sensing region 94 in front of the vehicle 1. On the other hand, the sensing region 95 has a narrower range in the left-right direction than the sensing region 94.

The sensing result in the sensing area 95 is used for, for example, ACC (Adaptive Cruise Control), emergency braking, collision avoidance, and the like.

Note that the sensing areas of the cameras 51, radar 52, LiDAR 53, and ultrasonic sensors 54 included in the external recognition sensor 25 may have various configurations other than those in FIG. 2. Specifically, the ultrasonic sensor 54 may be made to sense the side of the vehicle 1, or the LiDAR 53 may be made to sense the rear of the vehicle 1. Further, the installation position of each sensor is not limited to each of the above-mentioned examples. Further, the number of each sensor may be one or a plurality.

In the above configuration, when the direction (that is, the posture) of the camera 51 suddenly changes due to the inertial force when the vehicle starts, stops, turns, etc., or the vibration of the vehicle due to the unevenness of the road surface during traveling, the camera The image quality may be significantly deteriorated due to expansion and contraction or distortion of the image acquired by 51, or a sudden change in the position of an object appearing in the acquired image. Deterioration of image quality due to such distortion or position change can induce erroneous recognition or loss of an object in the recognition process, which is a factor of deteriorating recognition accuracy.

FIG. 3 is a diagram showing an example of an image taken by the front camera of the vehicle during normal driving, and FIG. 4 is a diagram showing an example of an image taken by the front camera of the vehicle during a sudden stop. When the vehicle 1 suddenly stops while traveling, a so-called nose dive occurs in which the front of the vehicle 1 sinks due to the moment of inertia. Then, as shown in FIGS. 3 to 4, the direction of the front camera of the vehicle 1 suddenly turns downward, whereby the subject (for example, the vehicle running in front) photographed during normal driving is exposed. It will move momentarily in the direction of arrow A1. When such a phenomenon occurs, erroneous recognition or loss of the subject occurs in the recognition process, and the recognition accuracy is lowered. Therefore, it may be difficult to track the subject normally in, for example, automatic driving.

Therefore, in the following embodiments, we propose an information processing device, an information processing system, and an information processing method capable of suppressing a decrease in recognition accuracy. For example, when a sensor such as an image sensor suddenly changes its posture based on an external impact, the environmental information such as image data acquired by the sensor shakes due to this external impact, thereby causing the quality of the environmental information ( If it is image data, the image quality) may deteriorate. Therefore, in the following embodiments, we propose an information processing device, an information processing system, and an information processing method capable of suppressing a decrease in recognition accuracy due to a decrease in information quality caused by such factors. The vehicle control system described above is merely an example of the application destination of the embodiment described below. That is, the embodiments described below can be applied to various devices, systems, methods, programs, and the like that involve the transfer of data such as image data.

2. 2. (1) Embodiment 2.1 Schematic configuration example of a recognition system FIG. 5 is a block diagram showing an outline of a recognition system according to the present embodiment. As shown in FIG. 5, the recognition system includes an image pickup device 100 and a recognition unit 120. The image pickup apparatus 100 may correspond to, for example, an example of an information processing apparatus within the scope of claims. Further, the recognition unit 120 may correspond to, for example, an example of a processing unit within the scope of claims.

The image pickup device 100 corresponds to, for example, the camera 51, the in-vehicle sensor 26, etc. described above with reference to FIG. 1, and generates and outputs image data of a color image or a monochrome image. The output image data is input to the recognition unit 120 via a predetermined network such as the communication network 41 described above with reference to FIG. 1.

The image pickup device 100 is connected to sensors such as the IMU 131 and the position sensor 132 that acquire information regarding the posture change of the image pickup device 100. For example, the IMU 131 corresponds to an acceleration sensor, an angular velocity sensor (gyro sensor), an IMU, etc. in the vehicle sensor 27 described above with reference to FIG. 1, and information on the detected acceleration and angular velocity (hereinafter referred to as acceleration / angular velocity information). ) Is output to the image pickup apparatus 100. Further, the position sensor 132 corresponds to, for example, a steering angle sensor, a yaw rate sensor, an accelerator sensor, a brake sensor, etc. in the vehicle sensor 27 described above with reference to FIG. 1, and captures odometry information detected by each sensor. Output to device 100. In addition to this, sensor information detected by various sensors mounted on the vehicle 1 as the vehicle sensor 27 may be input to the image pickup apparatus 100.

The recognition unit 120 corresponds to, for example, the recognition unit 73 or the like described above with reference to FIG. 1, and by executing a recognition process on the image data input from the image pickup apparatus 100, an object included in the image or an object or the like. Detect the background etc. The object may include a moving object such as a car, a bicycle, or a pedestrian, as well as a fixed object such as a building, a house, or a tree. On the other hand, the background may be a wide area located in a distant place such as the sky, mountains, plains, and the sea.

Further, the recognition unit 120 determines the area of the object or the area of the background obtained as a result of the recognition process for the image data as the ROI (Region of Interest) which is a part of the effective pixel area in the image sensor 101. You may. At that time, the recognition unit 120 may determine the resolution at which the image data is read from each ROI. In that case, the recognition unit 120 notifies the image pickup apparatus 100 of the determined ROI and resolution information (hereinafter referred to as ROI / resolution information), so that the ROI to be read and the image data are read from each ROI. The resolution may be set in the image pickup device 100.

Note that the ROI information may be, for example, information regarding the address of the pixel that is the starting point of the ROI and the size in the vertical and horizontal directions. In that case, each ROI is a rectangular area. However, the ROI is not limited to this, and the ROI may be a circle, an ellipse, or a polygon, or may be a region having an indefinite shape specified by information specifying a boundary (contour). Further, when the recognition unit 120 determines a plurality of ROIs, the recognition unit 120 may determine a different resolution for each ROI.

2.2 Schematic configuration example of the image pickup device FIG. 6 is a block diagram showing a schematic configuration example of the image pickup device according to the present embodiment. As shown in FIG. 6, the image pickup apparatus 100 includes an image sensor 101, a control unit 102, a signal processing unit 103, a storage unit 104, and an input / output unit 105. One or more of the control unit 102, the signal processing unit 103, the storage unit 104, and the input / output unit 105 may be provided on the same chip as the image sensor 101.

Although not shown, the image sensor 101 converts a pixel array unit in which a plurality of pixels are arranged in a two-dimensional grid, a drive circuit for driving the pixels, and a pixel signal read from each pixel into digital values. The image data read from the entire pixel array unit or individual ROIs is output to the signal processing unit 103. In this embodiment, a case where the image sensor 101 is a so-called rolling shutter type image sensor 101 in which image data is read out row by row from the pixel array unit is illustrated.

The signal processing unit 103 executes predetermined signal processing such as noise reduction and white balance adjustment on the image data output from the image sensor 101.

Further, as will be described in detail later, the signal processing unit 103 refers to the image data output from the image sensor 101 in line units (hereinafter referred to as line data), and the time required to read the line (read time information). Information about) is given. That is, in the image data output from the image pickup apparatus 100 in the present embodiment, the read time information is added to each pixel row. For this read time information, various information related to the time required for reading the row, such as the time from the read timing of the first pixel to the read timing of the last pixel in each row and the number of pixels to be read in each row, is used. It's okay.

Further, the signal processing unit 103 may add acceleration / angular velocity information input from the IMU 131 and / or odometry information input from the position sensor 132 to each row data. Hereinafter, for the sake of simplification of the explanation, the acceleration / angular velocity information and the odometry information are collectively referred to as sensor information. The sensor information added to each row data is the sensor information input from the IMU 131 and / or the position sensor 132 during the exposure period of each pixel row in the pixel array unit, and the IMU 131 at the timing when each row data is output from the image sensor 101. And / or sensor information input from the position sensor 132 may be used.

Furthermore, the signal processing unit 103 includes sensor information input from the IMU 131 and / or the position sensor 132 during the reading period (also referred to as a frame period) of one image data, and various information obtained from the sensor information (also referred to as a frame period). For example, speed information, etc., hereinafter also referred to as additional information) may be included in the image data.

Then, the signal processing unit 103 outputs the image data to which the predetermined signal processing is performed and the read time information and the sensor information (and additional information) are added to the input / output unit 105.

The storage unit 104 temporarily holds the image data processed or unprocessed by the signal processing unit 103, the sensor information input from the IMU 131 and / or the position sensor 132, and the like as needed.

The input / output unit 105 transmits the image data input via the signal processing unit 103 to the recognition unit 120 via a predetermined network (for example, the communication network 41).

The control unit 102 controls the operation of the image sensor 101. Further, the control unit 102 sets one or more ROIs (also referred to as a read target area) and the resolution of each ROI in the image sensor 101 based on the ROI / resolution information input via the input / output unit 105.

2.3 Correction of distortion caused by environmental information Next, the distortion caused by environmental information acquired by the sensor will be described. In the following, for the sake of clarification of the explanation, the environmental information acquired by the sensor is referred to as image data, but the image data is merely an example of the environmental information. Therefore, the environmental information is the image sensor 101 (camera 51). , Radar 52, LiDAR53, ultrasonic sensor 54, etc., may be variously changed depending on the type of sensor used and the like.

Factors that cause distortion in image data include abrupt changes in the image pickup direction (that is, posture) of the image sensor 101 (shaking due to an external impact), and when multiple ROIs are read out at the same time, a part of the ROIs is lined up. There are things such as overlapping with each other in the direction. Here, a case of correcting the distortion generated in the image data due to these factors will be described. In the following description, the image data that does not include the correction information is referred to as frame data in order to distinguish between the image data that includes the correction information for correcting the distortion generated in the image data and the image data that does not include the correction information. The included image data is called image data as it is. The frame data and the image data are also referred to as two-dimensional data because they have a two-dimensional data structure in the row direction and the column direction.

2.3.1 Distortion caused by abrupt changes in orientation FIG. 7 is a schematic diagram for explaining the reading operation of the rolling shutter method. FIG. 8 is a diagram for explaining an example of distortion of frame data that may occur when the image sensor suddenly points downward during reading by the rolling shutter method.

As shown in FIG. 7, in the reading operation of the rolling shutter method, the effective pixel area in the pixel array unit 101a of the image sensor 101 is a column in which the pixels (pixel rows) arranged in the row direction are in units of one pixel row. It is read out sequentially in the direction. Therefore, if the orientation of the image sensor 101 suddenly changes downward while reading one frame data from the image sensor 101, the frame data G2 read from the image sensor 101 becomes a frame data G2 as shown in FIG. Compared with the frame data G1 read when the posture of the image sensor 101 is constant, the image is extended downward.

Further, as described above with reference to FIGS. 3 and 4, when the direction of the image sensor 101 suddenly changes downward between the reading of the previous frame and the reading of the current frame, the front frame and the current frame are displayed. Then, the subject (for example, the vehicle running in front) reflected in the frame data momentarily moves in the direction of the arrow A1.

Therefore, in the present embodiment, the sensor information input from the IMU 131 and / or the position sensor 132 is added to the frame data while reading each line data of the frame data. This makes it possible to correct the distortion of the frame data based on the sensor information.

As a method of adding sensor information to frame data, a method of adding sensor information input while reading the row data for each row data (hereinafter referred to as the first method) or one frame data. Various methods such as a method of adding sensor information input while reading is added to a header or footer of frame data (hereinafter referred to as a second method) may be applied. The sensor information is one aspect of the correction information for correcting the distortion generated in the image data.

Further, as a method of adding correction information to the frame data, a method of adding the sensor information input from the IMU 131 or the position sensor 132 to the row data or the frame data as it is, or a method of adding speed, acceleration or angular speed from the input sensor information. A method of calculating information for specifying or correcting distortion generated in image data such as angular acceleration (hereinafter referred to as distortion information) and adding the calculated distortion information to the frame data can be considered. The distortion information is one aspect of the correction information for correcting the distortion generated in the image data.

(First method)
FIG. 9 is a flowchart for explaining an example of the reading operation according to the first method of the present embodiment, and FIG. 10 is a diagram for supplementing the flowchart shown in FIG.

As shown in FIGS. 9 and 10, in the first method, first, the control unit 102 of the image pickup apparatus 100 sets the variable L for managing the read row to '1' indicating the first row (step S101). .. Then, the control unit 102 causes the image sensor 101 to read the row data from the Lth row (step S102). When reading to the ROI, the first line indicated by L = 0 may be the highest line in the ROI.

Further, the control unit 102 inputs sensor information from the IMU 131 and / or the position sensor 132 while reading the row data in step S102 (step S103), and inputs the input sensor information as shown in FIG. It is added to the row data read in step S102 (step S104).

Next, the control unit 102 determines whether or not the variable L has reached the maximum value L_max (step S105), and if not (NO in step S105), the variable L1 is incremented by 1 (step S106). , Return to step S102, and continue the subsequent operations.

On the other hand, when the variable L reaches the maximum value L_max (YES in step S105), the control unit 102 connects the image data (see FIG. 10) to which the sensor information is added to each row data of the frame data to a predetermined network (for example,). , Output to the recognition unit 120 via the communication network 41) (step S107).

After that, the control unit 102 determines whether or not to end this operation (step S108), and if it ends (YES in step S108), ends this operation. On the other hand, if it does not end (NO in step S108), the control unit 102 returns to step S101 and executes the subsequent operations.

(Second method)
FIG. 11 is a flowchart for explaining an example of the reading operation according to the second method of the present embodiment. As shown in FIG. 11, in the second method, first, the control unit 102 of the image pickup apparatus 100 drives the image sensor 101 to read out the frame data (step S121).

Further, the control unit 102 inputs sensor information from the IMU 131 and the position sensor 132 to the signal processing unit 103 (step S122) while the frame data is being read out in step S121, and calculates distortion information in the signal processing unit 103. Is executed (step S123).

Next, the control unit 102 generates image data by adding the distortion information calculated in step S123 to the frame data read in step S121 (step S124), and the generated image data is used in a predetermined network. It is output to the recognition unit 120 via (for example, the communication network 41) (step S125).

After that, the control unit 102 determines whether or not to end this operation (step S126), and if it ends (YES in step S126), ends this operation. On the other hand, if it does not end (NO in step S126), the control unit 102 returns to step S121 and executes the subsequent operations.

2.3.2 Distortion due to overlapping ROIs in the row direction FIG. 12 is a diagram illustrating a case where two ROIs partially overlapping in the row direction are set in the pixel array portion, and FIG. 13 is a diagram. It is a figure for demonstrating the reading of the image data (hereinafter referred to as ROI data) from each ROI shown in FIG. 12, and FIG. 14 shows two ROIs partially overlapping in the row direction in the reading operation of the rolling shutter method. It is a figure which shows an example of the read start timing of each line at the time of reading ROI data from.

As shown in FIG. 12, when two ROI regions R11 and R12 are set in the pixel array unit 101a and a part of the two ROI regions R11 and R12 overlap each other in the row direction, in the reading operation of the rolling shutter method, the reading operation is performed. , There is a range R21 for reading only the row data in the ROI area R11, a range R22 for reading both the row data in the ROI area R11 and the ROI area R12, and a range R23 for reading only the row data in the ROI area R12. Therefore, as shown in FIG. 13, the number of pixels constituting the row data changes in each range R21 to R23. Specifically, for example, assuming that the number of pixels in the row direction in the ROI region R11 and the ROI region R12 is the same, the number of pixels in each row in the range R22 is twice the number of pixels in each row in the ranges R21 and R23. Become. Therefore, since the read time of each row changes between the range R22 and the ranges R21 and R23, as shown in FIG. 14, in the rolling shutter type read operation, the read start timing of each line in each of the ranges R21 to R23 changes. It will be. Such a change in the read start timing causes the ROI data to be distorted.

Therefore, in the present embodiment, the number of read pixels of each row in the ROI data (hereinafter, also referred to as the number of read pixels) is added to the ROI data. This makes it possible to correct the distortion of the ROI data based on the number of read pixels. The number of read pixels is an aspect of correction information for correcting the distortion generated in the image data.

FIG. 15 is a flowchart showing an example of the reading operation according to the present embodiment, and FIG. 16 is a diagram for supplementing the flowchart shown in FIG.

As shown in FIGS. 15 and 16, in this operation, first, the control unit 102 of the image pickup apparatus 100 sets a variable L for managing read rows for one or more ROIs to '1' indicating the first row. (Step S141). Then, the control unit 102 causes the image sensor 101 to read the row data from the Lth row in the range where the ROI exists (step S142). Then, as shown in FIG. 16, the control unit 102 adds the number of read pixels in the Lth row to the row data read in step S102 (step S143).

Next, the control unit 102 determines whether or not the variable L has reached the maximum value L_max (step S144), and if not (NO in step S144), the variable L1 is incremented by 1 (step S145). , Return to step S142, and continue the subsequent operations.

On the other hand, when the variable L reaches the maximum value L_max (YES in step S144), the control unit 102 connects the image data (see FIG. 10) to which the sensor information is added to each row data of the frame data to a predetermined network (for example, FIG. 10). , Output to the recognition unit 120 via the communication network 41) (step S146).

After that, the control unit 102 determines whether or not to end the main operation (step S147), and if it ends (YES in step S147), the control unit 102 ends the main operation. On the other hand, if it does not end (NO in step S147), the control unit 102 returns to step S141 and executes the subsequent operations.

The first method and the second method described above can be implemented in combination. In that case, sensor information and the number of read pixels may be added to each row data of the frame data.

Further, in this description, the imaging direction (that is, the posture) of the image sensor 101 changes abruptly, or when a plurality of ROIs are read out at the same time, some of the ROIs overlap each other in the row direction. The case of correcting the distortion generated in the frame data or the ROI data has been illustrated, but the present embodiment is not limited to this, and the correction information for correcting the distortion generated in the frame data on the image pickup apparatus 100 side for some reason. It is possible to apply various configurations as long as the configuration is such that the above is added.

3.3.3 Distortion correction In addition, there are cases where distortion generated in frame data becomes a problem, for example, when a sensor with a different reading method such as an image sensor and an EVS (Event Vision Sensor) is combined. do. For example, EVS, which detects a change in the brightness of each pixel as an event, has the same exposure period for each pixel as in the global shutter method, so that the distortion included in the generated image data (also referred to as a difference image) is small. .. Therefore, when the image data output from the image sensor and the difference image output from the EVS are integrated to perform a process such as reconstructing the image data of the current frame, the image data is output from the image sensor. The difference in distortion between the image data and the difference image output from the EVS can be a problem.

FIG. 17 is a diagram for explaining the difference in distortion when the image sensor and EVS are combined. When the image data reading method in the image sensor 101 is the rolling shutter method, a time difference D1 occurs in the reading timing between the highest pixel row and the lowest pixel row in the column direction, so that the read image data G31 is referred to as so-called. A distortion called rolling shutter distortion occurs. On the other hand, in EVS, an event is detected in each pixel by the same operation as the so-called global shutter method in which all pixels are simultaneously driven, so that the image data G32 output from EVS is not distorted or is not distorted. It is small enough to be ignored in the recognition process by the recognition unit 120.

As described above, the difference in distortion between the two image data caused by the different drive methods can be eliminated by using, for example, the first method and / or the second method described above.

2.3.4 Operation example Next, the operation based on the correction information added to the frame data as described above will be described. This operation may be executed, for example, by the recognition unit 120 in which image data is input via a predetermined network (for example, a communication network 41). FIG. 18 is a flowchart showing an example of the operation according to the present embodiment. FIG. 19 is a diagram for explaining an example of distortion correction shown in step S164 of FIG.

As shown in FIG. 18, when the recognition unit 120 inputs image data via a predetermined network (for example, communication network 41) (step S161), the recognition unit 120 receives correction information (for example, sensor information, distortion information) included in the image data. , Number of read pixels) (step S162).

Next, the recognition unit 120 corrects the distortion of the frame data based on the specified correction information (step S163). For example, when the frame data G2 is distorted as described above with reference to FIG. 8, the recognition unit 120 uses the sensor information added to each row data of the frame data G2 as shown in FIG. Then, the frame data G2 is corrected to the frame data G3 in which the distortion (for example, the delay) is reduced or eliminated.

When the distortion of the frame data is corrected in this way, the recognition unit 120 executes the recognition process for the frame data whose distortion has been corrected (step S164), and the result is the action planning unit 62, the operation control unit 63, etc. (FIG. FIG. 1) (see step S165).

After that, the recognition unit 120 determines whether or not to end the main operation (step S166), and if it ends (YES in step S166), the recognition unit 120 ends the main operation. On the other hand, if it does not end (NO in step S166), the control unit 102 returns to step S161 and executes the subsequent operations.

2.4 ROI setting As described above, in the present embodiment, the image data acquired by the recognition unit 120 may include sensor information and distortion information. For example, the speed of the vehicle 1 can be directly or indirectly specified from the sensor information and the distortion information included in the image data. Further, it is possible to directly or indirectly identify whether the vehicle 1 is traveling straight or turning from the sensor information and the distortion information included in the image data. Therefore, in the present embodiment, the recognition unit 120 may determine the ROI and the resolution in the next frame and thereafter based on the sensor information and the distortion information.

20 to 25 are diagrams for explaining the ROI determination method according to the present embodiment. FIG. 20 is a diagram showing a sensing region of the vehicle when traveling straight at a low speed, and FIG. 21 is a diagram showing an ROI corresponding to the sensing region shown in FIG. 20. FIG. 22 shows the sensing region of the vehicle when traveling straight at high speed, and FIG. 23 is a diagram showing the ROI corresponding to the sensing region shown in FIG. 22. FIG. 24 shows the sensing region of the vehicle when turning left, and FIG. 25 is a diagram showing the ROI corresponding to the sensing region shown in FIG. 24.

First, as shown in FIG. 20, when the vehicle 1 is traveling straight at a low speed, it is desirable that the sensing region SR101 covers a wide area in the vicinity in front of the vehicle 1. Therefore, when the vehicle 1 is traveling straight at a low speed, the recognition unit 120 may determine the ROI region R101 in the entire area or a wide area of the pixel array unit 101a as shown in FIG. Further, the object existing in the vicinity of the vehicle 1 is imaged as a large image. Therefore, when the vehicle 1 is traveling straight at a low speed, the recognition unit 120 may determine the resolution for the ROI to be low.

Further, as shown in FIG. 22, when the vehicle 1 is traveling straight at high speed, the sensing region SR102 may cover a distant narrow range in front of the vehicle 1. Therefore, when the vehicle 1 is traveling straight at high speed, the recognition unit 120 may determine the ROI region R102 in a part of the central region of the pixel array unit 101a as shown in FIG. 23. Further, an object existing in the distance of the vehicle 1 is imaged as a small image. Therefore, when the vehicle 1 is traveling straight at high speed, the recognition unit 120 may determine the resolution for ROI to be high resolution.

Further, as shown in FIG. 24, when the vehicle 1 is turning (turning to the left in FIG. 24), it is desirable that the sensing region SR103 covers the turning direction of the vehicle 1. Therefore, when the vehicle 1 is turning, the recognition unit 120 may determine the ROI region R103 in the region of the pixel array unit 101a that is closer to the turning direction (to the left in FIG. 25), as shown in FIG. 25. How much to shift in the turning direction may be determined based on, for example, odometry information (steering angle, etc.) input from the position sensor 132. Further, when the vehicle 1 is turning, it is considered that the vehicle 1 is traveling at an intersection, a corner, or the like. Therefore, the recognition unit 120 determines the resolution for the ROI to be low in order to execute the recognition process at high speed. You may.

2.5 Regarding super-resolution of image data Further, in the present embodiment, the image data may include sensor information (information regarding acceleration and angular velocity) indicating the attitude displacement of the image sensor 101. Therefore, in the present embodiment, it is possible to generate super-resolution image data higher than the maximum resolution of the image sensor 101 by utilizing the posture change of the image sensor 101 between frames.

Specifically, as shown in FIG. 26, the frame data G101, G102, G103, ... Input from the image sensor 101 are moved in the vertical and horizontal directions due to the posture change caused by the shaking or vibration of the image sensor 101 itself. It's shifting. Then, the direction in which the frame data is shifted can be specified from the sensor information added to the frame data G101, G102, G103, ....

Therefore, as shown in FIG. 27, the recognition unit 120 assumes that a certain frame data G112 is shifted by, for example, half a pixel in the left-right direction with respect to the frame data G111 of the previous frame based on the sensor information added to the frame data. By synthesizing the frame data G111 and G112, it is possible to make the resolution in the horizontal direction (row direction) twice the resolution of the frame data G111 and G112. Similarly, by synthesizing the frame data G111 and G112 assuming that a certain frame data G112 is, for example, half a pixel shifted in the vertical direction with respect to the frame data G111 of the previous frame, the resolution in the vertical direction (column direction) is frame data. It is possible to have twice the resolution of G111 and G112.

Further, as shown in FIG. 28, by setting the image data to be combined with the frame data G121 as a key frame as the ROI data G122, the image data G123 in which a part of the region (ROI) is super-resolution is generated. It is also possible to do.

2.6 Actions / Effects As described above, according to the present embodiment, since the image data includes the correction information for correcting the distortion, the recognition unit 120 has the distortion generated in the frame data based on the correction information. Can be corrected. As a result, in the present embodiment, it is possible to suppress a decrease in recognition accuracy.

In the above description, the case where the correction information is included in the frame data acquired by the image sensor 101 is illustrated, but the present disclosure is not limited to this, and for example, the radar 52, the LiDAR 53, the ultrasonic sensor 54, etc. 2 It is also possible to add correction information to two-dimensional data output from various sensors having a dimensional data structure.

Further, in the present embodiment, a case where the recognition unit 120 executes correction processing or the like on the image data acquired by one image pickup device 100 is illustrated, but the configuration is not limited to such a configuration. For example, the recognition unit 120 may execute correction processing or the like on the image data acquired by each of the two or more image pickup devices 100. In that case, the image data acquired by each of the two or more image pickup devices 100 may be integrated by the sensor fusion unit 72 and then input to the recognition unit 120. The sensor fusion unit 72 may correspond to an example of an integrated unit within the scope of claims.

3. 3. The vehicle control system 11 described above with reference to FIG. 1 may include, for example, a system structure based on a domain architecture as shown in FIG. 29. The system structure exemplified in FIG. 29 is configured such that domain controllers 311 to 315 that manage each of the front, left, left rear, right rear, and right side of the vehicle 1 are connected to each other via the gateway 301 and cooperate with each other. ing. Further, each domain controller 311 to 315 is connected to sensor groups 321 to 325 such as an external recognition sensor 25, an in-vehicle sensor 26, and a vehicle sensor 27, and the vehicle is based on sensor information acquired by each sensor group 321 to 325. Control each part of 1.

In such a configuration, each domain controller 311 to 315 may correspond to the vehicle control system 11 shown in FIG. One domain controller may input image data including correction information acquired by the external recognition sensor 25 in the other one or more domain controllers, and process the input image data in an integrated manner. Further, when one of the domain controllers 311 to 315 functions as a central controller (also referred to as a main controller), this central controller is the correction information acquired by the external recognition sensor 25 in the other one or more domain controllers. Image data including the above may be input and the input image data may be processed in an integrated manner.

Furthermore, at least one of the domain controllers 311 to 315 goes to the cloud via a mobile communication network such as LTE (Long Term Evolution) or 5G (5th Generation) or a predetermined network such as a wireless LAN (Local Area Network). If it is accessible, the image data including the correction information acquired by the external recognition sensor 25 in one or more domain controllers and the processing result obtained by processing this image data in an integrated manner are uploaded to the cloud. It may be configured to do so. In that case, the above-mentioned frame data distortion correction, recognition processing, and the like may be executed on the cloud side.

4. Hardware Configuration The recognition unit 120 according to the above-described embodiment, modifications thereof, and application examples can be realized by, for example, a computer 1000 having a configuration as shown in FIG. FIG. 30 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the information processing apparatus constituting the recognition unit 120. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, an HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. Each part of the computer 1000 is connected by a bus 1050.

The CPU 1100 operates based on the program stored in the ROM 1300 or the HDD 1400, and controls each part. For example, the CPU 1100 expands the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200, and executes processing corresponding to various programs.

The ROM 1300 stores a boot program such as a BIOS (Basic Input Output System) executed by the CPU 1100 when the computer 1000 is started, a program depending on the hardware of the computer 1000, and the like.

The HDD 1400 is a computer-readable recording medium that non-temporarily records a program executed by the CPU 1100 and data used by such a program. Specifically, the HDD 1400 is a recording medium for recording a projection control program according to the present disclosure, which is an example of program data 1450.

The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (for example, the Internet). For example, the CPU 1100 receives data from another device or transmits data generated by the CPU 1100 to another device via the communication interface 1500.

The input / output interface 1600 has a configuration including the above-mentioned I / F unit 18, and is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or mouse via the input / output interface 1600. Further, the CPU 1100 transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. Further, the input / output interface 1600 may function as a media interface for reading a program or the like recorded on a predetermined recording medium (media). The media is, for example, an optical recording medium such as DVD (Digital Versatile Disc) or PD (Phase change rewritable Disk), a magneto-optical recording medium such as MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. Is.

For example, the CPU 1100 of the computer 1000 functions as the recognition unit 120 according to the above-described embodiment by executing the program loaded on the RAM 1200. Further, the program and the like related to the present disclosure are stored in the HDD 1400. The CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program, but as another example, these programs may be acquired from another device via the external network 1550.

Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments as they are, and various changes can be made without departing from the gist of the present disclosure. In addition, components spanning different embodiments and modifications may be combined as appropriate.

Further, the effects in each embodiment described in the present specification are merely examples and are not limited, and other effects may be obtained.

The present technology can also have the following configurations.
(1)
Sensors that acquire environmental information and
A control unit that adds correction information to the environmental information for correcting shaking caused by an external impact generated in the environmental information acquired by the sensor, and a control unit.
Information processing device equipped with.
(2)
The information processing device according to (1) above, wherein the sensor is any one of an image sensor, a radar, a LiDAR, and an ultrasonic sensor.
(3)
The sensor is an image sensor and is an image sensor.
The environmental information is imaging data and is
The information processing apparatus according to (2), wherein the control unit adds the correction information for each row data to each row data constituting the imaging data.
(4)
The information processing apparatus according to (3) above, wherein the correction information includes at least one of acceleration, angular velocity, odometry information, and the number of pixels read out for each row.
(5)
The control unit adds the acceleration, the angular velocity, the odometry information, and the correction information calculated based on at least one of the acceleration, the angular velocity, and the odometry information to the environment information. The information processing apparatus according to (4).
(6)
The information processing apparatus according to any one of (1) to (5) above,
A processing unit connected to the information processing device via a predetermined network,
Equipped with
The information processing apparatus transmits the environmental information to which the correction information is added to the processing unit via the predetermined network.
The processing unit is an information processing system that corrects shaking due to the external impact generated in the environmental information based on the correction information added to the environmental information.
(7)
The information processing system according to (6), wherein the processing unit executes recognition processing for environmental information corrected based on the correction information.
(8)
The processing unit determines the read target area in the sensor and the resolution of the read target area based on the correction information, sets the determined read target area and the resolution in the control unit, and sets the determined read target area and the resolution in the control unit.
The information processing system according to (6) or (7), wherein the control unit drives the sensor based on the set read-out target area and the resolution.
(9)
Further equipped with an integration unit that integrates environmental information transmitted from two or more information processing devices.
The information processing system according to any one of (6) to (8) above, wherein the processing unit executes processing on the environmental information integrated by the integrated unit.
(10)
With the plurality of the processing units
A plurality of the information processing devices connected to each of the processing units on a one-to-one basis,
The information processing system according to any one of (6) to (9) above.
(11)
The information processing system according to (10), wherein one of the plurality of processing units integrally processes environment information output from two or more of the plurality of information processing devices.
(12)
At least one of the environment information output from the information processing apparatus, the environment information corrected by the processing unit, and the predetermined processing result for the corrected environment information by the processing unit is transmitted via a predetermined network. The information processing system according to (6) above, further comprising a transmission unit for transmission.
(13)
It is an information processing method executed by an information processing system.
Correction information for correcting the shaking caused by the external impact generated in the environmental information acquired by the sensor is added to the environmental information.
An information processing method including correcting a shake caused by an external impact generated in the environmental information received from an information processing apparatus via a predetermined network based on the correction information added to the environmental information.

100 Image pickup device 101 Image sensor 102 Control unit 103 Signal processing unit 104 Storage unit 105 Input / output unit 120 Recognition unit 131 IMU
132 Position sensor

Claims (13)

1. Sensors that acquire environmental information and
   A control unit that adds correction information to the environmental information for correcting shaking caused by an external impact generated in the environmental information acquired by the sensor, and a control unit.
   Information processing device equipped with.
2. The information processing device according to claim 1, wherein the sensor is any one of an image sensor, a radar, a LiDAR, and an ultrasonic sensor.
3. The sensor is an image sensor and is an image sensor.
   The environmental information is imaging data and is
   The information processing device according to claim 2, wherein the control unit adds the correction information for each row data to each row data constituting the imaging data.
4. The information processing apparatus according to claim 3, wherein the correction information includes at least one of acceleration, angular velocity, odometry information, and the number of pixels read out for each row.
5. The control unit adds the acceleration, the angular velocity, the odometry information, and the correction information calculated based on at least one of the acceleration, the angular velocity, and the odometry information to the environment information. Item 4. The information processing apparatus according to item 4.
6. The information processing apparatus according to claim 1 and
   A processing unit connected to the information processing device via a predetermined network,
   Equipped with
   The information processing apparatus transmits the environmental information to which the correction information is added to the processing unit via the predetermined network.
   The processing unit is an information processing system that corrects shaking due to the external impact generated in the environmental information based on the correction information added to the environmental information.
7. The information processing system according to claim 6, wherein the processing unit executes recognition processing for environmental information corrected based on the correction information.
8. The processing unit determines the read target area in the sensor and the resolution of the read target area based on the correction information, sets the determined read target area and the resolution in the control unit, and sets the determined read target area and the resolution in the control unit.
   The information processing system according to claim 6, wherein the control unit drives the sensor based on the set read-out target area and the resolution.
9. Further equipped with an integration unit that integrates environmental information transmitted from two or more information processing devices.
   The information processing system according to claim 6, wherein the processing unit executes processing on the environmental information integrated by the integrated unit.
10. With the plurality of the processing units
    A plurality of the information processing devices connected to each of the processing units on a one-to-one basis,
    The information processing system according to claim 6.
11. The information processing system according to claim 10, wherein one of the plurality of processing units integrally processes environment information output from two or more of the plurality of information processing devices.
12. At least one of the environment information output from the information processing apparatus, the environment information corrected by the processing unit, and the predetermined processing result for the corrected environment information by the processing unit is transmitted via a predetermined network. The information processing system according to claim 6, further comprising a transmission unit for transmitting information.
13. It is an information processing method executed by an information processing system.
    Correction information for correcting the shaking caused by the external impact generated in the environmental information acquired by the sensor is added to the environmental information.
    An information processing method including correcting a shake caused by an external impact generated in the environmental information received from an information processing apparatus via a predetermined network based on the correction information added to the environmental information.

Images