WO2019203000A1 - External environment recognition device - Google Patents

External environment recognition device Download PDF

Info

Publication number
WO2019203000A1
WO2019203000A1 PCT/JP2019/014915 JP2019014915W WO2019203000A1 WO 2019203000 A1 WO2019203000 A1 WO 2019203000A1 JP 2019014915 W JP2019014915 W JP 2019014915W WO 2019203000 A1 WO2019203000 A1 WO 2019203000A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
image
unit
external environment
recognition device
Prior art date
Application number
PCT/JP2019/014915
Other languages
French (fr)
Japanese (ja)
Inventor
健 志磨
琢馬 大里
裕史 大塚
Original Assignee
日立オートモティブシステムズ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日立オートモティブシステムズ株式会社 filed Critical 日立オートモティブシステムズ株式会社
Priority to CN201980025028.1A priority Critical patent/CN111937035A/en
Publication of WO2019203000A1 publication Critical patent/WO2019203000A1/en

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
    • B60W30/08—Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
    • B60W30/09—Taking automatic action to avoid collision, e.g. braking and steering
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/02—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to ambient conditions
    • B60W40/04—Traffic conditions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems

Definitions

  • the present invention relates to an external recognition device.
  • a vehicular image processing apparatus capable of improving the reliability of information analysis using camera images is known (see, for example, Patent Document 1).
  • Patent Document 1 states that “a region of interest including a preceding vehicle or a road white line or a sign and other non-attention regions are set in an image captured by an imaging unit that images the front of the host vehicle, and the imaging unit Luminance information in the attention area and non-attention area in the captured image is detected, and the vehicle is in a driving environment where it is difficult to analyze the situation ahead of the vehicle by image processing based on the luminance information of the attention area and the non-attention area. It is determined whether or not.
  • An object of the present invention is to provide an external recognition apparatus capable of suppressing a transfer delay when outputting image data to the outside.
  • the present invention provides an imaging unit that captures an image, a data generation unit that generates a plurality of types of data from the image captured by the imaging unit, and data from the plurality of types of data.
  • transfer delay can be suppressed when image data is output to the outside.
  • FIG. 1 is a configuration diagram of a system including a stereo camera according to an embodiment of the present invention. It is a block diagram of the stereo camera by embodiment of this invention. It is a block diagram which shows the function of the stereo camera by embodiment of this invention. It is a figure which shows the example of a communication interface. It is a flowchart which shows operation
  • FIG. 1 is a configuration diagram of a system 1 including a stereo camera 100 according to an embodiment of the present invention.
  • the system 1 includes a stereo camera 100, an automatic operation ECU 200 (Electronic Control Unit), a millimeter wave radar 300, a LIDAR 400 (Light Detection Detection And And Ranging), and the like.
  • the stereo camera 100 images the same target object from a plurality of different viewpoints, and calculates the distance to the target object from the deviation of the appearance (parallax) in the obtained image.
  • the stereo camera 100 is connected to the automatic operation ECU 200 with a LAN (Local Area Network) cable compliant with Ethernet (registered trademark).
  • Ethernet registered trademark
  • CAN Controller Area ⁇ ⁇ Network
  • the automatic driving ECU 200 controls automatic driving of the vehicle based on the distance, angle, relative speed, and the like of the target object detected by sensors such as the millimeter wave radar 300 and the LIDAR 400.
  • the automatic operation ECU 200 is connected to sensors such as the millimeter wave radar 300 and the LIDAR 400 via a CAN-compliant two-wire bus.
  • Millimeter wave radar 300 detects the distance, angle, relative speed, etc. of the target object using millimeter waves (electromagnetic waves).
  • the LIDAR 400 detects the distance, angle, relative speed, and the like of the target object using light waves (electromagnetic waves) such as ultraviolet rays, visible rays, and near infrared rays.
  • FIG. 2 is a configuration diagram of the stereo camera 100 according to the embodiment of the present invention.
  • the stereo camera 100 includes an imaging unit 101, a CPU 102 (Central Processing Unit), a memory 103, an image processing unit 104, and a communication circuit 105.
  • a CPU 102 Central Processing Unit
  • the imaging unit 101 includes a left imaging unit 101L and a right imaging unit 101R, and captures an image.
  • each of the left imaging unit 101L and the right imaging unit 101R includes an optical element such as a lens and an imaging element such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor).
  • the optical element refracts light and forms an image on the imaging element.
  • the image sensor receives an image of light refracted by the optical element and generates an image corresponding to the intensity of the light.
  • the CPU 102 implements various functions to be described later by executing a predetermined program.
  • CPU102 produces
  • the memory 103 includes, for example, a nonvolatile memory such as FLASH and a volatile memory such as a RAM (Random Access Memory), and stores various information.
  • a nonvolatile memory such as FLASH
  • a volatile memory such as a RAM (Random Access Memory)
  • the image processing unit 104 includes a circuit such as an FPGA (Field-Programmable Gate Array), a DSP (Digital Signal Processor), and an ASIC (Application Specific Integrated Circuit).
  • the image processing unit 104 performs image correction, parallax calculation, and the like, for example. Details of the function of the image processing unit 104 will be described later with reference to FIG.
  • the communication circuit 105 includes, for example, an Ethernet-compliant transceiver.
  • the communication circuit 105 transmits or receives various data.
  • FIG. 3 is a block diagram illustrating functions of the stereo camera 100 according to the embodiment of the present invention.
  • the left imaging unit 101L captures a first image of the target object.
  • the right imaging unit 101R captures a second image of the target object.
  • the captured first image and second image are temporarily stored, for example, in the memory 103 as “raw images” (RAW data).
  • the image processing unit 104 includes an image correction unit 1041, a parallax calculation unit 1042, a three-dimensional object extraction unit 1043, a three-dimensional grouping unit 1044, and an object identification unit 1045. Note that the image correction unit 1041, the parallax calculation unit 1042, the three-dimensional object extraction unit 1043, the three-dimensional grouping unit 1044, and the object identification unit 1045 may be configured separately.
  • the image correction unit 1041 corrects the first image captured by the left imaging unit 101L and the second image captured by the right imaging unit 101R. Specifically, the image correction unit 1041 corrects, for example, distortion and optical axis deviation between the first image and the second image.
  • the corrected first image and second image are temporarily stored, for example, in the memory 103 as “corrected images”.
  • the parallax calculation unit 1042 calculates parallax from the first image and the second image corrected by the image correction unit 1041 by performing stereo matching.
  • the parallax calculation unit 1042 calculates the distance from the parallax to the target object by triangulation.
  • the parallax calculation unit 1042 generates a parallax image in which a position on the image is associated with a distance corresponding to the parallax.
  • the parallax image (first data) is stored in the memory 103, for example.
  • the three-dimensional object extraction unit 1043 extracts an object located at an equivalent distance from the parallax image generated by the parallax calculation unit 1042 as one solid object.
  • the three-dimensional object raw information (second data) that is data of the extracted three-dimensional object is stored in the memory 103, for example.
  • the three-dimensional grouping unit 1044 groups the three-dimensional objects extracted by the three-dimensional object extraction unit 1043 based on luminance, edges, and the like.
  • the post-three-dimensional object grouping information (third data) that is data of the grouped three-dimensional object is stored in the memory 103, for example.
  • the object identification unit 1045 identifies what the target object is (a pedestrian, a vehicle, etc.) from the three-dimensional object grouped by the three-dimensional grouping unit 1044.
  • the CPU 102 functions as a vehicle control unit 1021 and a data extraction unit 1022 by executing a predetermined program.
  • the vehicle control unit 1021 generates a command for controlling the vehicle according to the type of the object identified by the object identification unit 1045. For example, if the vehicle control unit 1021 determines that the pedestrian is on the path of the vehicle, the vehicle control unit 1021 generates a command to decelerate the vehicle.
  • the data extraction unit 1022 extracts (selects) data stored in the memory 103 in response to a request from the automatic operation ECU 200 received by a request reception unit 1051 described later. That is, the data extraction unit 1022 extracts predetermined data from a plurality of types of data. In other words, the data extraction unit 1022 extracts data in response to a request.
  • the communication circuit 105 functions as a request receiving unit 1051 and a data output unit 1052.
  • Request receiving unit 1051 receives a request from external automatic driving ECU 200.
  • Data output unit 1052 outputs the data extracted by data extraction unit 1022 to external automatic operation ECU 200. In other words, the data output unit 1052 outputs data in response to a request.
  • the image correction unit 1041, the parallax calculation unit 1042, the three-dimensional object extraction unit 1043, and the three-dimensional grouping unit 1044 constitute a data generation unit 104a that generates a plurality of types of data from the image captured by the imaging unit 101.
  • FIG. 4 is a diagram illustrating an example of a communication interface.
  • FIG. 4 shows a “request” input from the automatic operation ECU 200 to the communication circuit 105.
  • the third row of the table of FIG. 4 shows “data according to request” output from the communication circuit 105 to the automatic operation ECU 200.
  • the request from the automatic operation ECU 200 includes a coordinate value on the image (information indicating a position on the image) as information indicating a region of interest, a type ID as information indicating output content (type of data), and an output resolution (resolution).
  • a coordinate value on the image information indicating a position on the image
  • a type ID information indicating output content
  • an output resolution resolution
  • the number of images to be thinned out or the number of output objects, and the number of outputs per unit time (seconds) as information indicating the output frequency are included.
  • the attention area is, for example, a part estimated by the automatic operation ECU 200 that there is a three-dimensional object based on the distance, angle, relative speed, and the like of the target object detected by sensors such as the millimeter wave radar 300 and the LIDAR 400.
  • the type ID indicates the type of data to be output.
  • data types for example, raw images (RAW data) captured by the imaging unit 101, parallax images (first data), three-dimensional object raw information (second data), and three-dimensional object grouping information (third data) are included. is there.
  • FIG. 5 is a flowchart showing the operation of the stereo camera 100 according to the embodiment of the present invention.
  • the stereo camera 100 receives the request shown in FIG. 4 from the automatic driving ECU 200 (S10).
  • the stereo camera 100 extracts (selects) data stored in the memory 103 in response to the request received in S10 (S15).
  • Stereo camera 100 outputs the data extracted in S15 (S20). For example, in response to the request illustrated in FIG. 4, the stereo camera 100 reads out the data of the type indicated by the “type ID” from the memory 103 for the attention area indicated by the “coordinate value on the image”. Data is output (transmitted) to the automatic operation ECU 200 at an output resolution indicated by “Decimation degree / number of outputs” and an output frequency indicated by “times / s”. Note that the stereo camera 100 may output synchronization information.
  • the data extraction unit 1022 extracts “coordinate values on the image” (position on the image indicated by the position information) included in the request. Thereby, the automatic driving ECU 200 can acquire data of only the attention area.
  • the data extraction unit 1022 extracts the type of data indicated by the “type ID” included in the request. As a result, the automatic operation ECU 200 can acquire only a desired type of data.
  • the data output unit 1052 outputs the data extracted by the data extraction unit 1022 to the external automatic operation ECU 200 at an output resolution (resolution) indicated by “Decimation degree / number of outputs” included in the request. Thereby, the automatic driving ECU 200 can acquire data of a desired output resolution.
  • the data output unit 1052 outputs the data extracted by the data extraction unit 1022 to the outside at an output frequency indicated by “times / s” included in the request. Thereby, automatic driving ECU200 can acquire data with desired output frequency.
  • the data output unit 1052 outputs the data extracted by the data extraction unit 1022 to the external automatic operation ECU 200 via Ethernet. As a result, the transfer speed is improved as compared with CAN or the like.
  • the data generation unit 104a generates a parallax image (first data) indicating parallax or a distance corresponding to the parallax, and uses the three-dimensional object data as the three-dimensional object raw information (second data) from the parallax image (first data). Extracted and generated three-dimensional object grouping information (third data) grouped from the three-dimensional object raw information (second data).
  • the data extraction unit 1022 converts the type of data indicated by the type ID into a raw image (RAW data), a parallax image (first data), three-dimensional object raw information (second data), and three-dimensional object grouping information (third data). Extract (select) from a raw image (RAW data), a parallax image (first data), three-dimensional object raw information (second data), and three-dimensional object grouping information (third data). Extract (select) from a raw image (RAW data), a parallax image (first data), three-dimensional object raw information (second data), and three-dimensional object grouping information (third data). Extract (select) from a raw image (RAW data), a parallax image (first data), three-dimensional object raw information (second data), and three-dimensional object grouping information (third data). Extract (select) from a raw image (RAW data), a parallax image (first data), three-dimensional object raw information (second data), and three-dimensional object grouping information (third
  • the object identification unit 1045 identifies an object from the image captured by the imaging unit 101.
  • the data extraction unit 1022 extracts data of an object associated with an importance level equal to or higher than a threshold value.
  • the system 1 includes a stereo camera 100 (external world recognition device), sensors such as millimeter wave radar 300 and LIDAR 400 that detect the position of an object using electromagnetic waves, and an automatic operation ECU 200 (control device).
  • a stereo camera 100 external world recognition device
  • sensors such as millimeter wave radar 300 and LIDAR 400 that detect the position of an object using electromagnetic waves
  • an automatic operation ECU 200 control device
  • the automatic operation ECU 200 converts the position of the object detected by the sensor into a position on the image, transmits a request including the position information of the position to the stereo camera 100 (external world recognition device), and a data extraction unit
  • the data extracted by 1022 is received from the stereo camera 100 (external recognition device), and the vehicle is controlled based on the received data.
  • the automatic operation ECU 200 can extract data of various layers (the imaging unit 101, the image correction unit 1041, the three-dimensional grouping unit 1044, and the like).
  • transfer delay can be suppressed when image data is output to the outside.
  • the volume of data output from a stereo camera with high pixels can be reduced.
  • FIG. 6 is a block diagram illustrating functions of a stereo camera 100A according to a modification. Note that the hardware configuration of the stereo camera 100A of this modification is the same as that of the stereo camera 100 shown in FIG.
  • the communication circuit 105 does not have the request reception unit 1051.
  • the data extraction unit 1022A of the present modification extracts data on the position (coordinate value) on the image of the object identified by the object identification unit 1045 and associated with an importance level equal to or higher than the threshold. That is, the data extraction unit 1022A extracts data of a highly important object (pedestrian, dangerous part).
  • the importance level and the object type are stored in association with the memory 103, for example.
  • FIG. 7 is a flowchart showing the operation of the modified stereo camera 100A.
  • Stereo camera 100A identifies an object (S13).
  • the stereo camera 100A extracts data of objects (pedestrians and dangerous parts) with high importance among the objects identified in S13 (S18).
  • the stereo camera 100A outputs the data extracted in S18 (S20).
  • the stereo camera 100 is adopted as the external recognition device, but the number of image sensors is arbitrary, and a monocular camera may be used.
  • millimeter wave radar and LIDAR are exemplified as the sensor, but the sensor may be a sonar that detects the distance, angle, relative speed, and the like of the target object using sound waves.
  • images such as raw images (RAW data), parallax images (first data), three-dimensional object raw information (second data), and three-dimensional object grouping information (third data).
  • RAW data raw images
  • first data parallax images
  • second data three-dimensional object raw information
  • third data three-dimensional object grouping information
  • the data related to the above is stored in the memory 103, but may be stored in the built-in memory of the image processing unit 104.
  • the image correction unit 1041, the parallax calculation unit 1042, the three-dimensional object extraction unit 1043, the three-dimensional grouping unit 1044, and the object identification unit 1045 are realized as functions of the image processing unit 104 as an example. It may be realized. Note that each of the image correction unit 1041, the parallax calculation unit 1042, the three-dimensional object extraction unit 1043, the three-dimensional grouping unit 1044, and the object identification unit 1045 may be configured as a circuit such as an ASIC.
  • Ethernet is adopted as a standard between the stereo camera 100 and the automatic operation ECU 200, but other standards may be used as long as a transfer rate equal to or higher than that of the Ethernet can be realized.
  • the “request” from the automatic operation ECU 200 includes the coordinate value on the image, but may include the coordinate value of another coordinate system such as the world coordinate system instead of the coordinate value on the image.
  • the CPU 102 of the stereo camera 100 converts coordinate values in the world coordinate system into coordinate values on the image.
  • the vehicle speed may be included instead of the coordinate value on the image.
  • the data extraction unit 1022 extracts (selects), for example, a high resolution image and a parallax image, and extracts (selects) a low resolution image and a parallax image.
  • the request from the automatic operation ECU 200 includes information specifying either or both of the first image data captured by the left imaging unit 101L and the second image data captured by the right imaging unit 101R. Also good.
  • the data extraction units 1022 and 1022A extract (select) the data of the designated image.
  • the vehicle speed is detected by a vehicle speed sensor.
  • the automatic operation ECU 200 refers to the parallax image (first data), the three-dimensional object raw information (second data), the three-dimensional object grouping information (third data), and the like received from the stereo camera 100, and there is a detected object candidate.
  • the coordinate value of the image coordinate system or the world coordinate system corresponding to the position as information indicating the region of interest may be included in the “request” and input to the stereo camera 100 again.
  • each of the above-described configurations, functions, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit.
  • Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by a processor (CPU).
  • Information such as programs, tables, and files for realizing each function can be stored in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.
  • SYMBOLS 1 DESCRIPTION OF SYMBOLS 1 ... System, 100, 100A ... Stereo camera, 101 ... Imaging part, 101L ... Left imaging part, 101R ... Right imaging part, 103 ... Memory, 104 ... Image processing part, 104a ... Data generation part, 105 ... Communication circuit, 300 ... millimeter wave radar, 400 ... LIDAR, 1021 ... vehicle control unit, 1022 ... data extraction unit, 1022A ... data extraction unit, 1041 ... image correction unit, 1042 ... parallax calculation unit, 1043 ... three-dimensional object extraction unit, 1044 ... three-dimensional grouping , 1045 ... object identification part, 1051 ... request reception part, 1052 ... data output part

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Automation & Control Theory (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Mathematical Physics (AREA)
  • Traffic Control Systems (AREA)
  • Control Of Driving Devices And Active Controlling Of Vehicle (AREA)
  • Image Processing (AREA)

Abstract

The present invention provides an external environment recognition device with which a transfer delay when externally outputting image data can be alleviated. A stereo camera 100 (external environment recognition device) according to the present invention comprises an image capture part 101, a data generation part 104a, a data extraction part 1022, and a data output part 1052. The image capture part 101 captures images. The data generation part 104a generates a plurality of types of data from the images captured by the image capture part 101. The data extraction part 1022 extracts data from the plurality of types of data. The data output part 1052 externally outputs the data extracted by the data extraction part 1022.

Description

外界認識装置External recognition device
　本発明は、外界認識装置に関する。 The present invention relates to an external recognition device.
　カメラ画像による情報分析の信頼性を向上させることができる車両用画像処理装置が知られている（例えば、特許文献１参照）。 A vehicular image processing apparatus capable of improving the reliability of information analysis using camera images is known (see, for example, Patent Document 1).
　特許文献１には、「自車前方を撮像する撮像手段により撮像された画像の中で、前方車両または道路白線または標識を含む注目領域とそれ以外の非注目領域とを設定し、撮像手段により撮像された画像の中の注目領域と非注目領域における輝度情報を検出する。そして、注目領域と非注目領域の輝度情報に基づいて画像処理による自車前方の状況分析が困難な走行環境にあるか否かを判定する。」と記載されている。 Patent Document 1 states that “a region of interest including a preceding vehicle or a road white line or a sign and other non-attention regions are set in an image captured by an imaging unit that images the front of the host vehicle, and the imaging unit Luminance information in the attention area and non-attention area in the captured image is detected, and the vehicle is in a driving environment where it is difficult to analyze the situation ahead of the vehicle by image processing based on the luminance information of the attention area and the non-attention area. It is determined whether or not.
特開2005-135308号公報JP 2005-135308 A
　近年、ステレオカメラ等の外界認識装置の高画素化に伴い、画像の容量が大きくなってきている。そのため、画像のデータを外部へ出力するときに転送遅延が生じるおそれがある。特許文献１では、このような転送遅延は考慮されていない。 In recent years, the capacity of images has increased with the increase in the number of pixels in external recognition devices such as stereo cameras. Therefore, there is a possibility that a transfer delay occurs when image data is output to the outside. In Patent Document 1, such a transfer delay is not considered.
　本発明の目的は、画像のデータを外部に出力するときに転送遅延を抑制することができる外界認識装置を提供することにある。 An object of the present invention is to provide an external recognition apparatus capable of suppressing a transfer delay when outputting image data to the outside.
　上記目的を達成するために、本発明は、画像を撮像する撮像部と、前記撮像部によって撮像された前記画像から複数種類のデータを生成するデータ生成部と、前記複数種類のデータからデータを抽出するデータ抽出部と、前記データ抽出部によって抽出されたデータを外部へ出力するデータ出力部と、備える。 To achieve the above object, the present invention provides an imaging unit that captures an image, a data generation unit that generates a plurality of types of data from the image captured by the imaging unit, and data from the plurality of types of data. A data extraction unit for extracting; and a data output unit for outputting the data extracted by the data extraction unit to the outside.
　本発明によれば、画像のデータを外部に出力するときに転送遅延を抑制することができる。上記した以外の課題、構成及び効果は、以下の実施形態の説明により明らかにされる。 According to the present invention, transfer delay can be suppressed when image data is output to the outside. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.
本発明の実施形態によるステレオカメラを含むシステムの構成図である。1 is a configuration diagram of a system including a stereo camera according to an embodiment of the present invention. 本発明の実施形態によるステレオカメラの構成図である。It is a block diagram of the stereo camera by embodiment of this invention. 本発明の実施形態によるステレオカメラの機能を示すブロック図である。It is a block diagram which shows the function of the stereo camera by embodiment of this invention. 通信インタフェースの例を示す図である。It is a figure which shows the example of a communication interface. 本発明の実施形態によるステレオカメラの動作を示すフローチャートである。It is a flowchart which shows operation | movement of the stereo camera by embodiment of this invention. 変形例のステレオカメラの機能を示すブロック図である。It is a block diagram which shows the function of the stereo camera of a modification. 変形例のステレオカメラの動作を示すフローチャートである。It is a flowchart which shows operation | movement of the stereo camera of a modification.
　以下、図面を用いて、本発明の実施形態によるステレオカメラ（外界認識装置）の構成及び動作について説明する。なお、各図において、同一符号は同一部分を示す。 Hereinafter, the configuration and operation of the stereo camera (external recognition device) according to the embodiment of the present invention will be described with reference to the drawings. In each figure, the same numerals indicate the same parts.
　（実施形態）
  初めに、図１を用いて、システムの構成を説明する。図１は、本発明の実施形態によるステレオカメラ１００を含むシステム１の構成図である。 (Embodiment)
First, the configuration of the system will be described with reference to FIG. FIG. 1 is a configuration diagram of a system 1 including a stereo camera 100 according to an embodiment of the present invention.
　システム１は、ステレオカメラ１００、自動運転ＥＣＵ２００（Electronic Control Unit）、ミリ波レーダ３００、ＬＩＤＡＲ４００（Light Detecting And Ranging）等を備える。 The system 1 includes a stereo camera 100, an automatic operation ECU 200 (Electronic Control Unit), a millimeter wave radar 300, a LIDAR 400 (Light Detection Detection And And Ranging), and the like.
　ステレオカメラ１００は、同一の対象物体を異なる複数の視点から撮像し、得られた画像における見え方のずれ（視差）から対象物体までの距離を算出する。ステレオカメラ１００は、一例として、イーサネット（登録商標）に準拠したＬＡＮ（Local Area Network）ケーブルで自動運転ＥＣＵ２００に接続される。ここで、イーサネット（登録商標）は、後述するＣＡＮ（Controller Area Network）よりも転送速度が速いため、大容量のデータを送信することに適している。 The stereo camera 100 images the same target object from a plurality of different viewpoints, and calculates the distance to the target object from the deviation of the appearance (parallax) in the obtained image. As an example, the stereo camera 100 is connected to the automatic operation ECU 200 with a LAN (Local Area Network) cable compliant with Ethernet (registered trademark). Here, since Ethernet (registered trademark) has a higher transfer speed than CAN (Controller Area 後 述 Network) described later, it is suitable for transmitting a large amount of data.
　自動運転ＥＣＵ２００は、ミリ波レーダ３００、ＬＩＤＡＲ４００等のセンサによって検出される対象物体の距離、角度、相対速度等に基づいて、車両の自動運転を制御する。
自動運転ＥＣＵ２００は、一例として、ＣＡＮに準拠した２線式バスでミリ波レーダ３００、ＬＩＤＡＲ４００等のセンサと接続される。 The automatic driving ECU 200 controls automatic driving of the vehicle based on the distance, angle, relative speed, and the like of the target object detected by sensors such as the millimeter wave radar 300 and the LIDAR 400.
For example, the automatic operation ECU 200 is connected to sensors such as the millimeter wave radar 300 and the LIDAR 400 via a CAN-compliant two-wire bus.
　ミリ波レーダ３００は、ミリ波（電磁波）を用いて、対象物体の距離、角度、相対速度等を検出する。ＬＩＤＡＲ４００は、紫外線、可視光線、近赤外線等の光波（電磁波）を用いて、対象物体の距離、角度、相対速度等を検出する。 Millimeter wave radar 300 detects the distance, angle, relative speed, etc. of the target object using millimeter waves (electromagnetic waves). The LIDAR 400 detects the distance, angle, relative speed, and the like of the target object using light waves (electromagnetic waves) such as ultraviolet rays, visible rays, and near infrared rays.
　次に、図２を用いて、ステレオカメラ１００の構成を説明する。図２は、本発明の実施形態によるステレオカメラ１００の構成図である。 Next, the configuration of the stereo camera 100 will be described with reference to FIG. FIG. 2 is a configuration diagram of the stereo camera 100 according to the embodiment of the present invention.
　ステレオカメラ１００は、一例として、撮像部１０１、ＣＰＵ１０２（Central Processing Unit）、メモリ１０３、画像処理部１０４、通信回路１０５から構成される。 As an example, the stereo camera 100 includes an imaging unit 101, a CPU 102 (Central Processing Unit), a memory 103, an image processing unit 104, and a communication circuit 105.
　撮像部１０１は、左撮像部１０１Ｌ、右撮像部１０１Ｒから構成され、画像を撮像する。詳細には、左撮像部１０１Ｌと右撮像部１０１Ｒはそれぞれ、レンズ等の光学素子とＣＣＤ（Charge Coupled Device）、ＣＭＯＳ（Complementary Metal Oxide Semiconductor）等の撮像素子から構成される。光学素子は、光を屈折させて、撮像素子上に像を結ぶ。
  撮像素子は、光学素子により屈折した光の像を受光し、その光の強さに応じた画像を生成する。 The imaging unit 101 includes a left imaging unit 101L and a right imaging unit 101R, and captures an image. Specifically, each of the left imaging unit 101L and the right imaging unit 101R includes an optical element such as a lens and an imaging element such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor). The optical element refracts light and forms an image on the imaging element.
The image sensor receives an image of light refracted by the optical element and generates an image corresponding to the intensity of the light.
　ＣＰＵ１０２は、所定のプログラムを実行することで、後述する種々の機能を実現する。本実施形態では、ＣＰＵ１０２は、例えば、対象物体の種別（歩行者、車両等）に基づいて車両制御に関する指令を生成する。 The CPU 102 implements various functions to be described later by executing a predetermined program. In this embodiment, CPU102 produces | generates the command regarding vehicle control based on the classification (pedestrian, vehicle, etc.) of a target object, for example.
　メモリ１０３は、例えば、ＦＬＡＳＨ等の不揮発性メモリ、ＲＡＭ（Random Access Memory）等の揮発性メモリから構成され、種々の情報を記憶する。 The memory 103 includes, for example, a nonvolatile memory such as FLASH and a volatile memory such as a RAM (Random Access Memory), and stores various information.
　画像処理部１０４は、ＦＰＧＡ（Field-Programmable Gate Array）、ＤＳＰ（Digital Signal Processor）、ＡＳＩＣ（Application Specific Integrated Circuit）等の回路で構成される。画像処理部１０４は、例えば、画像の補正、視差の算出等を行う。画像処理部１０４の機能の詳細は、図３を用いて後述する。 The image processing unit 104 includes a circuit such as an FPGA (Field-Programmable Gate Array), a DSP (Digital Signal Processor), and an ASIC (Application Specific Integrated Circuit). The image processing unit 104 performs image correction, parallax calculation, and the like, for example. Details of the function of the image processing unit 104 will be described later with reference to FIG.
　通信回路１０５は、一例として、イーサネットに準拠したトランシーバ等から構成される。通信回路１０５は、種々のデータを送信又は受信する。 The communication circuit 105 includes, for example, an Ethernet-compliant transceiver. The communication circuit 105 transmits or receives various data.
　次に、図３を用いて、ステレオカメラ１００の機能を説明する。図３は、本発明の実施形態によるステレオカメラ１００の機能を示すブロック図である。 Next, functions of the stereo camera 100 will be described with reference to FIG. FIG. 3 is a block diagram illustrating functions of the stereo camera 100 according to the embodiment of the present invention.
　左撮像部１０１Ｌは、対象物体の第１の画像を撮像する。右撮像部１０１Ｒは、対象物体の第２の画像を撮像する。撮像された第１の画像と第２の画像は、「生画像」（ＲＡＷデータ）として、例えば、メモリ１０３に一時的に記憶される。 The left imaging unit 101L captures a first image of the target object. The right imaging unit 101R captures a second image of the target object. The captured first image and second image are temporarily stored, for example, in the memory 103 as “raw images” (RAW data).
　画像処理部１０４は、画像補正部１０４１、視差算出部１０４２、立体物抽出部１０４３、立体グルーピング部１０４４、物体識別部１０４５を備える。なお、画像補正部１０４１、視差算出部１０４２、立体物抽出部１０４３、立体グルーピング部１０４４、物体識別部１０４５は、それぞれ別体に構成されていてもよい。 The image processing unit 104 includes an image correction unit 1041, a parallax calculation unit 1042, a three-dimensional object extraction unit 1043, a three-dimensional grouping unit 1044, and an object identification unit 1045. Note that the image correction unit 1041, the parallax calculation unit 1042, the three-dimensional object extraction unit 1043, the three-dimensional grouping unit 1044, and the object identification unit 1045 may be configured separately.
　画像補正部１０４１は、左撮像部１０１Ｌによって撮像された第１の画像と右撮像部１０１Ｒによって撮像された第２の画像を補正する。具体的には、画像補正部１０４１は、例えば、第１の画像と第２の画像の歪みや光軸ずれを補正する。補正された第１の画像と第２の画像は、「補正画像」として、例えば、メモリ１０３に一時的に記憶される。 The image correction unit 1041 corrects the first image captured by the left imaging unit 101L and the second image captured by the right imaging unit 101R. Specifically, the image correction unit 1041 corrects, for example, distortion and optical axis deviation between the first image and the second image. The corrected first image and second image are temporarily stored, for example, in the memory 103 as “corrected images”.
　視差算出部１０４２は、ステレオマッチングを行うことで、画像補正部１０４１によって補正された第１の画像と第２の画像から視差を算出する。視差算出部１０４２は、三角測量により視差から対象物体までの距離を算出する。視差算出部１０４２は、画像上の位置と視差に対応する距離とを関連付けた視差画像を生成する。視差画像（第１データ）は、例えば、メモリ１０３に記憶される。 The parallax calculation unit 1042 calculates parallax from the first image and the second image corrected by the image correction unit 1041 by performing stereo matching. The parallax calculation unit 1042 calculates the distance from the parallax to the target object by triangulation. The parallax calculation unit 1042 generates a parallax image in which a position on the image is associated with a distance corresponding to the parallax. The parallax image (first data) is stored in the memory 103, for example.
　立体物抽出部１０４３は、視差算出部１０４２によって生成された視差画像から、同等の距離に位置する物体を１つの立体物として抽出する。抽出された立体物のデータである立体物生情報（第２データ）は、例えば、メモリ１０３に記憶される。 The three-dimensional object extraction unit 1043 extracts an object located at an equivalent distance from the parallax image generated by the parallax calculation unit 1042 as one solid object. The three-dimensional object raw information (second data) that is data of the extracted three-dimensional object is stored in the memory 103, for example.
　立体グルーピング部１０４４は、立体物抽出部１０４３によって抽出された立体物を、輝度、エッジ等に基づいてグルーピングする。グルーピングされた立体物のデータである立体物グルーピング後情報（第３データ）は、例えば、メモリ１０３に記憶される。 The three-dimensional grouping unit 1044 groups the three-dimensional objects extracted by the three-dimensional object extraction unit 1043 based on luminance, edges, and the like. The post-three-dimensional object grouping information (third data) that is data of the grouped three-dimensional object is stored in the memory 103, for example.
　物体識別部１０４５は、立体グルーピング部１０４４によってグルーピングされた立体物から対象物体が何（歩行者、車両等）であるかを識別する。 The object identification unit 1045 identifies what the target object is (a pedestrian, a vehicle, etc.) from the three-dimensional object grouped by the three-dimensional grouping unit 1044.
　ＣＰＵ１０２は、所定のプログラムを実行することで、車両制御部１０２１とデータ抽出部１０２２として機能する。車両制御部１０２１は、物体識別部１０４５によって識別された物体の種別に応じて車両を制御する指令を生成する。例えば、車両制御部１０２１は、歩行者が車両の進路上にいると判定した場合、車両を減速する指令を生成する。 The CPU 102 functions as a vehicle control unit 1021 and a data extraction unit 1022 by executing a predetermined program. The vehicle control unit 1021 generates a command for controlling the vehicle according to the type of the object identified by the object identification unit 1045. For example, if the vehicle control unit 1021 determines that the pedestrian is on the path of the vehicle, the vehicle control unit 1021 generates a command to decelerate the vehicle.
　データ抽出部１０２２は、後述する要求受付部１０５１によって受け付けられた自動運転ＥＣＵ２００からの要求に応じてメモリ１０３に記憶されたデータを抽出（選択）する。すなわち、データ抽出部１０２２は、複数種類のデータから所定のデータを抽出する。
  換言すれば、データ抽出部１０２２は、要求に応じてデータを抽出する。 The data extraction unit 1022 extracts (selects) data stored in the memory 103 in response to a request from the automatic operation ECU 200 received by a request reception unit 1051 described later. That is, the data extraction unit 1022 extracts predetermined data from a plurality of types of data.
In other words, the data extraction unit 1022 extracts data in response to a request.
　通信回路１０５は、要求受付部１０５１とデータ出力部１０５２として機能する。要求受付部１０５１は、外部の自動運転ＥＣＵ２００から要求を受け付ける。データ出力部１０５２は、データ抽出部１０２２によって抽出されたデータを外部の自動運転ＥＣＵ２００へ出力する。換言すれば、データ出力部１０５２は、要求に応じてデータを出力する。 The communication circuit 105 functions as a request receiving unit 1051 and a data output unit 1052. Request receiving unit 1051 receives a request from external automatic driving ECU 200. Data output unit 1052 outputs the data extracted by data extraction unit 1022 to external automatic operation ECU 200. In other words, the data output unit 1052 outputs data in response to a request.
　なお、画像補正部１０４１、視差算出部１０４２、立体物抽出部１０４３、及び立体グルーピング部１０４４は、撮像部１０１によって撮像された画像から複数種類のデータを生成するデータ生成部１０４ａを構成する。 The image correction unit 1041, the parallax calculation unit 1042, the three-dimensional object extraction unit 1043, and the three-dimensional grouping unit 1044 constitute a data generation unit 104a that generates a plurality of types of data from the image captured by the imaging unit 101.
　次に、図４を用いて、通信回路１０５の通信インタフェースを説明する。図４は、通信インタフェースの例を示す図である。 Next, the communication interface of the communication circuit 105 will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of a communication interface.
　図４の表の２行目は、自動運転ＥＣＵ２００から通信回路１０５へ入力される「要求」を示す。図４の表の３行目は、通信回路１０５から自動運転ＥＣＵ２００へ出力される「要求に応じたデータ」を示す。 4 shows a “request” input from the automatic operation ECU 200 to the communication circuit 105. The third row of the table of FIG. 4 shows “data according to request” output from the communication circuit 105 to the automatic operation ECU 200.
　自動運転ＥＣＵ２００からの要求は、注目領域を示す情報としての画像上の座標値（画像上の位置を示す情報）、出力内容（データの種類）を示す情報としての種別ＩＤ、出力解像度（解像度）を示す情報としての画像の間引き度合若しくはオブジェクトの出力個数、出力頻度を示す情報としての単位時間（秒）あたりの出力回数（回／s）を含む。 The request from the automatic operation ECU 200 includes a coordinate value on the image (information indicating a position on the image) as information indicating a region of interest, a type ID as information indicating output content (type of data), and an output resolution (resolution). The number of images to be thinned out or the number of output objects, and the number of outputs per unit time (seconds) as information indicating the output frequency are included.
　注目領域は、例えば、ミリ波レーダ３００、ＬＩＤＡＲ４００等のセンサによって検出される対象物体の距離、角度、相対速度等に基づいて自動運転ＥＣＵ２００によって立体物があるだろうと推定された部位である。種別ＩＤは、出力されるべきデータの種別を示す。データの種別として、例えば、撮像部１０１で撮像された生画像（ＲＡＷデータ）、視差画像（第１データ）、立体物生情報（第２データ）、立体物グルーピング後情報（第３データ）がある。 The attention area is, for example, a part estimated by the automatic operation ECU 200 that there is a three-dimensional object based on the distance, angle, relative speed, and the like of the target object detected by sensors such as the millimeter wave radar 300 and the LIDAR 400. The type ID indicates the type of data to be output. As data types, for example, raw images (RAW data) captured by the imaging unit 101, parallax images (first data), three-dimensional object raw information (second data), and three-dimensional object grouping information (third data) are included. is there.
　次に、図５を用いて、ステレオカメラ１００の動作を説明する。図５は、本発明の実施形態によるステレオカメラ１００の動作を示すフローチャートである。 Next, the operation of the stereo camera 100 will be described with reference to FIG. FIG. 5 is a flowchart showing the operation of the stereo camera 100 according to the embodiment of the present invention.
　ステレオカメラ１００は、自動運転ＥＣＵ２００から図４に示した要求を受け付ける（Ｓ１０）。ステレオカメラ１００は、Ｓ１０で受け付けた要求に応じてメモリ１０３に記憶されたデータを抽出（選択）する（Ｓ１５）。 The stereo camera 100 receives the request shown in FIG. 4 from the automatic driving ECU 200 (S10). The stereo camera 100 extracts (selects) data stored in the memory 103 in response to the request received in S10 (S15).
　ステレオカメラ１００は、Ｓ１５で抽出したデータを出力する（Ｓ２０）。例えば、図４に示した要求に対して、ステレオカメラ１００は、「画像上の座標値」によって示される注目領域について、「種別ＩＤ」によって示される種類のデータを、メモリ１０３から読み出し、読み出したデータを「画像の間引き度合／出力個数」によって示される出力解像度かつ「回／ｓ」で示される出力頻度で自動運転ＥＣＵ２００へ出力（送信）する。なお、ステレオカメラ１００は、同期情報を出力してもよい。 Stereo camera 100 outputs the data extracted in S15 (S20). For example, in response to the request illustrated in FIG. 4, the stereo camera 100 reads out the data of the type indicated by the “type ID” from the memory 103 for the attention area indicated by the “coordinate value on the image”. Data is output (transmitted) to the automatic operation ECU 200 at an output resolution indicated by “Decimation degree / number of outputs” and an output frequency indicated by “times / s”. Note that the stereo camera 100 may output synchronization information.
　ここで、データ抽出部１０２２は、要求に含まれる「画像上の座標値」（位置情報が示す画像上の位置）のデータを抽出する。これにより、自動運転ＥＣＵ２００は、注目領域のみのデータを取得することができる。 Here, the data extraction unit 1022 extracts “coordinate values on the image” (position on the image indicated by the position information) included in the request. Thereby, the automatic driving ECU 200 can acquire data of only the attention area.
　データ抽出部１０２２は、要求に含まれる「種別ＩＤ」が示す種類のデータを抽出する。これにより、自動運転ＥＣＵ２００は、所望の種類のデータのみを取得することができる。 The data extraction unit 1022 extracts the type of data indicated by the “type ID” included in the request. As a result, the automatic operation ECU 200 can acquire only a desired type of data.
　データ出力部１０５２は、データ抽出部１０２２によって抽出されたデータを、要求に含まれる「画像の間引き度合／出力個数」によって示される出力解像度（解像度）で外部の自動運転ＥＣＵ２００へ出力する。これにより、自動運転ＥＣＵ２００は、所望の出力解像度のデータを取得することができる。 The data output unit 1052 outputs the data extracted by the data extraction unit 1022 to the external automatic operation ECU 200 at an output resolution (resolution) indicated by “Decimation degree / number of outputs” included in the request. Thereby, the automatic driving ECU 200 can acquire data of a desired output resolution.
　データ出力部１０５２は、データ抽出部１０２２によって抽出されたデータを、要求に含まれる「回／ｓ」で示される出力頻度で外部へ出力する。これにより、自動運転ＥＣＵ２００は、所望の出力頻度でデータを取得することができる。 The data output unit 1052 outputs the data extracted by the data extraction unit 1022 to the outside at an output frequency indicated by “times / s” included in the request. Thereby, automatic driving ECU200 can acquire data with desired output frequency.
　データ出力部１０５２は、データ抽出部１０２２によって抽出されたデータをイーサネットで外部の自動運転ＥＣＵ２００へ出力する。これにより、ＣＡＮ等に比較して転送速度が向上する。 The data output unit 1052 outputs the data extracted by the data extraction unit 1022 to the external automatic operation ECU 200 via Ethernet. As a result, the transfer speed is improved as compared with CAN or the like.
　データ生成部１０４ａは、視差、又は視差に対応する距離を示す視差画像（第１データ）を生成し、視差画像（第１データ）から立体物のデータを立体物生情報（第２データ）として抽出し、立体物生情報（第２データ）からグルーピングされた立体物グルーピング後情報（第３データ）を生成する。 The data generation unit 104a generates a parallax image (first data) indicating parallax or a distance corresponding to the parallax, and uses the three-dimensional object data as the three-dimensional object raw information (second data) from the parallax image (first data). Extracted and generated three-dimensional object grouping information (third data) grouped from the three-dimensional object raw information (second data).
　データ抽出部１０２２は、種別ＩＤが示す種類のデータを、生画像（ＲＡＷデータ）、視差画像（第１データ）、立体物生情報（第２データ）、立体物グルーピング後情報（第３データ）から抽出（選択）する。 The data extraction unit 1022 converts the type of data indicated by the type ID into a raw image (RAW data), a parallax image (first data), three-dimensional object raw information (second data), and three-dimensional object grouping information (third data). Extract (select) from
　物体識別部１０４５は、撮像部１０１によって撮像された画像から物体を識別する。データ抽出部１０２２は、閾値以上の重要度が関連付けられた物体のデータを抽出する。 The object identification unit 1045 identifies an object from the image captured by the imaging unit 101. The data extraction unit 1022 extracts data of an object associated with an importance level equal to or higher than a threshold value.
　システム１は、ステレオカメラ１００（外界認識装置）と、電磁波を用いて物体の位置を検出するミリ波レーダ３００、ＬＩＤＡＲ４００等のセンサと、自動運転ＥＣＵ２００（制御装置）と、を備える。 The system 1 includes a stereo camera 100 (external world recognition device), sensors such as millimeter wave radar 300 and LIDAR 400 that detect the position of an object using electromagnetic waves, and an automatic operation ECU 200 (control device).
　自動運転ＥＣＵ２００（制御装置）は、センサによって検出された物体の位置を画像上の位置に変換し、その位置の位置情報を含む要求をステレオカメラ１００（外界認識装置）へ送信し、データ抽出部１０２２によって抽出されたデータをステレオカメラ１００（外界認識装置）から受信し、受信したデータに基づいて車両を制御する。 The automatic operation ECU 200 (control device) converts the position of the object detected by the sensor into a position on the image, transmits a request including the position information of the position to the stereo camera 100 (external world recognition device), and a data extraction unit The data extracted by 1022 is received from the stereo camera 100 (external recognition device), and the vehicle is controlled based on the received data.
　このようにして、自動運転ＥＣＵ２００は、さまざまな階層（撮像部１０１、画像補正部１０４１から立体グルーピング部１０４４等）のデータを抽出することができる。 In this way, the automatic operation ECU 200 can extract data of various layers (the imaging unit 101, the image correction unit 1041, the three-dimensional grouping unit 1044, and the like).
　以上説明したように、本実施形態によれば、画像のデータを外部に出力するときに転送遅延を抑制することができる。また、高画素化されたステレオカメラから出力されるデータの容量を小さくすることができる。 As described above, according to this embodiment, transfer delay can be suppressed when image data is output to the outside. In addition, the volume of data output from a stereo camera with high pixels can be reduced.
　（変形例）
  次に、図６及び図７を用いて、変形例を説明する。図６は、変形例のステレオカメラ１００Ａの機能を示すブロック図である。なお、本変形例のステレオカメラ１００Ａのハードウェア構成は、図２に示したステレオカメラ１００と同じである。 (Modification)
Next, a modification will be described with reference to FIGS. FIG. 6 is a block diagram illustrating functions of a stereo camera 100A according to a modification. Note that the hardware configuration of the stereo camera 100A of this modification is the same as that of the stereo camera 100 shown in FIG.
　図６に示す本変形例では、通信回路１０５に要求受付部１０５１がない。本変形例のデータ抽出部１０２２Ａは、物体識別部１０４５によって識別され、閾値以上の重要度が関連付けられた物体の画像上の位置（座標値）のデータを抽出する。すなわち、データ抽出部１０２２Ａは、重要度の高い物体（歩行者、危険な部位）のデータを抽出する。なお、重要度と物体の種別は、例えば、メモリ１０３に関連付けて記憶される。 6, the communication circuit 105 does not have the request reception unit 1051. The data extraction unit 1022A of the present modification extracts data on the position (coordinate value) on the image of the object identified by the object identification unit 1045 and associated with an importance level equal to or higher than the threshold. That is, the data extraction unit 1022A extracts data of a highly important object (pedestrian, dangerous part). The importance level and the object type are stored in association with the memory 103, for example.
　図７は、変形例のステレオカメラ１００Ａの動作を示すフローチャートである。 FIG. 7 is a flowchart showing the operation of the modified stereo camera 100A.
　ステレオカメラ１００Ａは、物体を識別する（Ｓ１３）。ステレオカメラ１００Ａは、Ｓ１３で識別された物体のうち、重要度の高い物体（歩行者、危険な部位）のデータを抽出する（Ｓ１８）。ステレオカメラ１００Ａは、Ｓ１８で抽出したデータを出力する（Ｓ２０）。 Stereo camera 100A identifies an object (S13). The stereo camera 100A extracts data of objects (pedestrians and dangerous parts) with high importance among the objects identified in S13 (S18). The stereo camera 100A outputs the data extracted in S18 (S20).
　なお、本発明は上記した実施形態に限定されるものではなく、様々な変形例が含まれる。例えば、上述した実施形態は本発明を分かりやすく説明するために詳細に説明したものであり、必ずしも説明した全ての構成を備えるものに限定されるものではない。 Note that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to one having all the configurations described.
　上記実施形態では、外界認識装置としてステレオカメラ１００を採用したが、撮像素子の数は任意であり、単眼カメラであってもよい。 In the above embodiment, the stereo camera 100 is adopted as the external recognition device, but the number of image sensors is arbitrary, and a monocular camera may be used.
　上記実施形態では、センサとしてミリ波レーダ、ＬＩＤＡＲを例示したが、センサは、音波を用いて、対象物体の距離、角度、相対速度等を検出するソナーであってもよい。 In the above embodiment, millimeter wave radar and LIDAR are exemplified as the sensor, but the sensor may be a sonar that detects the distance, angle, relative speed, and the like of the target object using sound waves.
　上記実施形態では、理解を容易にするため、生画像（ＲＡＷデータ）、視差画像（第１データ）、立体物生情報（第２データ）、立体物グルーピング後情報（第３データ）等の画像に関するデータは、一例としてメモリ１０３に記憶されるものとしたが、画像処理部１０４の内蔵メモリに記憶されてもよい。 In the above embodiment, in order to facilitate understanding, images such as raw images (RAW data), parallax images (first data), three-dimensional object raw information (second data), and three-dimensional object grouping information (third data). As an example, the data related to the above is stored in the memory 103, but may be stored in the built-in memory of the image processing unit 104.
　上記実施形態では、画像補正部１０４１、視差算出部１０４２、立体物抽出部１０４３、立体グルーピング部１０４４、物体識別部１０４５は、一例として画像処理部１０４の機能として実現されるが、ＣＰＵ１０２の機能として実現されてもよい。なお、画像補正部１０４１、視差算出部１０４２、立体物抽出部１０４３、立体グルーピング部１０４４、物体識別部１０４５のそれぞれが、ＡＳＩＣ等の回路として構成されていてもよい。 In the above embodiment, the image correction unit 1041, the parallax calculation unit 1042, the three-dimensional object extraction unit 1043, the three-dimensional grouping unit 1044, and the object identification unit 1045 are realized as functions of the image processing unit 104 as an example. It may be realized. Note that each of the image correction unit 1041, the parallax calculation unit 1042, the three-dimensional object extraction unit 1043, the three-dimensional grouping unit 1044, and the object identification unit 1045 may be configured as a circuit such as an ASIC.
　上記実施形態では、ステレオカメラ１００と自動運転ＥＣＵ２００の間の規格としてイーサネットを採用したが、イーサネットと同等以上の転送速度を実現可能であれば他の規格であってもよい。 In the above embodiment, Ethernet is adopted as a standard between the stereo camera 100 and the automatic operation ECU 200, but other standards may be used as long as a transfer rate equal to or higher than that of the Ethernet can be realized.
　上記実施形態では、自動運転ＥＣＵ２００からの「要求」に画像上の座標値が含まれるが、画像上の座標値に代えて世界座標系等の他の座標系の座標値を含めてもよい。ステレオカメラ１００のＣＰＵ１０２は、世界座標系の座標値を画像上の座標値に変換する。また、画像上の座標値に代えて車速を含めてもよい。データ抽出部１０２２は、車速が閾値以上（高速）の場合、例えば、高解像度画像と視差画像を抽出（選択）し、低解像度画像と視差画像を抽出（選択）する。 In the above embodiment, the “request” from the automatic operation ECU 200 includes the coordinate value on the image, but may include the coordinate value of another coordinate system such as the world coordinate system instead of the coordinate value on the image. The CPU 102 of the stereo camera 100 converts coordinate values in the world coordinate system into coordinate values on the image. Further, the vehicle speed may be included instead of the coordinate value on the image. When the vehicle speed is equal to or higher than the threshold (high speed), the data extraction unit 1022 extracts (selects), for example, a high resolution image and a parallax image, and extracts (selects) a low resolution image and a parallax image.
　自動運転ＥＣＵ２００からの要求に、左撮像部１０１Ｌによって撮像された第１の画像のデータ、又は右撮像部１０１Ｒによって撮像された第２の画像のデータのいずれか又は両方を指定する情報を含めてもよい。データ抽出部１０２２、１０２２Ａは、指定された画像のデータを抽出（選択）する。なお、車速は車速センサによって検出される。 The request from the automatic operation ECU 200 includes information specifying either or both of the first image data captured by the left imaging unit 101L and the second image data captured by the right imaging unit 101R. Also good. The data extraction units 1022 and 1022A extract (select) the data of the designated image. The vehicle speed is detected by a vehicle speed sensor.
　自動運転ＥＣＵ２００は、ステレオカメラ１００から受信した視差画像（第１データ）、立体物生情報（第２データ）、立体物グルーピング後情報（第３データ）等を参照して、検知物体候補がある場合、注目領域を示す情報としてその位置に対応する画像座標系または世界座標系の座標値を「要求」に含めて、再度、ステレオカメラ１００に入力してもよい。 The automatic operation ECU 200 refers to the parallax image (first data), the three-dimensional object raw information (second data), the three-dimensional object grouping information (third data), and the like received from the stereo camera 100, and there is a detected object candidate. In this case, the coordinate value of the image coordinate system or the world coordinate system corresponding to the position as information indicating the region of interest may be included in the “request” and input to the stereo camera 100 again.
　また、上記の各構成、機能等は、それらの一部又は全部を、例えば集積回路で設計する等によりハードウェアで実現してもよい。また、上記の各構成、機能等は、プロセッサ（ＣＰＵ）がそれぞれの機能を実現するプログラムを解釈し、実行することによりソフトウェアで実現してもよい。各機能を実現するプログラム、テーブル、ファイル等の情報は、メモリや、ハードディスク、SSD（Solid State Drive）等の記録装置、または、ICカード、SDカード、DVD等の記録媒体に置くことができる。 In addition, each of the above-described configurations, functions, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by a processor (CPU). Information such as programs, tables, and files for realizing each function can be stored in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.
１…システム、１００、１００Ａ…ステレオカメラ、１０１…撮像部、１０１Ｌ…左撮像部、１０１Ｒ…右撮像部、１０３…メモリ、１０４…画像処理部、１０４ａ…データ生成部、１０５…通信回路、３００…ミリ波レーダ、４００…ＬＩＤＡＲ、１０２１…車両制御部、１０２２…データ抽出部、１０２２Ａ…データ抽出部、１０４１…画像補正部、１０４２…視差算出部、１０４３…立体物抽出部、１０４４…立体グルーピング部、１０４５…物体識別部、１０５１…要求受付部、１０５２…データ出力部 DESCRIPTION OF SYMBOLS 1 ... System, 100, 100A ... Stereo camera, 101 ... Imaging part, 101L ... Left imaging part, 101R ... Right imaging part, 103 ... Memory, 104 ... Image processing part, 104a ... Data generation part, 105 ... Communication circuit, 300 ... millimeter wave radar, 400 ... LIDAR, 1021 ... vehicle control unit, 1022 ... data extraction unit, 1022A ... data extraction unit, 1041 ... image correction unit, 1042 ... parallax calculation unit, 1043 ... three-dimensional object extraction unit, 1044 ... three-dimensional grouping , 1045 ... object identification part, 1051 ... request reception part, 1052 ... data output part

Claims (10)

  1. 　画像を撮像する撮像部と、
    　前記撮像部によって撮像された前記画像から複数種類のデータを生成するデータ生成部と、
    　前記複数種類のデータからデータを抽出するデータ抽出部と、
    　前記データ抽出部によって抽出されたデータを外部へ出力するデータ出力部と、
    　備えることを特徴とする外界認識装置。     An imaging unit that captures an image;
    A data generation unit that generates a plurality of types of data from the image captured by the imaging unit;
    A data extraction unit for extracting data from the plurality of types of data;
    A data output unit for outputting the data extracted by the data extraction unit to the outside;
    An external recognition apparatus comprising:
  2. 　請求項１に記載の外界認識装置であって、
    　外部からの要求を受け付ける要求受付部をさらに備え、
    　前記データ出力部は、前記要求に応じてデータを出力する
    　ことを特徴とする外界認識装置。     The external environment recognition device according to claim 1,
    It further includes a request accepting unit that accepts an external request,
    The said data output part outputs data according to the said request | requirement. The external field recognition apparatus characterized by the above-mentioned.
  3. 　請求項２に記載の外界認識装置であって、
    　前記データ抽出部は、前記要求に応じてデータを抽出する
    　ことを特徴とする外界認識装置。     The external environment recognition device according to claim 2,
    The data extraction unit extracts data in response to the request.
  4. 　請求項３に記載の外界認識装置であって、
    　前記要求は、画像上の位置を示す位置情報を含み、
    　前記データ抽出部は、
    　前記位置情報が示す画像上の位置のデータを抽出する
    　ことを特徴とする外界認識装置。     The external environment recognition device according to claim 3,
    The request includes position information indicating a position on the image,
    The data extraction unit
    An external environment recognition apparatus, wherein data of a position on an image indicated by the position information is extracted.
  5. 　請求項４に記載の外界認識装置であって、
    　前記要求は、
    　データの種類を示す種別ＩＤを含み、
    　前記データ抽出部は、前記種別ＩＤが示す種類のデータを抽出する
    　ことを特徴とする外界認識装置。     The external environment recognition device according to claim 4,
    The request is
    Including a type ID indicating the type of data,
    The external field recognition apparatus, wherein the data extraction unit extracts data of a type indicated by the type ID.
  6. 　請求項４に記載の外界認識装置であって、
    　前記要求は、
    　解像度を含み、
    　前記データ出力部は、前記データ抽出部によって抽出されたデータを前記解像度で外部へ出力することを特徴とする外界認識装置。     The external environment recognition device according to claim 4,
    The request is
    Including resolution,
    The outside recognition apparatus, wherein the data output unit outputs the data extracted by the data extraction unit to the outside at the resolution.
  7. 　請求項４に記載の外界認識装置であって、
    　前記要求は、出力頻度を含み、
    　前記データ出力部は、前記データ抽出部によって抽出されたデータを前記出力頻度で外部へ出力する
    　ことを特徴とする外界認識装置。     The external environment recognition device according to claim 4,
    The request includes an output frequency;
    The data output unit outputs the data extracted by the data extraction unit to the outside at the output frequency.
  8. 　請求項５に記載の外界認識装置であって、
    　前記データ生成部は、
    　視差、又は視差に対応する距離を示す第１データを生成し、
    　前記第１データから立体物のデータを第２データとして抽出し、
    　前記第２データからグルーピングされた第３データを生成し、
    　前記データ抽出部は、前記種別ＩＤが示す種類のデータを、前記画像のＲＡＷデータ、前記第１データ、前記第２データ、前記第３データから抽出する
    　ことを特徴とする外界認識装置。     The external environment recognition device according to claim 5,
    The data generator is
    Generating first data indicating a parallax or a distance corresponding to the parallax;
    Extracting three-dimensional object data from the first data as second data,
    Generating third data grouped from the second data;
    The external data recognition device, wherein the data extraction unit extracts data of a type indicated by the type ID from raw data of the image, the first data, the second data, and the third data.
  9. 　請求項１に記載の外界認識装置であって、
    　前記撮像部によって撮像された画像から物体を識別する物体識別部を備え、
    　前記データ抽出部は、閾値以上の重要度が関連付けられた前記物体のデータを抽出する
    　ことを特徴とする外界認識装置。     The external environment recognition device according to claim 1,
    An object identification unit for identifying an object from an image captured by the imaging unit;
    The data extraction unit extracts data of the object associated with an importance level equal to or higher than a threshold value.
  10. 　請求項４に記載の外界認識装置を含むシステムであって、
    　電磁波又は音波を用いて物体の位置を検出するセンサと、
    　前記センサによって検出された前記物体の位置を画像上の位置に変換し、その位置の位置情報を含む前記要求を前記外界認識装置へ送信し、前記データ抽出部によって抽出されたデータを前記外界認識装置から受信し、受信したデータに基づいて車両を制御する制御装置と、を備える
    　ことを特徴とする外界認識装置。     A system including the external environment recognition device according to claim 4,
    A sensor for detecting the position of an object using electromagnetic waves or sound waves;
    The position of the object detected by the sensor is converted into a position on an image, the request including the position information of the position is transmitted to the external environment recognition device, and the data extracted by the data extraction unit is recognized as the external environment And a control device that controls the vehicle based on the data received from the device.
PCT/JP2019/014915 2018-04-17 2019-04-04 External environment recognition device WO2019203000A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201980025028.1A CN111937035A (en) 2018-04-17 2019-04-04 External recognition device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018-079266 2018-04-17
JP2018079266A JP7106332B2 (en) 2018-04-17 2018-04-17 External recognition device

Publications (1)

Publication Number Publication Date
WO2019203000A1 true WO2019203000A1 (en) 2019-10-24

Family

ID=68238894

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/014915 WO2019203000A1 (en) 2018-04-17 2019-04-04 External environment recognition device

Country Status (3)

Country Link
JP (1) JP7106332B2 (en)
CN (1) CN111937035A (en)
WO (1) WO2019203000A1 (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014073322A1 (en) * 2012-11-08 2014-05-15 日立オートモティブシステムズ株式会社 Object detection device and object detection method
WO2017195650A1 (en) * 2016-05-13 2017-11-16 ソニー株式会社 Generation device and generation method, and reproduction device and reproduction method

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008059260A (en) * 2006-08-31 2008-03-13 Fujitsu Ltd Movement detection image creating device
JP2014130429A (en) * 2012-12-28 2014-07-10 Ricoh Co Ltd Photographing device and three-dimensional object area detection program
JP6098318B2 (en) * 2013-04-15 2017-03-22 オムロン株式会社 Image processing apparatus, image processing method, image processing program, and recording medium
JP6313646B2 (en) * 2014-04-24 2018-04-18 日立オートモティブシステムズ株式会社 External recognition device
JP2016019009A (en) * 2014-07-04 2016-02-01 日立オートモティブシステムズ株式会社 Information processing apparatus
WO2017130639A1 (en) * 2016-01-28 2017-08-03 株式会社リコー Image processing device, imaging device, mobile entity apparatus control system, image processing method, and program

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014073322A1 (en) * 2012-11-08 2014-05-15 日立オートモティブシステムズ株式会社 Object detection device and object detection method
WO2017195650A1 (en) * 2016-05-13 2017-11-16 ソニー株式会社 Generation device and generation method, and reproduction device and reproduction method

Also Published As

Publication number Publication date
JP2019185666A (en) 2019-10-24
JP7106332B2 (en) 2022-07-26
CN111937035A (en) 2020-11-13

Similar Documents

Publication Publication Date Title
TWI540462B (en) Gesture recognition method and electronic apparatus using the same
US11616900B2 (en) Electronic device for recording image as per multiple frame rates using camera and method for operating same
JP6156733B2 (en) Object recognition apparatus, device control system, vehicle, object recognition method, and object recognition program
JP6795027B2 (en) Information processing equipment, object recognition equipment, device control systems, moving objects, image processing methods and programs
EP2889641B1 (en) Image processing apparatus, image processing method, program and image processing system
CN106709895B (en) Image generation method and apparatus
US8224069B2 (en) Image processing apparatus, image matching method, and computer-readable recording medium
US20180348369A1 (en) Ranging module, ranging system, and method of controlling ranging module
JP6198113B2 (en) Object recognition apparatus, device control system, vehicle, object recognition method, and object recognition program
US20200322544A1 (en) Electronic device, method, and computer-readable medium for providing bokeh effect in video
CN110335302B (en) Depth map generated from a single sensor
KR102552923B1 (en) Electronic device for acquiring depth information using at least one of cameras or depth sensor
CN113170065A (en) Solid-state imaging element and imaging device
US20210117708A1 (en) Method for obtaining face data and electronic device therefor
KR102507746B1 (en) Method for generating plural information using camera to sense plural wave bandwidth and apparatus thereof
WO2019203000A1 (en) External environment recognition device
KR101420242B1 (en) vehicle detector and method using stereo camera
JP2010060371A (en) Object detection apparatus
CN111479078A (en) Image sensor chip, electronic device and method for operating image sensor chip
JP2018146495A (en) Object detection apparatus, object detection method, object detection program, imaging apparatus, and equipment control system
JP2006325094A (en) Image processor for vehicle
US20170078644A1 (en) Controller applied to a three-dimensional (3d) capture device and 3d image capture device
CN114731362A (en) Electronic device including camera and method thereof
JP6943092B2 (en) Information processing device, imaging device, device control system, moving object, information processing method, and information processing program
JP7379523B2 (en) image recognition device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19787580

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19787580

Country of ref document: EP

Kind code of ref document: A1