WO2020135730A1 - 车载控制单元、基于fpga的车辆自动驾驶方法及装置 - Google Patents

车载控制单元、基于fpga的车辆自动驾驶方法及装置 Download PDF

Info

Publication number
WO2020135730A1
WO2020135730A1 PCT/CN2019/129248 CN2019129248W WO2020135730A1 WO 2020135730 A1 WO2020135730 A1 WO 2020135730A1 CN 2019129248 W CN2019129248 W CN 2019129248W WO 2020135730 A1 WO2020135730 A1 WO 2020135730A1
Authority
WO
WIPO (PCT)
Prior art keywords
soc
fpga
arm
mcu
vehicle
Prior art date
Application number
PCT/CN2019/129248
Other languages
English (en)
French (fr)
Inventor
胡稼悦
Original Assignee
百度在线网络技术（北京）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 百度在线网络技术（北京）有限公司 filed Critical 百度在线网络技术（北京）有限公司
Priority to JP2020572668A priority Critical patent/JP7416731B2/ja
Priority to KR1020207024742A priority patent/KR102471010B1/ko
Priority to EP19903938.9A priority patent/EP3839686B1/en
Publication of WO2020135730A1 publication Critical patent/WO2020135730A1/zh
Priority to US17/024,137 priority patent/US20210001880A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/183—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00—Computer-aided design [CAD]
    • G06F30/30—Circuit design
    • G06F30/32—Circuit design at the digital level
    • G06F30/33—Design verification, e.g. functional simulation or model checking
    • G06F30/3308—Design verification, e.g. functional simulation or model checking using simulation
    • G06F30/331—Design verification, e.g. functional simulation or model checking using simulation with hardware acceleration, e.g. by using field programmable gate array [FPGA] or emulation
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W60/00—Drive control systems specially adapted for autonomous road vehicles
    • B60W60/001—Planning or execution of driving tasks
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G01S13/867—Combination of radar systems with cameras
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/93—Radar or analogous systems specially adapted for specific applications for anti-collision purposes
    • G01S13/931—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S15/00—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems
    • G01S15/86—Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S15/00—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems
    • G01S15/88—Sonar systems specially adapted for specific applications
    • G01S15/93—Sonar systems specially adapted for specific applications for anti-collision purposes
    • G01S15/931—Sonar systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/0088—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
    • G05D1/0251—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0255—Control of position or course in two dimensions specially adapted to land vehicles using acoustic signals, e.g. ultra-sonic singals
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0257—Control of position or course in two dimensions specially adapted to land vehicles using a radar
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/183—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
    • H04N7/185—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source from a mobile camera, e.g. for remote control
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
    • B60W2420/40—Photo or light sensitive means, e.g. infrared sensors
    • B60W2420/403—Image sensing, e.g. optical camera
    • B60W2420/408—
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
    • G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/93—Radar or analogous systems specially adapted for specific applications for anti-collision purposes
    • G01S13/931—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G01S2013/9323—Alternative operation using light waves
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/93—Radar or analogous systems specially adapted for specific applications for anti-collision purposes
    • G01S13/931—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G01S2013/9324—Alternative operation using ultrasonic waves

Definitions

  • the embodiments of the present application relate to the technical field of vehicle control, and in particular, to an on-board control unit and an FPGA-based vehicle automatic driving method and device.
  • Self-driving vehicles also known as unmanned vehicles, computer-driven vehicles, or wheeled mobile robots, are intelligent vehicles that realize unmanned driving through computer systems.
  • An electronic control unit is installed on the self-driving car.
  • the ECU is also called a trip computer and an on-board computer, and is the "brain" of an unmanned vehicle.
  • the on-board camera is used to collect the surrounding information of the automatic driving vehicle, and the collected information is sent to the ECU.
  • the ECU uses a perception algorithm to visually recognize the received information to generate a decision result, and generates a control command according to the decision result, and then based on The control command completes automatic driving.
  • the visual perception of the information collected by the on-board camera using the perception algorithm relies heavily on the ECU, which causes the ECU to increase the burden.
  • Embodiments of the present application provide an on-board control unit, an FPGA-based vehicle automatic driving method and device.
  • the first SoC is integratedly configured by FPGA and ARM, and the sensor data is processed by FPAG and ARM and sent to the MCU to reduce the burden of the MCU.
  • an embodiment of the present application provides an FPGA-based vehicle automatic driving method, which is suitable for a vehicle-mounted control unit.
  • the vehicle-mounted control unit includes a first system-on-chip SoC and a micro-control unit MCU.
  • the first SoC uses an FPGA It is integrated with the advanced reduced instruction set machine ARM.
  • the vehicle-mounted control unit is installed on an autonomous vehicle. The method includes:
  • the FPGA of the first SoC receives the video data sent by the vehicle camera
  • the FPGA of the first SoC uses a first neural network algorithm to perform visual perception on the video data to obtain first perception information
  • the FPGA of the first SoC sends the first perception information to the ARM of the first SoC;
  • the ARM of the first SoC processes the first perception information, obtains first decision information, and sends it to the MCU.
  • the ARM of the first SoC processes the first perception information to obtain the first decision information and sends it to the MCU, including:
  • the ARM of the first SoC receives radar data
  • the ARM of the first SoC fuses the first perception information and the radar data
  • the ARM of the first SoC processes the merged first perception information and the radar data to obtain the first decision information and sends it to the MCU.
  • the radar data includes at least one of ultrasonic radar data, millimeter wave radar data, and lidar data.
  • the vehicle-mounted control unit further includes: a second system-on-chip SoC, and the second SoC is integratedly set through an FPGA and an advanced reduced instruction set machine ARM, and the method further includes:
  • the FPGA of the second SoC receives the video data sent by the vehicle camera
  • the FPGA of the second SoC uses a second neural network algorithm to perform visual perception on the video data to obtain second perception information
  • the FPGA of the second SoC sends the second perception information to the ARM of the second SoC;
  • the ARM of the second SoC processes the second perception information to obtain second decision information and sends it to the MCU;
  • the MCU generates a control instruction according to the first decision information and the second decision information.
  • an embodiment of the present application provides a vehicle-mounted control unit, including: a first system-on-chip SoC and a micro control unit MCU, the first SoC is integrated with a field programmable gate array FPGA and an advanced reduced instruction set machine ARM Set, the first SoC and the MCU are connected through an Ethernet switching chip.
  • the above-mentioned vehicle-mounted control unit further includes: a second system-on-chip SoC, where an FPGA and an ARM are provided on the second SoC, and the FPGA and the ARM on the second SoC are connected by a bus, and the second The SoC and the MCU are connected through the Ethernet switching chip.
  • a second system-on-chip SoC where an FPGA and an ARM are provided on the second SoC, and the FPGA and the ARM on the second SoC are connected by a bus, and the second The SoC and the MCU are connected through the Ethernet switching chip.
  • the vehicle-mounted control unit further includes: a first synchronous dynamic random access memory SDRAM and a first flash memory Flash, the first SDRAM is connected to the first SoC, and the first Flash is connected to the first One SoC connection.
  • an embodiment of the present application provides an automatic driving device, which is suitable for a vehicle-mounted control unit.
  • the automatic driving device includes: a first system-on-chip SoC module and a micro control unit MCU module, and the first SoC module Through the integrated setting of FPGA and advanced reduced instruction set machine ARM, the vehicle-mounted control unit is set on an autonomous vehicle, and the first SoC includes a first FPGA unit and a first ARM unit, wherein,
  • the first FPGA unit is configured to receive video data sent by an on-board camera, perform visual perception on the video data using a first neural network algorithm, obtain first perception information, and send the first sensor to the ARM of the first SoC A perceived information;
  • the first ARM unit is configured to process the first perception information, obtain first decision information, and send it to the MCU module.
  • the first ARM unit is specifically used to receive radar data, fuse the first perception information and the radar data, and perform the fusion on the first perception information and the radar data Processing to obtain the first decision information and send it to the MCU.
  • the device further includes: a second SoC module, the second SoC module includes a second FPGA unit and a second ARM unit, wherein,
  • the second FPGA unit is configured to receive the video data sent by the on-board camera, and perform visual perception on the video data using a second neural network algorithm to obtain second perception information and send it to the ARM of the second SoC Sending the second perception information;
  • the second ARM unit is configured to process the second perception information, obtain second decision information, and send it to the MCU;
  • the MCU module is configured to generate a control instruction according to the first decision information and the second decision information.
  • the vehicle-mounted control unit and the FPGA-based vehicle automatic driving method and device provided in the embodiments of the present application.
  • the vehicle-mounted control unit includes an MCU and a first SoC implemented by FPGA and ARM integrated settings.
  • the vehicle-mounted control unit is provided on an automatic driving vehicle.
  • the FPGA of the first SoC receives the video data sent by the on-board sensors, uses the first neural network algorithm to visually perceive the video data to obtain the first perception information, and sends the first perception information to the ARM of the first SoC,
  • the ARM of the first SoC processes the first perception information to obtain the first decision information and sends it to the MCU.
  • the MCU generates a control instruction according to the first decision information and sends it to the corresponding execution mechanism, so that the control mechanism automatically performs the control according to the control execution. drive.
  • the first SoC is integrated with FPGA and ARM, and the sensor data is processed by FPAG and ARM and sent to the MCU to reduce the burden on the MCU.
  • Figure 1 is an ECU based on FPGA implemented in the prior art
  • FIG. 2 is a schematic structural diagram of a vehicle-mounted control unit provided by an embodiment of the present application.
  • FIG. 3 is a flowchart of an automatic driving method based on a field programmable gate array FPGA provided by an embodiment of the present application;
  • FIG. 4 is a schematic structural diagram of another vehicle-mounted control unit provided by an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of an automatic driving device provided by an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of another automatic driving device provided by an embodiment of the present application.
  • On-board control unit also called electronic control unit (ECU, ECU)
  • ECU electronice control unit
  • ECU electronice control unit
  • Multiple ECUs are installed on a car to be responsible for different functions. Information exchange is possible, and multiple ECUs form the control system of the car.
  • an ECU includes a micro controller unit (MCU), input and output interfaces, etc.
  • MCU micro controller unit
  • the ECU and other electronic components together form the central nervous system of the car, process the information sent by various sensors according to a preset program, generate decision results and then generate control instructions, and then send the control instructions to the actuator, which executes various Scheduled control functions.
  • Fig. 1 is an ECU implemented based on FPGA in the prior art. Please refer to FIG. 1, the FPGA-based ECU is to set the FPGA outside the ECU. The FPGA is connected to the external ECU, and is suitable for ECUs that do not have a large number of computing needs.
  • technologies such as assisted driving and automatic driving, more and more sensors such as on-board cameras and radars connected to the ECU are required, and the ECU is required to have strong computing power to process data from various sensors.
  • the data collected by the car camera is the most huge and difficult to process, and the current mainstream processing method for the video data collected by the car camera is the neural network algorithm for deep learning.
  • the neural network algorithm for deep learning needs to consume huge ECU CPU (central processing) unit resources, that is, the ECU needs to have strong computing power.
  • the existing FPGA-based ECU uses FPGA to process the video data sent by the on-board camera and send the processed data to an external ECU, the external ECU further processes the received data, generates a control command according to the processing result, and then completes the automatic driving according to the control command.
  • the data calculated by the FPGA needs to be further processed by an external ECU, resulting in an increased burden on the ECU.
  • the embodiments of the present application provide an FPGA based on a field programmable gate array, and the sensor data is processed by the FPGA integrated with ARM and sent to the MCU of the ECU to reduce the burden of the MCU.
  • the FPGA integrated with ARM is processed by the FPGA integrated with ARM and sent to the MCU of the ECU to reduce the burden of the MCU.
  • FIG. 2 is a schematic structural diagram of a vehicle-mounted control unit provided by an embodiment of the present application.
  • the vehicle-mounted control unit provided by the embodiment of the present application includes: a first system on chip (system on chip, SoC) and a micro control unit (micro controller unit (MCU), the first SoC uses a field programmable gate array
  • SoC system on chip
  • MCU micro controller unit
  • the FPGA and the advanced reduced instruction set machine (advanced RISC Machine, ARM) are integrated, and the first SoC and the MCU are connected through an Ethernet switching chip.
  • the first SoC integrates FPGA and ARM, and encapsulates the two into a single chip, which improves the integration and reduces the data connection between FPGA and ARM, so that the first SoC can do more operations. Improve the reliability while reducing the complexity of the ECU.
  • the first SoC is connected to the MCU through an Ethernet switching chip, thereby implementing data exchange between the first SoC and the MCU.
  • the Ethernet switching chip can also interact with external on-board Ethernet.
  • the MCU is responsible for the functional safety of the printed circuit board (PCB) of the ECU and the internal and external interfaces.
  • the MCU internally interacts with the first SoC through the Ethernet switch chip, and the global positioning system (global positioning system) system (GPS), inertial measurement unit (inertial measurement unit (IMU), temperature sensor, etc.) all interact with the MCU.
  • GPS global positioning system
  • IMU inertial measurement unit
  • the MCU leads the debugging interface and the vehicle control CAN interface.
  • the debugging interface is mainly responsible for the debugging of the board, and the CAN interface realizes the control function of the vehicle.
  • the vehicle-mounted control unit provided in the embodiment of the present application includes an MCU and a first SoC implemented by integrating FPGA and ARM settings.
  • the vehicle-mounted control unit is set on an autonomous vehicle.
  • the FPGA of the first SoC receives the sensor sent by the vehicle-mounted sensor.
  • For video data use the first neural network algorithm to visually perceive the video data to obtain first perception information, and send the first perception information to the ARM of the first SoC, and the ARM of the first SoC processes the first perception information to obtain the first
  • the decision information is sent to the MCU, and finally the MCU generates a control instruction according to the first decision information and sends it to the corresponding execution mechanism, so that the control mechanism performs automatic driving according to the control execution.
  • the first SoC is integrated with FPGA and ARM, and the sensor data is processed by FPAG and ARM and sent to the MCU to reduce the burden on the MCU.
  • FIG. 3 is a flowchart of an automatic driving method based on a field programmable gate array FPGA provided by an embodiment of the present application. This embodiment includes:
  • the FPGA of the first SoC receives video data sent by a vehicle sensor.
  • the vehicle-mounted control unit is provided on the self-driving car, and the vehicle-mounted control unit includes a first SoC and an MCU, and the first SoC is integratedly set through FPGA and ARM.
  • the on-board camera on the autonomous vehicle collects video data and sends it to the FPGA on the first SoC.
  • the FPGA on the first SoC receives the video data.
  • the FPGA of the first SoC performs visual perception on the video data using a first neural network algorithm to obtain first perception information.
  • the FPGA on the first SoC uses the first neural network algorithm to perform visual perception on the received video data to obtain first perception information.
  • the first neural network algorithm is a preset neural network algorithm based on deep learning, such as a convolutional neural network (convolutional neural networks, CNN) algorithm, etc.
  • a convolutional neural network convolutional neural networks, CNN
  • TDNN time delay network
  • SIANN shift-invariant artificial neural network
  • LeNet-5 neural network algorithm e.g., a neural network algorithm for example, a convolutional neural network (convolutional neural networks, SIANN), LeNet-5 neural network algorithm, VGGNet neural network algorithm, GoogLeNet neural network algorithm or ResNet Neural network algorithms, etc.
  • the FPGA of the first SoC sends the first perception information to the ARM of the first SoC.
  • the FPGA of the first SoC sends the first sensing information to the ARM of the first SoC through the FPGA internal bus.
  • the ARM of the first SoC processes the first perception information, obtains first decision information, and sends it to the MCU.
  • the ARM of the first SoC models the environment based on the first perception information, etc., and uses the decision algorithm to process the first perception information, and then sends the first decision information to the MCU.
  • the MCU After receiving the first decision information, the MCU generates a control instruction according to the first decision information and sends it to the corresponding execution mechanism, so that the control mechanism performs automatic driving according to the control execution.
  • the field-codeable gate array FPGA-based automatic driving method provided by an embodiment of the present application is applicable to an on-vehicle control unit.
  • the on-vehicle control unit includes an MCU and a first SoC implemented by FPGA and ARM integrated settings.
  • the on-vehicle control unit is set to automatic In the driving car, during the automatic driving process, the FPGA of the first SoC receives the video data sent by the vehicle sensor, uses the first neural network algorithm to visually perceive the video data, obtains the first perception information, and sends the first SoC's ARM The first perception information, the ARM of the first SoC processes the first perception information to obtain the first decision information and sends it to the MCU, and finally the MCU generates a control instruction according to the first decision information and sends it to the corresponding executing agency, so that the control agency This control is executed for automatic driving.
  • the first SoC is integrated with FPGA and ARM, and the sensor data is processed by FPAG and ARM and sent to the MCU to reduce the burden on the MCU.
  • the ARM of the first SoC when the ARM of the first SoC processes the first perception information to obtain the first decision information and sends it to the MCU, the ARM of the first SoC also receives radar data and fuse the first Perceive information and radar data, and then process the merged first perception information and radar data to obtain first decision information and send it to the MCU.
  • the ARM of the first SoC after receiving the first perception information sent by the FPGA of the first SoC, the ARM of the first SoC aggregates the first perception information and the data sent by other sensors to obtain fusion data, and uses the fusion data to build an environment After the first decision information is obtained, it is sent to the MCU.
  • the MCU After receiving the first decision information, the MCU generates a control instruction according to the first decision information and sends it to the corresponding execution mechanism, so that the control mechanism performs automatic driving according to the control execution.
  • the tactile sensor includes ultrasonic radar, millimeter wave radar, lidar, etc.
  • the data sent by the sensor includes at least one of ultrasonic radar data, millimeter wave radar data, and lidar data.
  • the vehicle-mounted control unit provided by the embodiment of the present application on the basis of the above-mentioned FIG. 2, further includes: a second SoC, which is integratedly set through an FPGA and an advanced reduced instruction set machine ARM.
  • the onboard camera in addition to sending the collected video data to the FPGA of the first SoC, the onboard camera also sends the video data to the FPGA of the second SoC.
  • the FPGA of the second SoC uses the second neural network algorithm to visually sense the video data to obtain the second perception information and sends the second perception information to the ARM of the second SoC through the FPGA internal bus.
  • the ARM of the second SoC After receiving the second perception information, the ARM of the second SoC processes the second perception information to obtain second decision information and sends it to the MCU. In this way, the MCU will receive the first decision information sent by the ARM of the first SoC and the second decision information sent by the second SoC. The MCU compares and analyzes the two decision-making information, selects appropriate strategy information from them, generates a control instruction according to the selected strategy information and sends it to the corresponding executing agency, so that the control agency performs automatic driving according to the control execution.
  • the FPGA of the first SoC uses the first neural network algorithm to visually perceive the video data to obtain first perception information; the FPGA of the second SoC uses the second neural network algorithm to visually perceive the video data To get the second perception information.
  • the first neural network algorithm and the second neural network algorithm are different neural network algorithms from each other.
  • the first neural network algorithm is a convolutional neural network algorithm
  • the second neural network algorithm is a Bp algorithm.
  • the first neural network algorithm is a time delay network algorithm
  • the second neural network algorithm is a ResNet neural network algorithm.
  • the first SoC, the second SoC, and the MCU obtain a highly integrated vehicle control unit architecture with huge processing capabilities, and use the first SoC FPGA and the second SoC FPGA to perform neural network algorithm on the video data.
  • Calculation, and the perception information obtained by calculating the video data by each FPGA is sent to the ARM corresponding to the SoC, and the ARM integrates the perception information and the radar data. Therefore, the vehicle-mounted control unit provided by the embodiment of the present application has stronger processing capabilities.
  • the first SoC and the second SoC are used to implement a dual SoC, thereby achieving a heterogeneous redundant structure, that is, the FPGA of the first SoC and the FPGA of the second SoC use different
  • the neural network algorithm processes the video data, which not only ensures the functional safety of the vehicle-mounted control unit but also can be used as the main and standby functions.
  • the vehicle-mounted control unit provided by the embodiment of the present application may further include a first synchronous dynamic random access memory (SDRAM) and a first flash memory, the first SDRAM is connected to the first SoC, The first Flash is connected to the first SoC.
  • the vehicle-mounted control unit provided in the embodiment of the present application may further include: a second synchronous dynamic random access memory (SDRAM) and a second flash memory, the second SDRAM is connected to the second SoC, and the second Flash Connect with the second SoC.
  • SDRAM synchronous dynamic random access memory
  • the second SoC loads the program from the second Flash. After the program is loaded, it waits for data input from the vehicle sensor and other sensors.
  • the FPGA of the first SoC uses the first neural network algorithm to visually perceive the video data to obtain first perception information; the FPGA of the second SoC uses the second neural network algorithm to visually perceive the video data, Get the second perception information.
  • the first perception information is sent by the FPGA of the first SoC to the ARM of the first SoC, and the ARM of the first SoC fuse the first perception information with the detection data of various radars to obtain the first fusion data, and use the first Fusion data is used to model the environment, and then the first decision information is generated according to the decision algorithm and sent to the MCU through the Ethernet switching chip;
  • the second perception information is sent to the second SoC's ARM by the second SoC's FPGA, and the second SoC's ARM fuses the second perception information and the detection data of various radars to obtain the second fusion data, and uses the second fusion data to model the environment, and then generates the second decision information according to the decision algorithm and exchanges the chip through the Ethernet Send to MCU.
  • the MCU After receiving the first decision information and the second decision information, the MCU analyzes and compares the two decision information, selects the appropriate decision information, generates a control command according to the selected decision information, and sends the control command to the CAN interface to The actuator, the actuator executes automatic driving according to the control instructions.
  • FIG. 5 is a schematic structural diagram of an automatic driving device provided by an embodiment of the present application.
  • the automatic driving device may be a vehicle-mounted control unit, or may be a chip applied inside the vehicle-mounted control unit.
  • the automatic driving device can be used to perform the functions of the vehicle-mounted control unit in the above-described embodiment.
  • the automatic driving device 100 may include: a first system-on-chip SoC module 11 and a micro-control unit MCU module 12, the first SoC module 11 is integrated with an advanced reduced instruction set machine ARM through FPGA,
  • the vehicle-mounted control unit is provided on an autonomous driving vehicle, and the first SoC module 11 includes a first FPGA unit 111 and a first ARM unit 112, wherein,
  • the first FPGA unit 111 is configured to receive video data sent by an on-board camera, perform visual perception on the video data using a first neural network algorithm, obtain first perception information, and send the first ARM unit 112 the First perception information;
  • the first ARM unit 112 is configured to process the first perception information, obtain first decision information, and send it to the MCU module 12.
  • the first ARM unit 112 is specifically used to receive radar data, fuse the first perception information and the radar data, and combine the first perception information and the radar data after fusion Perform processing to obtain the first decision information and send it to the MCU.
  • the automatic driving device provided in the embodiment of the present application is suitable for an on-board control unit.
  • the automatic driving device includes an MCU module and a first SoC module.
  • the first SoC module includes a first FPGA unit and a first ARM unit.
  • the first FPGA unit receives the video data sent by the vehicle sensor, uses the first neural network algorithm to visually perceive the video data to obtain the first perception information, and sends the first perception information to the first ARM unit, which is processed by the first ARM unit
  • the first perception information obtains the first decision information and sends it to the MCU module.
  • the MCU module generates a control instruction according to the first decision information and sends it to the corresponding execution mechanism, so that the control mechanism performs automatic driving according to the control execution.
  • the first SoC is integrated with FPGA and ARM, and the sensor data is processed by FPAG and ARM and sent to the MCU to reduce the burden on the MCU.
  • a second SoC module 13 the second SoC module 13 includes a second FPGA unit 131 and a second FARM unit 132, wherein,
  • the second FPGA unit 131 is configured to receive the video data sent by the vehicle-mounted camera, and perform visual perception on the video data using a second neural network algorithm to obtain second perception information and send the second perception information to the second ARM unit 132 sending the second perception information;
  • the second ARM unit 132 is configured to process the second perception information, obtain second decision information, and send it to the MCU;
  • the MCU module 12 is configured to generate a control instruction according to the first decision information and the second decision information.
  • An embodiment of the present application further provides a storage medium that stores computer-executed instructions, which are used by the processor to implement the above-mentioned field programmable gate array FPGA-based automatic driving method when executed by the processor .
  • the described device and method may be implemented in other ways.
  • the device embodiments described above are only schematic.
  • the division of the modules is only a division of logical functions.
  • there may be other divisions for example, multiple modules may be combined or integrated To another system, or some features can be ignored, or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or modules, and may be in electrical, mechanical or other forms.
  • modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • the functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may exist alone physically, or two or more modules may be integrated into one unit.
  • the unit formed by the above modules can be implemented in the form of hardware, or in the form of hardware plus software functional units.
  • the above-mentioned integrated modules implemented in the form of software function modules may be stored in a computer-readable storage medium.
  • the above software function modules are stored in a storage medium, and include several instructions to enable an electronic device (which may be a personal computer, server, or network device, etc.) or a processor (English: processor) to perform the embodiments described in this application Part of the method.
  • processor may be a central processing unit (central processing unit, CPU), or other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs) Wait.
  • the general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the invention may be directly embodied and completed by a hardware processor, or may be performed and completed by a combination of hardware and software modules in the processor.
  • the memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one magnetic disk storage, and may also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.
  • NVM non-volatile storage
  • the bus may be an industry standard architecture (ISA) bus, an external device interconnection (peripheral component, PCI) bus, or an extended industry standard architecture (extended Industry standard architecture, EISA) bus, etc.
  • ISA industry standard architecture
  • PCI peripheral component
  • EISA extended Industry standard architecture
  • the bus can be divided into address bus, data bus, control bus and so on.
  • the bus in the drawings of this application does not limit to only one bus or one type of bus.
  • the above storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable In addition to programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
  • SRAM static random access memory
  • EEPROM electrically erasable programmable read only memory
  • EPROM erasable In addition to programmable read only memory
  • PROM programmable read only memory
  • ROM read only memory
  • magnetic memory flash memory
  • flash memory magnetic disk or optical disk.
  • optical disk any available medium that can be accessed by a general-purpose or special-purpose computer.
  • An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium.
  • the storage medium may also be a component of the processor.
  • the processor and the storage medium may be located in application-specific integrated circuits (application specific integrated circuits, ASIC).
  • ASIC application specific integrated circuits
  • the processor and the storage medium may also exist as discrete components in the terminal or server.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices.
  • the computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium.
  • the computer instructions can be transmitted from a website site, computer, server, or data center via wire (e.g.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device including one or more available media integrated servers, data centers, and the like.
  • the available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk, SSD), and so on.
  • plural herein refers to two or more.
  • the term “and/or” in this article is just an association relationship that describes an associated object, indicating that there can be three relationships, for example, A and/or B, which can mean: A exists alone, A and B exist at the same time, exist alone B these three cases.
  • the character "/" in this article generally indicates that the related object is a "or” relationship; in the formula, the character "/" indicates that the related object is a "divide” relationship.

Abstract

一种车载控制单元、基于FPGA的车辆自动驾驶方法及装置，包括MCU以及通过FPGA和ARM集成设置实现第一SoC，车载控制单元设置在自动驾驶汽车上，第一SoC的FPGA接收车载传感器发送的视频数据（101），利用第一神经网络算法对视频数据进行视觉感知（102），得到第一感知信息后向第一SoC的ARM发送该第一感知信息（103），第一SoC的ARM处理该第一感知信息得到第一决策信息并发送给MCU（104），最终由MCU根据该第一决策信息生成控制指令并发送给相应的执行机构。该过程中，第一SoC上通过FPGA和ARM集成设置，由FPAG和ARM对传感器数据进行处理后发送给MCU，降低MCU的负担。

Description

车载控制单元、基于FPGA的车辆自动驾驶方法及装置
本申请要求于2018年12月28日提交中国专利局、申请号为201811625816.8、申请人为百度在线网络技术(北京)有限公司、申请名称为“车载控制单元、基于FPGA的车辆自动驾驶方法及装置”，以及2019年01月07日提交中国专利局、申请号为2019100131227、申请人为百度在线网络技术(北京)有限公司、申请名称为“车载控制单元、基于FPGA的车辆自动驾驶方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及车辆控制技术领域，尤其涉及一种车载控制单元、基于FPGA的车辆自动驾驶方法及装置。
背景技术
自动驾驶车辆(Self-driving Car)，又称无人驾驶车辆、电脑驾驶车辆、或轮式移动机器人，是一种通过计算机系统实现无人驾驶的智能车辆。
自动驾驶汽车上设置有电子控制单元(electronic control unit，ECU)，ECU也被称之为行车电脑、车载电脑，是无人驾驶车辆的“大脑”。自动驾驶过程中，利用车载摄像头采集自动驾驶车辆周边信息，将采集到的信息发送给ECU，由ECU利用感知算法对接收到信息进行视觉感知生成决策结果，并根据决策结果生成控制指令，进而根据控制指令完成自动驾驶。
上述自动驾驶过程中，利用感知算法对车载摄像头采集到的信息进行视觉感知严重依赖ECU，导致ECU负担加重。
发明内容
本申请实施例提供一种车载控制单元、基于FPGA的车辆自动驾驶方法及装置，第一SoC通过FPGA和ARM集成设置，由FPAG和ARM对传感器数据进行处理后发送给MCU，降低MCU的负担。
第一方面，本申请实施例提供一种基于FPGA的车辆自动驾驶方法，适用于车载控制单元，所述车载控制单元包括第一片上系统SoC和微控制单元MCU，所述第一SoC通过FPGA和进阶精简指令集机器ARM集成设置，所述车载控制单元设置在自动驾驶车辆上，所述方法包括：
所述第一SoC的FPGA接收车载摄像头发送的视频数据；
所述第一SoC的FPGA利用第一神经网络算法对所述视频数据进行视觉感知，得到第一感知信息；
所述第一SoC的FPGA向所述第一SoC的ARM发送所述第一感知信息；
所述第一SoC的ARM处理所述第一感知信息，得到第一决策信息并发送给所述MCU。
一种可行的设计中，所述第一SoC的ARM处理所述第一感知信息，得到第一决策信息并发送给所述MCU，包括：
所述第一SoC的ARM接收雷达数据；
所述第一SoC的ARM融合所述第一感知信息与所述雷达数据；
所述第一SoC的ARM对融合后的所述第一感知信息与所述雷达数据进行处理，得到所述第一决策信息并发送给所述MCU。
一种可行的设计中，所述雷达数据包括超声波雷达数据、毫米波雷达数据、激光雷达数据中的至少一个。
一种可行的设计中，所述车载控制单元还包括：第二片上系统SoC，所述第二SoC通过FPGA和进阶精简指令集机器ARM集成设置，所述方法还包括：
所述第二SoC的FPGA接收所述车载摄像头发送的所述视频数据；
所述第二SoC的FPGA利用第二神经网络算法对所述视频数据进行视觉感知，得到第二感知信息；
所述第二SoC的FPGA向所述第二SoC的ARM发送所述第二感知信息；
所述第二SoC的ARM处理所述第二感知信息，得到第二决策信息并发送给所述MCU；
所述MCU根据所述第一决策信息和所述第二决策信息，生成控制指令。
第二方面，本申请实施例提供一种车载控制单元，包括：第一片上系统SoC和微控制单元MCU，所述第一SoC通过现场可编程门阵列FPGA和进阶精简指令集机器ARM集成设置，所述第一SoC与所述MCU通过以太网交换芯片连接。
一种可行的设计中，上述的车载控制单元还包括：第二片上系统SoC，所述第二SoC上设置FPGA和ARM，所述第二SoC上的FPGA与ARM通过总线连接，所述第二SoC与所述MCU通过所述以太网交换芯片连接。
一种可行的设计中，上述的车载控制单元还包括：第一同步动态随机存储器SDRAM和第一闪存Flash，所述第一SDRAM与所述第一SoC连接，所述第一Flash与所述第一SoC连接。
第三方面，本申请实施例提供一种自动驾驶装置，该装置适用于车载控制单元，所述自动驾驶装置包括：第一片上系统SoC模块和微控制单元MCU模块，所述第一SoC模块通过FPGA和进阶精简指令集机器ARM集成设置，所述车载控制单元设置在自动驾驶车辆上，所述第一SoC包括第一FPGA单元和第一ARM单元，其中，
所述第一FPGA单元，用于接收车载摄像头发送的视频数据，利用第一神经网络算法对所述视频数据进行视觉感知，得到第一感知信息，向所述第一SoC的ARM发送所述第一感知信息；
所述第一ARM单元，用于处理所述第一感知信息，得到第一决策信息并发送给所述MCU模块。
一种可行的设计中，所述第一ARM单元，具体用于接收雷达数据，融合所述第 一感知信息与所述雷达数据，对融合后的所述第一感知信息与所述雷达数据进行处理，得到所述第一决策信息并发送给所述MCU。
一种可行的设计中，所述装置还包括：第二SoC模块，所述第二SoC模块包括第二FPGA单元和第二ARM单元，其中，
所述第二FPGA单元，用于接收所述车载摄像头发送的所述视频数据，利用第二神经网络算法对所述视频数据进行视觉感知，得到第二感知信息，向所述第二SoC的ARM发送所述第二感知信息；
所述第二ARM单元，用于处理所述第二感知信息，得到第二决策信息并发送给所述MCU；
所述MCU模块，用于根据所述第一决策信息和所述第二决策信息，生成控制指令。
本申请实施例提供的车载控制单元、基于FPGA的车辆自动驾驶方法及装置，车载控制单元上包括MCU以及通过FPGA和ARM集成设置实现的第一SoC，车载控制单元设置在自动驾驶汽车上，自动驾驶过程中，第一SoC的FPGA接收车载传感器发送的视频数据，利用第一神经网络算法对视频数据进行视觉感知，得到第一感知信息，并向第一SoC的ARM发送该第一感知信息，第一SoC的ARM处理该第一感知信息得到第一决策信息并发送给MCU，最终由MCU根据该第一决策信息生成控制指令并发送给相应的执行机构，使得控制机构根据该控制执行进行自动驾驶。该过程中，第一SoC通过FPGA和ARM集成设置，由FPAG和ARM对传感器数据进行处理后发送给MCU，降低MCU的负担。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作一简单地介绍，显而易见地，下面描述中的附图是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为现有技术中基于FPGA实现的ECU；
图2是本申请实施例提供的一种车载控制单元的架构示意图；
图3是本申请实施例提供的一种基于现场可编程门阵列FPGA的自动驾驶方法的流程图；
图4是本申请实施例提供的另一种车载控制单元的架构示意图；
图5为本申请实施例提供的一种自动驾驶装置的结构示意图；
图6为本申请实施例提供的另一种自动驾驶装置的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的 所有其他实施例，都属于本申请保护的范围。
本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”、“第三”及“第四”等(如果存在)是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本申请的实施例能够以除了在这里图示或描述的那些以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
车载控制单元，也称之为电子控制单元(electronic control unit，ECU，ECU)，是现代汽车电子的核心元件之一，一个汽车上设置多个ECU，用于负责不同的功能，各ECU之间可以进行信息交互，多个ECU形成汽车的控制系统。通常情况下，一个ECU包括微控制单元(micro controller unit，MCU)、输入输出接口等。ECU与其他电子元件一起组成汽车的大脑中枢神经系统，根据预先设置的程序处理各种传感器发送的信息，生成决策结果进而生成控制指令，然后将控制指令发送给执行机构，由执行机构执行各种预定的控制功能。
目前，各大厂商利用集成电路(integrated circuit，IC)实现ECU，也有部分厂商采用基于现场可编程门阵列(field－programmable gate array，FPGA)架构上实现ECU。图1为现有技术中基于FPGA实现的ECU。请参照图1，该基于FPGA的ECU是将FPGA设置在ECU的外部，该FPGA与外部的ECU连接，适用于没有大量计算需求的ECU。但是，随着辅助驾驶、自动驾驶等技术的演进，与ECU连接的车载摄像头、雷达等传感器越来越多，需要ECU具备强大的计算能力以处理各种传感器的数据。各种传感器中，车载摄像头采集的数据最为庞大且难以处理，而且，目前对车载摄像头采集的视频数据比较主流的处理方法是深度学习的神经网络算法，深度学习的神经网络算法需要消耗ECU巨大的处理器(central processing unit，CPU)资源，即需要ECU具备强大的计算能力，现有的基于FPGA的ECU，利用FPGA对车载摄像头发送的视频数据进行处理，并将处理后的数据发送给外部的ECU，由外部ECU对接收到的数据进行进一步的处理，根据处理结果生成控制指令，进而根据控制指令完成自动驾驶。
上述自动驾驶过程中，FPGA运算后的数据需要外部ECU进一步的进行处理，导致ECU负担加重。
有鉴于此，本申请实施例提供一种基于现场可编程门阵列的FPGA，通过集成了ARM的FPGA对传感器数据进行处理后发送给ECU的MCU，降低MCU的负担。示例性的，可参见图2。
图2是本申请实施例提供的一种车载控制单元的架构示意图。请参照图2，本申请实施例提供的车载控制单元包括：第一片上系统(system on chip，SoC)和微控制单元(micro controller unit，MCU)，该第一SoC通过现场可编程门阵列FPGA和进阶精简指令集机器(advanced RISC Machine，ARM)集成设置，所述第一SoC与所述MCU通过以太网交换芯片连接。
请参照图2，第一SoC将FPGA和ARM融合，将两者集中封装组成单个芯片， 提高集成度的同时减少了FPGA和ARM之间的数据连接，使得第一SoC可以做更多的操作，提高可靠性的同时降低了ECU的复杂度。该第一SoC通过以太网交换芯片与MCU连接，从而实现第一SoC与MCU之间的数据交换。以太网交换芯片还可以与外部车载以太网交互。MCU负责ECU的印刷电路板(printed circuit board，PCB)整板的功能安全处理以及对内对外接口，MCU对内通过以太网交换(switch)芯片与第一SoC交互，同时全球定位系统(global positioning system，GPS)、惯性测量单元(inertial measurement unit，IMU)、温度传感器等均与MCU交互。对外接口方面，MCU引出调试接口和车辆控制CAN接口，调试接口主要负责板卡的调试，CAN接口实现对车辆的控制功能。
本申请实施例提供的车载控制单元，包括MCU以及通过FPGA和ARM集成设置实现的第一SoC，车载控制单元设置在自动驾驶汽车上，自动驾驶过程中，第一SoC的FPGA接收车载传感器发送的视频数据，利用第一神经网络算法对视频数据进行视觉感知，得到第一感知信息，并向第一SoC的ARM发送该第一感知信息，第一SoC的ARM处理该第一感知信息得到第一决策信息并发送给MCU，最终由MCU根据该第一决策信息生成控制指令并发送给相应的执行机构，使得控制机构根据该控制执行进行自动驾驶。该过程中，第一SoC通过FPGA和ARM集成设置，由FPAG和ARM对传感器数据进行处理后发送给MCU，降低MCU的负担。
上述的车载控制单元设置在自动驾驶汽车上。下面，基于图2，对本申请实施例提供的基于FPGA的自动驾驶方法进行详细说明。示例性，可参见图3，图3是本申请实施例提供的一种基于现场可编程门阵列FPGA的自动驾驶方法的流程图。本实施例包括：
101、所述第一SoC的FPGA接收车载传感器发送的视频数据。
本申请实施例中，车载控制单元设置在自动驾驶汽车上，车载控制单元包括第一SoC和MCU，该第一SoC通过FPGA和ARM集成设置。本步骤中，自动驾驶车辆上的车载摄像头采集视频数据并向第一SoC上的FPGA发送。相应的，第一SoC上的FPGA接收该视频数据。
102、所述第一SoC的FPGA利用第一神经网络算法对所述视频数据进行视觉感知，得到第一感知信息。
本步骤中，第一SoC上的FPGA利用第一神经网络算法对接收到的视频数据进行视觉感知，得到第一感知信息。其中，第一神经网络算法为预设的基于深度学习的神经网络算法，如卷积神经网络(convolutional neural networks，CNN)算法等。例如，时间延迟网络(time delay neural network，TDNN)算法、平移不变人工神经网络(shift-invariant artificial neural networks，SIANN)、LeNet-5神经网络算法、VGGNet神经网络算法、GoogLeNet神经网络算法或ResNet神经网络算法等。
103、所述第一SoC的FPGA向所述第一SoC的ARM发送所述第一感知信息。
本步骤中第一SoC的FPGA通过FPGA内部总线，将第一感知信息发送给第一SoC的ARM。
104、所述第一SoC的ARM处理所述第一感知信息，得到第一决策信息并发送给所述MCU。
本步骤中，由第一SoC的ARM根据第一感知信息对环境进行建模等，并利用判决算法处理第一感知信息，得到第一决策信息后发送给MCU。MCU接收到该第一决策信息后，根据该第一决策信息生成控制指令并发送给相应的执行机构，使得控制机构根据该控制执行进行自动驾驶。
本申请实施例提供的基于现场可编码门阵列FPGA的自动驾驶方法，适用于车载控制单元，该车载控制单元上包括MCU以及通过FPGA和ARM集成设置实现的第一SoC，车载控制单元设置在自动驾驶汽车上，自动驾驶过程中，第一SoC的FPGA接收车载传感器发送的视频数据，利用第一神经网络算法对视频数据进行视觉感知，得到第一感知信息，并向第一SoC的ARM发送该第一感知信息，第一SoC的ARM处理该第一感知信息得到第一决策信息并发送给MCU，最终由MCU根据该第一决策信息生成控制指令并发送给相应的执行机构，使得控制机构根据该控制执行进行自动驾驶。该过程中，第一SoC通过FPGA和ARM集成设置，由FPAG和ARM对传感器数据进行处理后发送给MCU，降低MCU的负担。
再请参照图2，本申请实施例中，第一SoC的ARM处理第一感知信息，得到第一决策信息并发送给所述MCU时，该第一SoC的ARM还接收雷达数据，融合第一感知信息和雷达数据，然后对融合后的第一感知信息与雷达数据进行处理，得到第一决策信息并发送给MCU。该过程中，第一SoC的ARM接收到第一SoC的FPGA发送的第一感知信息后，将该第一感知信息与其他传感器发送的数据汇总融后得到融合数据，并利用融合数据进行环境建模和判决操作，得到第一决策信息后发送给MCU。MCU接收到该第一决策信息后，根据该第一决策信息生成控制指令并发送给相应的执行机构，使得控制机构根据该控制执行进行自动驾驶。其中，触感器包括超声波雷达、毫米波雷达、激光雷达等，相应的，传感器发送的数据包括超声波雷达数据、毫米波雷达数据、激光雷达数据中的至少一个。
图4是本申请实施例提供的另一种车载控制单元的架构示意图。请参照图4，本申请实施例提供的车载控制单元，在上述图2的基础上，进一步的还包括：第二SoC，该第二SoC通过FPGA和进阶精简指令集机器ARM集成设置。自动驾驶过程中，车载摄像头除了将采集到的视频数据发送给第一SoC的FPGA外，还将该视频数据发送给第二SoC的FPGA。第二SoC的FPGA接收到视频数据后，利用第二神经网络算法对该视频数据进行视觉感知，得到第二感知信息并通过FPGA内部总线，将该第二感知信息发送给第二SoC的ARM。第二SoC的ARM接收到第二感知信息后，对该第二感知信息进行处理，得到第二决策信息并发送给MCU。如此一来，MCU会接收到第一SoC的ARM发送的第一决策信息和第二SoC发送的第二决策信息。MCU对该两个决策信息进行比较和分析，从中选择出合适的策略信息，根据选择出的策略信息生成控制指令并发送给相应的执行机构，使得控制机构根据该控制执行进行自动驾驶。
上述图4实施例中，第一SoC的FPGA利用第一神经网络算法对视频数据进行视觉感知，得到第一感知信息；第二SoC的FPGA利用第二神经网络算法对所述视频数据进行视觉感知，得到第二感知信息。其中，第一神经网络算法和第二神经网络算法互为不同的神经网络算法。例如，第一神经网络算法为卷积神经网络算法，第二神经网络算法为Bp算法等。再如，第一神经网络算法为时间延迟网络算法，第二神经网 络算法为ResNet神经网络算法等。
本申请实施例中，通过第一SoC、第二SoC和MCU得到具有巨大处理能力、高度集成的车载控制单元架构，利用第一SoC的FPGA和第二SoC的FPGA对视频数据进行神经网络算法的计算，并且每个FPGA对视频数据进行计算得到的感知信息被发送给对应SoC的ARM，由ARM融合感知信息和雷达数据，因此，本申请实施例提供的车载控制单元具备更强的处理能力。而且，从汽车电子产品需要的功能安全角度出发，利用第一SoC和第二SoC实现双SoC，从而实现异构冗余的结构，即第一SoC的FPGA和第二SoC的FPGA分别采用不同的神经网络算法处理视频数据，既保证了车载控制单元的功能安全又可以作为主备功能使用。
再请参照图4，本申请实施例提供的车载控制单元，还可以包括第一同步动态随机存储器(synchronous dynamic random access memory，SDRAM)和第一闪存Flash，该第一SDRAM与第一SoC连接，第一Flash与第一SoC连接。同理，本申请实施例提供的车载控制单元还可以包括：第二同步动态随机存储器(synchronous dynamic random access memory，SDRAM)和第二闪存Flash，该第二SDRAM与第二SoC连接，第二Flash与第二SoC连接。基于该架构，自动驾驶过程中，ECU启动后，第一SoC从第一Flash加载程序，第二SoC从第二Flash加载程序，程序加载完成以后，等待车载传感器以及其他传感器输入数据。对于车载摄像头采集的视频数据，第一SoC的FPGA利用第一神经网络算法对视频数据进行视觉感知，得到第一感知信息；第二SoC的FPGA利用第二神经网络算法对视频数据进行视觉感知，得到第二感知信息。第一感知信息被第一SoC的FPGA发送给第一SoC的ARM，该第一SoC的ARM对第一感知信息和各种雷达的检测数据进行融合，得到第一融合数据，并利用该第一融合数据对环境进行建模，然后根据判决算法生成第一决策信息并通过以太网交换芯片发送给MCU；第二感知信息被第二SoC的FPGA发送给第二SoC的ARM，该第二SoC的ARM对第二感知信息和各种雷达的检测数据进行融合，得到第二融合数据，并利用该第二融合数据对环境进行建模，然后根据判决算法生成第二决策信息并通过以太网交换芯片发送给MCU。MCU接收到第一决策信息和第二决策信息后，对该两个决策信息进行分析比较，选择出合适的决策信息，并根据选择出的决策信息生成控制指令，将控制指令通过CAN接口发送给执行机构，由执行机构根据控制指令进行自动驾驶。
图5为本申请实施例提供的一种自动驾驶装置的结构示意图。本实施例所涉及的自动驾驶装置可以为车载控制单元，也可以为应用于车载控制单元内部的芯片。该自动驾驶装置可以用于执行上述实施例中车载控制单元的功能。如图5所示，该自动驾驶装置100可以包括：第一片上系统SoC模块11和微控制单元MCU模块12，所述第一SoC模块11通过FPGA和进阶精简指令集机器ARM集成设置，所述车载控制单元设置在自动驾驶车辆上，所述第一SoC模块11包括第一FPGA单元111和第一ARM单元112，其中，
所述第一FPGA单元111，用于接收车载摄像头发送的视频数据，利用第一神经网络算法对所述视频数据进行视觉感知，得到第一感知信息，向所述第一ARM单元112发送所述第一感知信息；
所述第一ARM单元112，用于处理所述第一感知信息，得到第一决策信息并发送给所述MCU模块12。
一种可行的设计中，所述第一ARM单元112，具体用于接收雷达数据，融合所述第一感知信息与所述雷达数据，对融合后的所述第一感知信息与所述雷达数据进行处理，得到所述第一决策信息并发送给所述MCU。
本申请实施例提供的自动驾驶装置，适用于车载控制单元，该自动驾驶装置上包括MCU模块以及第一SoC模块，该第一SoC模块包括第一FPGA单元和第一ARM单元，自动驾驶过程中，第一FPGA单元接收车载传感器发送的视频数据，利用第一神经网络算法对视频数据进行视觉感知，得到第一感知信息，并向第一ARM单元发送该第一感知信息，第一ARM单元处理该第一感知信息得到第一决策信息并发送给MCU模块，最终由MCU模块根据该第一决策信息生成控制指令并发送给相应的执行机构，使得控制机构根据该控制执行进行自动驾驶。该过程中，第一SoC通过FPGA和ARM集成设置，由FPAG和ARM对传感器数据进行处理后发送给MCU，降低MCU的负担。
图6为本申请实施例提供的另一种自动驾驶装置的结构示意图。本实施例提供的自动驾驶装置100，在上述图5的基础上，还包括：
第二SoC模块13，所述第二SoC模块13包括第二FPGA单元131和第二FARM单元132，其中，
所述第二FPGA单元131，用于接收所述车载摄像头发送的所述视频数据，利用第二神经网络算法对所述视频数据进行视觉感知，得到第二感知信息，向所述第二ARM单元132发送所述第二感知信息；
所述第二ARM单元132，用于处理所述第二感知信息，得到第二决策信息并发送给所述MCU；
所述MCU模块12，用于根据所述第一决策信息和所述第二决策信息，生成控制指令。
本申请实施例还提供一种存储介质，所述存储介质中存储有计算机执行指令，所述计算机执行指令被处理器执行时用于实现如上所述的基于现场可编程门阵列FPGA的自动驾驶方法。
在上述的实施例中，应该理解到，所描述的设备和方法，可以通过其它的方式实现。例如，以上所描述的设备实施例仅仅是示意性的，例如，所述模块的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个模块可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或模块的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的模块可以是或者也可以不是物理上分开的，作为模块显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能模块可以集成在一个处理单元中，也可以 是各个模块单独物理存在，也可以两个或两个以上模块集成在一个单元中。上述模块成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。
上述以软件功能模块的形式实现的集成的模块，可以存储在一个计算机可读取存储介质中。上述软件功能模块存储在一个存储介质中，包括若干指令用以使得一台电子设备(可以是个人计算机，服务器，或者网络设备等)或处理器(英文：processor)执行本申请各个实施例所述方法的部分步骤。
应理解，上述处理器可以是中央处理单元(central processing unit，CPU)，还可以是其他通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合发明所公开的方法的步骤可以直接体现为硬件处理器执行完成，或者用处理器中的硬件及软件模块组合执行完成。
存储器可能包含高速RAM存储器，也可能还包括非易失性存储NVM，例如至少一个磁盘存储器，还可以为U盘、移动硬盘、只读存储器、磁盘或光盘等。
总线可以是工业标准体系结构(industry standard architecture，ISA)总线、外部设备互连(peripheral component，PCI)总线或扩展工业标准体系结构(extended Industry standard architecture，EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。为便于表示，本申请附图中的总线并不限定仅有一根总线或一种类型的总线。
上述存储介质可以是由任何类型的易失性或非易失性存储设备或者它们的组合实现，如静态随机存取存储器(SRAM)，电可擦除可编程只读存储器(EEPROM)，可擦除可编程只读存储器(EPROM)，可编程只读存储器(PROM)，只读存储器(ROM)，磁存储器，快闪存储器，磁盘或光盘。存储介质可以是通用或专用计算机能够存取的任何可用介质。
一种示例性的存储介质耦合至处理器，从而使处理器能够从该存储介质读取信息，且可向该存储介质写入信息。当然，存储介质也可以是处理器的组成部分。处理器和存储介质可以位于专用集成电路(application specific integrated circuits，ASIC)中。当然，处理器和存储介质也可以作为分立组件存在于终端或服务器中。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行计算机程序指令时，全部或部分地产生按照本申请实施例的流程或功能。计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘Solid State Disk，SSD)等。
本文中的术语“多个”是指两个或两个以上。本文中术语“和/或”，仅仅是一种 描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系；在公式中，字符“/”，表示前后关联对象是一种“相除”的关系。
本领域普通技术人员可以理解：实现上述各方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成。前述的程序可以存储于一计算机可读取存储介质中。该程序在执行时，执行包括上述各方法实施例的步骤；而前述的存储介质包括：ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
最后应说明的是：以上各实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述各实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的范围。

Claims (10)

  1. 一种基于现场可编程门阵列FPGA的自动驾驶方法，其特征在于，适用于车载控制单元，所述车载控制单元包括第一片上系统SoC和微控制单元MCU，所述第一SoC通过FPGA和进阶精简指令集机器ARM集成设置，所述车载控制单元设置在自动驾驶车辆上，所述方法包括：
    所述第一SoC的FPGA接收车载摄像头发送的视频数据；
    所述第一SoC的FPGA利用第一神经网络算法对所述视频数据进行视觉感知，得到第一感知信息；
    所述第一SoC的FPGA向所述第一SoC的ARM发送所述第一感知信息；
    所述第一SoC的ARM处理所述第一感知信息，得到第一决策信息并发送给所述MCU。
  2. 根据权利要求1所述的方法，其特征在于，所述第一SoC的ARM处理所述第一感知信息，得到第一决策信息并发送给所述MCU，包括：
    所述第一SoC的ARM接收雷达数据；
    所述第一SoC的ARM融合所述第一感知信息与所述雷达数据；
    所述第一SoC的ARM对融合后的所述第一感知信息与所述雷达数据进行处理，得到所述第一决策信息并发送给所述MCU。
  3. 根据权利要求2所述的方法，其特征在于，所述雷达数据包括超声波雷达数据、毫米波雷达数据、激光雷达数据中的至少一个。
  4. 根据权利要求1～3任一项所述的方法，其特征在于，所述车载控制单元还包括：第二片上系统SoC，所述第二SoC通过FPGA和进阶精简指令集机器ARM集成设置，所述方法还包括：
    所述第二SoC的FPGA接收所述车载摄像头发送的所述视频数据；
    所述第二SoC的FPGA利用第二神经网络算法对所述视频数据进行视觉感知，得到第二感知信息；
    所述第二SoC的FPGA向所述第二SoC的ARM发送所述第二感知信息；
    所述第二SoC的ARM处理所述第二感知信息，得到第二决策信息并发送给所述MCU；
    所述MCU根据所述第一决策信息和所述第二决策信息，生成控制指令。
  5. 一种车载控制单元，其特征在于，包括：第一片上系统SoC和微控制单元MCU，所述第一SoC通过现场可编程门阵列FPGA和进阶精简指令集机器ARM集成设置，所述第一SoC与所述MCU通过以太网交换芯片连接。
  6. 根据权利要求5所述的车载控制单元，其特征在于，还包括：第二片上系统SoC，所述第二SoC上设置FPGA和ARM，所述第二SoC上的FPGA与ARM通过总线连接，所述第二SoC与所述MCU通过所述以太网交换芯片连接。
  7. 根据权利要求5或6所述的车载控制单元，其特征在于，还包括：第一同步动态随机存储器SDRAM和第一闪存Flash，所述第一SDRAM与所述第一SoC连接，所述第一Flash与所述第一SoC连接。
  8. 一种自动驾驶装置，其特征在于，该装置适用于车载控制单元，所述自动驾驶装置包括：第一片上系统SoC模块和微控制单元MCU模块，所述第一SoC模块通过FPGA和进阶精简指令集机器ARM集成设置，所述车载控制单元设置在自动驾驶车辆上，所述第一SoC包括第一FPGA单元和第一ARM单元，其中，
    所述第一FPGA单元，用于接收车载摄像头发送的视频数据，利用第一神经网络算法对所述视频数据进行视觉感知，得到第一感知信息，向所述第一SoC的ARM发送所述第一感知信息；
    所述第一ARM单元，用于处理所述第一感知信息，得到第一决策信息并发送给所述MCU模块。
  9. 根据权利要求8所述的装置，其特征在于，
    所述第一ARM单元，具体用于接收雷达数据，融合所述第一感知信息与所述雷达数据，对融合后的所述第一感知信息与所述雷达数据进行处理，得到所述第一决策信息并发送给所述MCU。
  10. 根据权利要求8或9所述的装置，其特征在于，所述装置还包括：第二SoC模块，所述第二SoC模块包括第二FPGA单元和第二ARM单元，其中，
    所述第二FPGA单元，用于接收所述车载摄像头发送的所述视频数据，利用第二神经网络算法对所述视频数据进行视觉感知，得到第二感知信息，向所述第二SoC的ARM发送所述第二感知信息；
    所述第二ARM单元，用于处理所述第二感知信息，得到第二决策信息并发送给所述MCU；
    所述MCU模块，用于根据所述第一决策信息和所述第二决策信息，生成控制指令。
PCT/CN2019/129248 2018-12-28 2019-12-27 车载控制单元、基于fpga的车辆自动驾驶方法及装置 WO2020135730A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2020572668A JP7416731B2 (ja) 2018-12-28 2019-12-27 車載制御ユニット、ｆｐｇａベースの車両自動運転方法及び装置
KR1020207024742A KR102471010B1 (ko) 2018-12-28 2019-12-27 차량 탑재 제어 유닛, fpga 기반 차량 자동 주행 방법 및 장치
EP19903938.9A EP3839686B1 (en) 2018-12-28 2019-12-27 Vehicle-mounted control unit, and fpga-based vehicle automatic driving method and device
US17/024,137 US20210001880A1 (en) 2018-12-28 2020-09-17 Vehicle-mounted control unit, and method and apparatus for fpga based automatic driving of vehicle

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN201811625816.8 2018-12-28
CN201811625816 2018-12-28
CN201910013122.7 2019-01-07
CN201910013122.7A CN109814552A (zh) 2018-12-28 2019-01-07 车载控制单元、基于fpga的车辆自动驾驶方法及装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/024,137 Continuation US20210001880A1 (en) 2018-12-28 2020-09-17 Vehicle-mounted control unit, and method and apparatus for fpga based automatic driving of vehicle

Publications (1)

Publication Number Publication Date
WO2020135730A1 true WO2020135730A1 (zh) 2020-07-02

Family

ID=66604077

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/129248 WO2020135730A1 (zh) 2018-12-28 2019-12-27 车载控制单元、基于fpga的车辆自动驾驶方法及装置

Country Status (6)

Country Link
US (1) US20210001880A1 (zh)
EP (1) EP3839686B1 (zh)
JP (1) JP7416731B2 (zh)
KR (1) KR102471010B1 (zh)
CN (1) CN109814552A (zh)
WO (1) WO2020135730A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112793481A (zh) * 2021-01-11 2021-05-14 宁波均联智行科技股份有限公司 一种座舱主动安全方法及系统

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109814552A (zh) * 2018-12-28 2019-05-28 百度在线网络技术（北京）有限公司 车载控制单元、基于fpga的车辆自动驾驶方法及装置
CN111025959B (zh) * 2019-11-20 2021-10-01 华为技术有限公司 一种数据管理的方法、装置、设备及智能汽车
CN110780608B (zh) * 2019-11-26 2023-03-10 北京百度网讯科技有限公司 仿真测试方法及装置
CN110955232A (zh) * 2019-12-13 2020-04-03 深圳市英博超算科技有限公司 一种自动驾驶系统架构
CN110928693B (zh) * 2020-01-23 2021-01-15 飞诺门阵(北京)科技有限公司 一种计算设备及资源分配方法
US11706076B2 (en) 2020-01-23 2023-07-18 Novnet Computing System Tech Co., Ltd. Computer system with computing devices, communication device, task processing device
JP7425615B2 (ja) * 2020-02-05 2024-01-31 マツダ株式会社 車両用制御システム
CN111290310B (zh) * 2020-02-13 2021-03-16 天津鹍骐科技有限公司 一种车载计算系统
US20220048525A1 (en) * 2020-08-14 2022-02-17 Nvidia Corporation Hardware fault detection for feedback control systems in autonomous machine applications
CN112527003B (zh) * 2021-02-18 2021-07-16 北京图森智途科技有限公司 数据传输装置和系统
CN113110407B (zh) * 2021-06-16 2021-09-10 奥特酷智能科技(南京)有限公司 基于区块网关电控单元的汽车控制器
CN113200003A (zh) * 2021-06-18 2021-08-03 北京理工华创电动车技术有限公司 一种商用车整车多域集成控制系统及方法
CN113721503A (zh) * 2021-08-16 2021-11-30 北京超星未来科技有限公司 一种车载计算平台、无人驾驶系统及车辆
CN113442938A (zh) * 2021-08-31 2021-09-28 国汽智控(北京)科技有限公司 车载计算系统、电子设备和车辆
CN114348018A (zh) * 2021-12-17 2022-04-15 际络科技(上海)有限公司 商用车辆的自动驾驶系统及方法
CN114407872A (zh) * 2022-02-17 2022-04-29 驭势科技（浙江）有限公司 自动驾驶系统及车辆
CN114625189A (zh) * 2022-02-28 2022-06-14 北京梧桐车联科技有限责任公司 车辆管理方法、装置及设备
CN114604260A (zh) * 2022-05-11 2022-06-10 青岛慧拓智能机器有限公司 用于无人车的域控制器及域控制装置

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180093538A1 (en) * 2017-11-07 2018-04-05 Michael Benn Rothschild Digital hardware method, system and device for preventing the initial onset of trailer oscillations and control thereof
CN108037756A (zh) * 2017-11-29 2018-05-15 江苏若博机器人科技有限公司 一种多传感器融合中速无人车探测避障系统
CN108162977A (zh) * 2017-11-25 2018-06-15 深圳市元征科技股份有限公司 驾驶辅助系统及控制方法
CN108205279A (zh) * 2017-12-08 2018-06-26 联创汽车电子有限公司 异构多芯片智能驾驶控制器
CN108508882A (zh) * 2017-02-28 2018-09-07 郑州宇通客车股份有限公司 一种车辆辅助驾驶控制平台
CN109814552A (zh) * 2018-12-28 2019-05-28 百度在线网络技术（北京）有限公司 车载控制单元、基于fpga的车辆自动驾驶方法及装置

Family Cites Families (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008021034A (ja) * 2006-07-11 2008-01-31 Fujitsu Ten Ltd 画像認識装置、画像認識方法、歩行者認識装置および車両制御装置
US20100063652A1 (en) * 2008-09-11 2010-03-11 Noel Wayne Anderson Garment for Use Near Autonomous Machines
US8478493B2 (en) * 2008-09-11 2013-07-02 Deere & Company High integrity perception program
US9188980B2 (en) * 2008-09-11 2015-11-17 Deere & Company Vehicle with high integrity perception system
US20130151088A1 (en) * 2011-11-16 2013-06-13 Flextronics Ap, Llc Method and system for vehicle data collection regarding traffic
NL2007104C2 (nl) * 2011-03-18 2012-09-19 Innovius B V Voertuigvolgsysteem voor toepassing in een voertuig.
US10380022B2 (en) * 2011-07-28 2019-08-13 Netlist, Inc. Hybrid memory module and system and method of operating the same
EP2973487B1 (en) * 2013-03-13 2019-07-31 Bosch Automotive Service Solutions Inc. Vehicle measurement apparatus having a system-on-a-chip device, a sensor and a wireless adapter
US20140267727A1 (en) * 2013-03-14 2014-09-18 Honda Motor Co., Ltd. Systems and methods for determining the field of view of a processed image based on vehicle information
US9132746B2 (en) * 2013-08-15 2015-09-15 Honda Motor Co., Ltd. Method and system for reducing range anxiety
US9720797B2 (en) * 2015-06-30 2017-08-01 Nxp Usa, Inc. Flash memory controller, data processing system with flash memory controller and method of operating a flash memory controller
EP3358369A4 (en) * 2015-09-30 2019-05-08 Sony Corporation INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING METHOD, AND PROGRAM
CN105335327B (zh) * 2015-10-13 2017-11-21 电子科技大学 基于Soc的可重构/双冗余VPX3U信号处理载板
US10003554B1 (en) * 2015-12-22 2018-06-19 Amazon Technologies, Inc. Assisted sideband traffic management
JP6858002B2 (ja) * 2016-03-24 2021-04-14 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカＰａｎａｓｏｎｉｃ Ｉｎｔｅｌｌｅｃｔｕａｌ Ｐｒｏｐｅｒｔｙ Ｃｏｒｐｏｒａｔｉｏｎ ｏｆ Ａｍｅｒｉｃａ 物体検出装置、物体検出方法及び物体検出プログラム
US11378954B2 (en) * 2016-06-28 2022-07-05 Faraday&Future Inc. Multi-processor SoC system
US10509407B2 (en) * 2016-07-01 2019-12-17 Samsung Electronics Co., Ltd. Apparatus and method for a vehicle platform
CN111897751A (zh) * 2017-01-26 2020-11-06 华为技术有限公司 一种数据传输的方法、装置、设备和系统
US9919702B1 (en) * 2017-01-27 2018-03-20 GM Global Technology Operations LLC Systems and methods for managing battery state of charge
CN108462726A (zh) * 2017-02-14 2018-08-28 广州市联奥信息科技有限公司 面向未知状况的车辆辅助驾驶决策系统和装置
KR20180097247A (ko) * 2017-02-23 2018-08-31 주식회사 만도 사용자 친화적 차량 제어 시스템 및 방법
JP6799590B2 (ja) * 2017-03-10 2020-12-16 バイドゥ ドットコム タイムス テクノロジー （ベイジン） カンパニー リミテッド 自律走行車のアンダーステアリングを解決するための自動的ステアリング制御の基準適用
CN106951627A (zh) * 2017-03-15 2017-07-14 北京百度网讯科技有限公司 车辆自动驾驶的仿真测试方法、装置、设备及计算机可读存储介质
US10816973B2 (en) * 2017-06-02 2020-10-27 Baidu Usa Llc Utilizing rule-based and model-based decision systems for autonomous driving control
WO2018230492A1 (ja) * 2017-06-16 2018-12-20 本田技研工業株式会社 情報処理装置、情報処理方法、及びプログラム
CN107404529A (zh) * 2017-07-26 2017-11-28 成都网烁信息科技有限公司 一种基于网络云的大数据分析系统及方法
CN107885214A (zh) * 2017-11-22 2018-04-06 济南浪潮高新科技投资发展有限公司 一种基于fpga的加速自动驾驶视觉感知的方法及装置
US10620266B2 (en) * 2017-11-29 2020-04-14 Intel Corporation System, apparatus and method for in-field self testing in a diagnostic sleep state
US11029744B2 (en) * 2017-12-29 2021-06-08 Intel Corporation System, apparatus and method for controlling a processor based on effective stress information
CN108196547B (zh) * 2018-01-08 2020-06-23 北京图森未来科技有限公司 一种自动驾驶系统
CN108445885A (zh) * 2018-04-20 2018-08-24 鹤山东风新能源科技有限公司 一种基于纯电动物流车的自动驾驶系统及其控制方法
CN108614790B (zh) * 2018-05-03 2021-02-05 浪潮集团有限公司 一种车载计算单元、车辆及车辆自动驾驶方法
CN108776472A (zh) * 2018-05-17 2018-11-09 驭势（上海）汽车科技有限公司 智能驾驶控制方法及系统、车载控制设备和智能驾驶车辆

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108508882A (zh) * 2017-02-28 2018-09-07 郑州宇通客车股份有限公司 一种车辆辅助驾驶控制平台
US20180093538A1 (en) * 2017-11-07 2018-04-05 Michael Benn Rothschild Digital hardware method, system and device for preventing the initial onset of trailer oscillations and control thereof
CN108162977A (zh) * 2017-11-25 2018-06-15 深圳市元征科技股份有限公司 驾驶辅助系统及控制方法
CN108037756A (zh) * 2017-11-29 2018-05-15 江苏若博机器人科技有限公司 一种多传感器融合中速无人车探测避障系统
CN108205279A (zh) * 2017-12-08 2018-06-26 联创汽车电子有限公司 异构多芯片智能驾驶控制器
CN109814552A (zh) * 2018-12-28 2019-05-28 百度在线网络技术（北京）有限公司 车载控制单元、基于fpga的车辆自动驾驶方法及装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3839686A4

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112793481A (zh) * 2021-01-11 2021-05-14 宁波均联智行科技股份有限公司 一种座舱主动安全方法及系统
CN112793481B (zh) * 2021-01-11 2021-10-15 宁波均联智行科技股份有限公司 一种座舱主动安全方法及系统

Also Published As

Publication number Publication date
US20210001880A1 (en) 2021-01-07
KR20200115594A (ko) 2020-10-07
EP3839686A1 (en) 2021-06-23
CN109814552A (zh) 2019-05-28
EP3839686A4 (en) 2022-05-04
EP3839686B1 (en) 2024-05-01
KR102471010B1 (ko) 2022-11-25
JP2021529388A (ja) 2021-10-28
JP7416731B2 (ja) 2024-01-17

Similar Documents

Publication Publication Date Title
WO2020135730A1 (zh) 车载控制单元、基于fpga的车辆自动驾驶方法及装置
CN109917765B (zh) 一种基于自动驾驶系统的网络架构的集散式域控制器系统
US11308739B2 (en) Automatic driving system, vehicle control method and device
US11378954B2 (en) Multi-processor SoC system
CN209842367U (zh) 一种基于自动驾驶系统的网络架构的集散式域控制器系统
CN110979314A (zh) 一种自主代客泊车方法、车载设备及存储介质
US20190132555A1 (en) Methods and systems to broadcast sensor outputs in an automotive environment
CN113655788B (zh) 车辆遥控泊车方法、系统、终端设备及可读存储介质
JP2023534048A (ja) データ送信方法および装置
KR101703500B1 (ko) 차량 유닛
US20220212692A1 (en) Automated driving control system and automated driving control apparatus
US11390243B2 (en) Method and apparatus for processing vehicle collision, vehicle, device and storage medium
WO2023124354A1 (zh) 一种车辆控制器、车辆和车辆控制方法
US20200114905A1 (en) Electronic Control Apparatus, Electronic Control System, and Electronic Control Method
EP4266184A1 (en) Method and device for processing sensor data
US20230085381A1 (en) Vehicle sensor data processing method and system
JP2019188958A (ja) 駐車支援装置
CN108243244B (zh) 车载通信系统以及其控制方法
CN111766866A (zh) 信息处理装置和包括信息处理装置的自动行驶控制系统
CN116533757A (zh) 电压故障判断方法、装置、存储介质、控制器及车辆
CN115503626A (zh) 车辆的控制系统、方法、电子设备及存储介质
CN113968227B (zh) 用于非自动驾驶车的控制方法和装置
EP3896915A1 (en) Data processing method and apparatus
JP6979630B2 (ja) 監視装置、監視方法及びプログラム
EP4144593A1 (en) Method and system for processing data of sensor of vehicle

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19903938

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 20207024742

Country of ref document: KR

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2020572668

Country of ref document: JP

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2019903938

Country of ref document: EP

Effective date: 20210317

NENP Non-entry into the national phase

Ref country code: DE