JP2005035396A - On-vehicle information processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an inexpensive and flexible on-vehicle information processing device which promptly cope with changes of a driving environment and has a small-scale circuit. <P>SOLUTION: A controlling portion 13 controls a dynamic reconstruction controlling portion 15 on the basis of information from a sensor 18 in a vehicle, and reconstructs a circuit of an image processing hardware 11, thereby always enabling image processing corresponding to the driving environment. The controlling portion 13 controls an exterior device 19 such as headlights and a brake on the basis of information to which image processing is carried out as the above. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an in-vehicle information processing apparatus (also referred to as an in-vehicle image processing apparatus) that supports driving of a vehicle, and particularly relates to coping with changes in the driving environment.

  In general, an image processing apparatus processes image data sent from an image sensor in software using a processor or the like, or arranges a circuit having a specific function and performs image processing in hardware. Alternatively, the image processing is realized by pre-processing the image with a circuit having a specific function in the previous stage of the processor and then performing image processing with the processor or the like.

  The current in-vehicle image processing system is also configured as described above, and it is common to deal with various changes in weather and driving conditions with software.

On the other hand, recently, dynamic reconfigurable circuit technology has been actively developed. For example, in Patent Document 1 below, there is a part common to each circuit in reconfiguring a circuit having another function on a programmable circuit. And a technique for reducing the time required for reconfiguration by reconfiguring only the non-common parts. Patent Document 2 below discloses an apparatus that performs image processing on an image such as an electron microscope using a reconfigurable circuit.
JP 2001-51826 A JP-A-11-144661

  In conventional in-vehicle image processing systems, image processing with the same algorithm is performed for various driving environments (which can be said to be the surrounding environment of the car), or image processing corresponding to the environment is performed by software .

  However, software image processing takes more time than hardware image processing, and image processing at video rates such as 60 fps and 30 fps can only be performed using a high-performance CPU, and processing is also performed to realize real-time performance. It will be limited.

  In addition, when creating hardware that supports various operating environments, a huge amount of circuits are required for daytime, nighttime, rainy weather, fine weather, fog, highway, and urban areas. is not.

  Furthermore, in the conventional system, the separation of the hardware processing and the software processing of the image processing is fixed, and it cannot be said that the system is efficient for various scene images.

  The present invention has been made under such circumstances, and it is an object of the present invention to provide an in-vehicle information processing apparatus that can respond quickly to changes in the operating environment, is inexpensive, and is flexible with a small-scale circuit. is there.

  In order to solve the above-described problems, in the present invention, an in-vehicle information processing apparatus is configured as described in the following (1) to (6).

(1) a first input terminal for inputting information from a camera for photographing outside the vehicle;
A reconfigurable image processing hardware unit for processing information from the camera input from the first input end;
An image processing software unit for processing information from the camera input from the first input terminal;
A dynamic reconfiguration control unit for reconfiguring the image processing hardware unit;
A second input for inputting information indicating the driving environment;
An output terminal for outputting information to an in-vehicle external device;
The dynamic reconfiguration control unit is controlled based on information from the second input terminal, and the external reconfiguration unit via the output terminal based on information from the image processing hardware unit and the image processing software unit. A control unit for controlling the device;
An in-vehicle information processing apparatus comprising:

(2) In the in-vehicle information processing apparatus according to (1),
The in-vehicle information processing apparatus, wherein the external device is a headlamp, and the control unit controls an optical axis and / or a light amount of the headlamp.

(3) In the in-vehicle information processing device according to (1),
It has a configuration ROM for storing circuit data,
The in-vehicle information processing apparatus, wherein the dynamic reconfiguration control unit reads required circuit data from the configuration ROM and reconfigures the image processing hardware unit based on the circuit data.

(4) In the in-vehicle information processing device according to (1),
The reconfigurable image processing hardware unit has at least one circuit area of a time parameter circuit area, a location parameter circuit area, a weather parameter circuit area, a vehicle speed parameter circuit area, and a road parameter circuit area, An in-vehicle information processing apparatus that reconfigures a circuit in each circuit area by changing.

(5) a first input terminal for inputting information from a camera for photographing outside the vehicle;
An FPGA that integrates an image processing hardware unit and an image processing software unit using an FPGA equipped with a CPU, which processes information input from the first input terminal;
A microcomputer integrated with a dynamic reconfiguration control unit and a control unit for reconfiguring the image processing hardware unit;
A second input for inputting information indicating the driving environment;
An output terminal for outputting information to an in-vehicle external device;
In-vehicle information processing apparatus equipped with
The control unit controls the dynamic reconfiguration control unit based on information from the second input terminal, and the output terminal based on information from the image processing hardware unit and the image processing software unit. An in-vehicle information processing apparatus that controls the external device via a computer.

(6) In the in-vehicle information processing apparatus according to any one of (1) to (5),
When the control unit compares the processing result generated by the image processing and the information from the second input end, generates a new parameter by adding past information, and the parameter changes In addition, the vehicle information processing apparatus is characterized by issuing a command for reconfiguring the circuit of the parameter designated to the dynamic reconfiguration control unit.

  As described above, according to the present invention, it is possible to provide an in-vehicle information processing apparatus that is inexpensive and flexible with a small-scale circuit.

  In other words, image parameters can be processed in more environments by changing the environmental parameters as needed depending on the country or region.

  In addition, when adding environmental parameters, it is possible to cope with the addition of circuit data and rewriting of software instead of adding the circuit (hardware) itself.

  Further, since no extra circuit is incorporated, power saving of the IC can be achieved.

  In addition, the processing speed can be dramatically increased compared to processing using software alone.

  Furthermore, the size can be drastically reduced (the circuit scale can be reduced) as compared with processing using only circuits (hardware).

  Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to examples.

  FIG. 1 is a block diagram illustrating functions of an “in-vehicle information processing apparatus” according to an embodiment. As shown in FIG. 1, the on-vehicle information processing apparatus includes a camera 10 that captures images outside the vehicle, an image processing hardware unit 11 that performs image processing and includes a circuit that is reconfigured by a dynamic reconfiguration control unit 15. The image processing software unit 12 that performs image processing and outputs the result to the control unit 13, and the dynamic reconfiguration control unit 15 based on the processing result of the image processing software unit 12 and information from the sensor 18 in the vehicle. A circuit reconfiguration command is issued, and a control unit 13 for controlling an external device 19 such as a headlamp, a circuit data storage ROM 14 storing circuit information of various functions, and circuit data storage by commands of the control unit 13 A dynamic reconfiguration control unit 15 that reads out circuit data from the ROM 14 and reconfigures the hardware of the image processing hardware unit 11, temporarily stores an image, and stores the image processing hardware. The image memory 16 accessed from at least one of the image processing unit 11 and the image processing software unit 12 and the image display command from at least one of the image processing hardware unit 11 and the image processing software unit 12 are displayed to display an image. The image display unit 17 is configured.

  The image processing hardware unit 11 uses a reconfigurable device such as an FPGA (Field Programmable Gate Array), for example, and performs part or all of the image processing. When an image memory controller circuit is configured in the image processing hardware unit 11 and an image memory access command is given from the image processing software unit 12, the image processing hardware unit 11 reads and writes image data from and to the image memory 16. Similarly, when a display control circuit is configured in the image processing hardware unit 11 and an output command such as an image is given from the image processing software unit 12, the image processing hardware unit 11 outputs an image or the like to the display unit 17. .

  The image processing software unit 12 outputs the image processing result to the control unit 13 based on the image processing hardware unit 11 or the result of its own image processing. Further, it is determined whether image processing, image memory access, image output, etc. are performed by hardware processing or software processing. When the processing is performed by hardware, a request for circuit reconfiguration is issued to the control unit 13 and performed by software. In some cases, the process is performed on its own.

  The control unit 13 calculates an image processing result of the image processing software unit 12, a circuit reconfiguration request, information on other sensors (vehicle speed sensor, temperature sensor, acceleration sensor, etc.) 18 in the vehicle, and a dynamic reconfiguration control unit A circuit reconfiguration command is issued to 15. Further, other systems 19 in the vehicle such as headlamps and brakes are controlled as necessary.

  As the circuit data storage ROM 14, a ROM for FPGA configuration is sold by an FPGA vendor, and a nonvolatile ROM such as Flash can be used.

  Arbitrary circuit data can be read by managing the address for each stored circuit by the dynamic reconfiguration control unit 15.

  The dynamic reconfiguration control unit 15 reads circuit data from one or more circuit data storage ROMs 14 and reconfigures the hardware. An IC having a reconfiguration control function or a CPLD having a reconfiguration control function. (Comp1ex Programmable Logic Device) or an external processor.

  The display unit 17 is a device that displays a result of some image processing to the driver, and outputs an image processed image, displays a caution or warning, or notifies the driver by voice.

  When a new circuit is added, circuit data is added to the circuit data storage ROM 14. The image processing circuit is also changed by rewriting circuit data or by the software unit 12. This makes it possible to change the image processing circuit flexibly even under new driving environment conditions and obtain an optimal image processing result. In addition, since the function can be added only by changing the circuit data and software, it is not necessary to change the device (substrate) itself. As a result, it is possible to make an inexpensive and flexible system with a small-scale circuit compared to the conventional method.

  Here, dynamic reconfiguration is to reconfigure (reconfigure) a part or the whole of the hardware part into a different circuit, leaving only the circuits that are used in common and only non-common circuits. Can also be reconfigured. Dynamic means that when a part of a circuit is reconfigured, the part that is not reconfigured remains functional. Examples of the dynamically reconfigurable device include Spartan-3 and Virtex-IIPRO of FPGA manufactured by Xi1inx. Depending on the circuit scale to be reconfigured and the speed of the clock used for reconfiguration, the time required for reconfiguration is generally about several tens of ms to several hundreds of ms.

  The dynamic reconfiguration will be described in detail. 2, 3 and 4 are block diagrams showing the dynamic reconfiguration function. The CPLD 22 in FIG. 2 is programmed with a circuit having a function for reconfiguring the FPGA 21. In the example of FIG. 2, the CPLD 22 is read from the ROM or the Flash 23 in which circuit data is stored by the Enable signal from the FPGA 21. The circuit data is read out from the FPGA, and the FPGA 21 is reconfigured. In FIG. 3, reconfiguration is performed using the microcomputer 32 instead of the CPLD. In FIG. 4, an FPGA 41 with a built-in CPU is used. The built-in CPU 42 reads circuit data from the external ROM 43 and reconfigures the programmable part of the FPGA 41.

  FIGS. 5A, 5B, and 5C show the internal state of the dynamically reconfigured FPGA. First, it is assumed that a circuit A52, a circuit B53, and a circuit C54 are configured in the FPGA 51 as shown in FIG. FIG. 5B shows a state in which dynamic reconfiguration is performed from this state and a circuit D55 is added. FIG. 5C is a diagram obtained by reconfiguring FIG. 5A and replacing the circuit C54 with a circuit E56.

  FIG. 6 is a block diagram showing the configuration of this embodiment. As shown in the figure, a camera 61 that is mounted in the vehicle and shoots the front, an FPGA 62 in which a hardware unit and a software unit for image processing are integrated using an FPGA with a CPU, a control unit, and dynamic reconfiguration control A microcomputer 65, an external image RAM 63, a configuration ROM 66 for storing circuit data, an in-car display 64, a vehicle speed sensor, a temperature sensor, a light switch, a radio clock, a GPS, a car navigation 67, and an optical axis. It consists of a headlight with variable light intensity and a brake 68.

  FIG. 7 is a diagram showing a list of operating environment parameters used in the present embodiment. There are roughly five environmental parameters: time, location, weather, vehicle speed, and road. Time parameters are daytime, evening (morning), nighttime, and location parameters are expressway, urban area, mountain road, residential area, Tunnel, weather parameters were fine weather, cloudy weather, rainy weather, snowfall, fog, vehicle speed parameter was divided into 80km or more every 20km, and road parameters were set as dry asphalt, wet asphalt, gravel road, snowy road.

  FIG. 8 shows an example of the internal configuration of the CPU built-in FPGA used in this embodiment. For the FPGA 62, for example, a device such as Xilinx-Virtex-IIPRO is used. Inside the FPGA 62, a time parameter circuit area 82, a location parameter circuit area 83, a weather parameter circuit area 84, a vehicle speed parameter circuit area 85, a road parameter circuit area 86, and a common circuit area 87 are provided, and by changing parameters, Reconfigure the circuits in each circuit area. For example, when it changes from daytime to evening, the circuit of the time parameter circuit area 82 is reconfigured from daytime to evening. The time required for reconfiguration is from several tens of ms to several hundreds of ms. However, if the headlamp and the brake are controlled as in the present embodiment, it is considered that the time required for the control is shorter than the time required for control. Since the frequency of parameter change is from several minutes to several hours in consideration of a general operating environment, real-time performance is not significantly reduced by dynamic reconfiguration. Further, the CPU 88 with a built-in FPGA processes the software section for image processing.

  The initial setting part of the processing flow is shown in FIG. If the power is ON (see step 91 (denoted as S91 in the figure), the same applies hereinafter), the initial environment parameter (daytime, urban area, clear sky, 0-20 km, dry asphalt) circuit is configured in the FPGA 62 (S92). ). When the configuration is completed (S93), the image processing is started (S94). Image processing is performed by the circuit inside the FPGA 62 and the CPU 88, and the result is sent to the microcomputer 65 sequentially. The access to the image memory 63 and the display on the in-vehicle display 64 are appropriately performed by the hardware unit and the software unit in the FPGA 62.

  FIG. 10 is a flowchart showing processing of the control unit. Information is obtained from the processing result generated by the image processing by the FPGA 62 and other sensors in the vehicle (S101). Then, the information is compared and calculated, and new parameters are generated in consideration of past information (S102). The reason for referring to past information is to prevent misjudgment. At the same time, the reliability of the generated result is obtained (S103). If the parameter changes (S104) and the reliability is equal to or higher than a specified value (S105), a command for reconfiguring the circuit of the parameter specified by the dynamic reconfiguration control unit is issued (S106). The calculated result and the reliability are used for the next calculation as past information (S107). If the results are inconsistent, the reliability will be low. If the decision is ambiguous in this way, the decision is carried over to the next image processing result.

  Further, as shown in FIG. 11, the optical axis and / or the light amount of the headlamp are changed using the calculation result separately from the dynamic reconfiguration process (S112, S113). Using the calculation result, it is possible to perform control such as emergency brake operation, change from two-wheel drive to four-wheel drive, and change from four-wheel drive to two-wheel drive.

  FIG. 12 is a flowchart showing the dynamic reconfiguration process. When a dynamic reconfiguration command is issued from the control unit to the dynamic reconfiguration control unit, the dynamic reconfiguration control unit determines whether the circuit block to be reconfigured in the FPGA 62 and the adjacent circuit blocks connected in circuit are connected. The interface is stopped (S121). Next, in order to start dynamic reconfiguration, a control signal is sent from the control pin (S122). Then, circuit data to be newly reconfigured is sent and reconfigured (S123). When the reconfiguration is completed, the operation of the interface between the reconfigured new circuit and the adjacent circuit block is started (S124).

  FIG. 13 shows an example of a circuit that is actually reconfigured for each parameter.

  Circuits that are always necessary and are commonly used include a preceding vehicle detection circuit, an area division circuit, an image memory controller, a display control circuit, and the like.

  With the time parameter, the image becomes dark at evening and night, so a circuit for correcting the contrast is required. Further, at night, a circuit for detecting the tail lamp and head lamp of the preceding vehicle and the oncoming vehicle is also required.

  For location parameters, there are adjacent lane monitoring circuits in multiple lanes such as highways, oncoming traffic detection circuits in urban areas and mountain roads, intersection monitoring circuits in places with many intersections, pedestrian detection circuits in hot water stations with many pedestrians, etc. I need it. In addition, the road width measurement circuit on mountainous roads with narrow roads, the obstacle detection circuit in residential areas with many street parking and obstacles, and the glare (halation) due to sunlight at the exit, mainly in the daytime, can be considered in tunnels. A tunnel exit dazzling prevention circuit is required.

  The weather parameter requires a sunlight dazzling prevention circuit that prevents halation due to direct sunlight reception in fine weather. When it rains, a raindrop removal circuit for removing raindrops on the windshield and a wiper recognition removal circuit for preventing the wiper from being recognized as an obstacle are required. During snowfall, a snow removal circuit that prevents the snow on the windshield from being recognized as an obstacle, a wiper recognition removal circuit, and the like are required. Further, since the image is blurred in white when foggy, an image sharpening circuit and a color correction circuit for correcting color information lost due to the fog are required.

  With the vehicle speed parameter, the closer to the edge of the image, that is, the closer the image becomes, the higher the speed, the smaller the image area to be processed, and the size of the rectangular area centered on the vanishing point can be reduced. A circuit to be changed is required.

  As for road parameters, a white line detection circuit is required on asphalt, and particularly on wet asphalt, road reflection light from oncoming vehicles and peripheral light sources becomes large, so a road reflection light removal circuit is required. Since white lines cannot be detected on gravel roads and snowy roads, a circuit that detects the roadway, especially a circuit that textures the gravel itself, is necessary on the gravel road, and image blurring due to vibrations on the gravel road is required. A circuit to prevent it is also necessary.

  In this way, by reconfiguring the circuits in each circuit area shown in FIG. 8 as illustrated in FIG. 13 in accordance with changes in the driving environment, image processing suitable for the driving environment can always be performed. Lights and brakes can be controlled quickly and appropriately.

  In addition to the above-described embodiment, the image processing unit hardware unit, the image processing software unit, the control unit, and the reconstruction control unit can be configured as follows.

  The image processing hardware unit is configured by a reconfigurable device such as an FPGA, and the image processing software unit, the control unit, and the reconfiguration control unit are similar to each other, and a reconfigurable device such as an FPGA. Are configured as an integrated device or as separate devices.

  The image processing software unit is configured with a microcomputer or an FPGA having a processor function, and the image processing hardware unit, the control unit, and the reconfiguration control unit are similarly configured with a microcomputer or an FPGA. , Configured as an integrated device or as a separate device.

  The control unit is configured by a microcomputer, CPLD, or FPGA, and when the image processing hardware unit, image processing software unit, and reconstruction control unit are similarly configured by a microcomputer, CPLD, or FPGA, It is configured as a device integrated with them or as a separate device.

  The reconfiguration control unit is configured with a microcomputer, CPLD, or FPGA, and the image processing hardware unit, image processing software unit, and control unit are configured with a microcomputer, CPLD, or FPGA as well. Are configured as a device integral with them or as separate devices.

Block diagram showing functions of the embodiment Block diagram showing the dynamic reconfiguration function Block diagram showing the dynamic reconfiguration function Block diagram showing the dynamic reconfiguration function A diagram showing the state of a dynamically reconfigured FPGA Block diagram showing the configuration of the embodiment A diagram showing a list of parameters used in the embodiment The block diagram which shows the internal circuit of FPGA used in an Example The flowchart which shows the initial setting process in an Example Flow chart showing processing of control unit Flow chart showing processing for headlamps Flow chart showing dynamic reconfiguration processing Diagram showing examples of circuits to be reconfigured for each parameter

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Camera 11 Image processing hardware part 12 Image processing software part 13 Control part 15 Dynamic reconfiguration control part 18 Sensor 19 in a vehicle External device

Claims (6)

A first input terminal for inputting information from a camera for photographing outside the vehicle;
A reconfigurable image processing hardware unit for processing information from the camera input from the first input end;
An image processing software unit for processing information from the camera input from the first input terminal;
A dynamic reconfiguration control unit for reconfiguring the image processing hardware unit;
A second input for inputting information indicating the driving environment;
An output terminal for outputting information to an in-vehicle external device;
The dynamic reconfiguration control unit is controlled based on information from the second input terminal, and the external reconfiguration unit via the output terminal based on information from the image processing hardware unit and the image processing software unit. A control unit for controlling the device;
An in-vehicle information processing apparatus comprising: The in-vehicle information processing apparatus according to claim 1,
The in-vehicle information processing apparatus, wherein the external device is a headlamp, and the control unit controls an optical axis and / or a light amount of the headlamp. The in-vehicle information processing apparatus according to claim 1,
It has a configuration ROM for storing circuit data,
The in-vehicle information processing apparatus, wherein the dynamic reconfiguration control unit reads required circuit data from the configuration ROM and reconfigures the image processing hardware unit based on the circuit data. The in-vehicle information processing apparatus according to claim 1,
The reconfigurable image processing hardware unit has at least one circuit area of a time parameter circuit area, a location parameter circuit area, a weather parameter circuit area, a vehicle speed parameter circuit area, and a road parameter circuit area, An in-vehicle information processing apparatus that reconfigures a circuit in each circuit area by changing the circuit. A first input terminal for inputting information from a camera for photographing outside the vehicle;
An FPGA that integrates an image processing hardware unit and an image processing software unit using an FPGA equipped with a CPU, which processes information input from the first input terminal;
A microcomputer integrated with a dynamic reconfiguration control unit and a control unit for reconfiguring the image processing hardware unit;
A second input for inputting information indicating the driving environment;
An output terminal for outputting information to an in-vehicle external device;
In-vehicle information processing apparatus equipped with
The control unit controls the dynamic reconfiguration control unit based on information from the second input terminal, and the output terminal based on information from the image processing hardware unit and the image processing software unit. An in-vehicle information processing apparatus that controls the external device via a computer. The in-vehicle information processing apparatus according to any one of claims 1 to 5,
When the control unit compares the processing result generated by the image processing and the information from the second input end, generates a new parameter by adding past information, and the parameter changes In addition, the vehicle information processing apparatus is characterized by issuing a command for reconfiguring the circuit of the parameter designated to the dynamic reconfiguration control unit.

Images