JP2015042004A - Imaging apparatus, imaging system, and vehicle - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus, an imaging system, and a vehicle which reduce costs by suppressing increase in circuit scale.SOLUTION: A coordinate transformation data switch unit 205 switches between first coordinate transformation data for an S polarization image and second coordinate transformation data for a P polarization image to be used for coordinate transformation processing by a coordinate calculation unit 206, on the basis of the coordinate values (X', Y') of an output image. The coordinate calculation unit 206 calculates the coordinate travel amount (ΔX, ΔY) of the coordinate values of a pixel in an input image with respect to the output pixel indicated by the coordinate values (X',Y') in the output image, as the coordinate transformation processing.

Description

  The present invention relates to an imaging device, an imaging system, and a vehicle.

  In recent years, driving of an inter-vehicle distance control system (ACC: Adaptive Cruise Control) and a forward collision warning system (FCW) for measuring the inter-vehicle distance with a preceding vehicle in front of the vehicle and maintaining a predetermined inter-vehicle distance. A support system has been developed. A stereo camera is already known as a technique for measuring the distance from the preceding vehicle in order to realize these systems. The stereo camera calculates position information of a subject by analyzing two captured images having parallax with respect to the subject.

  In general, since a stereo camera needs to calculate parallax with high accuracy from images captured by two imaging means, it accurately measures the positional relationship between the two incident portions and eliminates distortion caused by the optical system. The correction process is essential. In this distortion correction, coordinate conversion processing is performed inside the stereo camera. However, in a stereo camera constituted by two imaging means, coordinate conversion processing is performed at high speed on images captured by the respective imaging means. For this purpose, it is necessary to provide two coordinate conversion processing units. As such a stereo camera having two image pickup means, a camera that simultaneously acquires long-distance and short-distance stereo images and accurately detects the three-dimensional positions of long-distance and short-distance subjects has been proposed. (See Patent Document 1). The technique disclosed in Patent Document 1 is for a long distance from the image of the S-polarized component imaged by one of the two imaging units and the image of the S-polarized component imaged by the other imaging unit. A stereo image is formed, and a short-distance stereo image is formed from the image of the P-polarized component captured by the former imaging unit and the image of the P-polarized component captured by the latter imaging unit.

  In the technique described in Patent Document 1, since two imaging units are provided, in order to correct the positional relationship between the two imaging units and the distortion of the optical system, 2 for each high-speed correction calculation. It is assumed that it is necessary to provide two coordinate conversion processing units. However, when two coordinate conversion processing units are provided, there is a problem that the circuit scale increases and the cost increases.

  The present invention has been made in view of the above, and an object thereof is to provide an imaging apparatus, an imaging system, and a vehicle that suppress an increase in circuit scale.

  In order to solve the above-described problems and achieve the object, the present invention includes two incident portions that respectively receive light from a subject, and a first incident incident on one of the two incident portions. The first light component is extracted from the light, the second light component is extracted from the second incident light incident on the other incident portion, and the first light component and the second light component are combined into one path. A region dividing unit having a system, a plurality of first transmission regions that transmit the first light component, and a plurality of second transmission regions that transmit the second light component; An imaging unit that images the first light component and the second light component to generate one input image, and a first image that is included in the input image and generated from the first light component First correction data for correction, and the second light component included in the input image By switching between the second correction data for correcting the second image that has been al generates correction process is executed, characterized in that it comprises a correction processing unit for generating an output image.

  According to the present invention, an increase in circuit scale can be suppressed.

FIG. 1 is a diagram illustrating an example of an appearance of a vehicle including a stereo camera. FIG. 2 is a block diagram illustrating an example of the configuration of the stereo camera according to the first embodiment. FIG. 3 is a diagram illustrating an example of the structure of the image input unit. FIG. 4 is a diagram of the prism in the image input unit viewed from the incident side. FIG. 5 is a diagram illustrating an example of the structure of the optical filter and the imaging unit in the image input unit. FIG. 6 is a diagram illustrating the correspondence of the positional relationship between the optical filter and the image sensor in the image input unit. FIG. 7 is a block diagram illustrating an example of the configuration of the coordinate conversion processing unit in the processing unit. FIG. 8 is a flowchart showing the operation of the coordinate conversion data switching unit in the coordinate conversion processing unit. FIG. 9 is a block diagram illustrating an example of the configuration of the coordinate calculation unit in the coordinate conversion processing unit. FIG. 10 is a timing chart of operations of the initial value calculation unit and the recurrence formula calculation unit in the coordinate calculation unit. FIG. 11 is a diagram for explaining the operation of the read address calculation unit in the coordinate conversion processing unit. FIG. 12 is a conceptual diagram illustrating bilinear interpolation processing in the interpolator in the coordinate conversion processing unit. FIG. 13 is a diagram illustrating the operation of the polarization separation processing unit in the coordinate conversion processing unit. FIG. 14 is a flowchart showing the overall operation of the stereo camera according to the first embodiment. FIG. 15 is a block diagram illustrating a configuration of a coordinate calculation unit according to a modification of the first embodiment. FIG. 16 is a diagram for explaining an example of using an input image in various applications of the stereo camera according to the second embodiment.

  Hereinafter, embodiments of an imaging device, an imaging system, and a vehicle according to the present invention will be described in detail with reference to the drawings. In addition, the present invention is not limited by the following embodiments, and constituent elements in the following embodiments are easily conceivable by those skilled in the art, substantially the same, and so-called equivalent ranges. Is included. Furthermore, various omissions, substitutions, and changes of the components can be made without departing from the scope of the following embodiments.

(First embodiment)
<Configuration of vehicle with stereo camera>
FIG. 1 is a diagram illustrating an example of an appearance of a vehicle including a stereo camera. A configuration of the vehicle 1 including the stereo camera 10 will be described with reference to FIG.

  As shown in FIG. 1, the vehicle 1 includes a stereo camera 10, a steering wheel 12, and a brake pedal 13 in a vehicle compartment 3 that is a living room space in the main body 2. Furthermore, the vehicle 1 is provided with the vehicle control apparatus 11 inside the floor under the vehicle interior 3, for example.

  The stereo camera 10 is a camera that captures an image of the front of the vehicle 1, and is installed in the vicinity of the rearview mirror inside the front window of the vehicle 1, for example. And the stereo camera 10 calculates the positional information of solid objects, such as another vehicle, a pedestrian, and other structures on a road, by performing the process which mentions the image ahead of the imaged vehicle, and a white line, a traffic light or Recognition processing such as a sign is executed, and information on the processing result is transmitted to the vehicle control device 11.

  The vehicle control device 11 executes various vehicle controls based on the processing result information received from the stereo camera 10. As an example of vehicle control, the vehicle control device 11 provides a warning operation by voice to the driver or a display means using a speaker or the like when it is determined to be dangerous based on the position information of the preceding vehicle received from the stereo camera 10. A visual warning operation to the used driver is executed. As an example of vehicle control, the vehicle control device 11 controls a steering system including the steering wheel 12 to avoid an obstacle based on position information of an obstacle ahead, or a brake pedal 13. The brake control etc. which control and decelerate and stop the vehicle 1 are performed.

  As described above, the vehicle control system including the stereo camera 10 and the vehicle control device 11 performs vehicle control such as warning operation, steering control, or brake control when a preceding vehicle approaches, thereby Driving safety can be improved.

  As described above, the stereo camera 10 captures the front of the vehicle 1, but is not limited thereto. That is, the stereo camera 10 may be installed so as to capture the rear or side of the vehicle 1. In this case, the stereo camera 10 can detect the position of a succeeding vehicle behind the vehicle 1 or another vehicle that translates laterally. And the vehicle control apparatus 11 can detect the danger at the time of the lane change of the vehicle 1, a lane merge, etc., and can perform the above-mentioned vehicle control. In addition, when the vehicle control device 11 determines that there is a risk of collision based on the position information of the obstacle behind the vehicle 1 detected by the stereo camera 10 in the back operation when the vehicle 1 is parked or the like. The vehicle control described above can be executed.

<Configuration of stereo camera>
FIG. 2 is a block diagram illustrating an example of the configuration of the stereo camera according to the first embodiment. The configuration of the stereo camera 10 according to the present embodiment will be described with reference to FIG.

  As shown in FIG. 2, the stereo camera 10 includes an image input unit 20 and a processing unit 30. The image input unit 20 receives light from the front of the stereo camera 10, polarizes the light incident by the optical system, and captures an image using the image sensor. The image input unit 20 includes incident units 21 a and 21 b, an optical system 22, an optical filter 23 (region dividing unit), and an imaging unit 24.

  The incident portions 21 a and 21 b are arranged at a predetermined distance, and have a function of entering light from the front of the stereo camera 10 and guiding the light to the optical system 22. Note that the incident portions 21a and 21b are simply referred to as the “incident portion 21” when referred to without distinction or collectively.

  The optical system 22 has a function of condensing the light incident from the two incident units (21a, 21b) into an S (Senkrecht) polarization component and a P (Parallel) polarization component, respectively, and condensing the light on the imaging unit 24. . Specifically, the optical system 22 converts the light incident from the incident unit 21a into S-polarized light, converts the light incident from the incident unit 21b into P-polarized light, and combines the polarized light into one path to capture the image capturing unit 24. To collect light. Details of the structure of the optical system 22 will be described later.

  The optical filter 23 is a region that transmits only the S-polarized light component (first light component, first polarized light component) out of the light incident from the incident portion 21 and collected by the optical system 22 (hereinafter referred to as “S-polarized light transmission”). Area ”) (first transmission area) and an area that transmits only the P-polarized component (second light component, second polarization component) (hereinafter referred to as“ P-polarized transmission area ”) (second transmission area). This is a region-divided filter. Specifically, the optical filter 23 is region-divided so as to face each pixel column of the image sensor of the imaging unit 24 arranged in close contact with the rear stage side, and an S-polarized light transmission region and a P-polarized light transmission region for each pixel column. Are arranged alternately. A mode of area division of the optical filter 23 will be described later.

  The S-polarized light transmission region and the P-polarized light transmission region are divided in the optical filter 23 so as to face each pixel row of the image pickup element of the image pickup unit 24. However, the present invention is not limited to this. Absent. For example, the optical filter 23 may have a configuration in which an S-polarized light transmission region and a P-polarized light transmission region are alternately arranged for every two or more adjacent pixel columns of the image sensor of the imaging unit 24.

  The imaging unit 24 detects light incident from the incident unit 21, collected by the optical system 22, and transmitted through the optical filter 23, and captures it as an image. In addition, the imaging unit 24 simultaneously detects the light transmitted through the S-polarized light transmission region and the P-polarized light transmission region of the optical filter 23 and captures it as one image. Details of the configuration of the imaging unit 24 will be described later.

  The processing unit 30 is a part that performs various image processing on the image captured by the image input unit 20. The processing unit 30 includes an FPGA (Field-Programmable Gate Array) 31, a CPU (Central Processing Unit) 32, a ROM (Read Only Memory) 33, and a RAM (Random Access Memory) 34.

  The FPGA 31 is a hardware circuit that executes various types of image processing on the image captured by the image input unit 20. The FPGA 31 performs the operation functions of the image processing unit 41, the coordinate conversion processing unit 42 (correction processing unit), and the distance calculation unit 43. Note that the FPGA 31 is used to execute various image processing on the image captured by the image input unit 20, but is not limited to this, and is not limited to this, but is ASIC (Application Specific Integrated Circuit) or DSP (Digital Signal Processor). ).

  The image processing unit 41 executes various image processing such as γ correction and luminance conversion processing on the image captured by the image input unit 20.

  The coordinate conversion processing unit 42 receives the image processed by the image processing unit 41 as an input image, performs distortion correction on the input image to generate an output image, and generates an S-polarized component image from the output image. And a P-polarized component image are output separately. The distortion correction is image processing for eliminating the distortion of the image in which the distortion generated by the optical system 22 is reflected. Details of the configuration of the coordinate conversion processing unit 42 will be described later.

  The distance calculation unit 43 performs distortion correction by the coordinate transformation processing unit 42, separates the S-polarized component image and the P-polarized component image, and outputs the two output images to the subject as two output images. The corresponding pixel detection process (stereo matching process) is performed to calculate the parallax with respect to the subject, and the distance to the subject is calculated based on the principle of triangulation. The distance calculation unit 43 may calculate not only the distance to the subject but also information on the horizontal direction and the vertical direction (height direction) of the subject in the image output by the coordinate conversion processing unit 42.

  The CPU 32 is a hardware device that executes subject recognition processing, camera control, and various control processing. The CPU 32 realizes each operation function of the control unit 51 and the recognition processing unit 52.

  The control unit 51 performs camera control for instructing the imaging unit 24 with camera parameters such as an exposure value, a gain value, and a captured image size, and setting processing parameters for various processes in the FPGA 31. In addition, the control unit 51 receives vehicle information such as vehicle speed information, steering angle information, and yaw rate information from the vehicle control device 11 and reflects them in the recognition process by the recognition processing unit 52. Then, the control unit 51 obtains information on how to control the vehicle 1 based on the result information of the recognition processing by the recognition processing unit 52 and transmits the information to the vehicle control device 11.

  The recognition processing unit 52 is used to calculate the distance information to the subject calculated by the distance calculation unit 43, the distance to the subject, and an output image in which distortion correction is performed by the coordinate conversion processing unit 42, or image processing This is a hardware device that executes recognition processing for a subject from an image subjected to image processing by the unit 41. As the recognition process, the recognition processing unit 52 processes, for example, another vehicle, a pedestrian, a three-dimensional object such as a road structure such as a pole or a wall, a white line, a sign, a traffic light, and the like as a recognition process target. The recognition processing unit 52 also calculates information such as road surface unevenness, road surface inclination information, and the like as the recognition processing.

  The ROM 33 is a non-volatile storage device that stores a program read and executed by the CPU 32, circuit information of the FPGA 31, processing parameters used for various processes, and the like. The processing parameters used for various processes include a γ conversion table for γ conversion processing, a correction amount parameter for distortion correction processing, a processing parameter for distance calculation processing, and the like. The RAM 34 is a volatile storage device used as a work area for the CPU 32.

<Configuration of image input unit>
FIG. 3 is a diagram illustrating an example of the structure of the image input unit. FIG. 4 is a diagram of the prism in the image input unit viewed from the incident side. FIG. 5 is a diagram illustrating an example of the structure of the optical filter and the imaging unit in the image input unit. FIG. 6 is a diagram illustrating the correspondence of the positional relationship between the optical filter and the image sensor in the image input unit. The structure of the image input unit 20 will be described with reference to FIGS.

  As shown in FIG. 3, the stereo camera 10 includes an image input unit 20 in a housing 26, and light enters the image input unit 20 from incident units 21 a and 21 b formed on the side surface of the housing 26. The optical system 22 includes a cross prism 100, triangular prisms 110a and 110b, and an imaging lens 114.

  The cross prism 100 is a polarization selective prism. In the light from the −Y direction incident from the side surface 102a of the cross prism 100, the S polarization component is reflected by the polarization plane 101a (first surface) in the cross prism 100, and the P polarization component is transmitted through the polarization plane 101a. The S-polarized component reflected by the polarization plane 101a is emitted from the emission surface 103, which is the side surface of the cross prism 100 on the side where the imaging lens 114 is disposed, toward the imaging lens 114. The light from the + Y direction incident from the side surface 102b of the cross prism 100 is reflected by the polarization plane 101b (second surface) where the P polarization component intersects the polarization plane 101a in the cross prism 100, and the S polarization component is the polarization plane. 101b is transmitted. The P-polarized light component reflected by the polarization plane 101b exits from the exit plane 103 toward the imaging lens 114.

  The triangular prism 110 a is arranged such that one of two orthogonal side surfaces is in close contact with the side surface 102 a of the cross prism 100. The light from the + Z direction, which is incident from the incident surface 111a, which is the other of the two orthogonal side surfaces described above via the incident portion 21a, is totally reflected by the reflection surface 112a of the triangular prism 110a and is cross-prism from the side surface 102a. 100 is incident. Of the light incident on the triangular prism 110a from the incident surface 111a, the range on the incident surface 111a of the light detected by the imaging unit 24 via the cross prism 100 and the imaging lens 114 is the effective ray range 113a shown in FIG. is there.

  The triangular prism 110 b is arranged such that one of two orthogonal side surfaces is in close contact with the side surface 102 b of the cross prism 100. The light from the + Z direction that is incident on the incident surface 111b, which is the other of the two orthogonal side surfaces described above, via the incident portion 21b is totally reflected on the reflecting surface 112b of the triangular prism 110b and cross-prism from the side surface 102b. 100 is incident. Of the light incident on the triangular prism 110b from the incident surface 111b, the range of light on the incident surface 111b detected by the imaging unit 24 via the cross prism 100 and the imaging lens 114 is a light effective range 113b shown in FIG. is there.

  The imaging lens 114 is a lens that is disposed on the exit surface 103 side of the cross prism 100 and collects the light emitted from the exit surface 103 on the imaging unit 24. Specifically, the imaging lens 114 passes through the triangular prism 110a and the cross prism 100, and exits from the output surface 103 via the S polarization component of the light exiting from the exit surface 103 and the triangular prism 110b and the cross prism 100. The P-polarized component to be condensed is condensed on the imaging unit 24.

  The optical filter 23 is a filter disposed in close contact with an image sensor 24a of the imaging unit 24 described later, and is divided into an S-polarized light transmission region and a P-polarized light transmission region as described above. As shown in FIG. 5, the optical filter 23 includes a filter substrate 23a, a polarizing filter layer 23b, and a filling layer 23c.

  The filter substrate 23 a is a transparent substrate that transmits light incident on the optical filter 23 via the imaging lens 114. A polarizing filter layer 23b is formed on the surface of the filter substrate 23a on the imaging unit 24 side.

  The polarizing filter layer 23b is a filter layer that is divided into an S-polarized light transmitting region and a P-polarized light transmitting region, and transmits the S-polarized component of the light incident on the optical filter 23 in the S-polarized light transmitting region. The component is transmitted in the P-polarized light transmission region.

  The filling layer 23c is a protective layer that covers the polarizing filter layer 23b formed on the filter substrate 23a.

  The imaging unit 24 includes an imaging element 24a and a substrate 24b. The image pickup device 24a is mounted on the substrate 24b, detects light collected by the optical system 22 and transmitted through the optical filter 23, and picks up an image as an image. Note that the image sensor 24a may be a monochrome sensor or a color sensor. As described above, since the optical filter 23 (polarization filter layer 23b) is divided into an S-polarized light transmission region and a P-polarized light transmission region, the image sensor 24a includes an S-polarized component transmitted through the S-polarized light transmission region, A P-polarized light component transmitted through the P-polarized light transmission region is simultaneously detected. That is, the image sensor 24a simultaneously captures an image composed of the S-polarized component of the light incident from the detected incident portion 21a and an image composed of the P-polarized component of the light incident from the detected incident portion 21b. be able to. Here, an image composed of the S-polarized light component is referred to as “S-polarized image” (first image), and an image composed of the P-polarized light component is referred to as “P-polarized image” (second image). . The light incident from the incident portion 21a is P-polarized by the optical system 22 and captured as a P-polarized image by the imaging device 24a, and the light incident from the incident portion 21b is S-polarized by the optical system 22 and S by the imaging device 24a. It may be captured as a polarization image.

  As shown in FIG. 6, the image sensor 24a is configured by arranging a plurality of pixels in a matrix. The image pickup element portion 130 shown in FIG. 6 is obtained by extracting a part of a plurality of pixels arranged in a matrix of the image pickup element 24a, and is indicated by four pixels 130a. Further, as shown in FIG. 6, the polarizing filter layer 23b of the optical filter 23 arranged in close contact with the imaging unit 24 is configured by alternately arranging S-polarized light transmitting regions and P-polarized light transmitting regions in a strip shape. The S-polarized light transmitting region portion 120a and the P-polarized light transmitting region portion 120b of the polarizing filter portion 120 shown in FIG. 6 are obtained by extracting a part of the region-divided polarizing filter layer 23b. The S-polarized light transmission regions and the P-polarized light transmission regions alternately formed in a strip shape in the polarizing filter layer 23b are arranged so as to face each pixel row of the image sensor 24a. Specifically, in the paper view of FIG. 6, the upper left and upper right pixels 130a of the image sensor portion 130 face the S-polarized light transmission region portion 120a, and the lower left and lower right pixels 130a of the image sensor portion 130 are , Opposite to the P-polarized light transmission region portion 120b.

  The above-mentioned “pixel” indicates the minimum unit of the light detection unit in the image sensor 24a of the imaging unit 24, and is detected by the minimum unit of the detection unit in the image captured by the image sensor 24a. Let us denote the minimum unit of an image.

  As described above, since there is a certain distance between the effective ray range 113a of the triangular prism 110a and the effective ray range 113b of the triangular prism 110b, the S-polarized image and the P-polarized image captured by the imaging unit 24 are It is formed as an image with parallax. Therefore, the stereo camera 10 can generate a parallax image from an S-polarized image and a P-polarized image having parallax, and calculate the distance to the subject from the parallax image.

  Compared to a conventional stereo camera in which two sets of one imaging unit and one imaging lens are arranged in parallel, the stereo camera 10 of the present embodiment has one imaging unit and one imaging lens. Since it can be configured, the cost can be reduced.

  In addition, in a conventional stereo camera having two imaging lenses, there is an error in the calculation of the distance to the subject due to a change in the baseline length due to thermal expansion or aging due to a temperature change of the housing supporting each imaging lens. May occur. However, since the stereo camera 10 according to the present embodiment has one optical system including the imaging lens, the thermal expansion coefficient of the optical system is small with respect to the casing supporting the optical system, and the baseline length is long. The influence on the calculation of the distance to the subject due to the change can be suppressed. That is, highly accurate distortion correction can be realized without being affected by changes in the amount of distortion caused by the optical system accompanying changes in temperature or changes over time, and changes in the mounting position of the imaging unit. More specifically, since the configuration is such that two images having parallax are superimposed and imaged on the imaging unit via one optical system, distortion of the optical system has occurred due to temperature changes or changes over time. In this case, the influence of the distortion extends equally to each of the two images having parallax. Similarly, when the mounting position of the imaging unit is shifted due to temperature change or temporal change, two images having parallax change in the same manner on the light receiving surface of the imaging unit due to the shift. As a result, it is possible to obtain a stereo camera that is not affected by a change in the distortion amount of the optical system caused by a temperature change or a change with time and a change in the mounting position of the imaging unit.

<Configuration and operation of coordinate transformation processing unit>
FIG. 7 is a block diagram illustrating an example of the configuration of the coordinate conversion processing unit in the processing unit. FIG. 8 is a flowchart showing the operation of the coordinate conversion data switching unit in the coordinate conversion processing unit. FIG. 9 is a block diagram illustrating an example of the configuration of the coordinate calculation unit in the coordinate conversion processing unit. FIG. 10 is a timing chart of operations of the initial value calculation unit and the recurrence formula calculation unit in the coordinate calculation unit. FIG. 11 is a diagram for explaining the operation of the read address calculation unit in the coordinate conversion processing unit. FIG. 12 is a conceptual diagram illustrating bilinear interpolation processing in the interpolator in the coordinate conversion processing unit. FIG. 13 is a diagram illustrating the operation of the polarization separation processing unit in the coordinate conversion processing unit. The configuration and operation of the coordinate conversion processing unit 42 of the processing unit 30 will be described in detail with reference to FIGS.

  As shown in FIG. 7, the coordinate conversion processing unit 42 includes a line buffer 200, a write address calculation unit 201, an XY coordinate counter 202, an output timing generation unit 203, an X′Y ′ coordinate counter 204, A conversion data switching unit 205 (correction data switching unit), a coordinate calculation unit 206, a read address calculation unit 207, an interpolator 208, and a polarization separation processing unit 209 (separation processing unit) are provided.

  The line buffer 200 is the entire data (hereinafter referred to as “input image data”) of an image (hereinafter referred to as “input image”) captured by the imaging unit 24 and subjected to various types of image processing by the image processing unit 41. A memory that temporarily holds a predetermined amount of data. Here, the input image data specifically refers to an aggregate of pixel data detected by each pixel of the image sensor 24a of the imaging unit 24 and converted into a predetermined value. The line buffer 200 stores the pixel data of the input image based on the write address addr_w calculated by the write address calculation unit 201 at the time of writing. At this time, the line buffer 200 is overwritten with pixel data stored before a predetermined amount of data based on the write address addr_w calculated by the write address calculator 201, and is used like a ring buffer. In the line buffer 200, pixel data for four pixels is read by the interpolator 208 based on the read address addr_r calculated by the read address calculator 207 at the time of reading.

  The write address calculation unit 201 increments the write address addr_w based on the input synchronization signal synchronized with the input image data, and calculates the write address addr_w for writing the pixel data of the input image to the line buffer 200. specify. The write address calculation unit 201 passes the calculated write address addr_w to the line buffer 200 and the read address calculation unit 207.

  Based on the input synchronization signal, the XY coordinate counter 202 corresponds to which pixel of the input image the pixel data of the input image currently input corresponds to (which position of the image sensor 24a arranged in a matrix). ). That is, the XY coordinate counter 202 calculates the coordinate value (X, Y) of the pixel corresponding to the pixel data of the input image currently input. The XY coordinate counter 202 passes the calculated coordinate value (X, Y) to the read address calculation unit 207.

  The output timing generation unit 203 generates an output synchronization signal that delays the reading of pixel data while a predetermined amount of data of the input image data is written to the line buffer 200.

  The X′Y ′ coordinate counter 204 calculates the coordinate value (X ′, Y ′) of the output image constructed by the interpolator 208 based on the output synchronization signal generated by the output timing generation unit 203. That is, the coordinate value (X ′, Y ′) of the output image becomes a coordinate value that is shifted from the coordinate value (X, Y) of the input image by a predetermined processing delay calculated by the output timing generation unit 203. . The X′Y ′ coordinate counter 204 passes the calculated coordinate values (X ′, Y ′) to the coordinate conversion data switching unit 205, the coordinate calculation unit 206, and the read address calculation unit 207.

  The coordinate conversion data switching unit 205 uses the coordinate values (X ′, Y ′) of the output image as the coordinate conversion data used for the coordinate conversion processing by the subsequent coordinate calculation unit 206, and the first coordinates for the S-polarized image. Switching between using conversion data (first correction data) or using second coordinate conversion data (second correction data) for a P-polarized image is switched. In the present embodiment, for example, the odd-numbered lines in the input image are input to the line buffer 200 as pixel rows that form an S-polarized image, and the even-numbered lines are input to the line buffer 200 as a pixel row that forms a P-polarized image. The conversion data switching unit 205 switches the coordinate conversion data depending on which line the coordinate value (X ′, Y ′) indicates. Here, the coordinate conversion data is a coefficient parameter of a high-order polynomial for calculating an initial value for the coordinate movement amount (ΔX, ΔY) calculated by the subsequent coordinate calculation unit 206, and is stored in the ROM 33 or the like. Yes. The coordinate conversion data switching unit 205 passes the switched coordinate conversion data to the coordinate calculation unit 206.

  When a deviation occurs due to a temperature change or a change over time, the coordinate conversion data which is a coefficient of the above-described higher-order polynomial is recalculated by the CPU 32 or the like inside the stereo camera 10 and is rewritable nonvolatile such as an EEPROM. It may be held in the memory.

  Details of the operation of the coordinate conversion data switching unit 205 will be described with reference to FIG.

<< Step S11 >>
The coordinate conversion data switching unit 205 uses the X coordinate value X ′ of the coordinate value (X ′, Y ′) received from the X′Y ′ coordinate counter 204, and the coordinate value (X ′, Y ′) forms an image. It is determined whether or not the coordinate value indicates the first pixel of each line. That is, the start of the line is detected. When the coordinate conversion data switching unit 205 detects the start of a line, the process proceeds to step S12. Here, it is determined whether or not the first pixel of each line is indicated by the X coordinate value X ′ of the coordinate value (X ′, Y ′), but is not limited to this. That is, whether or not the coordinate conversion data switching unit 205 indicates the first pixel of each line by the horizontal synchronization signal (Hsync signal) or the line data valid signal (Line Valid signal) output from the output timing generation unit 203. May be determined.

<< Step S12 >>
The coordinate conversion data switching unit 205 determines from the Y coordinate value Y ′ of the coordinate value (X ′, Y ′) whether the pixel indicated by the coordinate value (X ′, Y ′) is an odd line or an even line in the image. It is determined whether it is. If it is determined that the line is an odd line (step S12: Yes), the process proceeds to step S13. If it is determined that the line is an even line (step S12: No), the process proceeds to step S14.

<< Step S13 >>
The coordinate conversion data switching unit 205 determines that the pixel indicated by the coordinate value (X ′, Y ′) belongs to the odd line of the output image, that is, the pixel indicated by the coordinate value (X ′, Y ′) is an S-polarized image. And the first coordinate conversion data is passed to the coordinate calculation unit 206. Then, the coordinate conversion data switching operation ends.

<< Step S14 >>
The coordinate conversion data switching unit 205 determines that the pixel indicated by the coordinate value (X ′, Y ′) belongs to the even line of the output image, that is, the pixel indicated by the coordinate value (X ′, Y ′) is a P-polarized image. And the second coordinate conversion data is passed to the coordinate calculation unit 206. Then, the coordinate conversion data switching operation ends.

  As described above, the coordinate conversion data switching unit 205 switches the coordinate conversion data used for the coordinate conversion processing by the coordinate calculation unit 206 for each line.

  The coordinate calculation unit 206 performs a coordinate conversion process on a pixel basis in order to perform distortion correction on the input image using a high-order polynomial and a recurrence formula. Specifically, the coordinate calculation unit 206 outputs the coordinate value of the pixel in the input image (the coordinate conversion source) to the pixel indicated by the coordinate value (X ′, Y ′) (the coordinate value of the coordinate conversion destination pixel) in the output image. The coordinate movement amount (ΔX, ΔY) of the pixel coordinate value) is calculated. That is, the coordinate calculation unit 206 calculates a coordinate conversion amount for correcting distortion. The coordinate calculation unit 206 passes the calculated coordinate movement amount (ΔX, ΔY) to the read address calculation unit 207.

  Here, the block configuration of the coordinate calculation unit 206 and the operation of each processing unit will be described with reference to FIGS. 9 and 10. As shown in FIG. 9, the coordinate calculation unit 206 includes an initial value calculation unit 220 and a recurrence formula calculation unit 221.

  The initial value calculation unit 220 calculates an initial value used in a recurrence formula (to be described later) of the input image by a high-order polynomial based on a coefficient that is changed according to the correction ratio with respect to the maximum correction amount of the input image based on the output synchronization signal. The calculation is performed for each line in the main scanning direction (the direction of the pixel column in which the S-polarized component or the P-polarized component is detected by the image sensor 24a). Here, when the coordinate value (X ′, Y ′) is a pixel belonging to the pixel column constituting the S-polarized image, the initial value calculation unit 220 receives the first coordinate conversion from the coordinate conversion data switching unit 205 as the above coefficient. Receive parameters. In addition, when the coordinate value (X ′, Y ′) is a pixel belonging to the pixel column constituting the P-polarized image, the initial value calculation unit 220 receives the second coordinate conversion parameter from the coordinate conversion data switching unit 205 as the above-described coefficient. Receive.

  Specifically, the initial value calculation unit 220 uses the following equation (1), which is a high-order polynomial using the coefficient received from the coordinate conversion data switching unit 205, to determine the X axis (main On the line in the scanning direction), correction values for the leftmost three pixels are obtained.

f (x) = cx (3) * X ^ 2 + cx (4) * X * Y + cx (5) * Y ^ 2 + cx (1) * X + cx (2) * Y + cx (0) (1)
x: 1 to 3 (number of repetitions for 3 pixels)
X: X coordinate Y: Y coordinate cx (0) to cx (5): coefficient of higher order polynomial ^: power

  Next, the initial value calculation unit 220 uses the correction amounts f (1) to f (3) for three pixels calculated by the equation (1), and uses the following equations (2) to (4) to describe later. The initial value (initial value at Y coordinate Y ′) of the recurrence formula is obtained.

d0 (1) = f (1) (2)
d1 (1) = f (2) -f (1) (3)
d2 = f (3) −2 × f (2) + f (1) (4)
d0 (1), d1 (1), d2: initial value of the recurrence formula

  Then, the initial value calculation unit 220 passes the initial values d0 (1), d1 (1), and d2 obtained by calculation using the equations (2) to (4) to the recurrence equation calculation unit 221. Note that the high-order polynomial shown in the equation (1) used by the initial value calculation unit 220 as described above is shown as a second-order polynomial, but is not limited to this, and is a higher-order (third-order or higher) polynomial. May be used.

  The recurrence equation calculation unit 221 uses the initial values d0 (1), d1 (1), and d2 calculated by the initial value calculation unit 220 based on the output synchronization signal, and uses the following equations (5) and ( The coordinate transformation amount for the coordinate transformation process is sequentially calculated for each pixel from the left end on the line in the main scanning direction by the recurrence formula based on the Newton forward difference method shown in 6).

d0 (n + 1) = d0 (n) + d1 (n) (5)
d1 (n + 1) = d1 (n) + d2 (6)
d0 (n): Coordinate conversion amount n: Indicates the number of pixels on the line from the left end (1 to n)

  The recurrence formula calculation unit 221 calculates the coordinates of the pixel in the input image with respect to the coordinate value (X ′, Y ′) in the output image by using d0 (n) calculated by the recurrence formula shown in the formulas (5) and (6). The coordinate movement amount ΔX with respect to the X coordinate of the value is set.

  Similarly, the initial value calculation unit 220 uses the high-order polynomial with a coefficient changed according to the correction ratio with respect to the maximum correction amount of the input image based on the output synchronization signal, and sets the initial value used in the recurrence formula as the input image. For each line in the sub-scanning direction (direction perpendicular to the main scanning direction). Specifically, the initial value calculation unit 220 obtains correction values for the top three pixels on the Y-axis (sub-scanning direction) line corresponding to the X coordinate value X ′ by a high-order polynomial.

  Next, the initial value calculation unit 220 uses the correction amount for three pixels calculated by a high-order polynomial, and the initial value (X coordinate X ′ ′) of the recurrence formula using the above formulas (2) to (4). (Initial value at). Then, the initial value calculation unit 220 passes the initial values d0 (1), d1 (1), and d2 obtained by calculation using the equations (2) to (4) to the recurrence equation calculation unit 221.

  The recurrence formula calculation unit 221 uses the initial values d0 (1), d1 (1), and d2 calculated by the initial value calculation unit 220 based on the output synchronization signal, and uses the above formulas (5) and ( The coordinate transformation amount for the coordinate transformation process is sequentially calculated for each pixel from the upper end on the line in the sub-scanning direction by the recurrence formula shown in 6). The recurrence formula calculation unit 221 calculates the coordinates of the pixel in the input image with respect to the coordinate value (X ′, Y ′) in the output image by using d0 (n) calculated by the recurrence formula shown in the formulas (5) and (6). The coordinate movement amount ΔY about the Y coordinate of the value is assumed.

  Through the operation as described above, the coordinate calculation unit 206 causes the coordinate value of the pixel in the input image (the coordinate value of the pixel as the coordinate conversion source) to the coordinate value (X ′, Y ′) (the coordinate value of the pixel as the coordinate conversion destination) in the output image. The coordinate movement amount (ΔX, ΔY) of (coordinate value) is calculated and passed to the read address calculation unit 207.

  Next, with reference to FIG. 10, an operation in which the initial value calculation unit 220 and the recurrence formula calculation unit 221 of the coordinate calculation unit 206 execute an initial value calculation and a recurrence formula calculation by pipeline processing, respectively, will be described. To do.

  The initial value calculation unit 220 calculates the initial value of the recurrence formula for each line in the main scanning direction in accordance with the timing of the output synchronization signal, and passes the calculated initial value to the recurrence formula calculation unit 221. The recurrence formula computing unit 221 computes the coordinate movement amount by the recurrence formula for each pixel in one line in the main scanning direction based on the received initial value. That is, the recurrence formula calculation unit 221 calculates the coordinate movement amount (ΔX, ΔY) (coordinate movement amount for each pixel) by reflecting the initial value calculated by the initial value calculation unit 220 in the recurrence formula calculation. To do.

  The calculation processing of the initial value calculation unit 220 and the recurrence formula calculation unit 221 as described above is executed in parallel by pipeline processing as shown in FIG. Specifically, while the recurrence formula calculating unit 221 calculates the coordinate movement amount on the n line in the main scanning direction, the initial value calculating unit 220 sets the initial recurrence formula in the n + 1 line in the main scanning direction. Find the value. Such a recursive equation calculation process by the recursive equation calculation unit 221 is executed with a delay of one line from the initial value calculation process by the initial value calculation unit 220, thereby enabling pipeline processing and coordinates. The speed of coordinate conversion processing by the calculation unit 206 can be increased.

  Note that the coordinate conversion processing of the coordinate calculation unit 206 described above may be realized as follows. For example, since the initial value calculation unit 220 executes a calculation including a multiplication process as shown in the above equation (1), although the process is complicated, the calculation process for three pixels in one line (three calculations) The processing time can be afforded. On the other hand, since the recurrence formula calculation unit 221 executes the calculation by addition processing as shown in the above formulas (5) and (6), although the processing is simple, the calculation is performed for every pixel in one line. Since the process is executed, there is no allowance for calculation time. Therefore, it is preferable that the coordinate calculation unit 206 realizes the calculation processing function of the initial value calculation unit 220 by software such as a program, and realizes the calculation processing function of the recurrence formula calculation unit 221 by a hardware circuit. It is. In this case, a program for realizing the calculation processing function of the initial value calculation unit 220 is provided by being incorporated in advance in a storage device such as the ROM 33. The program has a module configuration including the calculation processing function of the initial value calculation unit 220. When the CPU 32 reads the program from the ROM 33 and executes it, the program is loaded on the RAM 34, and the initial value calculation unit 220 is generated on the RAM 34. Is done.

  The read address calculation unit 207 receives the write address addr_w received from the write address calculation unit 201, the coordinate values (X, Y) received from the XY coordinate counter 202, and the coordinate values received from the X′Y ′ coordinate counter 204 (X ', Y') and the coordinate movement amount (ΔX, ΔY) received from the coordinate calculation unit 206, the read address addr_r for the line buffer 200 is calculated.

  Here, as described above, the input image input to the coordinate transformation processing unit 42 includes an S-polarized image by light incident from the incident unit 21a and a P-polarized image by light incident from the incident unit 21b for each line. Are arranged alternately. The S-polarized image has distortion due to transmission through the triangular prism 110a, the cross prism 100, and the imaging lens 114 in the optical system 22. On the other hand, the P-polarized image has distortion due to transmission through the triangular prism 110 b, the cross prism 100 and the imaging lens 114 in the optical system 22. Furthermore, the S-polarized image and the P-polarized image are images having parallax. Therefore, when the read address calculation unit 207 uses the coordinate movement amount (ΔX, ΔY) calculated by the coordinate calculation unit 206 as it is, the pixel data of the pixels constituting the P-polarized image is referred to for distortion correction of the S-polarized image. There is a risk that the pixel data of the pixels constituting the S-polarized image may be referred to for correcting the distortion of the P-polarized image. Therefore, the read address calculation unit 207 refers to only pixel data of pixels constituting the S-polarized image for distortion correction of the S-polarized image and calculates distortion of the P-polarized image in order to calculate the read address addr_r. For this purpose, it is necessary to adjust the coordinate movement amounts (ΔX, ΔY) so as to refer only to the pixel data of the pixels constituting the P-polarized image.

  FIG. 11A shows a case where the amount of coordinate movement (ΔX, ΔY) obtained by the coordinate calculation unit 206 is (−5.2, −4.5). That is, the transformation source coordinates 310a in the input image are 5.2 pixels in the X direction and 4.5 pixels in the Y direction with respect to the coordinates of the output pixel 301 indicated by the coordinate value (X ′, Y ′) in the output image 300. It shows that it has shifted. On the other hand, the coordinates of the output pixel 301 in the output image 300 are shifted by the coordinate movement amount (ΔX, ΔY) = (− 5.2, −4.5) with respect to the conversion source coordinate 310a in the input image. Show. However, in actuality, the coordinate movement amount (ΔX, ΔY) is a value including a decimal part, so the movement amount (−5, −4) is obtained by discarding the decimal part, and the output pixel 301 is a conversion source pixel in the input image. It is shifted by a moving amount (−5, −4) with respect to 302a. In FIG. 11, on the output image 300, for convenience of explanation, a conversion source pixel in the input image (a conversion source pixel 302a in FIG. 11A and a conversion source pixel 302b in FIG. 11B) is shown. .

  FIG. 11B shows a case where the coordinate movement amounts (ΔX, ΔY) obtained by the coordinate calculation unit 206 are (−5.2, −3.7). That is, the transformation source coordinates 310b in the input image are 5.2 pixels in the X direction and 3.7 pixels in the Y direction with respect to the coordinates of the output pixel 301 indicated by the coordinate value (X ′, Y ′) in the output image 300. It shows that it has shifted. Conversely, the coordinates of the output pixel 301 in the output image 300 are shifted by the coordinate movement amount (ΔX, ΔY) = (− 5.2, −3.7) with respect to the conversion source coordinate 310b in the input image. Show. Similarly, since the coordinate movement amount (ΔX, ΔY) is a value including a decimal part, the movement amount (−5, −3) is obtained by discarding the decimal part, and the output pixel 301 is a conversion source pixel in the input image. It is considered that the movement amount is shifted by (−5, −3) with respect to 302b. However, in this case, since the Y component of the movement amount (−5, −3) is an odd number, if the output pixel 301 is a pixel of an S-polarized image, the conversion source pixel 302b is a pixel of a P-polarized image. End up.

  From the above, it is assumed that the coordinate movement amount (ΔX, ΔY) is converted into the coordinate movement adjustment amount (ΔX ′, ΔY ′) according to the following rule. When the integer part of ΔY is an even number (including 0), ΔY ′ = ΔY, and when it is an odd number, ΔY ′ = ΔY + 1. Then, ΔX ′ = ΔX without any adjustment for ΔX.

  In the case of FIG. 11A, according to the above-mentioned law, the coordinate movement amount (ΔX, ΔY) and the coordinate movement adjustment amount (ΔX ′, ΔY ′) are the same, and (ΔX ′, ΔY ′) = (− 5.2, -4.5). On the other hand, in the case of FIG. 11B, the coordinate movement amount (ΔX, ΔY) = (− 5.2, −3.7) is equal to the coordinate movement adjustment amount (ΔX ′, ΔY ′) = ( -5.2, -2.7). Thus, by obtaining the coordinate movement adjustment amounts (ΔX ′, ΔY ′), as shown in FIG. 11B, the coordinates of the conversion source pixel 302b, which is the pixel of the S-polarized image, are converted into the coordinate movement adjustment amount ( The coordinates of the output image 300, which is a pixel of the S-polarized image, are obtained by shifting by an integer part of (ΔX ′, ΔY ′).

  From the above, the coordinate movement adjustment amounts (ΔX ′, ΔY ′) are the pixels in the input image (conversion source pixels 302a, 302b) with respect to the output pixel 301 indicated by the coordinate values (X ′, Y ′) in the output image 300. The amount of movement.

  Next, the read address calculation unit 207 calculates the read address addr_r in the line buffer 200 in which the pixel data of the conversion source pixels 302a and 302b is stored according to the following equation (7).

addr_r = addr_w− (Y−Y ′ + ΔY ′) × Linepix− (X−X ′ + ΔX ′) (7)
addr_r: read address addr_w: write address X, Y: coordinate value of pixel corresponding to input pixel data of input image X ′, Y ′: coordinate value of pixel in output image targeted for distortion correction ΔX ′ , ΔY ′: Coordinate movement adjustment amount of conversion source pixel in input image Linepix: number of pixels in one line

  The read address calculation unit 207 passes the read address addr_r calculated by Expression (7) to the line buffer 200, and data of the decimal part of the coordinate movement adjustment amount (ΔX ′, ΔY ′) (coordinate movement amount (ΔX, ΔY)). Is passed to the interpolator 208. Here, when the coordinate value of the conversion source pixel (302a, 302b) indicated by the read address addr_r is set to (X0, Y0), the following equation (8) is established.

  (X0, Y0) = (X′−ΔX ′, Y′−ΔY ′) (8)

  Based on the read address addr_r received from the read address calculation unit 207 and the coordinate value (X0, Y0) of the conversion source pixel, the line buffer 200 uses the coordinate values (X0, Y0), (X0 + 1, Y0), ( X0, Y0 + 2) and (X0 + 1, Y0 + 2), the conversion source pixel 302a and the element pixels 303a, 304a, and 305a (in FIG. 11B, the conversion source pixel 302b and the element pixels 303b, 304b, and 305b). Pixel data is read and passed to the interpolator 208.

  For example, in FIG. 11A, regarding the positional relationship between the conversion source pixel 302a and the element pixels 303a, 304a, and 305a, the conversion source pixel 302a corresponds to the upper left pixel in the paper view of FIG. It is not limited to this. That is, the conversion source pixel 302a may be in the lower left, upper right, or lower right in the positional relationship with the element pixels 303a, 304a, and 305a.

  The interpolator 208 includes the four pixel data received from the line buffer 200 and the decimal part of the coordinate movement adjustment amount (ΔX ′, ΔY ′) (coordinate movement amount (ΔX, ΔY)) received from the read address calculation unit 207. Based on the data, bilinear interpolation processing is executed to construct an output image.

  Details of the bilinear interpolation processing will be described with reference to FIG. As shown in FIG. 11A, it is assumed that the coordinate calculation unit 206 calculates the coordinate movement amount (ΔX, ΔY) = (− 5.2, −4.5). Further, as shown in FIG. 11A, the pixel data of the conversion source pixel 302a and the element pixels 303a, 304a, and 305a are respectively represented by D (X0, Y0), D (X0 + 1, Y0), D (X0, Y0 + 2), and Let D (X0 + 1, Y0 + 2). Note that the pixels of the P-polarized image between the conversion source pixel 302a and the element pixel 303a and the element pixels 304a and 305a are ignored. That is, the distance in the Y direction between the conversion source pixel 302a and the element pixel 303a and the element pixels 304a and 305a is assumed to be “1”. Further, in the decimal part of the coordinate movement amount (ΔX, ΔY), “0.2” as the X component is ax, and “0.5” as the Y component is ay.

  The interpolator 208 calculates the data D (X ′, Y ′) of the portion corresponding to the calculation target position 306 shown in FIG. 12 by the following equation (9) indicating the bilinear interpolation process. The calculation target position 306 corresponds to the conversion source coordinates 310a illustrated in FIG.

D (X ′, Y ′) = (1-ax) × (1-ay) × D (X0, Y0)
+ Ax × (1-ay) × D (X0 + 1, Y0)
+ (1-ax) × ay × D (X0, Y0 + 2)
+ Ax × ay × D (X0 + 1, Y0 + 2) (9)

  The interpolator 208 passes the data D (X ′, Y ′) calculated by Expression (9) to the polarization separation processing unit 209 as pixel data of the output pixel 301 in the output image 300. Thus, the output image data constituted by the pixel data calculated by the interpolator 208 becomes image data obtained by executing distortion correction processing on the input image data.

  When at least one of the element pixels 303a, 304a, and 305a falls outside the range of the input image, the pixel data of these pixels may be handled as “0” or a predetermined value.

  The polarization separation processing unit 209 separates the output image received from the interpolator 208 into an S-polarized image (first output image) and a P-polarized image (second output image). Specifically, as illustrated in FIG. 13, the polarization separation processing unit 209 separates the first output image 321a from the S component image region 320a in the output image 320 received from the interpolator 208, and generates a P component image. The second output image 321b is separated from the region 320b. Then, the polarization separation processing unit 209 passes the first output image data that is the data of the first output image 321 a and the second output image data that is the data of the second output image 321 b to the distance calculation unit 43.

  As described above, when the parallax with respect to the subject is calculated based on the first output image and the second output image simply separated from the output image by the polarization separation processing unit 209, Y in the first output image and the second output image is calculated. Since the corresponding pixel column in the direction is shifted by one pixel, an error may occur. Therefore, the polarization separation processing unit 209 inserts data that is linearly interpolated with the pixel data of the adjacent pixels between the adjacent pixels in the pixel column in the Y direction in the separated first output image and second output image. It is desirable. As a result, the linearly interpolated first output image and second output image have the same size as the input image or output image, and the distance calculation unit 43 calculates the distance to the subject with high accuracy from the two output images. can do. In the first output image and the second output image separated by the polarization separation processing unit 209, the method for interpolating between adjacent pixels in the pixel column in the Y direction is not limited to linear interpolation. It is also possible to use other interpolation processing.

<Overall flow of distortion correction processing>
FIG. 14 is a flowchart showing the overall operation of the stereo camera according to the first embodiment. The overall operation of the distortion correction processing in the coordinate conversion processing unit 42 will be described with reference to FIG.

<< Step S21 >>
The image input unit 20 receives light from the front of the stereo camera 10, and simultaneously detects the S-polarized light and P-polarized light by the optical system 22 and picks up the light as one image.

<< Step S22 >>
The image processing unit 41 performs various types of image processing such as γ correction on the image captured by the image input unit 20 and passes the processed image as an input image to the coordinate conversion processing unit 42.

<< Step S23 >>
The line buffer 200 of the coordinate conversion processing unit 42 temporarily holds a predetermined amount of data among input image data (input image data) from the image processing unit 41. In addition, the XY coordinate counter 202 of the coordinate conversion processing unit 42 calculates the coordinate value (X, Y) of the pixel corresponding to the pixel data of the input image currently input. The output timing generation unit 203 of the coordinate conversion processing unit 42 generates an output synchronization signal that delays the reading of the pixel data while a predetermined amount of data of the input image data is written to the line buffer 200.

<< Step S24 >>
The X′Y ′ coordinate counter 204 determines which pixel of the output image constructed by the interpolator 208 corresponds to the pixel data of the input image based on the output synchronization signal generated by the output timing generation unit 203. The indicated coordinate value (X ′, Y ′) is calculated.

<< Step S25 >>
The coordinate conversion data switching unit 205 uses the first coordinate conversion data for the S-polarized image or the P-polarized light for the coordinate conversion processing by the coordinate calculation unit 206 based on the coordinate values (X ′, Y ′) of the output image. Whether to use the second coordinate conversion data for the image is switched.

<< Step S26 >>
As the coordinate conversion process, the coordinate calculation unit 206 performs the coordinate movement amount of the pixel coordinate value (coordinate value of the coordinate conversion source pixel) in the input image with respect to the output pixel indicated by the coordinate value (X ′, Y ′) in the output image Calculate (ΔX, ΔY).

<< Step S27 >>
The read address calculation unit 207 receives the write address addr_w received from the write address calculation unit 201, the coordinate values (X, Y) received from the XY coordinate counter 202, and the coordinate values received from the X′Y ′ coordinate counter 204 (X ', Y') and the coordinate movement amount (ΔX, ΔY) received from the coordinate calculation unit 206, the read address addr_r for the line buffer 200 is calculated.

<< Step S28 >>
The line buffer 200 reads the four pixel data of the conversion source pixel and the three element pixels in the input image based on the read address addr_r and the coordinate value (X0, Y0) of the conversion source pixel and passes them to the interpolator 208. .

<< Step S29 >>
The interpolator 208 executes bilinear interpolation processing from the four pixel data, constructs an output image using the calculated data as pixel data of the output image, and passes it to the polarization separation processing unit 209.

<< Step S30 >>
The polarization separation processing unit 209 separates the output image received from the interpolator 208 into an S-polarized image (first output image) and a P-polarized image (second output image).

  As described above, since the stereo camera 10 according to the present embodiment can be configured with one imaging unit and one imaging lens, cost can be reduced.

  In addition, since the stereo camera 10 according to the present embodiment has one optical system including the imaging lens, the thermal expansion coefficient of the optical system is small with respect to the casing supporting the optical system, and the baseline length is long. The influence on the calculation of the distance to the subject due to the change can be suppressed. That is, highly accurate distortion correction can be realized without being affected by changes in the amount of distortion caused by the optical system accompanying changes in temperature or changes over time, and changes in the mounting position of the imaging unit. More specifically, since the configuration is such that two images having parallax are superimposed and imaged on the imaging unit via one optical system, distortion of the optical system has occurred due to temperature changes or changes over time. In this case, the influence of the distortion extends equally to each of the two images having parallax. Similarly, when the mounting position of the imaging unit is shifted due to temperature change or temporal change, two images having parallax change in the same manner on the light receiving surface of the imaging unit due to the shift. As a result, it is possible to obtain a stereo camera that is not affected by a change in the distortion amount of the optical system caused by a temperature change or a change with time and a change in the mounting position of the imaging unit.

  Further, the light incident from the two incident portions is picked up by one image pickup portion to form an image, and each image is obtained using coordinate conversion data corresponding to each of the S-polarized image and the P-polarized image included in the image. On the other hand, distortion correction can be performed by one coordinate conversion processing unit. Therefore, the stereo camera according to the present embodiment only needs to include one coordinate conversion processing unit as compared with the conventional stereo camera that includes two coordinate conversion processing units for distortion correction for each camera. Therefore, the circuit scale can be reduced and the cost can be reduced.

<Modification>
FIG. 15 is a block diagram illustrating a configuration of a coordinate calculation unit according to a modification of the first embodiment. The configuration and operation of the coordinate calculation unit 206a of this modification will be described with reference to FIG. The coordinate calculation unit 206 according to the first embodiment executes a coordinate conversion process for correcting distortion of an input image using a high-order polynomial and a recurrence formula from the coordinate values (X ′, Y ′), that is, The coordinate movement amount (ΔX, ΔY) was calculated. The coordinate calculation unit 206a according to the present modification calculates coordinate movement amounts (ΔX, ΔY) by referring to a lookup table (LUT).

  As shown in FIG. 15, the coordinate calculation unit 206 a includes an LUT reference unit 222. The LUT reference unit 222 receives data (LUT data) of coordinate movement amounts (ΔX, ΔY) uniquely determined from the coordinate values (X ′, Y ′) based on the output synchronization signal from the coordinate conversion data switching unit 205. The LUT data is obtained by calibration performed in advance, and is a value unique to the stereo camera 10. The coordinate conversion data switching unit 205 refers to the LUT stored in advance in the ROM 33, and reads LUT data that is data of coordinate movement amounts (ΔX, ΔY) uniquely determined from the input coordinate values (X ′, Y ′). The data is passed to the LUT reference unit 222. The coordinate calculation unit 206 a passes the coordinate movement amount (ΔX, ΔY), which is the LUT data, received from the coordinate conversion data switching unit 205 to the read address calculation unit 207.

  When a deviation occurs due to a temperature change or a change with time, the LUT data described above is recalculated by the CPU 32 or the like inside the stereo camera 10 and stored in a rewritable nonvolatile memory such as an EEPROM. That's fine.

  As described above, it is necessary to perform complicated calculation processing in the coordinate calculation unit 206a by using the LUT storing the data of the coordinate movement amounts (ΔX, ΔY) uniquely determined from the coordinate values (X ′, Y ′). There is no. Accordingly, it is possible to increase the speed of the distortion correction process.

(Second Embodiment)
The stereo camera according to the second embodiment will be described focusing on differences from the operation of the stereo camera 10 according to the first embodiment. In the first embodiment, the stereo camera 10 includes an S-polarized image incident from the incident unit 21 a and captured by the imaging unit 24, and a P-polarized image incident from the incident unit 21 b and captured by the imaging unit 24. The operation of acquiring the position information of the subject by using the two images has been described. In the present embodiment, in addition to a stereo application (hereinafter referred to as “stereo application”) that uses position information of a subject based on parallax detected from these two images, it is incident from one of two incident units 21. An operation for executing a monocular application (hereinafter, referred to as “monocular application”) using only the captured image will be described.

  Examples of the stereo application include a brake control application for detecting an obstacle by detecting position information of a subject to decelerate and stop the vehicle. Examples of the monocular application include applications that execute lane departure warning by white line recognition, lane keeping assist, sign recognition, headlight light distribution control, and the like.

  FIG. 16 is a diagram for explaining an example of using an input image in various applications of the stereo camera according to the second embodiment. The operation of executing the stereo application and the monocular application will be described with reference to FIG. An S-polarized image in which the S-polarized component of light incident from the incident unit 21a is captured by the imaging unit 24 is a first input image, and a P-polarized image in which the P-polarized component of light incident from the incident unit 21b is captured by the imaging unit 24 Is a second input image. In FIG. 16, the monocular application uses only the first input image, for example.

  The stereo camera 10 calculates parallax based on the first output image and the second output image generated by the coordinate conversion processing unit 42 from the first input image and the second input image, respectively, and obtains result information for the stereo application. The result information is transmitted to the vehicle control device 11. Further, the stereo camera 10 generates result information for the monocular application based on the first output image generated by the coordinate conversion processing unit 42 using only the first input image, and the result information is used as the vehicle control device. 11 to send.

  The vehicle control device 11 (see FIG. 2) stores software that is a stereo application and a monocular application in an internal ROM or the like, and an internal CPU reads each application from the ROM. Then, the vehicle control device 11 executes each application using the result information received from the stereo camera 10.

  As shown in FIG. 16, the vehicle control apparatus 11 considers the case where a 1st monocular application, a 2nd monocular application, and a stereo application are performed in order, for example for every 100 ms. In this case, each application is executed in about 33 ms.

  As described above, when the stereo application is executed in the vehicle control device 11, the stereo camera 10 generates the first output respectively generated by the distortion correction processing of the coordinate conversion processing unit 42 from the first input image and the second input image. Based on the image and the second output image, parallax is calculated and result information for the stereo application is generated.

  On the other hand, when the first monocular app or the second monocular app is executed in the vehicle control device 11, the stereo camera 10 generates a first output image by the distortion correction processing of the coordinate conversion processing unit 42 from only the first input image. The result information for the first monocular application or the second monocular application is generated based on the first output image. In this case, the stereo camera 10 does not use the second input image captured based on the light incident from the incident unit 21b, and performs the distortion correction process only on the first input image. Therefore, the coordinate conversion data switching unit 205 (see FIG. 7) of the coordinate conversion processing unit 42 distorts the first input image while the monocular single app or the second monocular app is being executed in the vehicle control device 11. The first coordinate conversion data for performing the correction process is passed to the coordinate calculation unit 206.

  As described above, when the monocular application is executed in the vehicle control device 11, the stereo camera 10 does not need to execute a distortion correction process on the second input image captured based on the light incident from the incident unit 21b. Good. Therefore, since the coordinate conversion processing unit 42 does not need to perform the correction processing operation during the period when the monocular application is being executed, power consumption can be reduced.

  Note that the monocular application uses only the first input image. However, the present invention is not limited to this, and only the second input image captured based on the light incident from the incident unit 21b is used. It is good also as what to do.

DESCRIPTION OF SYMBOLS 1 Vehicle 2 Main body 3 Car interior 10 Stereo camera 11 Vehicle control apparatus 12 Steering wheel 13 Brake pedal 20 Image input part 21a, 21b Incident part 22 Optical system 23 Optical filter 23a Filter substrate 23b Polarizing filter layer 23c Filling layer 24 Imaging part 24a Imaging Element 24b Substrate 26 Housing 30 Processing unit 31 FPGA
32 CPU
33 ROM
34 RAM
41 Image processing unit 42 Coordinate conversion processing unit 43 Distance calculation unit 51 Control unit 52 Recognition processing unit 100 Cross prisms 101a and 101b Polarizing surfaces 102a and 102b Side surfaces 103 Outgoing surfaces 110a and 110b Triangular prisms 111a and 111b Incident surfaces 112a and 112b Reflecting surfaces 113a, 113b Effective ray range 114 Imaging lens 120 Polarization filter portion 120a S-polarization transmission region portion 120b P-polarization transmission region portion 130 Imaging element portion 130a Pixel 200 Line buffer 201 Write address calculation unit 202 XY coordinate counter 203 Output timing generation unit 204 X'Y 'coordinate counter 205 Coordinate conversion data switching unit 206, 206a Coordinate calculation unit 207 Read address calculation unit 208 Interpolator 209 Polarization separation processing unit 220 Initial value calculation Unit 221 recursive equation calculation unit 222 LUT reference unit 300 output image 301 output pixel 302a, 302b conversion source pixel 303a, 303b element pixel 304a, 304b element pixel 305a, 305b element pixel 306 calculation target position 310a, 310b conversion source coordinate 320 output Image 320a S component image area 320b P component image area 321a First output image 321b Second output image addr_r read address addr_w write address

JP 2010-243463 A

Claims (12)

Two incident parts each for receiving light from the subject,
Of the two incident parts, a first light component is extracted from first incident light incident on one incident part, a second light component is extracted from second incident light incident on the other incident part, An optical system that combines one light component and the second light component into one path;
A region dividing section having a plurality of first transmission regions that transmit the first light component and a plurality of second transmission regions that transmit the second light component;
An imaging unit that images the first light component and the second light component that have passed through the region dividing unit to generate one input image;
First correction data for correcting the first image included in the input image and generated from the first light component, and included in the input image and generated from the second light component A correction processing unit that performs a correction process by switching the second correction data for correcting the second image, and generates an output image;
An imaging apparatus comprising:   The region dividing unit is configured such that the first transmission region and the second transmission region are alternately arranged to face each other in one or more pixel columns in a plurality of pixels arranged in the imaging unit. The imaging apparatus according to 1. The correction processing unit includes, from the output image, a first output image obtained by performing the correction process on the first image, and a second output image obtained by performing the correction process on the second image. A separation processing unit for separating and outputting
The imaging apparatus according to claim 1, further comprising a distance calculation unit that detects a parallax with respect to the subject from the first output image and the second output image and calculates a distance to the subject based on the parallax.   The separation processing unit, for each of the separated first output image and the second output image, performs linear interpolation on pixel data of pixels adjacent to each other in a direction in which the size of the output image is reduced. The imaging apparatus according to claim 3, wherein an image generated by being inserted between pixel data of pixels to be output is output as the first output image and the second output image. The correction processing unit
A correction data processing unit that switches and outputs the first correction data when the correction processing is performed on the first image, and the second correction data when the correction processing is performed on the second image;
An initial value calculator that calculates an initial value of a recurrence formula based on the first correction data or the second correction data for each pixel column in the first image or the second image;
A recurrence formula computing unit that computes a coordinate movement amount for moving the coordinates from the pixel of the input image to the pixel of the output image in the correction process by the recurrence formula using the initial value;
The imaging device according to claim 1, wherein:   The correction processing unit includes pixel data of pixels of the input image before coordinate movement based on the coordinate movement amount calculated by the recurrence formula calculation unit, pixel data of a plurality of pixels adjacent to the pixel, and The imaging apparatus according to claim 5, further comprising: an interpolation processing unit that uses the data calculated by executing bilinear interpolation processing using as a pixel data of the pixels of the output image whose coordinates are moved by the coordinate movement amount.   The interpolation processing unit performs the bilinear interpolation processing by assuming that pixel data of a pixel corresponding to a position outside the range of the input image among the plurality of adjacent pixels is a predetermined value. The imaging device described. The first light component is a first polarization component obtained by polarizing the first incident light in a first direction by the optical system,
8. The second light component is a second polarization component obtained by polarizing the second incident light in a second direction different from the first direction by the optical system. 9. Imaging device.   The optical system extracts light obtained by S-polarizing the first incident light with respect to the first surface as the first polarization component, and light obtained by P-polarizing the second incident light with respect to the second surface. The imaging device according to claim 8, wherein the imaging device extracts the second polarization component. The imaging device according to any one of claims 1 to 9,
Switch between a stereo application that uses distance information to the subject calculated from the parallax between the first image and the second image, and a monocular application that uses only the first image or only the second image. A vehicle control device,
An imaging system comprising:   A vehicle equipped with the imaging device according to claim 1.   A vehicle equipped with the imaging system according to claim 10.

Images