JP4710653B2 - In-vehicle image processing apparatus and vehicle image processing method - Google Patents

Description

  The present invention relates to an in-vehicle image processing apparatus and a vehicle image processing method for capturing an image of a vehicle periphery, converting the viewpoint of the image, and presenting an overhead image of the vehicle periphery to a driver or the like.

  Conventionally, a technique for converting image data captured by a camera attached to a vehicle into overhead image data and displaying the overhead image on a monitor in the vehicle is described in Patent Document 1 below. An in-vehicle image processing apparatus is known.

This in-vehicle image processing apparatus detects an angle between a road surface and a camera attached to a vehicle body as a pitch angle [deg] in a global coordinate system (camera coordinate system), and is imaged by the camera based on the pitch angle. An address conversion table for converting camera image data to overhead image data is created. That is, in this in-vehicle image processing device, an address conversion table is created based on a pitch angle in a global coordinate system (camera coordinate system) that represents the mounting state of the camera on the vehicle body, and the address conversion table created based on this pitch angle is used. By converting the image data of the camera to the overhead image data, it is possible to display an optimal overhead image according to the attachment state of the camera to the vehicle body on the monitor.
JP 2004-64441 A

  By the way, in the above-mentioned in-vehicle image processing device, in order to display an overhead image in which the entire area around the vehicle is looked down from a virtual viewpoint above the vehicle using four cameras having the imaging directions in the front-rear and left-right directions of the vehicle. It is necessary to replace the respective pixel data of the four cameras with the respective pixels constituting the overhead image using the address conversion table to create the overhead image data. That is, the address conversion table for creating a bird's-eye view looking down from the virtual viewpoint above the vehicle on the entire area around the vehicle is on the respective input buffers for inputting the camera numbers of the four cameras and the four camera image data. And the address on the output buffer for outputting the overhead image data to the monitor. In the above-described in-vehicle image processing apparatus, it is necessary to create such an address conversion table inside the apparatus.

  When creating a bird's-eye view looking down from the virtual viewpoint above the vehicle as described above, the four camera images are combined to create a complete bird's-eye view with matching seams between the images and without distortion. To do this, the actual mounting position of each camera on the vehicle body, the actual mounting direction of each camera, and the camera mounting position, camera mounting direction, and lens file calculated when creating the address conversion table are , It must match perfectly without error. In other words, when an overhead view image is created using an address conversion table that converts camera image data captured at the normal mounting position and mounting direction in a state where there is a shift in the camera mounting position and mounting direction, A mismatch or distortion of the seam in the overhead image obtained by converting each camera image data occurs. Such a shift in the camera mounting position and mounting direction that causes distortion of the overhead view image or the like is inevitably caused for each vehicle in a camera mounting operation in a factory or a dealer.

  On the other hand, in the in-vehicle image processing apparatus described in Patent Document 1 described above, an address conversion table corresponding to the mounting state of the camera with respect to the vehicle body is created inside the apparatus. In-vehicle image processing is used to detect camera mounting errors and recalculate the address conversion table inside the device based on the mounting errors. The processing load of the apparatus becomes high and a long calculation time is required. Therefore, the in-vehicle image processing apparatus described in Patent Document 1 needs to use an arithmetic device with high processing capability.

  In addition, even if a plurality of address conversion tables corresponding to camera mounting errors are stored in the apparatus in order to reduce the amount of calculation and any one of the address conversion tables is selected according to the actual mounting error, the number is small. Although it is possible to create a bird's-eye view image by correcting the camera mounting error with the amount of calculation, there is a problem that the memory capacity inside the apparatus becomes enormous and the apparatus becomes expensive.

  Therefore, the present invention has been proposed in view of the above-described circumstances, and the overhead capacity image is created from a plurality of images with a simple process without enlarging the memory capacity for creating the overhead image from the plurality of images. It is an object of the present invention to provide an in-vehicle image processing apparatus and a vehicle image processing method.

  In the present invention, a plurality of camera modules corresponding to a plurality of camera modules are used to convert viewpoints of camera image data generated by a plurality of camera modules that capture the periphery of the vehicle and display an overhead image around the vehicle on the display unit. An input means having an input buffer, an output buffer for storing an overhead image, an output means for outputting the overhead image stored in the output buffer to a display means, and an address conversion table used for creating the overhead image Storage means for extracting the camera image data from the input buffer according to the address conversion table, and storing the image data in the output buffer to store the overhead image.

And in this invention, the address conversion table memorize | stored in the memory | storage means is an address conversion table of the initial state for creating a bird's-eye view image using the camera image data in the state without the attachment error with respect to the vehicle of a camera module. an address conversion table is reconstructed by the table reconstruction means according to the parameters for correcting the mounting error with respect to the vehicle of the camera module, Ru is stored camera image data to the output buffer by the image processing means in accordance with the address conversion table. The table reconstruction means determines the address position specified by the input address information in the address conversion table in the initial state according to the parameter corresponding to the error amount of the yaw angle of the camera coordinate system, which is an attachment error of the camera module to the vehicle. Shift horizontally. Alternatively, the table reconstructing means determines the address position specified by the input address information in the address conversion table in the initial state according to the parameter corresponding to the error amount of the pitch angle of the camera coordinate system, which is an attachment error of the camera module to the vehicle. Shift in the vertical direction of the system. Alternatively, the table reconstructing means may image the address position specified by the input address information in the address conversion table in the initial state according to the parameter corresponding to the error amount of the roll angle of the camera coordinate system, which is an attachment error of the camera module to the vehicle. Move in the rotation direction with the arbitrary coordinate of the coordinate system as the rotation center.

In another aspect of the invention , the camera image data is used to convert the viewpoint of each of the camera image data generated by the plurality of camera modules that capture the vehicle periphery and to display the overhead image around the vehicle on the display means. An address conversion table for creating a bird's-eye view image, which is an initial state address conversion table for creating a bird's-eye view image using camera image data with no camera module mounting error. Storing the address conversion table reconstructed according to the parameter for correcting the mounting error of the vehicle in the storage means, and the camera image data generated by the plurality of camera modules in a plurality of input buffers corresponding to each camera module Each storing step and address conversion stored in the storage means The camera image data is extracted from the input buffer according to the table, the camera image data is stored in the output buffer according to the address conversion table stored in the storage means, and an overhead view image is created, and the initial address conversion table includes: The method includes a step of creating an address conversion table that is reconfigured according to a parameter for correcting an attachment error of the camera module to the vehicle and stored in the storage unit.
In the step of creating the address conversion table to be stored in the storage means, the input address information in the address conversion table in the initial state is changed according to the parameter for correcting the mounting error of the camera module to the vehicle, and is stored in the storage means The address position specified by the input address information in the address conversion table in the initial state is shifted in the horizontal direction of the image coordinate system according to the parameter corresponding to the error amount of the yaw angle of the camera coordinate system, which is an attachment error of the camera module to the vehicle. Let Alternatively, in the step of creating the address conversion table to be stored in the storage means, the input address in the address conversion table in the initial state is determined according to the parameter corresponding to the error amount of the pitch angle of the camera coordinate system, which is an attachment error of the camera module to the vehicle The address position specified by the information is shifted in the vertical direction of the image coordinate system. Alternatively, in the step of creating the address conversion table to be stored in the storage means, the input address information in the address conversion table in the initial state according to the parameter corresponding to the error amount of the roll angle of the camera coordinate system, which is an attachment error of the camera module to the vehicle Is moved in the rotation direction with an arbitrary coordinate in the image coordinate system as the rotation center.

  According to the present invention, it is possible to absorb the mounting error of the camera module with respect to the vehicle by a simple process of reconfiguring the address conversion table in the initial state without performing many processes when actually creating the overhead view image. Even if there is an attachment error in any of the plurality of camera modules, the overhead image created by converting the viewpoints of the plurality of camera image data can be an appropriate overhead image with reduced displacement and distortion. . In addition, since the address conversion table is reconstructed from the mounting error of the camera module unique to the vehicle, the memory capacity of the storage means for storing the address conversion table does not become enormous.

  According to another invention, the mounting error of the camera module with respect to the vehicle can be reduced by a simple process of adjusting the camera image data stored in the input buffer without performing many processes when actually creating the overhead view image. Appropriate bird's-eye view image with reduced misalignment and distortion, even if there is a mounting error in any of the camera modules It can be. Further, the adjustment of the camera image data stored in the input buffer absorbs the mounting error of the camera module with respect to the vehicle, so that the memory capacity of the storage means for storing the address conversion table does not become enormous.

[First Embodiment]
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings.

  The present invention is applied to an in-vehicle image processing apparatus 1 configured as shown in FIG. 1, for example. This in-vehicle image processing apparatus 1 uses camera modules 2A, 2B, 2C, and 2D to capture camera image data around the vehicle and input buffers 12A, 12B, 12C, and 2D corresponding to the camera modules 2A, 2B, 2C, and 2D, respectively. 12D, the CPU 13 and the image conversion unit 14 perform a viewpoint conversion process using the address conversion table stored in the table storage unit 15 to create a bird's-eye view image around the vehicle, and the overhead image data in the output buffer 16 Is stored and output to the monitor 3. The in-vehicle image processing apparatus 1 is a camera in which there is no error in mounting the default address conversion table, that is, the camera modules 2A, 2B, 2C, and 2D, for example, by function units 41 to 43 as shown in FIG. By reconstructing an address conversion table in an initial state for creating a bird's-eye view image using the image data, a bird's-eye view image distortion caused by an attachment error of the camera modules 2A, 2B, 2C, and 2D and a bird's-eye view image are configured. A shift between the images is suppressed, and an appropriate overhead image without any sense of incongruity is displayed on the monitor 3 as one continuous image.

  Specifically, as shown in FIG. 3, in order to create a bird's-eye view image 30 around the vehicle, the camera module 2A images the front camera image 31A, the camera module 2B images the rear camera image 31B, and the camera module. The left side camera image 31C is imaged by 2C, and the right side camera image 31D is imaged by the camera module 2D. Here, the camera image 31A includes the vehicle body 33 and white lines L1, L3, and L5 drawn on the road surface, the camera image 31B includes the vehicle body 33 and white lines L2, L3, and L5, and the camera image 31C includes the vehicle body 33 and the white lines L3 and L3. L4 is included, and the camera image 31D includes the vehicle body 33 and white lines L5 and L6. Note that white lines appearing in each camera image are drawn on the road surface and imaged by the camera modules 2A, 2B, 2C, and 2D in order to explain the distortion generated in the overhead image 30 and the shift between the images.

  The in-vehicle image processing apparatus 1 connects the front image 32A, the rear image 32B, the left side image 32C, and the right side image 32D obtained by converting the camera images 31A, 31B, 31C, and 31D into viewpoints from above. The own vehicle object 33 ′ made of computer graphics or the like is arranged at the center, and the overhead image 30 having no shift or distortion is created across the images constituting the overhead image 30.

  On the other hand, when there is an error in the mounting position of the camera modules 2A, 2B, 2C, 2D, as shown in FIG. 4, the white line L1 of the front image 32A after the viewpoint conversion is balanced with the host vehicle object 33 ′. As a result, the white line L3 may be distorted, or the white lines L1, L2, and L3 may be displaced across the images 32A, 32B, and 32C that form the overhead image 30. On the other hand, in the in-vehicle image processing apparatus 1 described below, a process for correcting the shift and distortion of the overhead image 30 caused by the mounting error of the camera modules 2A, 2B, 2C, 2D is performed, and the camera module 2A , 2B, 2C, 2D, even if there is an attachment error, the bird's-eye view image 30 that suppresses the shift or distortion of the bird's-eye view image 30 is created.

  In this way, the configuration and operation of the in-vehicle image processing apparatus 1 for creating the bird's-eye view image 30 in which distortion and deviation are suppressed from the camera images 31A, 31B, 31C, and 31D will be described below.

<Configuration and operation of in-vehicle image processing apparatus>
As shown in FIG. 1, the vehicle surrounding display system including the in-vehicle image processing apparatus 1 includes four camera modules 2A, 2B, 2C, and 2D (hereinafter simply referred to as generic names). The camera module 2 is connected to a monitor 3. The in-vehicle image processing apparatus 1 receives camera image data captured by each of the camera modules 2A, 2B, 2C, and 2D, and combines the respective camera image data to produce an overhead image 30 when viewed from above the vehicle. To create. In this example, as described above, the four camera modules 2A, 2B, 2C, and 2D having the respective imaging directions on the front, rear, right side, and left side of the vehicle body are provided. A case where the overhead image 30 is created will be described.

  The camera modules 2A, 2B, 2C, and 2D are installed in different parts of the vehicle body, and have different imaging directions that are fixed to each other. The camera module 2 includes an imaging lens 21 and a CCD (charge coupled device) 22. The camera module 2 includes, for example, an NTSC (National Television System Committee) camera, and outputs image data to the in-vehicle image processing apparatus 1 according to the NTSC system.

  The in-vehicle image processing apparatus 1 includes an input buffer 12A, 12B, 12C, and 12D (hereinafter collectively referred to as a unit "input buffer 12") connected to the internal bus 11 in each of the camera modules 2A, 2B, 2C, and 2D. And a CPU (Central Processing Unit) 13, an image conversion unit 14, a table storage unit 15, and an output buffer 16 connected to the monitor 3.

  The input buffer 12 is provided corresponding to the number of camera modules 2, the input buffer 12A is connected to the camera module 2A, the input buffer 12B is connected to the camera module 2B, and the input buffer 12C is connected to the camera module 2C. The input buffer 12D is connected to the camera module 2D. The input buffer 12 temporarily stores camera image data of the NTSC system, and the camera image data is read at the image conversion timing by the image conversion unit 14 under the control of the CPU 13.

  The table storage unit 15 stores address conversion tables 15a, 15b, 15c,... For each image layout presented to the driver. Here, in this example, a case where a single bird's-eye view image 30 is created by combining four camera image data will be described. Therefore, at least the input buffers 12A, 12B, 12C, and 12D are stored in the table storage unit 15. An address conversion table of an image layout for creating the overhead image 30 from each stored camera image data is stored.

  The address conversion table stored in the table storage unit 15 specifies a camera number for specifying the input buffer 12 corresponding to the camera module 2 and an address on the input buffer 12 for extracting camera image data for creating an overhead image. 3 is a table describing the correspondence between input address information to be output and output address information for designating an address on the output buffer 16 for storing camera image data extracted from an address on the input buffer 12. That is, the address conversion table includes the memory address on the output buffer 16, that is, the display coordinates of the monitor 3, and the memory address on the input buffer 12 corresponding to each camera module 2, that is, the coordinates on the camera image for each camera module 2. Is described. Such an address conversion table is created based on the specifications of the camera modules 2A, 2B, 2C, and 2D, the mounting position, the orientation in the imaging direction (optical axis), and the like.

  The image conversion unit 14 reads the address conversion table from the table storage unit 15 according to the control of the CPU 13, refers to the address conversion table, and converts the image data of the camera module 2 specified by the address conversion table into the address conversion table. The data is read from the input address on the designated input buffer 12 and stored in the output address on the output buffer 16 designated by the address conversion table. Here, since the address conversion table describes the memory address of the output buffer 16 according to the image layout, the overhead image data replaced by the output buffer 16 by the image conversion unit 14 has the same image layout as the used address conversion table. The image has been converted. The image conversion unit 14 may be configured by an LSI (Large Scale Integrated Circuit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), or the like, or may be performed by the CPU 13.

  As will be described later, the address conversion table depends on each mounting error of the camera module 2 (pitch angle, yaw angle, roll angle error in the camera coordinate system which is a global coordinate system) at the time of manufacturing the in-vehicle image processing apparatus 1 or the like. Thus, the default (initial state) address conversion table is reconfigured so as to display the overhead image 30 in which the mounting error is corrected. That is, camera image data in the input buffer 12 is read out by viewpoint conversion processing by the image conversion unit 14, and distortion of the overhead image data stored in the output buffer 16 and deviation between images constituting the overhead image 30 are suppressed. Thus, the address translation table is reconstructed.

  The output buffer 16 stores the bird's-eye view image data composed of the four pieces of camera image data whose viewpoint is converted by the image conversion unit 14, and outputs the bird's-eye view image data to the monitor 3 under the control of the CPU 13.

  The CPU 13 recognizes an image layout instruction or an image switching instruction determined by operating an operation input unit (not shown) by the driver, and determines an address conversion table used for image conversion processing in the image conversion unit 14. Further, the CPU 13 continuously stores the overhead image 30 in the output buffer 16 by controlling the image conversion timing from the input buffer 12 of the image conversion unit 14 and the data output timing of the output buffer 16 to the monitor 3. Then, the overhead image 30 to be continuously displayed on the monitor 3 is switched.

  Further, as shown in FIG. 2, the CPU 13 stores an error detection unit 41 that performs error detection performed at the time of shipment of the host vehicle, a parameter calculation unit 42 that calculates parameters for correcting a default address conversion table, and a table storage unit 15. And a table reconstruction unit 43 for reconstructing the stored address conversion table. In this example, the in-vehicle image processing apparatus 1 includes functional units for reconfiguring the address conversion table stored in the table storage unit 15 such as the error detection unit 41, the parameter calculation unit 42, and the table reconstruction unit 43. However, an externally reconstructed address conversion table may be stored in the table storage unit 15.

  The error detection unit 41 detects an installation error in the camera coordinate system of the camera modules 2A, 2B, 2C, and 2D at the time of vehicle shipment inspection, for example. At the time of this inspection, the error detection unit 41 stops the own vehicle at a predetermined position on the road surface on which a predetermined white line is drawn, and the camera modules 2A, 2B, 2C, 2D when the white line is imaged from the stop position. Each camera image data is acquired. Then, the error detection unit 41 compares each acquired camera image data with a template image to be imaged when there is no attachment error of the camera modules 2A, 2B, 2C, and 2D.

  If the white line extending in the vertical direction of the image of the camera image data and the white line extending in the vertical direction of the template image are shifted in the horizontal direction in the image, the amount of shift is calculated as an error of the yaw angle in the camera coordinate system [ deg]. In addition, when the white line extending in the horizontal direction in the image of the camera image data and the white line extending in the horizontal direction in the image serving as the template are shifted in the vertical direction in the image, the amount of shift is represented by the pitch angle in the camera coordinate system. Calculated as error [deg]. Further, when the camera image data and the template image are shifted in the rotation direction, that is, in the direction in which the image is rotated around a certain point in the image, the shift amount is determined as the roll angle error [deg] in the camera coordinate system. Calculate as As a result, the camera module 2A, 2B, 2C, 2D mounting error includes the camera coordinate system (global coordinate system) yaw angle error [deg], pitch angle error [deg], and roll angle error [deg]. Each can be detected.

  The parameter calculation unit 42 calculates the camera modules 2A, 2B, 2C, and the like from the yaw angle error in the camera coordinate system, the pitch angle error in the camera coordinate system, and the roll angle error in the camera coordinate system detected by the error detection unit 41. A correction amount (parameter) for an initial address conversion table for creating a bird's-eye view image, that is, a default address conversion table, is calculated using camera image data in a state where there is no 2D mounting error.

  At this time, the parameter calculation unit 42 obtains a left-right shift amount dx [pixel] that greatly shifts the address conversion target area of the camera image data in the address conversion table in the left-right direction as the yaw angle error increases. The parameter calculation unit 42 obtains a vertical shift amount dy [pixel] by which the address conversion target area of the camera image data by the address conversion table is largely shifted in the vertical direction as the pitch angle error is larger. Such shift amounts dx, dy [pixel] are parameters for correcting each of the mounting errors of the camera modules 2A, 2B, 2C, 2D.

  Further, the parameter calculation unit 42 obtains a rotation amount dθ [deg] for rotating the address conversion target area of the camera image data from the roll angle error [deg]. Here, in the graphics coordinate system (image coordinate system) of the camera image data, when an error of the roll angle of r [deg] occurs around an arbitrary coordinate (ox, oy), the camera image data Processing for obtaining the rotation amount dθ [deg] for rotating the address conversion target area will be described. The coordinates (ox, oy) are set to the center coordinates of the input buffers 12A, 12B, 12C, and 12D, so that pixel loss due to rotation can be minimized. In addition, the calculation can be simplified by making the center of rotation a unique point of the coordinate center regardless of the difference in camera displacement.

  First, before correcting the roll angle error, the coordinates (sx1, sy1) before the rotation appearing on the camera image due to the roll angle error are the coordinates after the rotation after correcting according to the roll angle error. Using (x1, y1), it can be obtained by the following equations 1 and 2.

sx1 = (x1−ox) * cos (−r) + (y1−oy) * sin (−r) + ox (Formula 1)
sy1 = (y1−oy) * cos (−r) − (x1−ox) * sin (−r) + oy (Formula 2)
Since the above formulas 1 and 2 can be applied to the post-rotation coordinates (x1, y1) and the post-rotation coordinates (x1 + 1, y) adjacent to each other in the x direction, the coordinates (sx1, sy1) before the rotation can be applied. The pre-rotation coordinates (sx2, sy2) that are adjacent in the x direction are
sx2 = (x1 + 1−ox) * cos (−r) + (y1−oy) * sin (−r) + ox (Formula 3)
sy2 = (y1−oy) * cos (−r) + (x1 + 1−ox) * sin (−r) + oy (Formula 4)
It is expressed.

Here, when dxx = sx2-sx1 and dyx = sy2-sy1 are set, dxx and dyx are obtained by using the above equations 1, 2, and 3, 4, respectively.
dxx = sx2-sx1 = cos (-r) (Formula 5)
dyx = sy2-sy1 = -sin (-r) (Formula 6)
It becomes. Then, the coordinates after rotation of coordinates (x1 + n, y1) continuous in the x direction are expressed using sx1, sy1, dxx, dyx,
sx = sx1 + dxx * n (Formula 7)
sy = sy1 + dyx * n (Formula 8)
It becomes. Therefore, it can be seen that the coordinates after rotation of the coordinates (x1 + n, y1) can be calculated by adding dxy and dyy to sx1 and sy1, respectively. Here, since Expressions 7 and 8 include a trigonometric function like Expressions 5 and 6, the rotation amount dθ [deg] including a decimal value is obtained.

Next, in the case of the coordinates (x1, y1 + 1) adjacent in the y direction of the coordinates (x1, y1) after rotation,
sx3 = (x1−ox) * cos (−r) + (y1 + 1−oy) * sin (−r) + ox (Formula 9)
sy3 = (y1 + 1-oy) * cos (-r)-(x1-ox) * sin (-r) + oy (Formula 10)
It is expressed. Here, if dxy = sx3−sx1, dyy = sy3−sy1, dxy and dyy are obtained by using the above equations 9 and 10, respectively.
dxx = sx3−sx1 = sin (−r)
dyx = sy3−sy1 = cos (−r)
Thus, the coordinates after rotation of coordinates (x1, y1 + n) continuous in the y direction can be calculated by adding dxy and dyy to sx1 and sy1, respectively. Further, the coordinates after rotation obtained in this way are the rotation amount dθ [deg] including a decimal value.

  From these, the above-mentioned post-rotation coordinates are calculated only once for the upper left coordinates of the graphics coordinate system, which is the coordinate system of the camera image data stored in the input buffers 12A, 12B, 12C, and 12D. Data pixels are rotated by performing addition calculation.

  Here, since the values of cos (−r) and sin (−r) in the above calculation are common, these two values are obtained only once, and the rotation amount dθ [deg] is calculated using the obtained results. Ask. If the correspondence between the pair of pre-rotation coordinates and the post-rotation coordinates among the coordinates after rotation is stored, it is a parameter for correcting the roll angle error without requiring the calculation of trigonometric functions. A certain rotation amount dθ [deg] can be obtained.

  Here, specific examples of operations in the error detection unit 41 and the parameter calculation unit 42 will be described with reference to FIGS. 5 and 6. FIG. 5 is a flowchart showing a series of procedures when the error detection unit 41 and the parameter calculation unit 42 detect an attachment error in the camera coordinate system of the camera module 2 and calculate a parameter for correcting the error. 6 is a diagram showing a camera image on the left side of the host vehicle captured by the camera module 2C for explaining a series of procedures shown in the flowchart of FIG.

  When detecting the mounting error in the camera coordinate system of the camera module 2 and calculating the parameters, first, in step S1, a template is prepared at an accurate position with respect to the vehicle. In this example, for example, a T-shaped line having a width of 50 mm drawn on the ground is used as the template. However, the position and shape of the template are not particularly limited. For example, the template has a lattice pattern such as a checkered flag. You may make it install a template on the ground. In this example, it is assumed that a T-shaped line template having a width of 50 mm drawn on the ground is reflected at a position P1 indicated by a solid line on the camera image on the left side of the host vehicle shown in FIG.

  Next, in step S2, the error detection unit 41 sets the marker line template at a position where the template should appear on the screen when there is no mounting error of the camera module 2C on the left-side camera image shown in FIG. And the marker line template is superimposed on the camera image on the left side of the host vehicle. In this example, it is assumed that the marker line template is superimposed on a position P2 indicated by a broken line on the camera image on the left side of the host vehicle shown in FIG.

  Next, in step S3, the error detection unit 41 rotates the marker line template superimposed on the camera image around the center of the camera image on the left side of the host vehicle and reflects it on the camera image. Make adjustments so that the template and the marker line template match. Then, the parameter calculation unit 42 calculates the rotation amount of the camera image at this time as a rotation amount dθ [deg] as a parameter for correcting a roll angle error in the camera coordinate system.

  Next, in step S4, the error detection unit 41 shifts the marker line template on the camera image on the left side of the host vehicle in the vertical and horizontal directions, and displays the template and marker line template reflected on the camera image. Adjust so that the position of is correct. Then, the parameter calculation unit 42 calculates the shift amount in the X-axis direction in the image coordinate system at this time as a left-right shift amount dx [pixel] as a parameter for correcting the error of the yaw angle in the camera coordinate system. The shift amount in the Y-axis direction in the image coordinate system is calculated as a vertical shift amount dy [pixel] as a parameter for correcting a pitch angle error in the camera coordinate system.

  As described above, in this example, the camera image of the position of the template actually reflected in the camera image captured by the camera module 2 and the marker line template drawn at the position on the camera image where the original template should be reflected. The positional deviation above, that is, the positional deviation between the template and the marker line template on the two-dimensional coordinate which is the image coordinate system is calculated as a parameter for correcting the mounting error of the camera module 2 in the camera coordinate system. In the above, the case where an attachment error has occurred in the camera module 2C that captures an image on the left side of the host vehicle has been illustrated. However, when an attachment error has occurred in another camera module, the attachment is performed in the same manner. Parameters for correcting the error can be calculated.

  The table reconstruction unit 43 corrects the left-right shift amount dx [pixel] for correcting the error of the yaw angle in the camera coordinate system obtained by the parameter calculation unit 42, and the pitch angle error in the camera coordinate system. Using the vertical shift amount dy [pixel] and the rotation amount dθ [deg] for correcting the roll angle error in the camera coordinate system, the default address conversion table stored in the table storage unit 15 is reconstructed. To create a new address conversion table. The table reconstruction unit 43 first shifts the address conversion target area by the horizontal shift amount dx [pixel] and shifts the address conversion target area by the vertical shift amount dy [pixel]. After that, the table reconstruction unit 43 rotates the address conversion target area shifted by the horizontal shift amount dx and the vertical shift amount dy by the number of pixels corresponding to the rotation amount dθ [deg]. As described above, the rotation operation may be performed by calculating the correspondence between the coordinates before rotation and the coordinates after rotation, or may be provided as a table.

  As described above, the table reconstruction unit 43 rotates the address conversion target area after shifting the address conversion target area in the horizontal direction and the vertical direction. As described above, the address conversion target area is shifted in the horizontal direction and the vertical direction first, and then rotated. The read addresses of the input buffers 12A, 12B, 12C, and 12D that are the address conversion target areas are real numbers. On the other hand, since the rotation amount dθ [deg] includes a decimal value, a rounding error occurs in the address conversion target area after the rotation with respect to the rotation amount dθ [deg]. Therefore, the table reconstruction unit 43 performs the left and right and up and down shifts that do not cause a rounding error first, and then rotates them to minimize the influence of the rounding error. In addition, when the address conversion target area is shifted vertically and horizontally after rotating the address conversion target area, the shift amount [pixel] is decomposed into the cos component and the sin component included in the previous rotation amount. Therefore, there is a concern that the shift amount [pixel] obtained by the parameter calculation unit 42 and the actual shift amount [pixel] may be different from each other. The above inconveniences can be avoided by performing the rotation operation. In addition, by performing the shift in the horizontal direction and the vertical direction first and then performing the rotation operation, the calculation can be easily performed with the rotation center as a unique point of the coordinate center regardless of the difference in the mounting error amount of the camera module 2. Can be

In the present embodiment, the address conversion table is reconstructed by calculating the rotation amount dθ of the address conversion target area in order to suppress image displacement and distortion caused by the roll angle error in the camera coordinate system. However, if the roll angle error is small, it is a matter of course that image shift and distortion caused by the yaw angle error and the pitch angle error can be suppressed only by shifting the address conversion target area up and down and left and right. As described above, the reconstruction of the address conversion table using only the vertical shift amount dy and the horizontal shift amount dx is performed by setting the output address as Addr, the number of vertical pixels in the input buffer 12 as Xi, and mounting the camera module 2. When defAddr is the horizontal input address in the initial state assuming no error,
Addr = (defAddr + dx) + Xi · dy
It can be expressed by simple addition and subtraction.

  Then, the table reconstruction unit 43 changes the address (input address information) on the input buffer 12 to be taken out as the address conversion target area among the input buffers 12A, 12B, 12C, and 12D by shifting the address conversion target area. The address conversion table is reconstructed and stored in the table storage unit 15.

  In this way, the process of reconfiguring the address conversion table is a default address conversion table as shown in FIG. 7 that is created on the assumption that there is no mounting error of the camera modules 2A, 2B, 2C, and 2D. To be done. In the example shown in FIG. 7, in order to explain the main points in an easy-to-understand manner, for example, among the input addresses on the input buffer 12 corresponding to an arbitrary camera module 2, the address conversion target area has input addresses “1” to “1”. It is assumed that “16” is a 4 image × 4 image block. The default address conversion table extracts camera image data from the input addresses “6”, “7”, “10”, and “11” among the input addresses “1” to “16”, and inputs the input address “6”. The camera image data fetched from the output buffer 16 is stored at the output address “1”, the camera image data fetched from the input address “7” is stored at the output address “2” of the output buffer 16, and the input address “ The camera image data extracted from “10” is stored in the output address “3” on the output buffer 16, and the camera image data extracted from the input address “11” is stored in the output address “4” on the output buffer 16. Show.

  Thereby, the camera image data of a specific place around the vehicle imaged by the camera module 2 can be stored in the output buffer 16 and displayed on the monitor 3.

  On the other hand, when there is an attachment error of the camera modules 2A, 2B, 2C, and 2D, an error occurs in the range imaged with the camera image data. Therefore, as shown in FIG. The camera image data that should be stored in the input addresses “6”, “7”, “10”, “11” of 7 shifts to the input addresses “1”, “2”, “5”, “6”. It will be stored. Similarly to the above, when the camera image data of the input addresses “6”, “7”, “10”, “11” are extracted and stored in the output addresses “1” to “4”, they are originally displayed on the monitor 3. The image of the place different from the specific place around the vehicle should be displayed.

  On the other hand, as described above, the vertical shift amount dy is set to -1 [one address], and the horizontal shift amount dx is determined to be -1 [one address] to shift the address conversion target area. As shown in FIG. 8B, the input addresses “1”, “2”, “5”, and “6” can be taken out. The above example is a case where the camera module 2 has an error in the yaw angle and the pitch angle. However, if an error in the roll angle is also generated, the address target area is shifted in the vertical and horizontal directions. The address target area may be rotated by the number of pixels corresponding to the rotation amount dθ according to the roll angle error described above.

  Such an address conversion table reconstruction process is performed by performing steps S11 to S13 as shown in FIG. 9 when a plurality of camera modules 2 are connected to the in-vehicle image processing apparatus 1 as described above. This is performed by repeating the number of camera modules 2 times.

  That is, in step S11, first, for an arbitrary camera module 2, the error detection unit 41 obtains the yaw angle error [deg], the pitch angle error [deg], and the roll angle error [deg] in the camera coordinate system, The parameter calculation unit 42 obtains the vertical shift amount dy [pixel], the horizontal shift amount dx [pixel], and the rotation amount dθ [deg] from each error [deg].

  In the next step S12, the address (input address) on the input buffer 12 where the camera image data captured by an arbitrary camera module 2 is stored in the vertical direction in the address conversion table for creating a default overhead image. Addition / subtraction is performed by the shift amount dy and the left / right shift amount dx, and then the entire input address (the entire address conversion target area) of the address conversion table is rotated by the rotation amount dθ based on the roll angle error [deg].

  In the next step S13, the address conversion table reconstructed in step S12 is stored in the table storage unit 15. Here, the table storage unit 15 may store the reconfigured address conversion table in a memory area different from the default address conversion table, and the reconfigured address conversion table in the default address conversion table. May be overwritten.

  As a result, the correction amount [pixel] and the rotation amount of each address conversion table are obtained from the mounting error [deg] of each camera module 2, and only the addition and subtraction and rotation in pixel units in a simple graphics coordinate system are performed. An error in the coordinate system (camera coordinate system) can be corrected.

<Effect of embodiment>
As described above in detail, according to the in-vehicle image processing apparatus 1 of the first embodiment to which the present invention is applied, an attachment error with respect to the own vehicle is generated in any of the plurality of camera modules 2 that capture the vehicle periphery. Even so, by reconstructing the default address conversion table stored in the table storage unit 15, it is possible to monitor a bird's-eye view image around the vehicle in which displacement and distortion between images due to mounting errors are effectively suppressed. 3 can be displayed.

  That is, according to this in-vehicle image processing apparatus 1, for example, as shown in FIG. 10, there is an error in the mounting position or direction of the camera module 2 (left side camera) with the left side of the host vehicle as the imaging direction. Due to the error, the overhead image as shown in FIG. 10A to be displayed on the monitor 3 is not displayed, and the white lines L3 and L4 are distorted as shown in FIG. It is possible to effectively suppress the inconvenience that an unpleasant overhead view image in which the white line L3 is shifted at the joint between the adjacent front image and the rear image is displayed on the monitor 3. For example, when the mounting error of the camera module 2 is a yaw angle error in the camera coordinate system, when an overhead image is created using the default address conversion table, the white line L3 is parallel as shown in FIG. A bird's-eye view image 30 in which the degree is not maintained is displayed on the monitor 3, but the input address of the default address conversion table is set according to the horizontal shift amount dx [pixel] which is a parameter corresponding to the error amount of the yaw angle. By simply shifting in the left-right direction, the overhead image 30 in which the parallelism of the white line L3 is maintained can be displayed on the monitor 3, as shown in FIG. Specifically, the camera module 2 that captures the left side of the host vehicle has an error in yaw angle of 3 degrees, and even if there is a deviation of 23 pixels on the image and an actual displacement of about 30 cm, the VGA monitor 3 In terms of (Y axis), the shift of 23/480 [pixel] can be suppressed to 1/480.

  In addition, according to the in-vehicle image processing apparatus 1 of the present embodiment, the default address conversion table for creating the bird's-eye view image 30 in the state where there is no attachment error of the camera module 2 to the vehicle, the attachment error of the camera module 2 to the vehicle. Since the camera image data is stored in the output buffer according to the reconstructed address conversion table, the overhead image 30 is actually created simply by changing the input address information in the address conversion table. At this time, an appropriate overhead image 30 can be created by a simple process without performing many processes. Further, since the address conversion table is reconstructed from the mounting error of the camera module 2 unique to the vehicle, the memory capacity of the table storage unit 15 that stores the address conversion table does not become enormous.

  Specifically, the table storage unit 15 designates the camera number for designating the input buffer 12 corresponding to the camera module 2 and the address on the input buffer 12 from which the camera image data for creating the overhead image 30 is extracted. An address conversion table describing a correspondence relationship between input address information and output address information for designating an address on the output buffer 16 for storing camera image data extracted from the address on the input buffer 12 is stored. The conversion unit 14 extracts the camera image data from the input buffer 12 according to the camera number and input address information specified in the address conversion table, and stores the camera image data at the address on the output buffer 16 specified in the address conversion table. Camera module 2 by processing only Distortion and displacement regardless attachment error can be displayed overhead image 30 was suppressed.

  Further, the in-vehicle image processing apparatus 1 according to the present embodiment includes the error detection unit 41, the parameter calculation unit 42, and the table reconstruction unit 43 illustrated in FIG. 2, thereby reducing the image coordinate system error caused by the camera coordinate system error. An address conversion table for correction can be reconfigured. That is, only by shifting the input address in the horizontal direction of the image coordinate system, it is possible to display the bird's-eye view image 30 that suppresses the shift caused by the error in the yaw angle of the camera coordinate system, which is an attachment error of the camera module 2 to the vehicle. it can. Moreover, the overhead image 30 in which the shift due to the error of the pitch angle, which is the mounting error of the camera module 2 with respect to the vehicle, is suppressed can be displayed only by shifting the input address in the vertical direction of the image coordinate system. Furthermore, the shift caused by the error in the roll angle of the camera coordinate system, which is the mounting error of the camera module 2 with respect to the vehicle, is suppressed only by moving the input address in the rotation direction with the arbitrary coordinates of the image coordinate system as the rotation center. The overhead image 30 can be displayed.

  Furthermore, according to the in-vehicle image processing apparatus 1 of the present embodiment, the image shift caused by the error of the yaw angle or the pitch angle of the camera module 2 and the error of the roll angle of the camera module 2 is prevented. When reconstructing the address conversion table so that it can be corrected together with the deviation, the shift correction in the horizontal direction according to the error amount of the yaw angle and the shift correction in the vertical direction according to the error amount of the pitch angle are performed first. Thereafter, by performing a rotation operation according to the roll angle error amount and reconfiguring the address conversion table, the calculation result of the rotation amount dθ [deg] based on the roll angle error [deg] includes a decimal value. Even in this case, it is possible to minimize the influence of the rounding error that appears due to the decimal value.

  The above-described embodiment is an example of the present invention. For this reason, the present invention is not limited to the above-described embodiment, and various modifications can be made depending on the design and the like as long as the technical idea according to the present invention is not deviated from this embodiment. Of course, it is possible to change.

  For example, the table storage unit 15 stores, for example, an address conversion table for displaying a right side image and a left side image on the monitor 3 on one screen, a right rear side image, a left rear side image, and a lower image. In addition, an address conversion table or the like may be stored. Even when a plurality of camera image data is converted into one continuous image by such an address conversion table, as described above, the error amount of the yaw angle and the error of the pitch angle, which are the mounting errors of the camera module 2. By reconstructing the address conversion table according to the parameters according to the amount and the error amount of the roll angle, it is possible to suppress the shift and distortion between images in the display image.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to the drawings.

  In this embodiment, as described in the first embodiment, the address conversion table is not reconfigured according to the parameter for correcting the attachment error of the camera module to the vehicle, but the default address conversion table is used for the viewpoint conversion processing. By adjusting the camera image data of each camera module stored in the input buffer according to the above parameters, it is possible to display an appropriate overhead image without any deviation or distortion even if an installation error occurs in the camera module. Is. In the following, description will be made centering on the characteristic parts of the present embodiment, and the same components as those in the first embodiment will be denoted by the same reference numerals and detailed description thereof will be omitted.

  FIG. 11 is a diagram showing a configuration of an in-vehicle image processing apparatus 50 according to the second embodiment to which the present invention is applied. The in-vehicle image processing apparatus 50 realizes the function units 41, 42, and 44 as shown in FIG. 12 by the CPU 13, for example, so that the camera image data input from each camera module 2 to the input buffer 12 is input to the input buffer 12. The shift and rotation adjustment above suppresses the shift and distortion on the overhead view image caused by the mounting error of the camera module 2 and causes the monitor 3 to display an appropriate overhead view image.

<Configuration and operation of in-vehicle image processing apparatus>
As shown in FIG. 12, the CPU 13 includes an image adjustment unit 44 that adjusts the camera image data on the input buffer 12 instead of the table reconstruction unit 43 described in the first embodiment. The error detection unit 41 and the parameter calculation unit 42 are basically the same as those in the first embodiment, but the parameter calculation unit 42 in the present embodiment is the camera module 2 detected by the error detection unit 41. Parameters for adjusting the camera image data of each camera module 2 on the input buffer 12 are calculated according to the mounting error.

  The operations of the error detection unit 41 and the parameter calculation unit 42 are the same as in the first embodiment, and the yaw angle [deg], pitch angle [deg], The error of the roll angle [deg] is determined by shifting the horizontal (horizontal) shift amount dx [pixel], the vertical (vertical) shift amount dy [pixel], and the center of the screen in the image coordinate system which is a two-dimensional coordinate system. The rotation amount is calculated as dθ [deg].

  The image adjustment unit 44 corrects the horizontal shift amount dx for correcting the yaw angle error obtained by the parameter calculation unit 42, the vertical shift amount dy for correcting the pitch angle error, and the roll angle error. The camera image data input from each camera module 2 and stored in the input buffer 12 is adjusted using the respective positive and negative values of the rotation amount dθ for the adjustment. The image adjustment unit 44 first shifts the camera image data stored in the input buffer 12 by the left-right direction shift amount −dx and shifts the camera image data stored in the input buffer 12 by the vertical direction shift amount −dy. Thereafter, the image adjustment unit 44 rotates the camera image data in the input buffer 12 shifted by the horizontal shift amount −dx and the vertical shift amount −dy by the number of pixels corresponding to the rotation amount −dθ. That is, the image adjustment unit 44 shifts the camera image data in the input buffer 12 in the horizontal direction and the vertical direction, and then rotates the shifted camera image data. The reason why the camera image data is first shifted in the left-right direction and the up-down direction and then rotated is the same as in the first embodiment.

  In the present embodiment, the camera image data is adjusted by calculating the rotation amount dθ of the camera image data in the input buffer 12 in order to suppress the image shift and distortion caused by the roll angle error in the camera coordinate system. If the roll angle error is small, image shift and distortion due to yaw angle error and pitch angle error can be suppressed only by shifting the camera image data up and down and left and right on the input buffer 12. Of course.

  An image of processing for adjusting camera image data on the input buffer 12 is schematically shown in FIG. In the example shown in FIG. 13, the state of FIG. 7 described in the first embodiment is set to a state where there is no mounting error of the camera module 2, and the yaw angle and the camera module 2 are compared with the state of FIG. This is an example when a pitch angle error occurs.

  If there is an attachment error in the camera module 2, an error occurs in the range captured by the camera module 2, and therefore, as shown in FIG. 13A, the input address “ The camera image data that should be stored in “6”, “7”, “10”, and “11” are stored in a state shifted to the input addresses “1”, “2”, “5”, and “6”. Become. In this state, when the camera image data of the input addresses “6”, “7”, “10”, “11” is extracted according to the address conversion table and stored in the output addresses “1” to “4”, An image of a place different from a specific place around the vehicle to be displayed on the monitor 3 is displayed.

  On the other hand, as described above, from the parameters dx = −1 and dy = −1, the vertical shift amount −dy, which is the opposite value, is +1 [one address], and the horizontal shift amount −dx is +1 [ 1 address] and the pixel information in the input buffer 12 is moved, the input addresses “1” and “2” are originally used while using the default address conversion table as shown in FIG. , “5”, “6” can be changed so as to take out the camera image data. In this example, the entire pixel is shifted so that the camera image data “R” stored at the input address “1” before adjustment is stored at the input address “6” after adjustment. Note that a portion where information is lost due to movement of camera image data is dealt with by storing color information, for example. Further, the above example is a case where the camera module 2 has errors in the yaw angle and the pitch angle, but when there is also an error in the roll angle, the pixel information in the input buffer 12 is changed in the vertical and horizontal directions. After the shift to, the camera image data may be rotated on the input buffer 12 by the number of pixels corresponding to the reciprocal -dθ of the rotation amount corresponding to the roll angle error described above.

  Here, in the in-vehicle image processing apparatus 50 according to the present embodiment, an outline of an operation for adjusting camera image data input from the camera module 2 on the input buffer 12 and creating an overhead image using a default address conversion table. Will be described with reference to FIGS. 14 and 15. 14 is a flowchart showing a flow of a series of operations, and FIG. 15 is a diagram showing an image of processing in the image adjustment unit 44.

  When the flow of FIG. 14 is started, first, in step S21, the error detection unit 41 obtains the yaw angle error [deg], the pitch angle error [deg], and the roll angle error [deg] in the camera coordinate system. The parameter calculation unit 42 obtains the vertical shift amount dy [pixel], the horizontal shift amount dx [pixel], and the rotation amount dθ [deg] from each error [deg]. The processing contents by the error detection unit 41 and the parameter calculation unit 42 are the same as those in the first embodiment described above.

  Next, in step S22, camera image data input from the camera module 2 and stored in the input buffer 12 in the in-vehicle image processing device 50 is detected by the image adjustment unit 44 on the input buffer 12 in step S21. Shifting is performed by the positive and negative values (−dx, −dy) of the parameters dx, dy. This processing is realized by calculation in the “shift adjustment” portion in FIG.

  Next, in step S23, the image adjustment unit 44 rotates the camera image data subjected to the shift adjustment in step S22 by the reciprocal (−dθ) of the parameter dθ detected in step S21 on the input buffer 12. I do. This processing is realized by calculation in the “rotation operation” portion in FIG. The operation of rotating the camera image data may be calculated as the correspondence between the coordinates before the rotation and the coordinates after the rotation as described in the first embodiment, or provided as a table. Also good. Further, the adjusted camera image data may be overwritten on the buffer area originally stored on the input buffer 12 or may be stored in a new area.

  The processes in steps S21 to S23 are repeatedly performed on the camera image data input from all the camera modules 2 in which the attachment error has occurred. That is, in step S24, it is determined whether or not the adjustment for the camera image data from all the camera modules 2 in which the mounting error has occurred is completed, and the camera image data from all the camera modules 2 in which the mounting error has occurred is determined. When the adjustment is completed, the process proceeds to the next step S25.

  In step S25, the image conversion unit 14 reads a default address conversion table stored in the table storage unit 15 according to the control of the CPU 13, and each camera image adjusted on the input buffer 12 according to the address conversion table. Data is read out and stored in the output buffer 16 to create an overhead image. Then, an overhead image is output from the output buffer 16 to the monitor 6.

  Through the above-described series of processing, the shift amount [pixel] and the rotation amount of each camera image data on the input buffer 12 are obtained from the mounting error [deg] of each camera module 2, and in a simple graphics coordinate system. An error in the global coordinate system (camera coordinate system) can be corrected only by addition / subtraction and rotation in units of pixels, and an appropriate overhead image without any deviation or distortion can be displayed on the monitor 6.

  The above-described shift processing and rotation processing of the camera image data in the input buffer 12 may be integrated and performed at a time. The above-described image adjustment processing is not performed in the in-vehicle image processing apparatus 50 but in the camera module 2, and the in-vehicle image processing apparatus 50 stores the camera image data adjusted in the camera module 2 in the input buffer 12. It may be stored and the conversion process may be performed according to a default address conversion table. In this case, at least the function as the image adjustment unit 44 among the functions of the error detection unit 41, the parameter calculation unit 42, and the image adjustment unit 44 realized by the CPU 13 is set in each camera module 2 or in each camera module 2. What is necessary is just to implement | achieve with the control apparatus connected to this, and the effect similar to the example mentioned above can be acquired.

<Effect of embodiment>
As described above in detail, according to the in-vehicle image processing apparatus 50 according to the second embodiment to which the present invention is applied, an attachment error with respect to the own vehicle occurs in any of the plurality of camera modules 2 that capture the vehicle periphery. Even if the camera image data of the camera module 2 is adjusted on the input buffer 12, the default address conversion table stored in the table storage unit 15 is used to shift the image between the images due to the mounting error. It is possible to create a bird's-eye view image around the vehicle in which distortion and distortion are effectively suppressed and display them on the monitor 3.

  Moreover, according to the vehicle-mounted image processing apparatus 50 of this embodiment, camera image data is adjusted according to the attachment error of the camera module 2 intrinsic | native to a vehicle, and a bird's-eye view image is used as an address conversion table using a default address conversion table. Since it is created, the memory capacity of the table storage unit 15 for storing the address conversion table does not become enormous.

  In addition, the in-vehicle image processing apparatus 50 according to the present embodiment includes the error detection unit 41, the parameter calculation unit 42, and the image adjustment unit 44 illustrated in FIG. 12, thereby correcting an error in the image coordinate system due to an error in the camera coordinate system. can do. That is, a bird's-eye view image in which a shift caused by an error in the yaw angle of the camera coordinate system, which is an attachment error of the camera module 2 to the vehicle, is suppressed only by shifting the camera image data of the camera module 2 horizontally on the input buffer 12. 30 can be displayed. Further, only by shifting the camera image data of the camera module 2 in the vertical direction on the input buffer 12, the overhead image 30 is displayed in which a shift due to a pitch angle error, which is an attachment error of the camera module 2 to the vehicle, is suppressed. be able to. Furthermore, the camera coordinate system roll, which is an attachment error of the camera module 2 with respect to the vehicle, can be obtained simply by moving the camera image data of the camera module 2 on the input buffer 12 in the rotation direction with an arbitrary coordinate of the image coordinate system as the rotation center. It is possible to display the bird's-eye view image 30 in which the shift due to the corner error is suppressed.

  Furthermore, according to the in-vehicle image processing apparatus 50 of the present embodiment, the image shift caused by the error in the yaw angle or the pitch angle of the camera module 2 and the error in the roll angle of the camera module 2 are detected. When the camera image data is adjusted on the input buffer 12 so that it can be corrected together with the shift, the shift in the left-right direction according to the yaw angle error amount or the vertical shift according to the pitch angle error amount is performed first. After that, by performing a rotation operation according to the roll angle error amount, even if the calculation result of the rotation amount dθ [deg] due to the roll angle error [deg] includes a decimal value, The influence of the rounding error that appears due to the decimal value can be minimized.

It is a block diagram which shows the structure of the vehicle-mounted image processing apparatus of 1st Embodiment to which this invention is applied. It is a block diagram which shows the structure for reconfiguring an address conversion table. It is a figure for demonstrating the process which performs a viewpoint conversion process with respect to the camera image data from which each imaging direction differs, and produces a bird's-eye view image. It is a figure for demonstrating the subject when it is a process which performs a viewpoint conversion process with respect to the camera image data from which each imaging direction differs, and produces a bird's-eye view image, and there exists an attachment error of a camera module. It is a flowchart which shows a series of procedures at the time of detecting the attachment error in the camera coordinate system of a camera module and calculating the parameter which corrects the said error by an error detection part and a parameter calculation part. It is a figure for demonstrating the process which calculates the parameter which detects the attachment error in the camera coordinate system of a camera module, and corrects the said error, and is a figure which shows the camera image of the left side of the own vehicle imaged with the camera module. is there. It is a figure for demonstrating the creation process of the bird's-eye view image using the address conversion table when there is no attachment error of a camera module. It is a figure for demonstrating the process which reconfigure | reconstructs an address conversion table and produces a bird's-eye view image, when there exists a mounting error of a camera module. It is a flowchart which shows the process sequence of the reconstruction process of an address conversion table. It is a figure for demonstrating the bird's-eye view image displayed when there exists a yaw angle error, Comprising: It is a figure for demonstrating the effect of embodiment. It is a block diagram which shows the structure of the vehicle-mounted image processing apparatus of 2nd Embodiment to which this invention is applied. It is a block diagram which shows the structure for adjusting camera image data on an input buffer. It is a figure for demonstrating the process which adjusts camera image data on an input buffer, and produces a bird's-eye view image, when there exists a mounting error of a camera module. It is a flowchart which shows the flow of a series of operation | movement at the time of adjusting the camera image data input from a camera module on an input buffer, and producing a bird's-eye view image. It is a figure showing the image of the process which adjusts camera image data by an image adjustment part.

Explanation of symbols

1,50 In-vehicle image processing device 2A, 2B, 2C, 2D Camera module 3 Monitor 12A, 12B, 12C, 12D Input buffer 13 CPU
DESCRIPTION OF SYMBOLS 14 Image conversion part 15 Table memory | storage part 16 Output buffer 30 Overhead image 41 Error detection part 42 Parameter calculation part 43 Table reconstruction part 44 Image adjustment part

Claims (12)

An in-vehicle image processing apparatus that is connected to a plurality of camera modules that capture a vehicle periphery and generate camera image data, converts the viewpoint of each of the camera image data, and displays an overhead image of the vehicle periphery on a display unit. ,
Input means comprising a plurality of input buffers for storing camera image data generated by the plurality of camera modules;
An output buffer for storing an overhead image created using camera image data extracted from each of the plurality of input buffers, and an output means for outputting the overhead image to the display means;
Extracted from the camera number specifying the input buffer corresponding to the camera module, the input address information specifying the address on the input buffer from which the camera image data for creating the overhead image is extracted, and the address on the input buffer Storage means for storing an address conversion table describing a correspondence relationship with output address information for designating an address on an output buffer for storing camera image data;
Camera image data is extracted from the input buffer according to the address conversion table stored in the storage means, the camera image data is stored in the output buffer according to the address conversion table stored in the storage means, and the overhead image is stored. Image processing means for creating,
The address conversion table stored in the storage means is an initial state address conversion table for creating the bird's-eye view image using camera image data in a state where there is no attachment error of the camera module to the vehicle. address conversion table der which was reconstituted according to the parameters to correct mounting error for the vehicle of the camera module is,
A table reconstructing means for reconfiguring the address conversion table in the initial state according to a parameter for correcting an attachment error of the camera module to the vehicle and storing the table in the storage means;
The table reconstruction means changes the input address information in the address conversion table in the initial state according to a parameter for correcting an attachment error of the camera module to the vehicle, and re-stores the address conversion table to be stored in the storage means. The image coordinate system is configured to specify an address position specified by the input address information in the address conversion table in the initial state in accordance with a parameter according to a yaw angle error amount of the camera coordinate system, which is an installation error of the camera module to the vehicle. A vehicle-mounted image processing apparatus characterized by shifting in the horizontal direction . An in-vehicle image processing apparatus that is connected to a plurality of camera modules that capture a vehicle periphery and generate camera image data, converts the viewpoint of each of the camera image data, and displays an overhead image of the vehicle periphery on a display unit. ,
Input means comprising a plurality of input buffers for storing camera image data generated by the plurality of camera modules;
An output buffer for storing an overhead image created using camera image data extracted from each of the plurality of input buffers, and an output means for outputting the overhead image to the display means;
Extracted from the camera number specifying the input buffer corresponding to the camera module, the input address information specifying the address on the input buffer from which the camera image data for creating the overhead image is extracted, and the address on the input buffer Storage means for storing an address conversion table describing a correspondence relationship with output address information for designating an address on an output buffer for storing camera image data;
Camera image data is extracted from the input buffer according to the address conversion table stored in the storage means, the camera image data is stored in the output buffer according to the address conversion table stored in the storage means, and the overhead image is stored. Image processing means for creating,
The address conversion table stored in the storage means is an initial state address conversion table for creating the bird's-eye view image using camera image data in a state where there is no attachment error of the camera module to the vehicle. An address conversion table reconfigured according to a parameter for correcting an attachment error of the camera module to the vehicle ,
A table reconstructing means for reconfiguring the address conversion table in the initial state according to a parameter for correcting an attachment error of the camera module to the vehicle and storing the table in the storage means;
The table reconstruction means changes the input address information in the address conversion table in the initial state according to a parameter for correcting an attachment error of the camera module to the vehicle, and re-stores the address conversion table to be stored in the storage means. The image coordinate system is configured to specify an address position specified by the input address information in the address conversion table in the initial state in accordance with a parameter according to a pitch angle error amount of the camera coordinate system, which is an installation error of the camera module to the vehicle. car mounting image processing apparatus, wherein the shifting in the longitudinal direction of. An in-vehicle image processing apparatus that is connected to a plurality of camera modules that capture a vehicle periphery and generate camera image data, converts the viewpoint of each of the camera image data, and displays an overhead image of the vehicle periphery on a display unit. ,
Input means comprising a plurality of input buffers for storing camera image data generated by the plurality of camera modules;
An output buffer for storing an overhead image created using camera image data extracted from each of the plurality of input buffers, and an output means for outputting the overhead image to the display means;
Extracted from the camera number specifying the input buffer corresponding to the camera module, the input address information specifying the address on the input buffer from which the camera image data for creating the overhead image is extracted, and the address on the input buffer Storage means for storing an address conversion table describing a correspondence relationship with output address information for designating an address on an output buffer for storing camera image data;
Camera image data is extracted from the input buffer according to the address conversion table stored in the storage means, the camera image data is stored in the output buffer according to the address conversion table stored in the storage means, and the overhead image is stored. Image processing means for creating,
The address conversion table stored in the storage means is an initial state address conversion table for creating the bird's-eye view image using camera image data in a state where there is no attachment error of the camera module to the vehicle. An address conversion table reconfigured according to a parameter for correcting an attachment error of the camera module to the vehicle,
A table reconstructing means for reconfiguring the address conversion table in the initial state according to a parameter for correcting an attachment error of the camera module to the vehicle and storing the table in the storage means ;
The table reconstruction means changes the input address information in the address conversion table in the initial state according to a parameter for correcting an attachment error of the camera module to the vehicle, and re-stores the address conversion table to be stored in the storage means. The address position specified by the input address information in the address conversion table in the initial state according to a parameter according to a roll angle error amount of the camera coordinate system, which is an installation error of the camera module with respect to the vehicle. car mounting image processing apparatus is characterized in that moving the arbitrary coordinate system in the direction of rotation and the rotation center. The table reconstructing means is an address position specified by the input address information in the address conversion table in the initial state according to a parameter corresponding to a pitch angle error amount of a camera coordinate system, which is an attachment error of the camera module to the vehicle. vehicle image processing apparatus according to claim 1, characterized in Rukoto is shifted in the vertical direction of the image coordinate system. The table reconstructing means is an address position designated by the input address information in the address conversion table in the initial state according to a parameter corresponding to an error amount of a roll angle of a camera coordinate system that is an attachment error of the camera module to the vehicle. the claim 1, 2, characterized in that to move the arbitrary coordinates of the image coordinate system into the rotational direction of the rotation center, the in-vehicle image processing apparatus according to any one of 4. The table reconstruction means is a process for shifting the address position specified by the input address information in the address conversion table in the initial state in the horizontal direction of the image coordinate system in accordance with a parameter corresponding to the error amount of the yaw angle of the camera coordinate system. and shifting the address position in the longitudinal direction of the image coordinate system in which the input address information is specified in the address conversion table of the previous SL initial state in accordance with the parameter corresponding to the error amount of the pitch angle of / or pre-listen camera coordinate system After performing the processing, according to the parameter corresponding to the roll angle error amount of the camera coordinate system, the address position specified by the input address information in the address conversion table in the initial state is rotated to an arbitrary coordinate of the image coordinate system. according to claim 3 or 5, characterized in that the process of moving in the rotational direction around In-vehicle image processing apparatus. An image processing method for a vehicle that converts a viewpoint of each of camera image data generated by a plurality of camera modules that capture the vehicle periphery and displays an overhead image of the vehicle periphery on a display means,
An address conversion table for creating the bird's-eye view image using the camera image data for creating the bird's-eye view image using camera image data in a state where there is no attachment error of the camera module to the vehicle. Storing the address conversion table in the initial state in the storage means, the address conversion table reconfigured according to the parameter for correcting the mounting error of the camera module to the vehicle;
Storing camera image data generated by the plurality of camera modules in a plurality of input buffers corresponding to each camera module;
The camera image data is extracted from the input buffer according to the address conversion table stored in the storage means, and the camera image data is stored in the output buffer according to the address conversion table stored in the storage means to create the overhead image. And steps to
Reconfiguring the address conversion table in the initial state according to a parameter for correcting an attachment error of the camera module to the vehicle, and creating an address conversion table to be stored in the storage unit;
In the step of creating an address conversion table to be stored in the storage means, the input address information in the address conversion table in the initial state is changed according to a parameter for correcting an attachment error of the camera module to the vehicle, and the storage means The address specified by the input address information in the initial address conversion table according to a parameter corresponding to an error amount of a yaw angle of the camera coordinate system, which is an attachment error of the camera module to the vehicle vehicular image processing how to said shifting position in the transverse direction of the images coordinate system. An image processing method for a vehicle that converts a viewpoint of each of camera image data generated by a plurality of camera modules that capture the vehicle periphery and displays an overhead image of the vehicle periphery on a display means,
An address conversion table for creating the bird's-eye view image using the camera image data for creating the bird's-eye view image using camera image data in a state where there is no attachment error of the camera module to the vehicle. Storing the address conversion table in the initial state in the storage means, the address conversion table reconfigured according to the parameter for correcting the mounting error of the camera module to the vehicle;
Storing camera image data generated by the plurality of camera modules in a plurality of input buffers corresponding to each camera module;
The camera image data is extracted from the input buffer according to the address conversion table stored in the storage means, and the camera image data is stored in the output buffer according to the address conversion table stored in the storage means to create the overhead image. And steps to
Reconfiguring the address conversion table in the initial state according to a parameter for correcting an attachment error of the camera module to the vehicle, and creating an address conversion table to be stored in the storage unit;
In the step of creating an address conversion table to be stored in the storage means, the input address information in the address conversion table in the initial state is changed according to a parameter for correcting an attachment error of the camera module to the vehicle, and the storage means The address specified by the input address information in the address conversion table in the initial state according to a parameter corresponding to an error amount of the pitch angle of the camera coordinate system, which is an attachment error of the camera module to the vehicle vehicular image processing how to said shifting position in the longitudinal direction of the image coordinate system. An image processing method for a vehicle that converts a viewpoint of each of camera image data generated by a plurality of camera modules that capture the vehicle periphery and displays an overhead image of the vehicle periphery on a display means,
An address conversion table for creating the bird's-eye view image using the camera image data for creating the bird's-eye view image using camera image data in a state where there is no attachment error of the camera module to the vehicle. Storing the address conversion table in the initial state in the storage means, the address conversion table reconfigured according to the parameter for correcting the mounting error of the camera module to the vehicle;
Storing camera image data generated by the plurality of camera modules in a plurality of input buffers corresponding to each camera module;
The camera image data is extracted from the input buffer according to the address conversion table stored in the storage means, and the camera image data is stored in the output buffer according to the address conversion table stored in the storage means to create the overhead image. the method comprising the steps of,
Wherein the address conversion table in the initial state, and reconstituted according to the parameters to correct mounting error for the vehicle of the camera module, have a step of creating an address conversion table to be stored in the storage means,
In the step of creating an address conversion table to be stored in the storage means, the input address information in the address conversion table in the initial state is changed according to a parameter for correcting an attachment error of the camera module to the vehicle, and the storage means The address specified by the input address information in the address conversion table in the initial state according to a parameter corresponding to an error amount of a roll angle of the camera coordinate system, which is an attachment error of the camera module to the vehicle position, vehicular image processing method comprising Rukoto move any coordinate of an image coordinate system into the rotational direction of the rotation center. In the step of creating an address conversion table to be stored in the storage means, the address conversion table in the initial state according to a parameter corresponding to an error amount of a pitch angle of a camera coordinate system that is an attachment error of the camera module to the vehicle. vehicular image processing method according to claim 7, characterized in Rukoto shifting the address where the input address information is designated in the vertical direction of the image coordinate system. In the step of creating an address conversion table to be stored in said storage means, in accordance with the parameters according to the error amount of the roll angle of the camera coordinate system is a mounting error with respect to the vehicle in front Symbol camera module, in the address conversion table of the initial state the vehicle according to the address position where the input address information is specified, claim 7, 8, characterized in Rukoto move any coordinate of an image coordinate system into the rotational direction of the rotation center, in any one of the 10 Image processing method. In the step of creating an address conversion table to be stored in the storage means, the input address information is specified in the address conversion table of the previous SL initial state in accordance with the parameter corresponding to the error amount of the yaw angle before hear camera coordinate system The address position specified by the input address information in the address conversion table in the initial state is imaged according to a process for shifting the address position in the horizontal direction of the image coordinate system and / or a parameter corresponding to the error amount of the pitch angle of the camera coordinate system. After performing the process of shifting in the vertical direction of the coordinate system, the address position specified by the input address information in the address conversion table in the initial state is determined according to the parameter corresponding to the roll angle error amount of the camera coordinate system. Processing to move in the rotation direction around the arbitrary coordinate of the coordinate system Vehicular image processing method according to claim 9 or 11, characterized in that.