JP7110592B2 - Image processing device - Google Patents

Description

An embodiment of the present invention relates to an image processing apparatus.

Conventionally, a plurality of imaging units (cameras) provided in a vehicle capture images of the situation around the vehicle, perform image processing (e.g., viewpoint conversion) on the multiple captured images, and combine the images to form a surrounding image (e.g., a bird's-eye view). image) is known. In such an image processing apparatus, there are cases where the image pickup positions are displaced due to an installation error of each image pickup unit, a shock after installation, or the like, and the seams of the images may become discontinuous. Therefore, for example, it is checked whether or not the same object is shown in adjacent images forming the bird's-eye view image, and if the same object is shown, the contour of the object is detected. There is also a technique for ensuring continuity when a plurality of images are joined by correcting images so that the contours of adjacent images of imaging units are connected.

WO2015/129280

However, in the case of the conventional technology, since each image is corrected so that the contours of the object are connected, the position of the object may not correspond to the actual position. That is, image processing such as rotation processing, translation processing, scaling correction, or the like is performed on one image or both images in order to join the contours of the object. As a result, stitching may be performed based on an image that is shifted from the actual position of the object, or both images may move relative to each other and stitching may be performed at a position different from the actual position of the object. there were. In such a case, there is a possibility that the driver who sees the generated image may mistakenly recognize the distance to the object.

Therefore, one of the objects of the present invention is to provide an image processing apparatus that can display an object or the like to be displayed so as to correspond to an actual position when performing image processing on an image captured by an imaging unit and displaying the image. to provide.

An image processing apparatus according to an embodiment of the present invention includes, for example, an acquisition unit that acquires a first reference position that is set on a part of the outer surface of a vehicle and is stored in advance, and an imaging unit that is provided in the vehicle. a detection unit that detects a second reference position corresponding to the first reference position on the captured image data that has been obtained; and a correction unit that corrects the recognition position of the imaging unit. According to this configuration, for example, when the captured image data is subjected to image processing such as viewpoint conversion and displayed, the deviation of the image based on the installation error of the imaging unit is determined in advance for a part of the vehicle. Since the correction is made with reference to the stored first reference position, the position of the displayed image is fixed on the basis of the vehicle. As a result, it is possible to reduce the possibility that an object or the like appearing in the image is displayed with deviation from its actual position.
In the image processing apparatus according to the embodiment of the present invention, the first reference position may be, for example, a position where the front wheels of the vehicle touch the ground. According to this configuration, for example, it becomes easier to recognize the reference position for correction on the image based on the captured image data captured by the imaging unit. As a result, correction processing can be facilitated and correction accuracy can be improved.

The correction unit of the image processing apparatus according to the embodiment of the present invention may correct the recognition position of the imaging unit based on the amount of displacement between the first reference position and the second reference position, for example. According to this configuration, for example, it is possible to correct the recognition position of the imaging unit based on the positional relationship between the first reference position and the second reference position.

The correcting unit of the image processing apparatus according to the embodiment of the present invention may correct the projection transformation parameter based on the corrected recognized position of the imaging unit, for example. According to this configuration, for example, it is possible to smoothly perform projection processing of an image captured by the imaging unit.

The imaging unit of the image processing apparatus according to the embodiment of the present invention includes, for example, a first imaging unit that images a side area of the vehicle and a second imaging unit that images a traveling direction area of the vehicle. The correction unit corrects the recognition position of one of the first imaging unit and the second imaging unit, for example, and performs imaging of the one imaging unit with the corrected recognition position. an image processing unit that corrects the captured image data of the other imaging unit so as to maintain the continuity of the image content when connecting the captured image data of the other imaging unit to the image based on the image data. You may do so. According to this configuration, for example, other images are spliced based on an image in which an object or the like is shown at a position corresponding to the actual position. Alignment is performed following the image captured by the . As a result, the deviation of the image is eliminated as a whole, and the quality of the composite image (for example, the bird's-eye view image) can be improved.

FIG. 1 is a schematic top view showing an example of a vehicle on which an image processing device according to an embodiment can be mounted; FIG. 2 is an exemplary block diagram of the configuration of an image processing system including the image processing apparatus according to the embodiment; 3 is an exemplary block diagram of the configuration of the CPU of the image processing apparatus according to the embodiment; FIG. FIG. 4 is a bird's-eye view showing an example of a first reference position used in the image processing apparatus according to the embodiment; FIG. 5 is a schematic diagram illustrating an example of an image captured by an imaging unit on the right side, showing an example of a first reference position used in the image processing apparatus according to the embodiment; FIG. 6 is an exemplary schematic diagram for explaining the relationship between the coordinate system on the captured image (camera image) and the coordinate system on the road surface in the image processing apparatus according to the embodiment. FIG. 7 is an exemplary schematic diagram for explaining deviation correction of recognition position of an imaging unit in image processing in the image processing apparatus according to the embodiment. FIG. 8 shows, in the image processing apparatus according to the embodiment, as a result of a deviation between the actual position of the imaging unit and the recognized position of the imaging unit in image processing, when a bird's-eye view image is generated in that state, FIG. 11 is a schematic diagram illustrating an example in which the position of the white line to be displayed is shifted from the actual position; FIG. 9 is a schematic diagram for explaining an example in which the position of a white line matches the actual position when a bird's-eye view image is generated by correcting the recognition position deviation in the image processing of the imaging unit by the image processing apparatus according to the embodiment; It is a diagram. FIG. 10 is a flowchart illustrating an example of a processing procedure for generating a bird's-eye view image using the image processing apparatus according to the embodiment;

Illustrative embodiments of the invention are disclosed below. The configurations of the embodiments shown below and the actions, results, and effects brought about by the configurations are examples. The present invention can be realized by configurations other than those disclosed in the following embodiments, and at least one of various effects based on the basic configuration and derivative effects can be obtained. .

FIG. 1 is a schematic plan view of a vehicle 10 on which an image processing device of this embodiment is mounted. The vehicle 10 may be, for example, an automobile (internal combustion engine automobile) using an internal combustion engine (engine, not shown) as a drive source, or an automobile (electric automobile) using an electric motor (motor, not shown) as a drive source. , a fuel cell vehicle, etc.), or a vehicle (hybrid vehicle) using both of them as a driving source. In addition, the vehicle 10 can be equipped with various transmission devices, and can be equipped with various devices (systems, parts, etc.) necessary for driving the internal combustion engine and the electric motor. Further, the system, number, layout, etc. of the devices involved in driving the wheels 12 (the front wheels 12F and the rear wheels 12R) of the vehicle 10 can be set variously.

As illustrated in FIG. 1, the vehicle 10 is provided with, for example, four imaging units 14a to 14d as the plurality of imaging units 14. As shown in FIG. The imaging unit 14 is, for example, a digital camera incorporating an imaging element such as a CCD (Charge Coupled Device) or a CIS (CMOS image sensor). The imaging unit 14 can output moving image data (captured image data) at a predetermined frame rate. The imaging units 14 each have a wide-angle lens or a fish-eye lens, and can photograph a horizontal range of, for example, 140° to 220°. In some cases, the optical axis of the imaging unit 14 is set obliquely downward. Therefore, the imaging unit 14 can detect road surfaces on which the vehicle 10 can move, marks attached to the road surfaces (including arrows, lane markings, lines indicating parking spaces, lane separation lines, etc.) and objects (such as obstacles such as walking). The surrounding environment outside the vehicle 10 including people, vehicles, etc.) is sequentially captured and output as captured image data.

The imaging unit 14 is provided on the outer periphery of the vehicle 10 . The imaging unit 14a is provided, for example, at the right end of the vehicle 10, for example, the right door mirror 10a, and captures a right side image including an area centered on the right side of the vehicle 10 (for example, an area from the front right to the rear right). can be imaged. The imaging unit 14b is provided, for example, at the left end of the vehicle 10, for example, at the left door mirror 10b, and captures a left side image including an area centered on the left side of the vehicle 10 (for example, an area from the front left to the rear left). can be imaged. Further, the imaging unit 14c is provided, for example, on the front side of the vehicle 10, that is, on the front side in the vehicle front-rear direction and at the substantially central end portion in the vehicle width direction, for example, on the front bumper 10c or the front grille. (For example, the front bumper 10c) can be captured. The imaging unit 14d is provided, for example, on the rear side of the vehicle 10, that is, on the rear side in the vehicle front-rear direction and substantially in the center in the vehicle width direction, for example, on the rear bumper 10d. It is possible to image the rear area including the . In addition, the imaging unit 14 a and the imaging unit 14 b can be referred to as a first imaging unit that images the side area of the vehicle 10 . Further, the imaging units 14 c and 14 d can be referred to as a second imaging unit that captures an image of the traveling direction area of the vehicle 10 .

Arithmetic processing and image processing are performed based on captured image data obtained by a plurality of imaging units 14 to generate an image with a wider viewing angle, or to generate a virtual bird's-eye view image (planar image) of the vehicle 10 viewed from above. ) can be generated. It should be noted that the captured image data (images) captured by the respective imaging units 14 may be provided with an overlapping region that overlaps with each other. For example, the front end of the captured image data captured by the imaging unit 14a overlaps the right end of the captured image data captured by the imaging unit 14c. Then, when connecting (combining) two images, a blending process may be performed in which the images are combined using a% of each of the captured image data of the right side image and the captured image data of the front image. By executing the blending process, the image on the right side and the image on the front side are combined so as to gradually change, and the boundary line caused by the difference in brightness and color tone can be made inconspicuous. Similarly, by blending the front image and the left image, the left image and the rear image, and the rear image and the right image, it is possible to make the boundary line inconspicuous in the entire combined peripheral image.

FIG. 2 is an exemplary block diagram of the configuration of an image processing system 100 including an image processing device mounted on the vehicle 10. As shown in FIG. A display device 16 and an audio output device 18 are provided in the interior of the vehicle 10 . The display device 16 is, for example, an LCD (liquid crystal display), an OELD (organic electroluminescent display), or the like. The audio output device 18 is, for example, a speaker. Also, the display device 16 is covered with a transparent operation input unit 20 such as a touch panel. A passenger (for example, a driver) can visually recognize an image displayed on the display screen of the display device 16 via the operation input unit 20 . Further, the occupant can perform an operation input by touching, pressing, or moving the operation input unit 20 with a finger or the like at a position corresponding to the image displayed on the display screen of the display device 16. can. The display device 16, the audio output device 18, the operation input unit 20, and the like are provided, for example, in a monitor device 22 positioned at the center of the dashboard of the vehicle 10 in the vehicle width direction, that is, in the left-right direction. The monitor device 22 can have operation input units (not shown) such as switches, dials, joysticks, and push buttons. The monitor device 22 can also be used, for example, as a navigation system or an audio system.

Further, as illustrated in FIG. 2, the image processing system 100 (image processing device) includes an ECU 24 (electronic control unit) in addition to the imaging section 14 (14a to 14d) and the monitor device 22. FIG. In the image processing system 100 (image processing device), the ECU 24 and the monitor device 22 are electrically connected via an in-vehicle network 26 as an electric communication line. The in-vehicle network 26 is configured as, for example, a CAN (controller area network). The ECU 24 can control various systems by sending control signals through the in-vehicle network 26 . In addition, the ECU 24 can receive, via the in-vehicle network 26, operation signals of the operation input unit 20 and various switches, detection signals of various sensors (not shown), and the like.

The ECU 24 acquires captured image data from the imaging section 14 . The ECU 24 transmits to the monitor device 22 data relating to the peripheral image and sound generated based on the captured image and the like. The ECU 24 includes, for example, a CPU 24a (central processing unit), a ROM 24b (read only memory), a RAM 24c (random access memory), a display control section 24d, an audio control section 24e, and an SSD 24f (solid state drive, flash memory). ing.

The CPU 24a reads a program stored (installed) in a non-volatile storage device such as the ROM 24b, and executes arithmetic processing according to the program. The ROM 24b stores each program and parameters necessary for executing the program. The CPU 24a includes various modules as shown in FIG. 3, for example, and executes processing related to images displayed on the display device 16. For example, the CPU 24a executes correction processing, arithmetic processing, image processing, and the like on the captured image data captured by the imaging unit 14, and generates a peripheral image (for example, a bird's-eye view image) by connecting a plurality of images. Details of the CPU 24a will be described later.

The RAM 24c temporarily stores various data used in calculations by the CPU 24a. In addition, the display control unit 24 d mainly performs data conversion of images for display to be displayed on the display device 16 among the arithmetic processing in the ECU 24 . In addition, the audio control unit 24e mainly executes processing of audio data output from the audio output device 18 among the arithmetic processing in the ECU 24. FIG. The SSD 24f is a rewritable non-volatile storage unit, and can store data even when the power of the ECU 24 is turned off. The CPU 24a, ROM 24b, RAM 24c, etc. can be integrated in the same package. Further, the ECU 24 may have a configuration in which another logical operation processor such as a DSP (digital signal processor), a logic circuit, or the like is used instead of the CPU 24a. A HDD (hard disk drive) may be provided instead of the SSD 24f, or the SSD 24f and the HDD may be provided separately from the ECU 24.

In the present embodiment, the ECU 24 controls image generation processing of the vehicle 10 through cooperation of hardware and software (control program). When the ECU 24 performs image processing, such as viewpoint conversion processing, on the captured image data (image) captured by the imaging unit 14 and causes the display device 16 to display the captured image data (image), the vehicle 10 (own vehicle) may be affected by surrounding objects. After correcting the recognition position of the imaging unit 14 in the image processing so that the image is displayed at the correct position (the position corresponding to the actual position), the image processing is executed and the display image is output. For example, when displaying a peripheral image (for example, a bird's-eye view image) by joining a plurality of captured image data (images) captured by each of the imaging units 14 (14a to 14d), the vehicle 10 (own vehicle) is surrounded by Captured image data is corrected so that an object or the like is displayed at a correct position, and the data are joined together to generate a peripheral image. In the present embodiment, the correction of the recognition position in the image processing of the imaging unit 14 may be performed on any one imaging unit 14 or may be performed on a plurality of imaging units 14. good. In the following description, when the installation position and orientation of the imaging unit 14a for imaging the right side of the imaging unit 14 of the vehicle 10 is deviated from the designed position and orientation, recognition of the imaging unit 14a in image processing will be described. An example of correcting the entire image to be displayed by correcting the position will be described.

A CPU 24a included in the ECU 24 has various modules for executing the above-described correction processing. Various modules are implemented by the CPU 24a reading programs installed and stored in a storage device such as the ROM 24b and executing the programs. For example, the CPU 24a includes modules such as an acquisition unit 28, a projection processing unit 30, a detection unit 32, a correction unit 34, and an image processing unit 36, as shown in FIG. The acquisition unit 28 includes an image acquisition unit 28a, a first reference position acquisition unit 28b, a parameter acquisition unit 28c, etc. The correction unit 34 includes a displacement acquisition unit 34a, a recognition position correction unit 34b, a parameter correction unit 34c, etc. including.

The acquisition unit 28 acquires various kinds of information necessary for correcting the recognized position in the image processing of the imaging unit 14 .

The image acquisition unit 28a sequentially acquires captured image data captured by each imaging unit 14 (14a to 14d) via the display control unit 24d.

The first reference position acquisition unit 28 b acquires a first reference position set on a portion of the outer surface of the vehicle 10 . The first reference position set on a part of the outer surface of the vehicle 10 is pre-stored, for example, in the ROM 24b as a position on a road surface coordinate system (X, Y, Z), which will be described later. FIG. 4 is a bird's-eye view showing an example of the first reference position P. FIG. FIG. 5 is a schematic diagram showing an example of an image taken by the imaging section 14a (right side imaging section) showing an example of the first reference position P. As shown in FIG. The first reference position P, as shown in FIGS. 4 and 5, can be, for example, the ground contact point on the outer surface side of the right front wheel 12F. The first reference position P is not limited to the outer peripheral surface of the front wheel 12F, and may be any position on the outer surface of the vehicle 10 that is included in the imaging range of the imaging unit 14 (for example, the imaging unit 14a) to be corrected. . In another embodiment, the first reference position P may be set at a portion of the body of the vehicle 10, such as the bottom corner of a door, with similar effects. As described above, when the image is captured by the image capturing unit 14a, the first reference position P is the ground contact position of the front wheel 12F, which is relatively close to the image capturing unit 14a and can be easily recognized even when recognizing the image. By doing so, the recognition accuracy can be improved.

Note that the first reference position P may be determined based on the reference position O set on the vehicle 10 . By setting the reference position O, the position of the first reference position P in the physical space and the position of the first reference position P on the image can be easily recognized, and a coordinate system can be used. This simplifies the processing when performing calculations using The reference position O can be, for example, the central position in the longitudinal direction (vehicle width direction) of the rear wheel axle 12a that supports the left and right rear wheels 12R of the vehicle 10 . In another embodiment, the reference position O may be a position on the vehicle center CL extending in the longitudinal direction of the vehicle 10, for example. In this case, for example, the reference position O may be the intersection of perpendicular lines drawn from the first reference position P to the vehicle center CL. Since the actual distance from the reference position O to the first reference position P is known, the separation distance (the number of dots) on the image when the reference position O and the first reference position P are projected onto the image By detecting , it becomes easy to associate the deviation of one dot on the image with the deviation of the distance in the physical space. For example, if there is a 10-dot shift on the image, it can be associated with a 10-cm shift in the physical space. By storing the distance corresponding to one dot on the image in advance in the ROM 24b or the like, the deviation of one dot on the image may be associated with the deviation of the distance in the physical space.

この場合、式１における係数部分が投影変換パラメータとなる。ＲＯＭ２４ｂまたはＳＳＤ２４ｆには、現在の撮像部１４ａの画像処理上の認識位置（例えば原点）に対応する極座標（ρ，θ，φ）が保存されてもよいし、式１にρ，θ，φを代入した係数とした投影変換パラメータとして記憶しておいてもよい。 The parameter acquisition unit 28c acquires projection transformation parameters used when projectively transforming an image captured by the imaging unit 14 from a storage unit such as the ROM 24b or the SSD 24f. For example, as shown in FIG. 6, the coordinate system on the road surface is (X, Y, Z), and the coordinate system (camera coordinate system) on the camera image of the imaging unit 14a is (X _{0 ,} Y ₀ , Z ₀ ). , the distance and angle between the coordinates on the road surface and the coordinate system on the camera image are represented by polar coordinates (.rho., .theta., .phi.). ρ is the distance from the origin of the coordinate system on the road surface (corresponding to the reference position O in FIG. 4, for example) to the origin _T0 of the camera coordinate system. θ is the angle between the X axis and the straight line connecting the origin of the coordinate system on the road surface and the position of the point T projected from the origin _T0 of the camera coordinate system onto the XY plane of the coordinate system on the road surface. φ is the angle formed by the Z-axis and the straight line connecting the origin of the coordinate system on the road surface and the origin _T0 of the camera coordinate system.

In this case, the coefficient part in Equation 1 becomes the projection transformation parameter. The ROM 24b or SSD 24f may store the polar coordinates (ρ, θ, φ) corresponding to the current recognition position (for example, the origin) in the image processing of the imaging unit 14a. It may be stored as a projection transformation parameter as a substituted coefficient.

The projection processing unit 30 applies the projection parameters acquired by the parameter acquisition unit 28c to each pixel (camera coordinate system (X _{0 ,} Y ₀ , Z ₀ )) can be sequentially projected onto the coordinate system (X, Y, Z) on the road surface. For example, a pixel positioned at the origin T ₀ of the camera coordinate system (X _{0 ,} Y ₀ , Z ₀ ) can be projected onto a point T on the coordinate system (X, Y, Z) on the road surface. In this case, the imaging unit 14 is fixed with respect to the vehicle 10 . For example, since the imaging unit 14a is fixed to the lower surface of the right side door mirror 10a, the recognition position of the imaging unit 14a in image processing can be fixed at a fixed position. Therefore, when the installation position and mounting posture of the imaging unit 14a are maintained at the design positions and postures, the pixels shown in the imaged image data, for example, corresponding to the contact position of the front wheel 12F, are on the road surface. On the coordinate system (X, Y, Z), it is projected and transformed to a position that matches the first reference position P acquired by the first reference position acquisition unit 28b.

On the other hand, the installation position and installation posture of the imaging unit 14a may deviate from the design position and posture due to some cause, for example, vibration given by running the vehicle 10 or contact with other objects. may be lost. In such a case, for example, an image corresponding to the actual ground contact position of the front wheel 12F (substantially the same position as the first reference position P) is expressed in the camera coordinate system (X _{0 ,} Y ₀ , Z ₀ ) will be imaged on the camera coordinate system (X _{1 ,} Y ₁ , Z ₁ ) deviated from the camera coordinate system. In this case, in terms of image processing, the recognition position of the imaging unit 14a remains the recognition position on the camera coordinate system (X _{0 ,} Y ₀ , Z ₀ ). The imaged image data captured by is projected and transformed using Equation 1 to which the projection transformation parameters corresponding to the polar coordinates (ρ, θ, φ) are applied. As a result, the pixels corresponding to the contact position of the front wheel 12F are projected onto the coordinate system (X, Y, Z) on the road surface after being moved by an amount corresponding to the deviation of the installation position and posture of the imaging unit 14a. become. That is, it is projected to the second reference position Q shifted from the first reference position P. FIG. FIG. 8 shows an example in which a bird's-eye view image (a bird's-eye view image of the right side of the vehicle 10) is generated based on captured image data captured when the installation position and posture of the imaging unit 14a are shifted. is a schematic diagram for explaining an example in which the position of is deviated from the actual position of the white line _L0 . When the driver recognizes the surroundings of the vehicle 10 based on such a bird's-eye view image, the position of the white line L existing on the side of the vehicle 10 is relative to the vehicle icon 40 displayed at the center of the image, for example. is displayed farther than it actually is. In other words, the driver may be given a sense of distance that is different from the actual distance to the white line _L0 . As a result, the vehicle 10 may not be able to pull the vehicle 10 to the side as expected, or the distance to an obstacle may be mistakenly recognized and the vehicle may come into contact with the obstacle.

Therefore, the detection unit 32 of the image processing system 100 of the present embodiment detects the first reference position P Detect a second reference position Q corresponding to . For example, on the camera coordinate system (X _{1 ,} Y ₁ , Z ₁ ) of the imaging unit 14a whose installation position and orientation are shifted, the projection transformation parameters (initial values) determined in a state where there is no shift are represented by (X _0, Y ₀ , Z ₀ ) to (X _{1 ,} Y ₁ , Z ₁ ). Then, the ground contact position of the front wheel 12F imaged by the imaging unit 14a is projected onto the coordinate system (X, Y, Z) on the road surface. That is, the detection unit 32 corresponds to the first reference position P acquired by the first reference position acquisition unit 28b (the position where the front wheel 12F is grounded if there is no deviation in the installation position or posture of the imaging unit 14a). A second reference position Q can be detected, which is the position.

Then, the correction unit 34 detects the difference between the first reference position P and the second reference position Q detected by the detection unit 32, that is, the displacement amount (difference, difference distance), based on the imaging unit 14 (for example, the imaging unit 14a ) is corrected in image processing. Correction of the recognition position can be executed, for example, by correcting the projection transformation parameters when performing the projection transformation.

The displacement acquisition unit 34a detects, for example, the amount of displacement (deviation) between the first reference position P and the second reference position Q projected onto the coordinate system (X, Y, Z) on the road surface. For example, when an image of an object is captured by the imaging unit 14a and projected onto the coordinate system (X, Y, Z) on the road surface using the image processing system 100 of the present embodiment, the ROM 24b can display the image with 10 dots, for example. Displacement reference value data indicating the magnitude (length, distance, etc.) is stored together with the data of the first reference position P. FIG. The displacement acquisition unit 34a can acquire the amount of displacement between the first reference position P and the second reference position Q based on the displacement reference value data. Note that the displacement reference value data may be determined based on the relationship between the actual distance between the reference position O and the first reference position P and the distance (the number of dots) on the screen, as described with reference to FIG.

Based on the displacement amount acquired by the displacement acquisition unit 34a, the recognition position correction unit 34b corrects the camera coordinate system (X _1, Y ₁ , Z ₁ ) are corrected. For example, the second reference position Q is corrected to the recognition position of the imaging unit 14a in image processing so that it matches the first reference position P. FIG. That is, the camera coordinate system ₍ X1 _, Y1, _Z1 ) is corrected to the camera coordinate system ₍ X2 _, Y2, _Z2 ). At this time, when the amount of displacement between the first reference position P and the second reference position Q is 10 dots, the imaging unit 14a recognizes the second reference position Q so that it matches the first reference position P. By moving the position by an amount corresponding to 10 dots, the recognition position of the imaging unit 14a can be corrected.

The parameter correction unit 34c sets the coordinate system on the road surface to (X, Y, Z), sets the camera coordinate system obtained as a result of correcting the recognition position to (X _{2 ,} Y ₂ , Z ₂ ), and sets the coordinates on the road surface to and corrected polar coordinates (ρ ₂ , θ ₂ , φ ₂ ) indicating the distance and angle of the coordinate system on the camera image. Then, when the polar coordinates (ρ ₂ , θ ₂ , φ ₂ ) are applied to Equation 1 in which (X _{0 ,} Y ₀ , Z ₀ ) are read as (X _{2 ,} Y ₂ , Z ₂ ), the coefficient part becomes is the projection transformation parameter of That is, for example, the projection transformation parameters of the SSD 24f are updated. In this case, the polar coordinates (ρ ₂ , θ ₂ , φ ₂ ) corresponding to the recognition position in the image processing of the imaging unit 14a after correction may be stored in the SSD 24f, or (X _{0 ,} Y ₀ , Z ₀ ) may be replaced with (X _{2 ,} Y ₂ , Z ₂ ) and ρ ₂ , θ ₂ , and φ ₂ are substituted into Equation 1, and the corrected projection transformation parameters may be stored and saved in the SSD 24f.

Then, the projection processing unit 30 projects and transforms the captured image data captured by the imaging unit 14a using the corrected projection transformation parameter, so that the first reference position P and the second reference position Q match. A projection image can be obtained.

The image processing unit 36 generates an image obtained by correcting the recognition position, for example, based on captured image data captured by the imaging unit 14a, and a captured image captured by the other imaging unit 14 (for example, the front imaging unit 14c). Performs processing for concatenating images based on data. In this case, the captured image data of the imaging unit 14c is corrected so as to maintain continuity between the corrected image content of the imaging unit 14a and the image content of the imaging unit 14c. For example, as shown in FIG. 9, the image processing unit 36 performs viewpoint conversion of the captured image data captured by each of the image capturing units 14 (14a to 14d), and joins them together so that the surroundings of the vehicle 10 (own vehicle icon 40) are displayed. A bird's-eye view image is generated. In this case, a region R1 obtained by correcting the captured image data captured by the imaging unit 14a using the projection conversion parameters after correction by the correction unit 34, and a region _R1 using the initially set projection conversion parameters without correcting the projection conversion parameters. The region R ₂ (captured image data of the imaging unit 14c), the region R ₃ (captured image data of the imaging unit 14b), and the region R ₄ (captured image data of the imaging unit 14d) which have been projected and transformed are combined.

As described above, the white line L1 appearing in the region _R1 where the image based on the captured image data of the imaging unit 14a, which has been subjected to the projection conversion by correcting the projection conversion parameters, is drawn is the position where the actual white line _{L0 exists} . exists corresponding to On the other hand, the right white line _L2R appearing in the area _R2 where the image based on the captured image data of the imaging unit 14c that has undergone the projection conversion without correcting the projection conversion parameters is drawn is shifted from the actual position. and may not maintain continuity with the white line _L1 . _In this case, the white line _L2R can be aligned with the corrected white line L1 by performing rotation processing, translation processing, enlargement/reduction processing, and the like. That is, the white line _L2R is also processed so as to be drawn at the correct position. In this case, since the white line L2L included in the region _R2 exists in the same region _captured by the image pickup unit 14c, the drawing position of the white line _L2L is also affected by the correction of the white line _L1 , and the drawing position is accurate. It will be drawn. Similarly, based on the white line _L2L of the area _R2 drawn at the correct position based _on the white line L1 _, the area R3 is drawn at the correct position by performing rotation processing, translation processing, scaling processing, etc. be done. The area _R4 can also be drawn at an accurate position by performing rotation processing, parallel movement, scaling processing, etc. _on the area _R4 based _on the white line L3 of the area R3 drawn at an accurate position. As _a result, the white line _{L4R coincides with the white line L3, and the white line L4L coincides with} the white line L1.

In this way, by correcting the projection transformation parameter for the captured image data captured by the imaging unit 14 at one location of the imaging unit 14, an accurate position centered on the vehicle 10 (own vehicle icon 40) can be obtained. A bird's-eye view image showing the relationship can be generated by simple processing. In the above example, the recognition position in image processing is corrected (projection transformation parameter correction) for any one of the imaging units 14, and the images of the other imaging units 14 are adjusted to the corrected image. I showed an example of processing like this. In another embodiment, correction of recognized positions in image processing (correction of projection transformation parameters) may be performed for a plurality of imaging units 14 . In this case, it is possible to improve the drawing accuracy of the synthesized bird's-eye view image.

FIG. 10 is a flowchart for explaining an example of image processing when generating a bird's-eye view image using the image processing system 100 (image processing apparatus) described above. Note that the installation position and posture of the imaging unit 14 do not shift frequently. This occurs only when the door mirror 10b) comes into contact with an external object, when it is repaired or replaced, or when it changes over time, and basically does not come off. Therefore, in the present embodiment, the correction of the recognition position in the image processing of the imaging unit 14 by correcting the projection transformation parameter can be performed at relatively long intervals, for example, regularly or irregularly.

First, the CPU 24a confirms whether display (creation) of a bird's-eye view image is requested. A navigation screen and an audio screen are displayed as normal screens on the display device 16, but when the driver operates the operation input unit 20 to request display of a bird's-eye view image, the CPU 24a displays the bird's-eye view image. The display mode is entered (Yes in S100). When a bird's-eye view display request is made, the image acquisition unit 28a acquires captured image data (peripheral images) captured by the imaging units 14 (14a to 14d) via the display control unit 24d (S102). At this time, if it is time to readjust (correct) the recognition position of the imaging unit 14 in image processing (Yes in S104), the first reference position acquisition unit 28b acquires the first reference position stored in the ROM 24b, for example. Information about the position P is acquired (S106). The timing of readjustment (correction) of the recognition position in image processing is, for example, when the ignition switch is turned on, or when a certain period of time (for example, one month) has passed since the previous readjustment. For example, when requesting adjustment.

When the timing for readjusting the recognition position in the image processing of the imaging unit 14 comes, the projection processing unit 30 acquires the currently valid projection transformation parameters from the ROM 24b or the SSD 24f via the parameter acquisition unit 28c, and performs a provisional bird's-eye view. An image is drawn (S108). For example, when the captured image data to be processed is captured by the imaging unit 14a, a bird's-eye view image of the right side is generated. At this time, if the installation position or posture of the imaging unit 14 to be corrected (for example, the imaging unit 14a) is misaligned, as shown in FIG. A second reference position Q is projected (drawn) at a position different from the position P. The detection unit 32 detects the second reference position Q (the contact position of the front wheel 12F) displayed in the temporary bird's-eye view image (coordinate system (X, Y, Z) on the road surface) (S110). Note that when the installation position and orientation of the imaging unit 14 to be corrected (for example, the imaging unit 14a) are not deviated, the first reference position P and the second reference position Q are drawn so as to substantially match. .

Subsequently, the displacement acquisition unit 34a calculates (acquires) the amount of displacement between the first reference position P and the second reference position Q on the coordinate system (X, Y, Z) on the road surface (S112), Based on the displacement amount acquired by the displacement acquisition unit 34a, the recognition position correction unit 34b corrects the recognition position of the imaging unit 14a in image processing so that the first reference position P and the second reference position Q match. As shown in FIG. 7, for example, it is corrected to the camera coordinate system (X _{2 ,} Y ₂ , Z ₂ ) (S114). Then, the parameter correction unit 34c calculates the polar coordinates (ρ ₂ , θ ₂ , φ ₂ ) in the corrected recognition position (camera coordinate system (X _{2 ,} Y ₂ , Z ₂ )), and (X _{0 ,} Y ₀ , Z ₀ ) are replaced with (X _{2 ,} Y ₂ , Z ₂ ) to change (update) the projection transformation parameters for the imaging unit 14a (S116). The changed projection transformation parameters are stored in, for example, the SSD 24f, and are used for projection transformation of captured image data captured by the imaging unit 14a until the projection transformation parameters are updated next time.

Subsequently, the projection processing unit 30 draws a side bird's-eye view image (S118). In this case, for example, the right bird's-eye view image is drawn as a whole by applying the latest projection transformation parameter corrected by the parameter correction unit 34c to each pixel of the captured image data captured by the imaging unit 14a and performing projection transformation. . On the other hand, the left side bird's-eye view image is projected and transformed by applying the projection transformation parameters determined when the imaging unit 14b is installed and stored in the ROM 24b to each pixel of the captured image data captured by the imaging unit 14b. is drawn by Also, the projection processing unit 30 draws the front and rear overhead images (S120). In this case, the front bird's-eye view image is projection-transformed by applying the projection transformation parameters, which are stored in the ROM 24b or the like and are determined when the imaging unit 14c is installed, to each pixel of the captured image data captured by the imaging unit 14c. It is drawn by Similarly, the rear bird's-eye view image is projected and transformed by applying the projection transformation parameters determined when the imaging unit 14d is installed and stored in the ROM 24b or the like to each pixel of the captured image data captured by the imaging unit 14d. It is drawn by

Then, the image processing unit 36 extracts a common point with continuity appearing in each drawn bird's-eye view image (S122). For example, as shown in FIG. 9, a white line L1 appearing in _a bird's-eye view image of a region _R1 based on the captured image data captured by the imaging unit 14a and a bird's-eye view of a region _R2 based on the captured image data captured by the imaging unit 14c. The white lines _L2R appearing in the image are extracted. Similarly, a white line L _2L appearing in the bird's-eye view image of the region R ₂ based on the captured image data captured by the imaging unit 14c, and a white line appearing in the bird's-eye view image of the region R ₃ based on the captured image data captured by the imaging unit 14b. L ₃ , white lines L _4L , L _4R , etc. appearing in the bird's-eye view image of the region R ₄ based on the captured image data captured by the imaging unit 14d are extracted.

Then, the image processing unit 36 synthesizes a bird's-eye view image based on the captured image data captured by each imaging unit 14 and displays it on the display device 16 (S124). Specifically, _first , the image processing unit 36 rotates and translates the front bird's _- eye view image displayed in the region _R2 with respect to the white line L1 so that the white line L1 and the white line _L2R are continuous. Processing, scaling, etc. are performed to adjust the drawing position. Subsequently, the image processing unit 36 rotates, translates, and scales the bird's-eye view image _on the left side of the white line _L2L displayed in the area R3 so that the white line _L2L and the white line L3 _are continuous. etc. to adjust the drawing position. Similarly, the image processing unit 36 _{performs rotation processing, translation processing, scaling processing, etc. of the bird's-eye view image displayed in the area R4 behind the white line L3 so that the white line L3 and the white line L4L are continuous .} to adjust the drawing position. As a result, as shown in FIG. 9, _each bird's-eye view image is aligned with reference to the white line L1 drawn at the correct position with respect to the own vehicle icon 40, and accurate drawing is realized. That is, it is possible to display an accurate bird's-eye view image on the display device 16 in which the sense of distance to the display objects around the own vehicle icon 40 (vehicle 10) corresponds to the sense of distance to the actual object.

After the bird's-eye view image is displayed on the display device 16, the CPU 24a confirms whether or not a request to end the display of the bird's-eye view image has been made (S126). For example, when the operation input unit 20 is operated and a request to end the display of the bird's-eye view image is made (Yes in S126), the CPU 24a changes the display content of the display device 16 from the currently displayed bird's-eye view image to the normal screen. After switching to a certain navigation screen or audio screen (S128), this flow is terminated.

In S126, if a request to end the display of the bird's-eye view image has not been issued (No in S126), the CPU 24a returns to the process of S102, acquires the peripheral image to be used in the next processing cycle via the image acquisition unit 28a, Repeat the following process. In this case, if it is not the time to readjust (correct) the recognition position in image processing in S104 (No in S104), the processes of S106 to S116 are skipped, and the image pickup unit 14a updated and stored in the current SSD 24f. and the projection transformation parameters initially set for the imaging units 14b, 14c, and 14d stored in the ROM 24b, the bird's-eye view image is drawn in and after S118. In other words, it prevents the projection transformation parameter from being excessively updated, thereby contributing to the reduction of the processing load on the CPU 24a. In S100, if creation of a bird's-eye view image (display on the display device 16) is not requested (No in S100), the processing of S102 to S126 is skipped and the normal screen (navigation screen or audio screen) continues to be displayed. do.

Note that the flowchart shown in FIG. 10 is only an example, and if substantially the same processing can be executed, the processing steps can be appropriately increased, decreased, replaced, or changed, and similar effects can be obtained. In the above example, the projection transformation parameter for the captured image data captured by the imaging unit 14a is corrected by correcting the recognized position in the image processing of the imaging unit 14a. In another embodiment, the recognition position in the image processing of the other imaging unit 14 may be corrected in the same manner, and the projection transformation parameters for the captured image data may be corrected. The imaging unit 14 to be corrected may be changed each time. When the target imaging unit 14 is changed each time correction processing is performed, the recognition position of the imaging unit 14a is first corrected at the readjustment (correction) timing of S104, and the recognition position of the imaging unit 14b is corrected at the next processing cycle. correction. Then, the recognition position of the imaging unit 14c is corrected in the next processing cycle, and the recognition position of the imaging unit 14d is corrected in the next processing cycle. In other words, the processing at the readjustment (correction) timing is considered to be completed when all the corrections of the recognition positions of the imaging units 14 are completed. In this case, since the recognition position of each imaging unit 14 is corrected, a more accurate bird's-eye view image can be generated.

In the case of the imaging unit 14c that captures an image of an area in front, the first reference position P may be set by forming or attaching a mark or the like on the front bumper 10c included in the imaging area of the imaging unit 14c. . Similarly, in the case of the imaging unit 14d that images the rear area, the first reference position P may be set by forming or attaching a mark or the like on the rear bumper 10d included in the imaging area of the imaging unit 14d. .

Further, in order to improve the recognizability of the second reference position Q corresponding to the first reference position P, for example, a detection mark such as a sticker or an LED may be provided on a part of the vehicle 10 . By using a light-emitting body such as an LED as the mark, the detection accuracy of the position of the second reference position Q can be improved, and the correction accuracy of the recognized position in the image processing of the imaging unit 14 can be improved. In addition, even when the surroundings of the vehicle 10 are dark (for example, at night), accurate correction processing can be performed.

The program for image processing executed by the CPU 24a of this embodiment is a file in an installable format or an executable format, and can may be configured to be recorded and provided on a computer-readable recording medium.

Furthermore, the image processing program may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Further, the image processing program executed in this embodiment may be configured to be provided or distributed via a network such as the Internet.

While embodiments and variations of the invention have been described, these embodiments and variations are provided by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

DESCRIPTION OF SYMBOLS 10... Vehicle 12... Wheel 12F... Front wheel 14, 14a, 14b, 14c, 14d... Imaging part 16... Display device 20... Operation input part 24... ECU, 24a... CPU, 24b... ROM, 24d... Display control unit 24f SSD 28 acquisition unit 28a image acquisition unit 28b first reference position acquisition unit 28c parameter acquisition unit 30 projection processing unit 32 detection unit 34 correction unit , 34a... Displacement acquisition unit, 34b... Recognition position correction unit, 34c... Parameter correction unit, 36... Image processing unit, 40... Vehicle icon, 100... Image processing system, L... White line, P... First reference position, Q...Second reference position.

Claims (4)

an acquisition unit configured to acquire a pre-stored first reference position set on a portion of the outer surface of the vehicle;
a detection unit that detects a second reference position corresponding to the first reference position on captured image data acquired by an imaging unit provided in the vehicle;
a correction unit that corrects the recognition position of the imaging unit in image processing so that the second reference position matches the first reference position ,
The first reference position is the ground contact position of the front wheels of the vehicle.
Image processing device. The image processing apparatus according to claim 1, wherein the correction section corrects the recognition position of the imaging section based on the amount of displacement between the first reference position and the second reference position. 3. The image processing apparatus according to claim 1, wherein the correcting section corrects the projection transformation parameter based on the corrected recognized position of the imaging section. The imaging unit includes a first imaging unit that images a side area of the vehicle and a second imaging unit that images a traveling direction area of the vehicle,
The correction unit corrects the recognition position of one of the first imaging unit and the second imaging unit,
When connecting the captured image data of the other imaging unit to the image based on the captured image data of the one of the imaging units in which the recognition position is corrected, the other 4. The image processing apparatus according to any one of claims 1 to 3 , comprising an image processing section that corrects captured image data of the imaging section.

Images