JP2019114965A - Image processing apparatus - Google Patents

Abstract

To provide an image processing apparatus that, when displaying images picked up by imaging units while performing image processing on the images, can project an object to be displayed or the like to correspond to an actual position.SOLUTION: An image processing apparatus comprises: an acquisition unit that acquires a first reference position set to a part of an outer surface of a vehicle and stored in advance; a detection unit that detects a second reference position corresponding to the first reference position on picked-up image data acquired by imaging units provided on the vehicle; and a correction unit that corrects a position recognized by the imaging unit in image processing so that the second reference position matches the first reference position.SELECTED DRAWING: Figure 3

Description

  Embodiments of the present invention relate to an image processing apparatus.

  Conventionally, a plurality of imaging units (cameras) provided in a vehicle picks up a situation around the vehicle, performs image processing (for example, viewpoint conversion) of the plurality of picked up images, and connects each image to a surrounding image (for example, An image processing apparatus that generates an image) is known. In such an image processing apparatus, an attachment position of an imaging unit may be shifted due to an attachment error of each imaging unit, an impact after attachment, or the like, and a seam of an image may be discontinuous. Therefore, for example, it is confirmed whether or not an object recognized as identical in adjacent images constituting a bird's-eye view image appears, and when the same object is imaged, the contour of the object is detected. Then, there is a technique for ensuring continuity when connecting a plurality of images by correcting the images so that the contours are connected between the images of the adjacent imaging units.

WO 2015/129280

  However, in the case of the prior art, since the correction of each image is performed so that the outline of the object is connected, the position of the object may not correspond to the actual position. That is, in order to connect the contours of the object, image processing such as rotation processing, parallel movement processing, and enlargement / reduction correction is performed on one or both images. As a result, the connection may be performed based on the image shifted from the position of the actual object, or both images may move relative to each other and the connection may be performed at a position different from the position of the actual object. there were. In such a case, there is a possibility that the driver who has seen the generated image may erroneously recognize the distance to the object.

  Therefore, one of the problems of the present invention is an image processing apparatus capable of displaying an object or the like to be displayed so as to correspond to an actual position when image processing is performed on an image captured by an imaging unit and displayed. It is to provide.

  The image processing apparatus according to the embodiment of the present invention is acquired by, for example, an acquisition unit configured to acquire a first reference position stored in advance and set on a part of the outer surface of the vehicle, and an imaging unit provided to the vehicle A detection unit for detecting a second reference position corresponding to the first reference position on the captured image data, and the image processing such that the second reference position matches the first reference position And a correction unit that corrects the recognition position of the imaging unit. According to this configuration, for example, when image processing such as viewpoint conversion is performed on the captured image data captured and displayed, the displacement of the image based on the mounting error or the like of the imaging unit is determined in part of the vehicle. Since the correction is made on the basis of the stored first reference position, the position of the displayed image is fixedly determined on the basis of the vehicle. As a result, it can be reduced that an object or the like shown in an image is displayed being shifted from the actual position.

  The correction unit of the image processing apparatus according to the embodiment of the present invention may correct the recognition position of the imaging unit based on, for example, the displacement amount between the first reference position and the second reference position. According to this configuration, for example, the recognition position of the imaging unit can be corrected based on the positional relationship between the first reference position and the second reference position.

  The correction unit of the image processing apparatus according to the embodiment of the present invention may correct, for example, the projection conversion parameter based on the corrected recognition position of the imaging unit. According to this configuration, for example, projection processing of an image captured by the imaging unit can be smoothly performed.

  In the image processing apparatus according to the embodiment of the present invention, the first reference position may be, for example, a position at which the front wheel of the vehicle is in contact with the ground. According to this configuration, for example, it is easy to recognize the reference position at the time of correction on the image based on the captured image data captured by the imaging unit. As a result, correction processing can be facilitated and correction accuracy can be improved.

  The imaging unit of the image processing apparatus according to the embodiment of the present invention includes, for example, a first imaging unit for imaging the side area of the vehicle, and a second imaging unit for imaging the traveling direction area of the vehicle. The correction unit corrects a recognition position of one of the first imaging unit and the second imaging unit, for example, and captures an image of the one imaging unit in which the recognition position is corrected. When connecting captured image data of the other imaging unit to an image based on image data, the image processing unit corrects the captured image data of the other imaging unit so that continuity of the image content is maintained. You may do so. According to this configuration, for example, another image is connected based on an image in which an object or the like appears at a position corresponding to an actual position, so that an image pickup unit whose recognition position has been corrected Alignment is performed according to the image captured in As a result, the displacement of the image is eliminated as a whole, and the quality of the composite image (for example, the overhead image) can be improved.

FIG. 1 is a schematic top view showing an example of a vehicle on which the image processing apparatus according to the embodiment can be mounted. FIG. 2 is an exemplary block diagram of an image processing system including the image processing apparatus according to the embodiment. FIG. 3 is an exemplary block diagram of the configuration of the CPU of the image processing apparatus according to the embodiment. FIG. 4 is a perspective view showing an example of a first reference position used in the image processing apparatus according to the embodiment. FIG. 5 is a schematic view showing an example of an image captured by an imaging unit on the right side, showing an example of a first reference position used in the image processing apparatus according to the embodiment. FIG. 6 is an exemplary schematic view illustrating a relationship between a coordinate system on a captured image (camera drawing) and a coordinate system on a road surface in the image processing apparatus according to the embodiment. FIG. 7 is an exemplary schematic view illustrating misalignment correction of a recognition position of an imaging unit in image processing in the image processing apparatus according to the embodiment. In the image processing apparatus according to the embodiment, as a result of the actual position of the imaging unit and the recognition position of the imaging unit on the image processing being shifted in the image processing apparatus according to the embodiment, as shown in FIG. It is a schematic diagram explaining the example by which the position of the white line to be displayed shifts | deviates from an actual position, and is displayed. FIG. 9 is a schematic diagram for explaining an example in which the position of the white line coincides with the actual position when the displacement of the recognition position on the image processing of the imaging unit by the image processing apparatus according to the embodiment is corrected to generate the overhead image. FIG. FIG. 10 is a flowchart for explaining an example of a processing procedure in the case of generating a bird's-eye view image by the image processing apparatus according to the embodiment.

  In the following, exemplary embodiments of the present invention are disclosed. The configurations of the embodiments shown below, and the operations, results, and effects provided by the configurations are examples. The present invention can be realized by configurations other than the configurations disclosed in the following embodiments, and at least one of various effects based on the basic configuration and derivative effects can be obtained. .

  FIG. 1 is a schematic plan view of a vehicle 10 on which the image processing apparatus of the present embodiment is mounted. The vehicle 10 may be, for example, an automobile (internal combustion engine automobile) driven by an internal combustion engine (engine, not shown) or an automobile (electric automobile) driven by an electric motor (motor, not shown) , A fuel cell vehicle, etc.) or a vehicle (hybrid vehicle) using both of them as a driving source. In addition, the vehicle 10 can mount various transmissions, and can mount various devices (systems, components, etc.) necessary to drive an internal combustion engine or a motor. In addition, the system, number, layout, and the like of devices involved in driving the wheels 12 (front wheels 12F, rear wheels 12R) in the vehicle 10 can be set variously.

  As illustrated in FIG. 1, the vehicle 10 is provided with, for example, four imaging units 14 a to 14 d as the plurality of imaging units 14. The imaging unit 14 is, for example, a digital camera that incorporates an imaging device such as a charge coupled device (CCD) or a CMOS image sensor (CIS). The imaging unit 14 can output moving image data (captured image data) at a predetermined frame rate. The imaging units 14 each have a wide-angle lens or a fish-eye lens, and can capture a range of, for example, 140 ° to 220 ° in the horizontal direction. In addition, the optical axis of the imaging unit 14 may be set obliquely downward. Therefore, the imaging unit 14 may use a road surface on which the vehicle 10 can move or a mark (including an arrow, a dividing line, a line indicating a parking space, a lane separation line, etc.) or an object (for example, walking as an obstacle). The surrounding environment outside the vehicle 10 including the person, the vehicle, etc.) is sequentially photographed, and is output as photographed image data.

  The imaging unit 14 is provided on the outer peripheral portion of the vehicle 10. The imaging unit 14a is provided at, for example, the right end of the vehicle 10, for example, the right side door mirror 10a, and includes a region centered on the right side of the vehicle 10 (for example, a right front to right rear region) Can be taken. The imaging unit 14b is provided at, for example, the left end of the vehicle 10, for example, the left side door mirror 10b, and includes a left side image including an area centered on the left side of the vehicle 10 (for example, an area from left front to left rear) Can be taken. The imaging unit 14c is provided, for example, on the front side of the vehicle 10, that is, on the front end in the vehicle longitudinal direction and at the substantially central end in the vehicle width direction, for example, the front bumper 10c or the front grille. It is possible to capture a forward image including (for example, the front bumper 10c). The imaging unit 14d is provided, for example, on the rear side of the vehicle 10, that is, on the rear end side of the vehicle in the approximate center in the vehicle width direction, for example, the rear bumper 10d, and the rear end of the vehicle 10 (for example, rear bumper 10d) It is possible to image the rear area including. The imaging unit 14 a and the imaging unit 14 b can be referred to as a first imaging unit that images a side area of the vehicle 10. In addition, the imaging unit 14 c and the imaging unit 14 d can be referred to as a second imaging unit that images the traveling direction area of the vehicle 10.

  Arithmetic processing and image processing are executed based on captured image data obtained by a plurality of imaging units 14 to generate an image with a wider viewing angle, or a virtual overhead image (planar image when the vehicle 10 is viewed from above Can be generated. The captured image data (image) captured by each imaging unit 14 may be provided with overlapping regions overlapping each other. For example, the front end of the captured image data captured by the imaging unit 14a and the end on the right side of the captured image data captured by the imaging unit 14c overlap. Then, when connecting (combining) two images, blending processing may be performed in which images are synthesized using a% of the captured image data of the right side image and the captured image data of the front image. By performing the blending process, the right side image and the front image are synthesized so as to gradually change, and it is possible to make the boundary line generated due to the difference in the brightness and the hue inconspicuous. Similarly, by performing blending processing also on the front image and the left side image, the left side image and the rear image, and the rear image and the right side image, it is possible to make the borderline inconspicuous in the entire synthesized peripheral image.

  FIG. 2 is an exemplary block diagram of a configuration of an image processing system 100 including an image processing device mounted on a vehicle 10. As shown in FIG. A display device 16 and an audio output device 18 are provided in the compartment of the vehicle 10. The display device 16 is, for example, a liquid crystal display (LCD), an organic electroluminescent display (OELD), or the like. The audio output device 18 is, for example, a speaker. The display device 16 is covered with a transparent operation input unit 20 such as a touch panel, for example. An occupant (for example, a driver) can visually recognize the image displayed on the display screen of the display device 16 via the operation input unit 20. In addition, the occupant may execute the operation input by operating the operation input unit 20 by touching or pushing or moving the operation input unit 20 with a finger or the like at a position corresponding to the image displayed on the display screen of the display device 16. it can. The display device 16, the audio output device 18, the operation input unit 20, and the like are provided, for example, in the monitor device 22 positioned at the center in the vehicle width direction of the dashboard of the vehicle 10. The monitor device 22 can have an operation input unit (not shown) such as a switch, a dial, a joystick, or a push button. The monitor device 22 can be used, for example, as a navigation system or an audio system.

  Further, as illustrated in FIG. 2, the image processing system 100 (image processing apparatus) includes an ECU 24 (electronic control unit) in addition to the imaging units 14 (14 a to 14 d) and the monitor device 22. In the image processing system 100 (image processing apparatus), the ECU 24 and the monitor device 22 are electrically connected via an in-vehicle network 26 as a telecommunication line. The in-vehicle network 26 is configured, for example, as a controller area network (CAN). The ECU 24 can execute control of various systems by transmitting control signals through the in-vehicle network 26. Further, the ECU 24 can receive, via the in-vehicle network 26, operation signals and the like of the operation input unit 20 and various switches and detection signals and the like of various sensors (not shown).

  The ECU 24 acquires captured image data from the imaging unit 14. The ECU 24 transmits, to the monitor device 22, data related to a surrounding image or sound generated based on a captured image or the like. The ECU 24 has, for example, a CPU 24a (central processing unit), a ROM 24b (read only memory), a RAM 24c (random access memory), a display control unit 24d, a voice control unit 24e, an SSD 24f (solid state drive, flash memory) and the like. ing.

  The CPU 24a reads a program stored (installed) in a non-volatile storage device such as the ROM 24b, and executes arithmetic processing according to the program. The ROM 24 b stores each program and parameters required for the execution of the program. The CPU 24a includes, for example, various modules as shown in FIG. 3, and executes processing related to the image displayed on the display device 16. For example, the CPU 24a executes correction processing, arithmetic processing, image processing, and the like on the captured image data captured by the imaging unit 14 to generate a peripheral image (for example, an overhead image) in which a plurality of images are connected. Details of the CPU 24a will be described later.

  The RAM 24 c temporarily stores various data used in the calculation in the CPU 24 a. The display control unit 24 d mainly performs data conversion of an image for display to be displayed on the display device 16 among the arithmetic processing in the ECU 24. Further, the voice control unit 24 e mainly performs processing of voice data output from the voice output device 18 among the calculation processing in the ECU 24. The SSD 24 f is a rewritable non-volatile storage unit, and can store data even when the power supply of the ECU 24 is turned off. The CPU 24a, the ROM 24b, the RAM 24c, and the like can be integrated in the same package. Further, the ECU 24 may be configured to use another logical operation processor such as a DSP (digital signal processor) or a logic circuit instead of the CPU 24a. In addition, a hard disk drive (HDD) may be provided instead of the SSD 24f, and the SSD 24f and the HDD may be provided separately from the ECU 24.

  In the present embodiment, the ECU 24 manages image generation processing of the vehicle 10 by cooperation of hardware and software (control program). The ECU 24 performs image processing, for example, viewpoint conversion processing, and displays the captured image data (image) captured by the imaging unit 14 on the display device 16, the surrounding objects and the like with respect to the vehicle 10 (own vehicle) After the recognition position of the imaging unit 14 in image processing is corrected so as to be displayed at the correct position (the position corresponding to the actual position), the image processing is performed to output the display image. For example, in the case where a plurality of captured image data (images) captured by the respective imaging units 14 (14a to 14d) are connected to display a surrounding image (for example, a bird's-eye view image) The captured image data is corrected so that an object or the like is displayed at the correct position, and then connected to generate a peripheral image. In the present embodiment, the correction of the recognition position on the image processing of the imaging unit 14 may be performed on any one imaging unit 14 or may be performed on a plurality of imaging units 14. Good. In the following description, when the installation position and orientation of the imaging unit 14a that images, for example, the right side in the imaging unit 14 of the vehicle 10 deviates from the designed position and orientation, recognition of the imaging unit 14a in image processing An example in which the correction of the entire image to be displayed is performed by correcting the position will be described.

  The CPU 24a included in the ECU 24 includes various modules for executing the correction process as described above. The various modules are realized by the CPU 24a reading a program installed and stored in a storage device such as the ROM 24b and executing the program. For example, as illustrated in FIG. 3, the CPU 24a includes modules such as an acquisition unit 28, a projection processing unit 30, a detection unit 32, a correction unit 34, and an image processing unit 36. The acquisition unit 28 includes an image acquisition unit 28a, a first reference position acquisition unit 28b, a parameter acquisition unit 28c, and the like. The correction unit 34 includes a displacement acquisition unit 34a, a recognition position correction unit 34b, a parameter correction unit 34c, and the like. including.

  The acquisition unit 28 acquires various pieces of information necessary to execute the process of correcting the recognition position on the image processing of the imaging unit 14.

  The image acquisition unit 28 a sequentially acquires captured image data captured by each of the imaging units 14 (14 a to 14 d) via the display control unit 24 d.

  The first reference position acquisition unit 28 b acquires a first reference position set on part of the outer surface of the vehicle 10. The first reference position set on part of the outer surface of the vehicle 10 is stored in advance in, for example, the ROM 24 b as a position on a road surface coordinate system (X, Y, Z) described later. FIG. 4 is a perspective view showing an example of the first reference position P. As shown in FIG. FIG. 5 is a schematic view showing an example of an image captured by the imaging unit 14a (right-side imaging unit) showing an example of the first reference position P. As shown in FIG. The first reference position P can be, for example, a grounding point on the outer surface side of the right front wheel 12F as shown in FIGS. 4 and 5. The first reference position P is not limited to the outer peripheral surface of the front wheel 12F, and may be any position as long as it is the outer surface of the vehicle 10 included in the imaging range of the imaging unit 14 (for example, imaging unit 14a) to be corrected . In another embodiment, the first reference position P may be set to a part of the body of the vehicle 10, for example, a lower end corner portion of a door or the like, and similar effects can be obtained. As described above, when imaging by the imaging unit 14a, the first reference position P is a contact position of the front wheel 12F which is relatively close to the imaging unit 14a and can be easily recognized even when the image is recognized. This can improve the recognition accuracy.

  The first reference position P may be determined based on the reference position O set in the vehicle 10. By setting the reference position O, the position of the first reference position P in real space and the position of the first reference position P on the image can be easily recognized, and a coordinate system is used. Processing becomes easy when performing the same calculation. The reference position O can be, for example, a central position in the longitudinal direction (vehicle width direction) of the rear wheel shaft 12a that supports the left and right rear wheels 12R of the vehicle 10. In another embodiment, the reference position O may be, for example, a position on the vehicle center CL extending in the front-rear direction of the vehicle 10. In this case, for example, the intersection of perpendiculars drawn from the first reference position P to the vehicle center CL may be used as the reference position O. In addition, since the actual distance from the reference position O to the first reference position P is known, the separation distance (the number of dots) on the image when the reference position O and the first reference position P are projected on the image By detecting V.sub.x, it becomes easy to associate the displacement of one dot on the image with the displacement of the distance in the real space. For example, in the case where there is a shift of 10 dots on the image, it is possible to perform matching in the real space, such as a shift of 10 cm. The distance corresponding to one dot on the image may be stored in advance in the ROM 24b or the like to associate the deviation of one dot on the image with the deviation of the distance in the physical space.

この場合、式１における係数部分が投影変換パラメータとなる。ＲＯＭ２４ｂまたはＳＳＤ２４ｆには、現在の撮像部１４ａの画像処理上の認識位置（例えば原点）に対応する極座標（ρ，θ，φ）が保存されてもよいし、式１にρ，θ，φを代入した係数とした投影変換パラメータとして記憶しておいてもよい。 The parameter acquisition unit 28c acquires, from a storage unit such as the ROM 24b or the SSD 24f, for example, a projection conversion parameter used when performing projection conversion on the image captured by the imaging unit 14. For example, as shown in FIG. 6, let (X, Y, Z) be the coordinate system on the road surface, and (X _0, Y ₀ , Z ₀ ) be the coordinate system (camera coordinate system) on the camera image of the imaging unit 14a. When the distance between the coordinates on the road surface and the coordinate system on the camera image and the angle are represented by polar coordinates (ρ, θ, φ), the transformation of the coordinate system can be represented by the following Equation 1. Note that ρ is the distance from the origin of the coordinate system on the road surface (for example, corresponding to the reference position O in FIG. 4) to the origin T ₀ of the camera coordinate system. Further, theta is the angle between the origin and the camera coordinate system lines and X-axis origin T ₀ connecting the position of the point T that is projected on the XY plane of the coordinate system on the road surface coordinate system on the road surface. Further, φ is an angle formed by a straight line connecting the origin of the coordinate system on the road surface and the origin T ₀ of the camera coordinate system and the Z axis.

In this case, the coefficient part in Equation 1 is a projection conversion parameter. Polar coordinates (ρ, θ, φ) corresponding to the recognition position (for example, the origin) of the current image processing of the imaging unit 14 a may be stored in the ROM 24 b or the SSD 24 f. It may be stored as a projection conversion parameter as a substituted coefficient.

The projection processing unit 30 applies each of the projection parameters acquired by the parameter acquisition unit 28 c to each pixel of the captured image data captured by the imaging unit 14 (for example, the imaging unit 14 a) (camera coordinate system (X _0, Y ₀ , Pixels on Z ₀ ) can be projected sequentially to the coordinate system (X, Y, Z) on the road surface. For example, a pixel located at the origin T ₀ of the camera coordinate system (X _0, Y ₀ , Z ₀ ) can be projected to a point T on the road coordinate system (X, Y, Z). In this case, the imaging unit 14 is fixed to the vehicle 10. For example, since the imaging unit 14a is fixed to the lower surface of the right side door mirror 10a, the recognition position on the image processing of the imaging unit 14a can be set to a fixed position. Therefore, when the installation position and mounting posture of the imaging unit 14a are maintained at the designed position and posture, for example, the pixels corresponding to the ground contact position of the front wheel 12F in the captured image data captured are on the road surface. The projection conversion is performed on the coordinate system (X, Y, Z) to a position coincident with the first reference position P acquired by the first reference position acquisition unit 28 b.

On the other hand, the installation position and mounting posture of the imaging unit 14a are shifted with respect to the designed position and posture due to some cause, for example, vibration given by the traveling of the vehicle 10 or contact with other objects. There are times when In such a case, for example, as shown in FIG. 7, the image corresponding to the contact position of the actual front wheel 12F (the position substantially the same as the first reference position P) is the camera coordinate system (X _0, Y ₀ , Z ₀₎ from the shifted camera coordinate system on _{(X 1, Y 1, Z} 1), to be imaged. In this case, in image processing, the recognition position of the imaging unit 14a remains the recognition position on the camera coordinate system (X _0, Y ₀ , Z ₀ ), so that the position or posture of the imaging unit 14a is shifted. The picked-up image data picked up by the above are projection-transformed using Formula 1 to which a projection transformation parameter corresponding to polar coordinates (.rho., .Theta., .Phi.) Is applied. As a result, the pixel corresponding to the contact position of the front wheel 12F is projected on the coordinate system (X, Y, Z) on the road surface while being moved by an amount corresponding to the displacement of the installation position or posture of the imaging unit 14a. become. That is, it is projected to the second reference position Q shifted from the first reference position P. FIG. 8 is an example in which an overhead image (an overhead image of the right side of the vehicle 10) is generated based on captured image data captured in a state in which the installation position and posture of the imaging unit 14a are shifted. Is a schematic diagram for explaining an example in which the position of s shifts from the actual position of the white line L ₀ . When the driver recognizes the surrounding condition of the vehicle 10 based on such a bird's-eye view image, the position of the white line L present on the side of the vehicle 10 corresponds to, for example, the vehicle icon 40 displayed at the image center position. For example, it is displayed far from the actual. That is, in some cases giving the driver a different sense of distance from the actual distance to the white line L _0. As a result, there is a possibility that the vehicle 10 can not be moved in the width direction or the like according to the image, or the distance to the obstacle may be erroneously recognized and touched.

Therefore, the detection unit 32 of the image processing system 100 according to the present embodiment is configured to detect the first reference position P on the captured image data acquired by the imaging unit 14 (for example, the imaging unit 14a) whose installation position or posture has shifted. A second reference position Q corresponding to is detected. For example, on the camera coordinate system (X _1, Y ₁ , Z ₁ ) of the imaging unit 14a in which the installation position or the orientation is shifted, a projection conversion parameter (initial value) determined in a state without displacement is set to (X _0, Y _{0, Z 0)} is applied to equation 1 read as the _{(X 1, Y 1, Z} 1). Then, the contact position of the front wheel 12F imaged by the imaging unit 14a is projected on the coordinate system (X, Y, Z) on the road surface. That is, the detection unit 32 corresponds to the first reference position P acquired by the first reference position acquisition unit 28b (a position at which the front wheel 12F will be in the ground position if there is no shift in the installation position or posture of the imaging unit 14a). A second reference position Q, which is a position, can be detected.

  Then, the correction unit 34 detects the imaging unit 14 (for example, the imaging unit 14a) based on the deviation between the first reference position P and the second reference position Q detected by the detection unit 32, that is, the displacement amount (difference, difference distance). Correct the recognition position on the image processing of The correction of the recognition position can be performed, for example, by correcting a projection conversion parameter at the time of projection conversion.

  The displacement acquisition unit 34a detects, for example, the displacement (displacement) between the first reference position P and the second reference position Q projected on the coordinate system (X, Y, Z) on the road surface. The ROM 24b can display, for example, 10 dots when the object is imaged by the imaging unit 14a and projected on a road coordinate system (X, Y, Z) using the image processing system 100 according to the present embodiment. The displacement reference value data indicating the size (length, distance, etc.) is stored together with the data of the first reference position P. The displacement acquisition unit 34a can acquire the displacement amounts of the first reference position P and the second reference position Q based on the displacement reference value data. The displacement reference value data may be determined by the relationship between the actual distance between the reference position O and the first reference position P and the distance on the screen (the number of dots), as described in FIG.

Recognizing the position correcting unit 34b, based on the amount of displacement displacement acquiring unit 34a acquires, for example, as shown in FIG. 7, the camera coordinate system is used by the imaging unit 14a to the installation position and posture is displaced (X _1, Y Correct ₁ , Z ₁ ). For example, it is corrected to the recognition position of the imaging unit 14a in image processing so that the second reference position Q coincides with the first reference position P. That is, the correction camera coordinate system _{(X 1, Y 1, Z} 1) in the camera coordinate system _{(X 2, Y 2, Z} 2). At this time, when the displacement amount between the first reference position P and the second reference position Q is 10 dots, the recognition of the imaging unit 14a is performed such that the second reference position Q coincides with the first reference position P. The recognition position of the imaging unit 14a can be corrected by moving the position by an amount corresponding to 10 dots.

The parameter correction unit 34 c sets the coordinate system on the road surface as (X, Y, Z), and the camera coordinate system obtained as a result of correcting the recognition position as (X _2, Y ₂ , Z ₂ ). coordinate system distance on the camera image, the polar coordinate after correction illustrating the angle _{(ρ 2, θ 2, φ} 2) is calculated. Then, when polar coordinates ( _{2 2} , θ ₂ , φ ₂ ) are applied to equation 1 in which (X _0, Y ₀ , Z ₀ ) is replaced by (X _2, Y ₂ , Z ₂ ), the coefficient part is corrected It becomes a projection conversion parameter of That is, for example, the projection conversion parameter of the SSD 24 f is updated. In this case, polar coordinates (ρ ₂ , θ ₂ , φ ₂ ) corresponding to the recognition position on the image processing of the imaging unit 14 a after correction may be stored in the SSD 24 f or (X _0, Y ₀ , Z ₀ ) The corrected projection conversion parameters may be stored in the SSD 24 f using the coefficients obtained by substituting ρ ₂ , θ ₂ , and φ ₂ into Equation 1 in which X is replaced by (X _2, Y ₂ , Z ₂ ).

  Then, the projection processing unit 30 performs projection conversion on the captured image data captured by the imaging unit 14a using the corrected projection conversion parameter, whereby the first reference position P and the second reference position Q coincide with each other. A projected image can be obtained.

The image processing unit 36 is, for example, an image based on captured image data captured by the imaging unit 14a obtained by correcting the recognition position and a captured image captured by the other imaging unit 14 (for example, the imaging unit 14c for the front) A process is performed to connect the data based image. In that case, the captured image data of the imaging unit 14c is corrected so that the continuity between the corrected image content of the imaging unit 14a and the image content of the imaging unit 14c is maintained. For example, as illustrated in FIG. 9, the image processing unit 36 performs viewpoint conversion of captured image data captured by each of the image capturing units 14 (14 a to 14 d), and connects them together to surround the vehicle 10 (own vehicle icon 40). Generate a bird's-eye view image. In this case, using the region R ₁ by the imaging unit 14a is corrected using the projective transformation parameter corrected by the correction unit 34 the captured image data obtained by imaging, without correction of the projective transformation parameters, the projection transformation parameters initially set region and projection transformation Te _{R 2} (captured image data of the imaging unit 14c), (captured image data of the image pickup unit 14b) regions _{R 3,} to synthesize the regions _{R 4} (captured image data of the image pickup unit 14d).

As described above, white lines L ₁ to the image by the captured image data of the projected transformed captured portion 14a projected transformation parameter is corrected is reflected to a region R ₁ to be drawn, the position where the white line L ₀ of the real exists Exists corresponding to On the other hand, the white line L _2R on the right side image by capturing the image data of the imaging unit 14c which projected transformation parameters are projection transformation without being corrected is reflected in a region R ₂ to be drawn is drawn offset from the actual position it may have, in some cases continuity white lines L ₁ can not be maintained. In this case, the white line L _{2 R} can be made to coincide with the corrected white line L ₁ by performing rotation processing, parallel movement, enlargement / reduction processing, and the like. That is, the white line _{L2R is} also processed so as to be drawn at the correct position. In this case, the white line L _2L included in the region R _2, due to the presence in the same region taken by the imaging unit 14c, the drawing position of the white line L _2L affected correcting the white line L _1, in the correct position It will be drawn. Drawing Similarly, rotation processing or translation criteria to the region R ₃ white lines L _2L of drawn in the correct position based on the white line L ₁ region R _2, the exact position by drawing is subjected to scaling processing and the like Be done. The exact rotation process and translate the white line L ₃ of the drawing area R ₃ in the region R ₄ to reference the position, can be drawn to the region R ₄ also exact location by performing scaling processing or the like. As a result, white lines _{L 4R} is consistent with the white line _{L 3,} white lines _{L 4L} is consistent with the white line _{L 1.}

  As described above, correction of the projection conversion parameter is performed on the imaging unit 14 at one place with respect to the captured image data captured by the imaging unit 14 to obtain an accurate position centered on the vehicle 10 (the vehicle icon 40). The bird's-eye view image showing the relationship can be generated by easy processing. In the above example, correction of the recognition position on the image processing (correction of the projection conversion parameter) is performed on any one of the imaging units 14, and the images of the other imaging units 14 are adjusted to the corrected image. An example of processing is shown. In another embodiment, the correction of the recognition position on the image processing (correction of the projection conversion parameter) may be performed on the plurality of imaging units 14. In this case, the drawing accuracy of the overhead image to be synthesized can be improved.

  FIG. 10 is a flowchart for explaining an example of image processing in the case of generating a bird's-eye view image by the image processing system 100 (image processing apparatus) described above. The installation position and posture of the imaging unit 14 do not shift frequently, for example, when traveling on a rough road where the vibration of the vehicle 10 becomes large for a long time, or the installation position of the imaging unit 14 (for example, the door mirror 10a or When the door mirror 10b) comes in contact with an external object, the door mirror 10b is generated only when it is repaired or replaced, or changes over time, etc., and it basically does not deviate. Therefore, in the present embodiment, the correction of the recognition position on the image processing of the imaging unit 14 by the correction of the projection conversion parameter can be performed at relatively long intervals, for example, periodically or irregularly.

  First, the CPU 24a confirms whether display (creation) of the overhead image is requested. A navigation screen or an audio screen is displayed on the display device 16 as a normal screen, but when the driver requests the display of the bird's-eye view image by operating the operation input unit 20, for example, the CPU 24a displays the bird's-eye view image The display mode is entered (Yes in S100). When the display request for the overhead image is made, the image acquisition unit 28a acquires captured image data (peripheral image) captured by each of the imaging units 14 (14a to 14d) via the display control unit 24d (S102). At this time, if it is time to re-adjust (correct) the recognition position of the imaging unit 14 in image processing (Yes in S104), the first reference position acquisition unit 28b may, for example, make a first reference stored in the ROM 24b. Information on the position P is acquired (S106). Note that the readjustment (correction) timing of the recognition position on the image processing is, for example, when the ignition switch is turned on or when a predetermined period (for example, one month) or more has passed since the previous readjustment, the driver It is when you request adjustment.

  When it is time to re-adjust the recognition position on the image processing of the imaging unit 14, the projection processing unit 30 acquires a currently effective projection conversion parameter from the ROM 24b or the SSD 24f via the parameter acquisition unit 28c, An image is drawn (S108). For example, when the imaged image data to be processed is imaged by the imaging unit 14a, the bird's-eye view image on the right side is generated. At this time, if the installation position or posture of the imaging unit 14 (for example, the imaging unit 14a) to be corrected is deviated, as shown in FIG. 7, the first reference in the coordinate system (X, Y, Z) on the road surface The second reference position Q is projected (drawn) to a position different from the position P. The detection unit 32 detects the second reference position Q (the contact position of the front wheel 12F) displayed on the temporary overhead view image (coordinate system (X, Y, Z) on the road surface) (S110). Note that the first reference position P and the second reference position Q are drawn so as to substantially match when the installation position or posture of the imaging unit 14 (for example, the imaging unit 14a) to be corrected is not shifted. .

Subsequently, the displacement acquisition unit 34a calculates (acquires) displacement amounts of the first reference position P and the second reference position Q on the coordinate system (X, Y, Z) on the road surface (S112), The recognition position correction unit 34b determines the recognition position of the imaging unit 14a in image processing so that the first reference position P and the second reference position Q coincide with each other based on the displacement amount acquired by the displacement acquisition unit 34a. for example the camera coordinate system as shown in FIG. _{7 (X 2, Y 2,} Z 2) to be corrected (S114). Then, the parameter correction unit 34 c calculates polar coordinates (ρ ₂ , θ ₂ , φ ₂ ) at the corrected recognition position (camera coordinate system (X _2, Y ₂ , Z ₂ )), and (X _0, Y ₀₎ , Z ₀ ) are replaced by (X _2, Y ₂ , Z ₂ ), and the projection conversion parameter for the imaging unit 14a is changed (updated) using the equation 1 (S116). The changed projection conversion parameter is stored, for example, in the SSD 24 f and is used for projection conversion of captured image data captured by the imaging unit 14 a until the projection conversion parameter is updated next time.

  Subsequently, the projection processing unit 30 draws a side bird's eye view (S118). In this case, for example, the entire right side view image is drawn by applying the latest projection conversion parameter corrected by the parameter correction unit 34c to each pixel of the captured image data captured by the imaging unit 14a and performing projection conversion on the entire image. . On the other hand, the left-side view image is subjected to projection conversion by applying the projection conversion parameter determined when the imaging unit 14b stored in the ROM 24b or the like is installed to each pixel of the captured image data captured by the imaging unit 14b. It is drawn by. In addition, the projection processing unit 30 draws the overhead images before and after (S120). In this case, the front bird's-eye view image applies a projection conversion parameter determined when the imaging unit 14c stored in the ROM 24b or the like is installed to each pixel of captured image data captured by the imaging unit 14c. It is drawn by doing. Similarly, for the overhead bird's-eye view image, a projection conversion parameter determined when the imaging unit 14d stored in the ROM 24b or the like is installed is applied to each pixel of captured image data captured by the imaging unit 14d It is drawn by doing.

Then, the image processing unit 36 extracts the common points having continuity appearing in the drawn overhead images (S122). For example, as shown in FIG. 9, the overhead region R ₂ based on the captured image data captured by a white line L ₁ and the imaging unit 14c that is reflected in the overhead image region R ₁ based on the captured image data captured by the imaging unit 14a It extracts the white lines L _2R contained in the image. Similarly, the white line L 2 _L in the overhead image of the region R ₂ based on the captured image data captured by the imaging unit 14 c and the white line in the overhead image of the region R ₃ based on the captured image data captured by the imaging unit 14 b L ₃ , white lines L _{4 L} and L _{4 R and the} like appearing in the overhead image of the area R ₄ based on the captured image data captured by the imaging unit 14 d are extracted.

Then, the image processing unit 36 synthesizes an overhead image based on the captured image data captured by each imaging unit 14 and causes the display device 16 to display the combined image (S124). Specifically, first, the image processing unit 36 performs rotation processing or parallel movement of the bird's-eye view image displayed in the region R ₂ with respect to the white line L ₁ so that the white line L ₁ and the white line L _{2 R} are continuous. Perform processing, enlargement / reduction processing, etc. to adjust the drawing position. Then the image processing unit 36, as the white line L _2L and the white line L ₃ are continuous, rotation processing and translation processing of the left side of the overhead image displayed on the region R ₃ relative to the white line L _2L, scaling processing Apply and adjust the drawing position. Similarly, the image processing unit 36, so that the white line L ₃ and the white lines L _4L are continuous, rotation processing and translation processing of the rear of the overhead image displayed on the region R ₄ relative to the white line L _3, scaling processing, etc. To adjust the drawing position. As a result, as shown in FIG. 9, the white line L ₁ as a reference drawn in the correct position relative to the vehicle icon 40, the alignment of each of the overhead image is performed, precise rendering is achieved. In other words, it is possible to display on the display device 16 an accurate overhead image in which the sense of distance to the display object around the vehicle icon 40 (vehicle 10) corresponds to the sense of distance to the real object.

  When the bird's-eye view image is displayed on the display device 16, the CPU 24a confirms whether a request for ending the display of the bird's-eye view image is made (S126). For example, when the operation input unit 20 is operated and the display end request for the overhead image is issued (Yes in S126), the CPU 24a displays the display content of the display device 16 from the display screen of the overhead image currently displayed. It switches to a certain navigation screen or audio screen (S128), and this flow is once ended.

  In S126, when the overhead image display end request is not made (No in S126), the CPU 24a returns to the process of S102 and performs acquisition of the peripheral image to be used in the next processing cycle via the image acquisition unit 28a. Execute the following process repeatedly. In this case, if it is not the timing of readjustment (correction) timing of the recognition position on the image processing in S104 (No in S104), the processing of S106 to S116 is skipped, and for the imaging unit 14a currently updated and stored in the SSD 24f. The overhead view image is drawn in S118 and subsequent steps using the latest projection conversion parameters and the projection conversion parameters initially set for the imaging unit 14b, the imaging unit 14c, and the imaging unit 14d stored in the ROM 24b. In other words, excessive execution of the projection conversion parameter updating process is suppressed, which contributes to the reduction of the processing load of the CPU 24a. In S100, when creation of a bird's-eye view image (display on the display device 16) is not requested (No in S100), processing of S102 to S126 is skipped and display of a normal screen (navigation screen or audio screen) is continued. Do.

  Note that the flowchart shown in FIG. 10 is an example, and if substantially the same processing can be executed, it is possible to appropriately increase, decrease, replace, and change the processing steps, and to obtain the same effect. In the above-described example, correction of the projection conversion parameter is performed on the captured image data captured by the imaging unit 14a by correcting the recognition position on the image processing of the imaging unit 14a. In another embodiment, the correction of the projection conversion parameter may be performed together with the captured image data captured by similarly correcting the recognition position on the image processing of the other imaging unit 14. The imaging unit 14 to be corrected may be changed each time. When changing the target imaging unit 14 every time the correction processing is performed, the recognition position of the imaging unit 14a is first corrected at the readjustment (correction) timing of S104, and the recognition position of the imaging unit 14b is performed in the next processing cycle. Make corrections for Then, the recognition position of the imaging unit 14c is corrected in the next processing cycle, and the recognition position of the imaging unit 14d is corrected in the next processing cycle. That is, when all the corrections of the recognition position of each imaging unit 14 are completed, it is considered that the processing at the readjustment (correction) timing is completed. In this case, since the correction of the recognition position of each imaging unit 14 is performed, it is possible to generate a bird's eye image more accurately.

  In addition, in the case of the imaging unit 14c that images the area in front, for example, a mark or the like may be formed or attached on the front bumper 10c included in the imaging area of the imaging unit 14c to set the first reference position P. . Similarly, in the case of the imaging unit 14d that images the rear area, a first reference position P may be set by, for example, forming or attaching a mark or the like on the rear bumper 10d included in the imaging area of the imaging unit 14d. .

  Further, in order to improve the recognizability of the second reference position Q corresponding to the first reference position P, for example, a mark for detection, such as a seal or an LED, may be provided on a part of the vehicle 10. By using a light emitter such as an LED as the mark, the detection accuracy of the position of the second reference position Q can be improved, and the correction accuracy of the recognition position on the image processing of the imaging unit 14 can be improved. In addition, even when the surroundings of the vehicle 10 are dark (for example, at night), the correction process can be performed with high accuracy.

  The program for image processing executed by the CPU 24a of the present embodiment is a file of an installable format or an executable format, and is a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), etc. The present invention may be configured to be provided by being recorded on a computer readable recording medium.

  Furthermore, the image processing program may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Further, the image processing program executed in the present embodiment may be provided or distributed via a network such as the Internet.

  While the embodiments and variations of the present invention have been described, these embodiments and variations are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and the gist of the invention, and are included in the invention described in the claims and the equivalent scope thereof.

  DESCRIPTION OF SYMBOLS 10 ... Vehicle, 12 ... Wheel, 12F ... Front wheel, 14, 14a, 14b, 14c, 14d ... Imaging part, 16 ... Display apparatus, 20 ... Operation input part, 24 ... ECU, 24a ... CPU, 24b ... ROM, 24d ... Display control unit, 24f: SSD, 28: Acquisition unit, 28a: Image acquisition unit, 28b: First reference position acquisition unit, 28c: Parameter acquisition unit, 30: Projection processing unit, 32: Detection unit, 34: Correction unit , 34a: displacement acquisition unit, 34b: recognition position correction unit, 34c: parameter correction unit, 36: image processing unit, 40: vehicle icon, 100: image processing system, L: white line, P: first reference position, Q: Second reference position.

Claims (5)

An acquisition unit configured to acquire a first reference position which is set on a part of the outer surface of the vehicle and stored in advance;
A detection unit that detects a second reference position corresponding to the first reference position on captured image data acquired by an imaging unit provided in the vehicle;
A correction unit configured to correct a recognition position of the imaging unit in image processing such that the second reference position matches the first reference position;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the correction unit corrects a recognition position of the imaging unit based on a displacement amount between the first reference position and the second reference position.   The image processing apparatus according to claim 1, wherein the correction unit corrects the projection conversion parameter based on the corrected recognition position of the imaging unit.   The image processing apparatus according to any one of claims 1 to 3, wherein the first reference position is a position at which a front wheel of the vehicle is in contact with the ground. The imaging unit includes a first imaging unit that images a side area of the vehicle, and a second imaging unit that images a traveling direction area of the vehicle.
The correction unit corrects a recognition position of one of the first imaging unit and the second imaging unit.
When the captured image data of the other imaging unit is connected to the image based on the captured image data of the one imaging unit whose recognition position has been corrected, the other of the other is maintained so as to maintain the continuity of the image content. The image processing apparatus according to any one of claims 1 to 4, further comprising an image processing unit configured to correct captured image data of the imaging unit.

Images