JP5959264B2 - Image processing apparatus and method, and computer program - Google Patents

Description

  The present invention relates to an image processing apparatus and method, and more particularly to a technique for generating an image from an arbitrary viewpoint by synthesizing images obtained by photographing with a plurality of cameras. Such image processing can be used, for example, as an aid for safety confirmation when driving a vehicle. The present invention also relates to a computer program for causing a computer to execute the above image processing method.

  An image displayed by processing the image from the in-vehicle camera to make it easier for the driver to look down on the white line on the ground from the top of the vehicle or to check objects that are difficult to see behind or on the side of the vehicle There is a processing device.

  For example, in the image processing apparatus described in Patent Document 1, an image obtained by imaging with a plurality of cameras is projected onto a spatial model, and the projected image is converted into an image viewed from an arbitrary viewpoint, and a display unit This makes it possible for the driver to more easily check the situation around the vehicle. As a spatial model for projecting an image, a three-dimensional surface shape such as a cylindrical shape or a saddle shape consisting of a curved surface and a plane is used, and this makes it possible to display peripheral images that could not be displayed by projection onto a plane corresponding to a conventional road surface. Can also be displayed.

International Publication No. 2000/64175 (page 10, FIG. 1)

  However, in the above-described conventional image processing apparatus, since a space model for projecting a camera image uses a predetermined shape such as a cylindrical shape having a curved surface and a flat surface, or a bowl shape, the wall portion (rises from the bottom portion). There is a problem that it is difficult to grasp the distance to the object being picked up even if the image portion projected onto the image is projected.

  The present invention has been made to solve the above-described problems of the prior art, and an object of the present invention is to enable generation of an image in which the distance to an object around the vehicle can be easily grasped.

An imaging device according to a first aspect of the present invention includes:
A plurality of cameras for imaging objects in the vicinity of the host vehicle;
A distance information generation unit that generates distance information to the object imaged by the camera;
A model shape determining unit that determines the shape of the space model having a three-dimensional surface corresponding to a distance from the own vehicle based on the distance information generated by the distance information generating unit to the object,
By projecting a plurality of images obtained by imaging with the plurality of cameras onto the spatial model having the shape determined by the model shape determining unit, and further converting the projected image into a screen coordinate system, a screen coordinate system A projection conversion unit for generating an image of
An image composition unit that extracts a part or all of the image of the screen coordinate system generated by the projection conversion unit to generate a composite image;
An information transmission unit that causes the image display unit to display the combined image generated by the image combining unit ;
The model shape determination unit reflects only the distance information of the object in the traveling direction of the host vehicle and closest to the host vehicle, or the portion thereof, based on the distance information generated by the distance information generation unit. It characterized that you determine the shape of the space model Te.
The imaging device according to the second aspect of the present invention is:
A plurality of cameras for imaging objects in the vicinity of the host vehicle;
A distance information generation unit that generates distance information to the object imaged by the camera;
A model shape determining unit that determines the shape of a spatial model having a three-dimensional surface according to the distance from the host vehicle to the object based on the distance information generated by the distance information generating unit;
By projecting a plurality of images obtained by imaging with the plurality of cameras onto the spatial model having the shape determined by the model shape determining unit, and further converting the projected image into a screen coordinate system, a screen coordinate system A projection conversion unit for generating an image of
An image composition unit that extracts a part or all of the image of the screen coordinate system generated by the projection conversion unit to generate a composite image;
An information transmission unit for causing the image display unit to display the combined image generated by the image combining unit;
With
The model shape determination unit is based on the distance information generated by the distance information generation unit, and is in the traveling direction of the host vehicle and closest to the host vehicle or a portion thereof, and the closest on the image The shape of the space model is determined by reflecting only the distance information about the object or a part located around the part.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to produce | generate the image which is easy to grasp | ascertain the distance to the target object of a vehicle periphery.

1 is a block diagram schematically showing a configuration of an image processing apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram schematically showing the configuration of the camera in FIG. 1. It is a vertical sectional view of a conventional saddle-shaped space model. It is a top view of the conventional saddle type space model. FIG. 5 is a horizontal cross-sectional view of an object that is a part of the spatial model of FIG. FIG. 5 is a horizontal sectional view of an object forming a part of the space model of FIG. 4 and an object located at a position closer thereto. FIG. 5 is a vertical cross-sectional view of a region and an object forming part of the space model of FIG. 4. It is a figure which shows an example of the image of a texture coordinate system. It is a figure which shows an example of the image of a screen coordinate system. It is a figure which shows the relationship between a world coordinate system, an imaging viewpoint coordinate system, and a virtual viewpoint coordinate system. It is a figure which shows the conversion from an imaging viewpoint coordinate system to a world coordinate system, and the conversion from a world coordinate system to a virtual viewpoint coordinate system. It is a horizontal sectional view which shows the area | region which makes a part of example of the space model formed by this invention. (A)-(c) is a figure which shows the vertical cross section of a different example of the space model formed according to the distance to the other vehicle ahead of the own vehicle, and this other vehicle. It is a figure which shows an example of the synthesized image formed using an example of the space model by this invention. It is a figure which shows an example of the synthesized image formed using the conventional space model. It is a figure which shows an example of the synthesized image formed using the other example of the space model by this invention. It is a perspective view which shows the other example of the space model by this invention. It is a figure which shows an example of the synthesized image formed using the space model of FIG. It is a figure which shows the method of measuring the distance to a target object using a stereo camera.

  FIG. 1 schematically shows a configuration of an image processing apparatus 100 according to an embodiment of the present invention. The illustrated image processing apparatus 100 is mounted on a vehicle, and includes first to fourth cameras 1a to 1d (represented simply by “1” when there is no need to distinguish each), Thru | or 4th sensing part 2a-2d (When it is not necessary to distinguish each, it represents only with a code | symbol "2"), the distance information calculation part 3, the model shape determination part 4, the camera control part 5, A projection conversion unit 6, an image composition unit 7, and an information transmission unit 8 are provided. An image display unit 80, a speaker 82, a host vehicle state detection unit 90, and an operation unit 92 are connected to the image processing apparatus 100.

The cameras 1a to 1d take images around the host vehicle.
In the present embodiment, as shown in FIG. 4, the first camera 1a is the front of the host vehicle, the second camera 1b is the rear of the host vehicle, the third camera 1c is the left side of the host vehicle, and the fourth camera. 1d is installed on the right side of the host vehicle, and images the front, rear, left side, and right side of the host vehicle, respectively. Image information (first image information) of the image captured by the camera 1a, information of the second image (second image information) captured by the second camera 1b, and image captured by the third camera 1c Information on the third image (third image information) and information on the fourth image captured by the fourth camera 1d (fourth image information) are transmitted via the data bus 9 to the camera control unit. 5 and the projection conversion unit 6.

FIG. 2 is a block diagram schematically showing the configuration of the camera 1. The camera 1 includes an optical system 11, an image sensor 12, an AD converter 13, and a sensor drive circuit 14.
The optical system 11 is composed of, for example, a wide-angle lens, for example, a fish-eye lens having an angle of view of 180 degrees or more, and the lens may be a single lens or a combination of a plurality of lenses. .
The image sensor 12 converts light obtained from the optical system 11 into an analog electrical signal. The imaging element 12 may be, for example, either a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled Device) type. The AD converter 13 converts an analog electric signal obtained from the image sensor 12 into digital image information. The sensor drive circuit 14 controls the operation of the image sensor 12.

  Returning to FIG. 1, the sensing unit 2 is configured by a sensing device that detects an object in the vicinity of the host vehicle and acquires information about the object. The sensing unit 2 includes, for example, a sonar sensor that measures a distance by emitting an ultrasonic wave in each direction and measuring a time until a reflected wave from the object is received. By changing (scanning) the direction in which the ultrasonic waves are emitted by a predetermined angle, information for obtaining the distance to the object in each direction within the predetermined range (scanning range) with the own vehicle as the center is obtained. Can do.

  The first sensing unit 2a is used to measure the distance to the object imaged by the first camera 1a, and similarly, the second sensing unit 2b is up to the object imaged by the second camera 1b. The third sensing unit 2c measures the distance to the object imaged by the third camera 1c, and the fourth sensing unit 2d measures the distance to the object imaged by the fourth camera 1d. Used for.

  The distance information calculation unit 3 calculates the distance from each sensing unit to the object based on the reception of the ultrasonic wave radiation and the reflected wave by the first to fourth sensing units 2a to 2d. In order to calculate the distance, each of the first to fourth sensing units 2a to 2d controls the timing of emitting an ultrasonic wave at each angle and measures the time until the reflected wave returns, and further measures the distance. Calculate the distance from the time to the object.

Further, the distance information calculation unit 3 further determines the target (the target object) from a specific position of the host vehicle based on the distances measured as described above (the distances from the sensing units 2a to 2d to the target, respectively). The distance to each part) is calculated.
By calculating the distance to the object, distance information (depth information) can be obtained for the entire object imaged by the camera 1. The distance calculated in this way is used to know the three-dimensional shape of the object seen from the virtual viewpoint.

  The above-mentioned “specific position” of the own vehicle may be a position determined in common with respect to all the sensing units 2a to 2d, for example, the center of the own vehicle, and instead to each sensing unit 2a to 2d. Alternatively, the positions may be determined separately. For example, with respect to the sensing unit 2a provided in front of the host vehicle, the front end of the host vehicle is set as a “specific position”, the distance from the front end is calculated, and the sensing unit 2b provided behind the host vehicle is The rear end of the vehicle may be set as a “specific position” and the distance from the rear end may be calculated. Alternatively, the distance from the camera may be calculated by setting the position of the corresponding camera for each sensing unit as a “specific position”. Furthermore, the distance from each of the sensing units 2a to 2d may be used as it is as the distance from the host vehicle without calculating the distance from the “specific position”.

  The sensing unit 2 and the distance information calculating unit 3 constitute a distance information generating unit 10 that generates information indicating the distance to the object imaged by the cameras 1a to 1d, that is, the distance from the own vehicle to each part of the object. Is done.

  The sensing unit 2 may be a sensing device that performs distance measurement by an active method, and may be, for example, a radar using millimeter waves, lasers, infrared rays, or the like configured to be able to acquire two-dimensional distance information of an object.

  The own vehicle state detection unit 90 detects the state of the own vehicle, for example, the traveling direction and speed, and outputs information representing the traveling direction and speed of the own vehicle to the model shape determining unit 4. The operation unit 92 is used for operation input by the user.

  The model shape determining unit 4 determines a three-dimensional surface shape to project a plurality of images captured by the cameras 1a to 1d based on the distance information generated by the distance information generating unit 10, and a space having the determined three-dimensional surface shape Generate a model. This space model is virtually formed in a virtual space corresponding to the real space. The model shape determination unit 4 refers to, for example, the detection result of the state of the host vehicle obtained from the host vehicle state detection unit 90, and is located in the traveling direction of the host vehicle and close to the target object or part thereof. The solid surface shape is determined by reflecting only the distance information. At this time, the shape of the space model may be adjusted according to an operation input from the user via the operation unit 92.

  In the conventional example, the space model is formed in a predetermined shape, for example, a cylindrical shape or a saddle shape. In the case of a saddle type, when generating a composite image in which the surroundings of the host vehicle are viewed from an arbitrary viewpoint position, all the objects above the road surface are projected onto the walls of the same space model, so the objects are crushed. It looks like.

  For this reason, the user (for example, the driver) cannot recognize the unevenness of the object in the image. For example, when the vehicle travels backward and parks, there is an object having a portion protruding toward the vehicle. However, since the protruding portion is difficult to recognize, there is a risk of colliding with the protruding portion.

  On the other hand, in the present invention, a three-dimensional space model taking into account the distance to the object is generated based on the distance information generated by the distance information generation unit 10, and an image is projected onto the three-dimensional space model. As a result, it becomes easier for the user to recognize the unevenness of the object with an image, so that collision can be prevented and safety is improved.

The camera control unit 5 includes a brightness adjustment unit 5a and a camera parameter setting unit 5b.
The brightness adjusting unit 5a matches the brightness of the first to fourth images captured by the first to fourth cameras 1a to 1d. That is, each of the first to fourth images is formed so as to include a common area with at least one other image as shown in FIG. 19 described later, for example, and the average level of the pixel values in these areas is the same. Then, the camera parameter setting unit 5b is controlled.

The camera parameter setting unit 5b increases / decreases the screen brightness level of the image captured by each of the cameras 1a to 1d by controlling the drive signal of the image sensor 12 used by the cameras 1a to 1d.
The brightness adjustment unit 5a and the camera parameter setting unit 5b can prevent the brightness level difference between the plurality of images, and the visibility for the user is improved.

  The projection conversion unit 6 performs projection conversion and coordinate conversion in order to generate an image from an arbitrary viewpoint. That is, a plurality of images obtained by imaging with the cameras 1a to 1d are projected onto a space model having the shape determined by the model shape determination unit 4, and the image on the space model is further viewed from an arbitrary virtual viewpoint in an arbitrary direction. Is projected onto an image (virtual viewpoint image) in the screen coordinate system representing the image viewed. In such projection, coordinate conversion is performed between different coordinate systems.

  The image composition unit 7 extracts a part or all of the screen coordinate system image generated by the projection conversion unit 6 and generates a display image.

The information transmission unit 8 generates a video signal based on the image generated by the image synthesis unit 7 and supplies the video signal to the image display unit 80. The image display unit 80 is provided in a dashboard, for example, and is configured by a liquid crystal display, for example. The image display unit 80 may be used for other purposes, for example, an image display unit for car navigation.
The information transmission unit 8 is further connected to a speaker 82, receives information indicating the state of the host vehicle from the host vehicle state detection unit 90, receives distance information from the distance information generation unit 10, and based on these information, such as a collision When it is determined that the danger is imminent, an audio signal is supplied to the speaker 82 to generate a warning sound.

  In order to generate an image from an arbitrary viewpoint, in the present invention, the space of the shape generated by the model shape determination unit 4 based on the distance information generated by the distance information generation unit 10 instead of the space model of a predetermined shape. A model, that is, a spatial model having a shape corresponding to the distance from the host vehicle to each part of the object is used, but since this spatial model is a modification of a spatial model having a predetermined shape, Below, while explaining the case where the space model of a predetermined shape is used, the difference in the case of using the space model as a modification will be described.

  As shown in FIG. 4, the space model is formed so as to have a bottom portion that substantially matches the road surface below the vehicle and a wall portion that rises from the bottom portion.

  If the space model is composed of only the horizontal surface of the ground level, the white line on the ground can be accurately reproduced. In this case, the height of other vehicles, trees, buildings around the vehicle The target object is displayed with large distortion. Therefore, in the conventional example, a saddle type model that has a flat shape corresponding to the ground in the host vehicle and its surroundings and rises gently from the host vehicle at a certain horizontal distance is used in many cases. If the object is a three-dimensional object, it becomes easier to recognize the actual shape of the object as the rising angle of the wall surface increases.

  3 shows a vertical cross section of the saddle-shaped space model 20, and FIG. 4 is a view of the space model 20 as viewed from above. The horizontal circular portion 21 of the space model 20 is referred to as a bottom portion of the model, and the other portion is referred to as a wall portion of the model. The saddle type space model 20 is formed such that the host vehicle 110 is arranged at the center of the bottom thereof. The wall portion includes a front portion 31, a rear portion 32, a left side portion 33, and a right side portion 34. The front portion 31 constitutes a region for pasting a front captured image captured by the camera 1a, the rear portion 32 constitutes a region for pasting a rear captured image captured by the camera 1b, and a left side portion 33. Constitutes a region for pasting the left side captured image captured by the camera 1c, and the right side portion 34 configures a region for pasting the right side captured image captured by the camera 1d.

  The pasting of the captured image to the space model is an image on the imaging surface (the surface on which the photoelectric conversion elements of the image sensor 12 are arranged) obtained by capturing an image of the object in the real space with a camera. This is performed by projecting onto a space model arranged in a virtual space corresponding to the real space. This will be described with reference to FIGS.

  5 and 6 show a horizontal section of the region 31 of the spatial model in FIG. 4 at a certain height (indicated by reference numeral Hc in FIG. 3) together with the camera 1a. 5 and 6 show horizontal sections of the object Os at the same height Hc. The object Os has a simple shape for the sake of simplicity.

  FIG. 5 shows a case where the object Os is farther than the area 31, and FIG. 6 shows a case where the imaged part of the object Os is closer to the area 31. In any case, the captured image of the object Os in the real space is formed on the imaging surface CP of the camera 1a, and the image formed on the imaging surface CP is virtually projected onto the area 31 of the space model. Therefore, the points indicated by the symbols Os01, Os02, Os03, Os04, and Os05 of the object Os are the intersection points Ps01, Ps02, Ps03, Ps04, and the line 31 connecting the respective points and the lens center Oc of the camera 1a with the region 31. Projected on Ps05.

  FIG. 7 shows a vertical section of the region 31 of the spatial model of FIG. 4 in a fixed orientation (indicated by reference numeral Vc in FIG. 4) together with the camera 1a. As described with reference to FIGS. 5 and 6, points Os11 and Os12 of the object Os are points of intersection Ps11 and Ps12 between the line 31 connecting the respective points and the center Oc of the lens of the camera 1a and the region 31. Projected on.

  The above assumes a case where the spatial model has a shape different from the surface to be imaged of the object, or is formed at a different position. When the space model has the same shape as the imaging surface of the object and is formed at the same position, each point on the imaging surface of the object in the real space is virtually formed as it is. In this case, an image obtained by imaging with the camera is projected onto the spatial model.

  The texture information of each point of the object Os is acquired by the pixel at the corresponding position on the imaging surface by imaging with the camera 1a, and each of the points on the imaging surface is projected when the captured image is projected and converted to the space model. Texture information is pasted (mapped) from the pixel to the corresponding point on the spatial model.

  Next, each point on the space model is associated with the position of the screen coordinate system S that defines the position in the output image (virtual viewpoint image), and the texture information of each point on the space model is A virtual viewpoint image (image on the screen coordinate system) is formed by being pasted (mapped) at a corresponding position. In the above association, as shown in FIG. 5 as an example, a plane (virtual viewpoint image plane) SP of the screen coordinate system S is placed between the space model and the virtual viewpoint, and each point of the space model and the virtual viewpoint Oe are set. Intersections between the connecting straight line and the virtual viewpoint image plane SP are positions corresponding to the above points of the space model.

Hereinafter, a coordinate system used in the above-described projection conversion will be described with reference to FIGS.
FIG. 8 shows an example of an image on the texture coordinate system T for defining the position in the image input from the camera 1. Although FIG. 8 shows an image viewed from the host vehicle, that is, an image acquired by the camera 1a, an image viewed from the other direction, that is, an image acquired by the other cameras 1b to 1d is also shown. Similar texture coordinate system data is generated. When the captured image is projected onto the space model, texture coordinate system data corresponding to each point of the space model is formed.

In the texture coordinate system T, the horizontal direction is the x-axis, the vertical direction is the y-axis, the lower left is (0, 0) and the upper right end is (1, 1) regardless of the image size (number of pixels). This is a coordinate system representing a texture using an arbitrary position as a parameter.
The texture information is information representing the color (brightness, hue, saturation) of each part or pixel of the image, and the information (texture value) representing the color of each part or pixel corresponds to the coordinate position in the texture coordinate system. It is attached.
The texture value is stored as texture data at a corresponding address in the memory 6a of the projection conversion unit 6, and is read when the texture is pasted on the space model or when drawing is performed in the screen coordinate system S. Used.

  FIG. 9 shows an example of an image of the screen coordinate system S. The image shown in FIG. 9 is an image obtained by looking forward from the rear of the host vehicle including the host vehicle. An illustration image or a real image prepared in advance is used as the image of the host vehicle. The screen coordinate system S in FIG. 9 is a two-dimensional coordinate system that represents an arbitrary position of an image on which a spatial model is projected as a parameter. The horizontal direction is the x axis, the vertical direction is the y axis, and the lower left corner is (0, 0), and the upper right corner is represented as (1, 1).

FIG. 10 shows a three-dimensional world coordinate system W that defines a position in a space for constructing a space model, an imaging viewpoint coordinate system C that defines a three-dimensional position with the position of the camera (center of the lens) as an origin Oc, and a virtual A virtual viewpoint coordinate system E that defines a three-dimensional position with the viewpoint position as the origin Oe is shown.
The Zc axis of the imaging viewpoint coordinate system C coincides with the optical axis of the camera. The Ze axis of the virtual viewpoint coordinate system E coincides with the direction of the line of sight when viewing the space model from the virtual viewpoint.

Hereinafter, projection conversion from a captured image to a spatial model will be described in more detail.
FIG. 11 shows the interrelationship between the coordinate systems C, W, and E shown in FIG. 10, and the imaging plane CP of the camera and a plane (virtual viewpoint image plane) SP that forms a virtual viewpoint image.
It can be considered that an image obtained by capturing an image while viewing the direction of the Zc axis from the viewpoint Oc with the camera is on the imaging surface CP (surface on which photoelectric conversion elements are arranged). The imaging surface is actually on the side opposite to the imaging target with respect to the origin (lens center) Oc of the viewpoint coordinate system, but is shown on the same side for convenience.
On the other hand, an image obtained by viewing an arbitrary (desired) direction Ze from an arbitrary (desired) virtual viewpoint Oe can be considered to be formed on the virtual viewpoint image plane SP.
Two-dimensional coordinates on the imaging plane CP are represented by (u, v), and coordinates on the virtual viewpoint image plane SP are represented by (x _S , y _S ).

式（１ａ）、（１ｂ）において、ｆ_Ｃは、原点Ｏｃから撮像面ＣＰまでの距離（レンズの焦点距離）である。
式（１ａ）、（１ｂ）により、撮像面上の各画素の位置と、空間モデル上の各点とが対応付けられ、この対応関係に基づいて、それぞれの画素で得られたテクスチャを、空間モデル上の対応する位置に割り当てることができる。 The coordinates (x _C , y _C , X) of the imaging viewpoint coordinate system C of the intersection point Pm between the line connecting the coordinates (u, v) of the position Pc on the imaging plane CP and the origin Oc and the plane constituting the spatial model. z _C ) has the following relationship.

In equations (1a) and (1b), f _C is the distance from the origin Oc to the imaging surface CP (focal length of the lens).
By the expressions (1a) and (1b), the position of each pixel on the imaging surface and each point on the space model are associated with each other, and based on this correspondence relationship, the texture obtained by each pixel is expressed as a space. Can be assigned to the corresponding position on the model.

Although only one imaging viewpoint coordinate system C is shown in FIGS. 10 and 11, there are actually as many imaging viewpoint coordinate systems C as the number of cameras.
When pixels of a plurality of captured images are associated with the same point of the spatial model, one of the captured image pixels is preferentially selected in accordance with a predetermined rule, and the texture of the selected captured image pixel is determined. Give.
For example, a camera that has priority for each area of the space model may be determined in advance. For example, priority is given to the captured image of the camera 1a imaging the front for the area 31 ahead of the spatial model, and priority is given to the captured image of the camera 1b imaging the rear for the area 32 behind the spatial model, For the region 33 on the left side of the spatial model, priority is given to the captured image of the camera 1c that is imaging the left side, and for the region 34 on the right side of the spatial model, priority is given to the captured image of the camera 1d that is capturing the right side. Let
Instead of selecting the pixels of one camera in this way, the textures at the corresponding positions on the spatial model may be determined by weighting and averaging the textures of the pixels of a plurality of cameras.

  A predetermined texture information, for example, information indicating that the color is black is assigned to a portion (position) that is not associated with any camera pixel in the spatial model.

  As the spatial model, a model constructed as a set of triangular polygons is generally used. In this case, it is determined which of the first to fourth cameras 1a to 1d should be assigned to the texture of each polygon, and the texture information for each vertex is determined based on the determination result. Make assignments.

  When the spatial model has a predetermined shape, information indicating the position of each point of the spatial model or information necessary for calculating the position of each point is stored in the memory 6a in advance. If the space model is formed according to the distance to the object, information representing the position of each point of the space model is calculated and stored in the memory 6a when the space model is formed.

式（２）において、（ｔｃｘ，ｔｃｙ，ｔｃｚ）は、ワールド座標系の原点Ｏｗに対する撮像視点座標系の原点Ｏｃの相対位置を表す。
Ａ_ＣＷは、撮像視点座標系をワールド座標系の向きに合わせるための３行３列の回転行列であり、仮想空間内でのカメラ視点の位置姿勢から算出される。 The coordinates (x _C , y _C , z _C ) of the imaging viewpoint coordinate system C and the coordinates (x _W , y _W , z _W ) of the world coordinate system W are expressed by the following equation (2). There is a relationship.

In Expression (2), (tcx, tcy, tcz) represents the relative position of the origin Oc of the imaging viewpoint coordinate system with respect to the origin Ow of the world coordinate system.
A _CW is a 3-by-3 rotation matrix for aligning the imaging viewpoint coordinate system with the orientation of the world coordinate system, and is calculated from the position and orientation of the camera viewpoint in the virtual space.

  The coordinates of the imaging viewpoint coordinate system C of each point of the space model can be converted into the coordinates of the world coordinate system W using the above equation (2).

式（３）において、（ｔｅｘ，ｔｅｙ，ｔｅｚ）は、ワールド座標系の原点Ｏｗに対する仮想視点座標系の原点Ｏｅの相対位置を表す。
Ａ_ＷＥは、ワールド座標系の向きを仮想視点座標系の向きに合わせるための３行３列の回転行列であり、仮想空間内での仮想視点の位置姿勢から算出される。 The coordinates (x _W , y _W , z _W ) of the world coordinate system W and the coordinates (x _E , y _E , z _E ) of the virtual viewpoint coordinate system E are expressed by the following expression (3). There is a relationship.

In Expression (3), (tex, tey, tez) represents the relative position of the origin Oe of the virtual viewpoint coordinate system with respect to the origin Ow of the world coordinate system.
A _WE is a 3-by-3 rotation matrix for matching the orientation of the world coordinate system with the orientation of the virtual viewpoint coordinate system, and is calculated from the position and orientation of the virtual viewpoint in the virtual space.

  Using the above equation (3), the coordinates of the world coordinate system W of each point of the space model can be converted into the coordinates of the virtual viewpoint coordinate system E.

式（４）で、Ａ_ＥＳは変換行列と呼ばれるものであり、仮想視点の視野角と出力画像（スクリーン座標系の画像）の解像度から算出される。 The transformation from the coordinates (x _E , y _E , z _E ) of the virtual viewpoint coordinate system representing each position on the space model to the coordinates (x _S , y _S ) of the screen coordinate system is expressed by the following equation (4). The

In Equation (4), _AES is called a transformation matrix, and is calculated from the viewing angle of the virtual viewpoint and the resolution of the output image (screen coordinate system image).

  The texture coordinate system texture formed corresponding to the spatial model is associated with each pixel constituting the image of the screen coordinate system S obtained as described above, and drawing is performed. By such processing, an image (screen coordinate system image) in which an arbitrary direction is viewed from an arbitrary viewpoint is obtained.

When the spatial model is constructed by a set of polygons, the position on the screen coordinate system S corresponding to each vertex of the polygon is obtained, and the corresponding position on the texture coordinate system (x _T , y _T ) color information is copied, and for pixels other than the pixel corresponding to the vertex, a texture value is obtained by interpolation and assigned.

  In the above example, texture information is given to the corresponding point of the spatial model from the imaging plane, and is given to the corresponding point of the virtual viewpoint image from the spatial model, but at the time of projection conversion from the imaging plane to the spatial model Is associated with each point of the space model as a reference information only for the corresponding position on the imaging surface, and thus the information representing the coordinates of the texture coordinate system, and when projecting from the space model to the image of the screen coordinate system, When the reference information associated with each point in the model is copied to the corresponding point in the screen coordinate system and rendered in the screen coordinate system, it is represented by the coordinates in the texture coordinate system specified by the reference information. It is also possible to obtain texture information from the position to be attached and paste the texture.

In such a configuration, when the space model is constructed of a set of polygons, the image is obtained by any one of the first to fourth cameras 1a to 1d for each vertex of each polygon. Decide which pixel of the image to assign, information indicating the image containing the assigned pixel (or the camera that captured the image), and the location of the assigned pixel in the image, ie in the corresponding texture coordinates Hold the coordinates. In forming an image of the screen coordinate system, the color information of the corresponding position (x _T , y _T ) on the texture coordinate system is copied to each pixel of the screen coordinate system S corresponding to each vertex of the polygon. For pixels other than the pixels corresponding to the vertices, texture values are obtained by interpolation and assigned.

  In this case, when coordinates of a plurality of texture coordinate systems are associated with each pixel of the screen coordinate system S, selection or weighting of any one of the texture information of the corresponding coordinates of the plurality of texture coordinate systems The average is done to determine the texture to paste.

  In order to prevent the influence of the aberration of the camera lens from appearing in the composite image (the image displayed on the image display unit 80), the texture coordinates corrected for distortion may be given to the vertex of the model. By doing so, a calculation for correcting distortion is not required at the time of synthesis, and a composite image from which the influence of lens aberration is removed can be obtained by applying the image acquired from the camera as it is.

  The image synthesizing unit 7 extracts part or all of the image in such a screen coordinate system, performs enlargement, reduction, and rotation as necessary, determines the position in the display screen, and further others as necessary A display image is formed by combining these images.

  For example, when generating an image looking down from above the host vehicle, an illustration image of the host vehicle is displayed in the center, and an image of the screen coordinate system is displayed so that a road surface image is displayed in the periphery (front, rear, left and right). Process. On the other hand, when generating an image viewed from behind the vehicle, the image of the screen coordinate system is processed so that only the captured image obtained by the camera is displayed without displaying the illustration image of the own vehicle.

FIG. 9 shows an example of an image in the screen coordinate system formed as a result of the above processing. In the example illustrated in FIG. 9, the image is a front view of the host vehicle 110 when viewed from the rear, and a portion surrounded by the curve 131 corresponds to an image portion projected on the region 31 of the spatial model, and is represented by a curve 133. The surrounded part corresponds to the image part projected on the area 33 of the space model, and the part surrounded by the curve 134 corresponds to the image part projected on the area 34 of the space model and is surrounded by the curve 121. The corresponding portion corresponds to the image portion projected on the bottom 21 of the space model.
A joint part between the image part 131 and the image part 133 is indicated by reference numeral 40a, and a joint part between the image part 131 and the image part 134 is indicated by reference numeral 40b.

Whether to generate an image viewed in which direction from which viewpoint as an image in the screen coordinate system may be automatically switched according to the state (traveling direction, traveling speed) of the host vehicle, or may be selectable by the user.
For example, when moving backward for parking, it is often desired to display an image viewed from behind the host vehicle or an image viewed from above the host vehicle. In this case, it is detected that the vehicle is moving backward based on the output of the own vehicle state detection unit 90, and an image viewed from behind or an image viewed from above is automatically selected. It may be possible to switch to another image by the operation of 92.

Switching of the virtual viewpoint can be realized by changing the origin of the virtual viewpoint coordinate system in the projection conversion in the projection conversion unit 6.
The virtual viewpoint may be selected from any one prepared in advance, and after the user automatically selects any one prepared in advance, the user operates the operation unit 92. Adjustment may be possible.
The change of the position of the virtual viewpoint is performed by changing the relative position (tex, tey, tez) of the origin Oe of the virtual viewpoint coordinate system E with respect to the origin Ow of the world coordinate system W described above. This is performed by changing the direction of the Ze axis of the virtual viewpoint coordinate system E.

As described above, in the present invention, instead of a space model having a predetermined shape, an image of the object is projected onto a space model having a shape corresponding to the distance from the host vehicle to each part of the object. .
Hereinafter, this will be described with reference to FIGS.
FIG. 12 is a horizontal cross-sectional view at the same position as in FIG. 5 and shows the object Os and the camera 1a.

However, the region 31 of the spatial model in FIG. 5 is not shown in FIG. In the present invention, instead of the area 31 of the spatial model, a surface Os1s (shown by a solid line) captured by the camera 1a of the object Os1 is a part of the spatial model. Of the object Os1, a portion (shown by a dotted line) Os1h that is hidden by another portion is not a part of the space model, and a line (shown by a chain line) Os1c that connects both ends of the hidden portion is a part of the space model. .
As shown in the figure, when another object Os2 is simultaneously imaged, the imaged surface Os2s is also a part of the spatial model.

  The above is based on the condition that distance information can be acquired as a result of distance measurement by the distance sensor for the entire surface imaged by the camera 1a. Of the surface imaged by the camera 1a, an area where distance information cannot be obtained may not be configured, and it corresponds by interpolation based on a space model part formed based on the surrounding part. A space model portion may be formed. For example, as shown in FIG. 12, when distance information from the own vehicle is not obtained in the region between the objects Os1 and Os2, the object Os1 is located at a predetermined distance from the own vehicle as indicated by a chain line Osc. The surface connecting Os2 may be configured.

  FIGS. 13A and 13B show examples of spatial model generation when an object different from that shown in FIG. 12 is imaged. FIG. 13A is an elevation view showing a state in which another camera 120 in front of the camera 1a is being imaged. In this case as well, as described with reference to FIG. 12, the space model 51 (FIG. 13 (b) corresponding to the specific position of the vehicle 120 as the object, for example, the distance from the camera 1a to each part of the object. )). FIG. 13B shows a vertical cross section of the space model 51 to be formed.

  In the example shown in FIG. 13A, the protruding end 50b of the luggage 50 protruding from the loading platform 120a of the vehicle 120 is closest to the host vehicle (camera 1a), and the distance is calculated as L50b. When the host vehicle is traveling in the direction of the vehicle 120, the most important thing to consider is the protruding portion of the luggage 50 (the portion protruding toward the host vehicle) 50a. In the projected image, it is difficult to recognize that the luggage 50 has the protruding portion 50a. For this reason, there is a risk of collision with the protruding portion 50a of the luggage 50 when driving based on the image displayed on the image display unit 80.

  In the present embodiment, in order to make it easy to recognize the distance to the protruding portion 50a of the luggage 50, particularly the protruding end 50b, the distance information generated by the distance information generating unit 10 is used instead of the predetermined space model. Based on the space model 51 generated by the model shape determination unit 4 based on the distance from the host vehicle to each part of the object (and thus the shape corresponding to the shape of the imaging surface of the object) (see FIG. 13 (b)). The space model 51 has a protruding portion 51a corresponding to the protruding portion 50a of the luggage 50, and the protruding end 51b is formed at a position corresponding to the protruding end 50b of the luggage.

An example of an image (an image displayed on the image display unit 80) formed using such a space model 51 is shown in FIG. For comparison, FIG. 15 shows an image formed using a conventional saddle type space model. In these figures, each part in the image is indicated by the same reference numeral as each part of the corresponding object.
In FIG. 15, as a result of projection onto the saddle-shaped space model, the image is distorted (a crushed image) and the perspective is lost, whereas in FIG. 14, there is no distortion. An image with a sense of perspective is obtained. For this reason, it is easy to recognize the protruding portion 50a of the luggage 50, and it is easy to recognize the danger of collision of the luggage 50 with the protruding portion 50a.

  As described above, in this embodiment, since a space model having a shape corresponding to the distance from the host vehicle to each part of the object is used without using a predetermined saddle-shaped space model, an arbitrary viewpoint is used. When generating a composite image that looks at the periphery of the host vehicle from the position, the object in the vicinity of the host vehicle does not appear to have been crushed, and it is easy to recognize the unevenness of the target object, especially the protruding part toward you Is formed, which helps prevent collisions.

  In the above embodiment, as described with reference to FIG. 13B, the distance information generated by the distance information generation unit 10 is used in accordance with the distance from the host vehicle to each part of the target object. When forming a spatial model of a shape, the amount of calculation for forming the spatial model may be extremely large. Although it is good when the arithmetic processing unit of the model shape determining unit 4 has high arithmetic capability, it is necessary to reduce the arithmetic load when using an arithmetic processing unit with low arithmetic capability.

  Therefore, among the plurality of objects, the object closest to the own vehicle, or a part of the object closest to the own vehicle (object part), for example, an object located in the traveling direction of the own vehicle It is also conceivable to use a shape that takes distance into consideration only for the portion closest to the host vehicle and to use a predetermined shape, for example, a saddle type space model, for the other portions (and other objects). Information about the traveling direction of the host vehicle is obtained from the host vehicle state detection unit 90. Data representing the above-mentioned predetermined shape, for example, a saddle-shaped space model shape, is stored in advance in the memory 4a in the model shape determining unit 4, and determination of the shape of the model for the “other portion” is performed. In this case, the stored data is used. By doing in this way, the safety improvement for collision avoidance can be aimed at, reducing calculation amount.

FIG. 13C shows an example of a spatial model having a shape in which the distance is taken into consideration only for the portion closest to the host vehicle when the object is another vehicle shown in FIG. FIG.13 (c) has shown the vertical cross section of the space model 52 similarly to FIG.13 (b).
Of the space model 52, the part indicated by reference numeral 52a corresponds to the distance from the own vehicle 110 to each part of the part 50a protruding forward of the luggage 50 of the other vehicle 120, and the other parts are 椀Has the shape of a mold.

In the illustrated example, the rear end 120b of the loading platform 120a of the vehicle 120 (the rear end 120b and a portion in front thereof) is not a protruding portion, and is rearward from the rear end 120b of the loading platform 120a of the load 50. Only the protruding portion 50a is treated as a protruding portion (portion closest to the host vehicle), and the space model 52 is formed with a portion 52a having a shape corresponding to the portion 50a.
The protruding end 52b of the protruding portion 52a is formed at a position corresponding to a distance L50b from the own vehicle to the protruding end 50b of the luggage 50 (the end closest to the own vehicle).
Parts of the space model 52 other than the part 52a are formed so as to have a shape continuous to the part 52a. That is, the radius 21a of the bottom portion 52d, the curvature of the wall portion 52c, and the like are determined based on the position of the protruding portion 52a (distance from the host vehicle).

  An example of the display image formed using the above space model is shown in FIG. 16 is the same as FIG. 15 except for the protruding portion 50a. However, only the protruding portion 50a has a distortion and a perspective image similar to FIG. For this reason, it is easy to confirm the protruding portion 50a of the luggage 50.

  Further, the protruding amount of the protruding portion 52a may not coincide with the protruding portion 50a of the luggage, and may be limited to a predetermined length or less, for example.

  Further, the portion corresponding to the surrounding portion together with the protruding portion of the object may be shaped in consideration of the distance, and the other portion may be formed as a predetermined saddle-shaped space model. The overall shape of the space model in this case is as shown in FIG. 17, and an example of the display image in this case is as shown in FIG.

  In FIG. 17, the same reference numerals as those in FIG. 4 are assigned to the bottom, front region, rear region, left side region, and right side region of the space model. As a result of detecting the protruding portion of the object at a position corresponding to a part of the front area 31, only the range 36 including the part and its periphery has a shape corresponding to the distance from the own vehicle to each part of the object. Has been. In the image display unit 80, the image projected in the above range 36 becomes a portion surrounded by a frame 54 in FIG.

In FIG. 18, an image similar to FIG. 15 is formed outside the frame 54, and an image similar to FIG. 14 is formed inside the frame 54. Even in such an image, the protruding portion 50a of the luggage 50 can be easily confirmed.
In addition, in order to avoid the discontinuity of the image between the inside and the outside of the frame 54, image processing for blurring the boundary portion may be performed, and conversely, the difference can be made by changing the display color only inside the frame. It may be emphasized to call attention to danger.

  In the example of FIG. 13 (c), the portion up to the rear end 120b of the loading platform 120a is processed as a portion protruding toward the own vehicle among the objects (that is, a portion closest to the own vehicle). You may treat the closest object or the end portion 50b of the portion closest to the host vehicle to a predetermined range, for example, 1 m ahead (distant) as the portion closest to the host vehicle.

In addition, it may be possible to set in advance which range is processed as a portion protruding toward the own vehicle (the portion closest to the own vehicle), based on the state of the own vehicle, for example, the traveling direction and the traveling speed. Switching may be performed automatically, or the user may be able to adjust while viewing the display image.
Similarly, the dimension of each part of the space model (the radius 21a of the bottom 52d, the curvature of the wall 52c, etc.) may be arbitrarily set in advance by the user, or may be adjusted while the user is viewing the display image.

  Furthermore, only when the distance to the object is closer than a predetermined threshold value, the shape of the spatial model may be set according to the distance to each part of the object. Further, the predetermined threshold value may be changed according to the traveling direction and traveling speed of the host vehicle. Furthermore, the shape of the space model may be a shape corresponding to the distance to each part of the object only in a predetermined direction, for example, the rear region, and may be a predetermined shape, for example, a bowl shape, in the other directions.

  Although the bottom portion has been described as a circular shape, it may be an ellipse, and the user can determine an arbitrary value. For example, a line segment passing through two focal points drawn inside an ellipse is defined as a major axis, a line segment obtained by drawing a vertical bisector of a major axis inside an ellipse is defined as a minor axis, and these values are set by the user. The shape of the bottom of the space model can be determined.

  The “predetermined shape” is not limited to a saddle shape, and may be, for example, a cylindrical shape or a quadrangular prism shape. Furthermore, the shape (partition type | mold) which provided the wall part only in the direction selected among front and back, right and left, for example, one or two sides may be sufficient. Further, in any case, the bottom may be omitted except when it is necessary to form an image of the road surface.

  Further, an area 40 surrounded by a dotted line in FIG. 9 indicates the boundary portion of the image in the screen coordinate system. However, due to the shift of the boundary portion, the originally visible eyelid image may not be seen (a blind spot occurs). In order to prevent the generation of blind spots, two adjacent images may be combined at the boundary between the images by semi-transparent composition (alpha blend) using a coefficient (alpha value). This processing not only makes the boundary between images less noticeable, but also improves visibility by eliminating blind spots.

  In addition, in the above description, it has been described that the distance information can be obtained for the entire image of the object captured by the camera 1, but this means that distance measurement is not necessarily performed for each pixel of the captured image. Does not mean that. Generally, since the resolution of sensing by the sensing unit 2 is lower than the resolution of imaging by the camera 1, only one distance information can be obtained for each of a plurality of pixels. In this case, the distance information obtained for each of the plurality of pixels may be used as common distance information for the plurality of pixels, and the pixel closest to the angle from which the distance information is obtained among the plurality of pixels. The distance information may be obtained by interpolation calculation for other pixels.

  Further, since the scanning range of the sonar sensor used as the sensing unit 2 is not as wide as the angle of view of the camera 1, it is difficult to radiate ultrasonic waves to all the objects reflected in the image captured by the camera 1. Therefore, for example, each of the sensing units 2 can be configured by a plurality of sonar sensors, and the angles of the plurality of sonar sensors can be shifted to arrange all the angles of view of the camera 1a within the radiation range. With such a configuration, distance information about the object can be obtained by the distance information generation unit 10 wherever the object imaged by the camera 1 appears in the image, which is effective from the viewpoint of collision prevention. .

Furthermore, the distance measurement using only the camera and the distance measurement using the sensing unit 2 may be performed without measuring the distance to the object around the host vehicle only by the sensing unit 2.
FIG. 19 shows a view from above the vehicle when a saddle type space model is used for easy explanation. When each of the first to fourth cameras 1a to 1d includes a fisheye lens having an angle of view of 180 degrees, the object existing in the region 60 is both in the first camera 1a and the fourth camera 1d. Since imaging is possible, the distance to the object can be measured by the passive stereo method using these two cameras. Similarly, it is possible to measure the distance to the object using the camera 1b and the camera 1d for the area 61, the camera 1b and the camera 1c for the area 62, and the camera 1a and the camera 1c for the area 63. Furthermore, since a camera is used, high-resolution distance measurement is possible.

  For objects in areas other than the areas 60 to 63 in FIG. 19, it is necessary to measure the distance based on the data obtained by the sensing unit 2, but all the angles of view are used as the detection target range of the sensing device without omission. Compared with the above, since the number of sensing devices can be reduced, the cost can be reduced.

  Furthermore, you may comprise the sensing part 2 with the light projector which projects not only a sensing device but a slit light and a two-dimensional pattern on a target object using LED and a laser as a light source. In that case, the distance can be obtained from the image captured by the camera 1 by the principle of triangulation by the active stereo method.

  Although the present invention has been described above as an image processing apparatus, an image processing method implemented by the image processing apparatus also forms part of the present invention, and allows a computer to execute the functions of the image processing apparatus. A computer program for causing a computer to execute and a recording medium storing the computer in a form readable by the computer also form a part of the present invention.

  1a to 1d camera, 2a to 2d sensing unit, 3 distance information calculation unit, 4 model shape determination unit, 5 camera control unit, 5a brightness adjustment unit, 5b camera parameter setting unit, 6 projection conversion unit, 7 image composition unit, 8 Information transmission unit, 9 data bus, 11 optical system, 12 imaging device, 13 AD transducer, 20 spatial model, 21 bottom of spatial model, 10 distance information generation unit, 31, 32, 33, 34 wall of spatial model 50 luggage, 50a protruding part, 50b protruding part of protruding part, 51, 52, 53 space model, 51a, 52a, 53a protruding part of space model, 51b, 52b, 53b protruding part of protruding part, 52c, 53c space model Wall part, 80 image display part, 82 speaker, 90 own vehicle state detection part, 92 operation , 100 image processing apparatus, 110 vehicle, 120 other vehicles.

Claims (12)

A plurality of cameras for imaging objects in the vicinity of the host vehicle;
A distance information generation unit that generates distance information to the object imaged by the camera;
A model shape determining unit that determines the shape of the space model having a three-dimensional surface corresponding to a distance from the own vehicle based on the distance information generated by the distance information generating unit to the object,
By projecting a plurality of images obtained by imaging with the plurality of cameras onto the spatial model having the shape determined by the model shape determining unit, and further converting the projected image into a screen coordinate system, a screen coordinate system A projection conversion unit for generating an image of
An image composition unit that extracts a part or all of the image of the screen coordinate system generated by the projection conversion unit to generate a composite image;
An information transmission unit that causes the image display unit to display the combined image generated by the image combining unit ;
The model shape determination unit reflects only the distance information of the object in the traveling direction of the host vehicle and closest to the host vehicle, or the portion thereof, based on the distance information generated by the distance information generation unit. the image processing apparatus characterized by that determine the shape of the space model Te. The model shape determination unit is configured as the closest object or portion thereof in the vehicle, reflecting the distance information about the part from the object or the end closest to the vehicle part thereof up to a predetermined distance range The image processing apparatus according to claim 1 , wherein the shape of the space model is determined. A plurality of cameras for imaging objects in the vicinity of the host vehicle;
A distance information generation unit that generates distance information to the object imaged by the camera;
A model shape determining unit that determines the shape of a spatial model having a three-dimensional surface according to the distance from the host vehicle to the object based on the distance information generated by the distance information generating unit;
By projecting a plurality of images obtained by imaging with the plurality of cameras onto the spatial model having the shape determined by the model shape determining unit, and further converting the projected image into a screen coordinate system, a screen coordinate system A projection conversion unit for generating an image of
An image composition unit that extracts a part or all of the image of the screen coordinate system generated by the projection conversion unit to generate a composite image;
An information transmission unit for causing the image display unit to display the combined image generated by the image combining unit;
With
The model shape determination unit, based on the distance information generated by the distance information generating unit, located in the traveling direction of the vehicle, and the closest on the closest object or portion thereof in the vehicle, and an image object or image picture processor to reflect only the distance information about the portion located around you and determining a shape of the space model portion thereof. The model shape determining unit is configured to assume a predetermined shape for a portion other than a portion determined by reflecting distance information to the object or the portion of the space model. The image processing apparatus according to any one of 1 to 3 . The image processing apparatus according to claim 4 , wherein the predetermined shape is a bowl shape. The model shape determining unit determines the size of the predetermined shape so that a step does not occur between a portion that reflects the object or distance information to the portion and a portion that is connected thereto. The image processing apparatus according to claim 4 , wherein the image processing apparatus is an image processing apparatus. The image synthesizing unit, in the composite image, according to claim 3 for the processing for blurring the boundary of the nearest object or image portion corresponding to the portion thereof in the vehicle and the image portion of the surrounding Image processing device. Each of the plurality of cameras, at least one other camera and the image processing apparatus according to any one of claims 1 to 7 in which the imaging range, wherein the partially overlapping of the plurality of cameras . A brightness adjustment unit for matching brightness levels for each image captured by the plurality of cameras;
Any one of claims 1 to 8, further comprising a camera parameter setting section for controlling the brightness level of the image captured by each of the plurality of cameras based on the control signal from the brightness adjusting portion An image processing apparatus according to 1. Imaging an object in the vicinity of the host vehicle with a plurality of cameras;
A distance information generating step for generating distance information to an object imaged by the camera;
A model shape determining step of determining the shape of the space model having a three-dimensional surface corresponding to the distance of the based on the distance information generated by the distance information generating step from vehicle to the object,
By projecting a plurality of images obtained by imaging with the plurality of cameras onto the spatial model having the shape determined in the model shape determining step, and further converting the projected image into a screen coordinate system, a screen coordinate system A projection transformation step for generating an image of
An image synthesis step of extracting a part or all of the image of the screen coordinate system generated by the projection conversion step to generate a synthesized image;
An information transmission step of causing the image display unit to display the combined image generated in the image combining step ;
The model shape determining step reflects only the distance information of an object in the traveling direction of the host vehicle and closest to the host vehicle, or a portion thereof, based on the distance information generated in the distance information generating step. image processing method characterized by that determine the shape of the space model Te.   Imaging an object in the vicinity of the host vehicle with a plurality of cameras;
  A distance information generating step for generating distance information to an object imaged by the camera;
  A model shape determining step for determining a shape of a spatial model having a three-dimensional surface according to the distance from the host vehicle to the object based on the distance information generated in the distance information generating step;
  By projecting a plurality of images obtained by imaging with the plurality of cameras onto the spatial model having the shape determined in the model shape determining step, and further converting the projected image into a screen coordinate system, a screen coordinate system A projection transformation step for generating an image of
  An image synthesis step of extracting a part or all of the image of the screen coordinate system generated by the projection conversion step to generate a synthesized image;
  An information transmission step of causing the image display unit to display the combined image generated in the image combining step;
  With
  The model shape determining step is based on the distance information generated in the distance information generating step, and is in the traveling direction of the host vehicle and the closest object to the host vehicle or a portion thereof, and the closest on the image The shape of the space model is determined by reflecting only the distance information about the object or a part located around the part.
  An image processing method. The computer program for making a computer perform each step of the image processing method of Claim 10 or 11.

Images