JP2003244688A - Image processing system for vehicle - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce the sense of anxiety of a driver about a blind spot by image-processing a past image imaged by a camera into a combination pseudo- image at a blind spot region of a current image obtained by imaging the periphery of a vehicle. <P>SOLUTION: A camera 13 for imaging the periphery of a vehicle is provided, images taken by the camera 13 are stored, and a processed image B' obtained by processing a past image BX imaged before reaching a current position is overlaid on a part of a current image A, which is currently imaged, being a blind spot region by being hidden by a car body or the like so that a combination image A'+B' is created. A contour line 6 symbolizing the car body is combined with the combination image A'+B' and the image A'+B' is displayed. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for a vehicle, and more particularly to an image processing apparatus for generating an image displaying a blind spot in a driver's field of view.

[0002]

2. Description of the Related Art When a driver operates a vehicle, the field of view that can be recognized from the viewpoint of the driver sitting in the driver's seat is:
There is an invisible area (blind spot area) that is obscured by the car body, pillars, etc. Conventionally, such a blind spot region is configured to be recognized to some extent by a room mirror, a side mirror, or the like. Further, recently, a vehicle has been proposed in which a camera that projects the outside of the vehicle is mounted and an image acquired by the camera is projected on a monitor in the vehicle. For example, a camera that projects the rear of the vehicle is mounted, and when the vehicle moves backward, the blind spot behind the vehicle is displayed on the in-vehicle monitor to assist the rear confirmation when entering the garage. The prior art with such a configuration includes the following.

[0003]

[Patent Document 1] Japanese Patent Laid-Open No. 10-264723.

[0004]

On the other hand, when a driver operates a vehicle in which the surroundings of the vehicle cannot be recognized by a conventional camera image or the like, an external obstacle coming into the field of view,
At the same time, the driver can intuitively understand the relative position and distance between the external obstacle and the vehicle, based on the view of the interior of the vehicle (window frame, trunk outline, etc.). is there.

However, as in the invention described in Japanese Patent Application Laid-Open No. 2004-242242, the one displaying a camera image displays the image displayed on the camera on the monitor as it is, and therefore the image displayed on the monitor is the image of the driver. It is far from the image seen with the naked eye. This is because the position of the camera viewpoint is different from the viewpoint position when the driver operates,
The reason is that the size of the obstacle that can be visually recognized is different from the size of the obstacle displayed on the monitor. Also, in the past, a driver could recognize the scenery outside the vehicle and the part of the vehicle body (pillars, bonnets, etc.) that are in sight at the same time and intuitively grasp their relative positions in the image. Is also due to the fact that only the outside scenery is projected. In other words, in the conventional monitor image, it is difficult for the driver, who has conventionally perceived the sense of distance, to grasp the positional relationship between the outline of the vehicle and surrounding objects based on the conventional perception. There was a problem.

The present invention provides a composite image for a blind spot area (area hidden by the vehicle body) around the vehicle to reduce anxiety about the blind spot, and to reduce the anxiety about the blind spot from a position close to the driver's viewpoint. By generating a captured image,
The purpose is to make it easier for the driver to understand the positional relationship between the vehicle and the surrounding obstacles through the image.

[0007]

The above objects are achieved by the present invention described below. (1) Image pickup means for picking up an image in the traveling direction of the vehicle,
A body region specifying unit that specifies a body region hidden by a body portion in an image captured by the image capturing unit;
A vehicle body image area acquiring means for acquiring an image area corresponding to the vehicle body area assumed to be visible if the vehicle body portion specified by the vehicle body area specifying means is transparent; and an image captured by the image capturing means. An image of a vehicle including image creating means for creating an image by replacing the vehicle body area with the image area acquired by the vehicle body image area acquiring means, and display means for displaying the image created by the image creating means Processing equipment.

(2) Image pickup means for picking up an image in the traveling direction of the vehicle, picked-up image conversion means for changing the viewpoint of the picked-up image, such as an image seen from the driver's viewpoint, and image pickup by the image pickup means. Body region specifying means for specifying a body region hidden by the body portion in the captured image, and a vehicle body for acquiring an image region corresponding to the body region which is supposed to be visible if the body portion is transparent An image is obtained by replacing the vehicle body area portion specified by the vehicle body area specifying means with the image area acquired by the vehicle body image area acquiring means in the image that has been subjected to viewpoint conversion by the captured image converting means. An image processing apparatus for a vehicle, comprising: an image creating unit to create; and a display unit to display an image created by the image creating unit.

(3) A moving distance detecting means for detecting a moving distance of the vehicle, wherein the picked-up image converting means has a distance from a driver's position to a position where the image pickup means is installed, and the moving distance. When the vehicle moving distance from the position imaged by the image capturing means detected by the detecting means becomes the same,
The vehicle image processing apparatus according to (2) above, wherein the captured image is subjected to viewpoint conversion.

(4) The image area assumed to be visible if the vehicle body is transparent uses the image area previously imaged by the imaging means.
Alternatively, the vehicle image processing device according to (2).

(5) Vehicle speed detecting means for detecting the speed of the vehicle, and when the speed detected by the vehicle speed detecting means is equal to or lower than a predetermined speed, the image created by the image creating means is displayed. The image processing device for a vehicle according to (1) or (2) above.

(6) A moving distance detecting means for detecting a moving distance of the vehicle, and the vehicle body image area acquiring means corresponds to the vehicle body area when the detected distance is equal to or more than a predetermined distance. The vehicle image processing apparatus according to (1) or (2) above, which acquires an image region.

(7) Vehicle body data storage means for storing vehicle body data of the vehicle, wherein the predetermined distance is determined based on the data stored in the vehicle body data storage means. The image processing device for a vehicle according to 6).

(8) Vehicle body data storage means for storing vehicle body data, and the vehicle body area specifying means specifies a vehicle body area hidden by a vehicle body portion based on the vehicle body data. Above (1) or (2)
The image processing device for a vehicle according to.

(9) A contour line of a vehicle body shape including at least one of a vehicle body, a bumper, a light, a wiper, an instrument panel, a steering wheel, a mirror, a tire, a seat, and a window is formed by the image creating means. The vehicle image processing apparatus according to (1) or (2) above, further including a vehicle body shape image synthesizing unit that adds a synthesized image to the created image.

[0016]

BEST MODE FOR CARRYING OUT THE INVENTION Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. FIG. 1 is a block diagram showing the configuration of an image processing apparatus 1 of the present invention. The image processing device 1 includes a processing device 12 for performing image processing, a camera 13 as an image pickup means attached to a vehicle body, an image display device 14 as display means for displaying an image, a current position detection device 15, and an image. Storage device 1 as storage means
6 and these devices are connected to each other via a system bus 17.

The camera 13 functioning as an image pickup means is provided, for example, in the interior of an automobile toward the traveling direction of the vehicle, and is attached at a height position corresponding to the driver's viewpoint position. In this embodiment, the rear-view mirror is used. It is located at the mounting position. The camera 13 is connected to the system bus 17 via the A / D converter 131. The image signal output from the camera 13 is converted into a digital signal by the A / D converter 131 in order to perform image processing. Further, if the connected camera 13 is capable of outputting a digital signal, the A / D converter is not necessary. Camera 1
3 is arranged toward the front of the vehicle and the other is arranged toward the rear of the vehicle. In the following description, the rear-mounted camera 13 used when moving to the rear where the field of view is most limited will be described as an example.

The present position detecting device 15 includes a steering angle sensor 15
1, a vehicle speed sensor 152, a GPS receiver 153, a direction sensor 154, and a distance sensor 155. The steering angle sensor 151 detects the steering angle of the vehicle. The steering angle is obtained by detecting the rotation angle of the steering wheel or the angle of the front wheels. The vehicle speed sensor 152, which functions as a vehicle speed detection unit, detects the traveling speed of the vehicle. The traveling speed is also detected when the vehicle is moved backward. The GPS receiver 153 detects the absolute position of the vehicle. The direction sensor 154 detects the direction of the vehicle. The distance sensor 155 detects the moving distance of the vehicle.

The moving distance of the vehicle can be obtained by the various sensors described above. That is, the moving distance can be detected by the distance sensor 155. Further, the moving distance can be detected by the vehicle speed and time detected by the vehicle speed sensor 152. Further, the moving distance can be detected by the locus of the position detected by the GPS receiving device 153. Further, when the direction of the vehicle changes, the traveling distance can be detected more accurately by the steering angle sensor 151, the direction sensor 154, the distance sensor 155 or the vehicle speed sensor 152. As described above, the moving distance detecting means is the distance sensor 155, the vehicle speed sensor 152, the GPS receiver 153, the steering angle sensor 151, the direction sensor 154, the distance sensor 155, or the vehicle speed sensor 152. It can be configured by. In this way, the current position detecting device 15 forms the moving distance detecting means of the present invention.

The storage device 16 is a device for storing the image projected by the camera 13, and the viewpoint conversion image data memory 161 in which the images output from the camera 13 are continuously stored.
And a composite work image data memory 162 and a pseudo transparent image data memory 163. In the viewpoint conversion image data memory 161, the images captured by the camera 13 are continuously recorded together with the position and time detected by the current position detection device 15. In the combined work image data memory 162, images corrected to the viewpoint position are temporally continuously stored. The pseudo transparent image data memory 163 stores a pseudo transparent image in which a past image is combined in the blind spot area.

The image display device 14 is composed of, for example, a liquid crystal display or the like, and displays the pseudo transparent image stored in the pseudo transparent image data memory 163. Image display device 1
4 is converted into an analog signal via the D / A converter 141. Further, when the image display device 14 can input a digital signal, that is, when the image display device 14 has a built-in converter, the D / A converter is not necessary.

The processing unit 12 is a central processing unit [CPU
(Central Processing Unit)] 121, a read-only memory [ROM] 122, and a random access memory [RAM] 123. Central processing unit 1
Reference numeral 21 acquires information about the moving distance and the direction of the vehicle obtained from the current position detection device 15, and the storage device 1
This is for performing various processes such as obtaining viewpoint conversion image data and image data for synthesizing work from 6 and generating pseudo transparent image data from these data.

In the ROM 122, for example, software for the central processing unit 121 to perform image processing, image data of the contour of the vehicle body, vehicle body data (vehicle height, vehicle width, installation position of the camera 13, camera image). Angles, relative distances between the vehicle body parts, etc. are stored and function as vehicle body data storage means. The RAM 123 is, for example, the central processing unit 121.
The data is used as a working area for temporarily storing data for performing the data processing.

The image processing apparatus of the present invention operates as follows. FIG. 2 is an overall side view of a vehicle 22 equipped with the image processing apparatus of the present invention, showing the relative positions of the vehicle 22 before and after the vehicle 22 is moving in the rearward direction. It is shown. The camera 13 that projects the rear of the vehicle functions as an image capturing unit and is arranged inside the vehicle (in FIG. 2, for ease of explanation,
Although the camera 13 is depicted as being provided outside the vehicle, it is actually provided inside the vehicle. ), The rear part of the vehicle is projected on a part of the screen, and a blind spot region b1 which is a vehicle body part exists, and the visible region a1 is actually projected on the screen. Here, the blind spot area refers to a scene outside the vehicle that is assumed to be projected by a part of the vehicle when the outside of the vehicle is viewed from inside the vehicle by the camera 13 and is supposed to be projected if there is no vehicle. Area. Here, a part of the vehicle means a side mirror of the vehicle body when visually recognizing the front,
It is the frame of the front window, the hood, etc. When recognizing the rear, it is a rear seat, a rear window frame, a rear trunk, or the like. Further, the visible region is a view of the outside of the vehicle that is directly projected when the outside of the vehicle is viewed from inside the vehicle by the camera 13. This view is displayed via, for example, the front window or the rear window of the vehicle.

As a method of specifying the blind spot area b1 (vehicle body area) by the vehicle body area specifying means, there are the following methods. For example, since the camera installation position, the size of the vehicle body, and the like are stored in the vehicle body data stored in the ROM 122, the vehicle body occupied in the image in advance can be determined from the relative distance between each component, the camera angle of view, and the like. The area to be displayed (that is, the blind spot area) can be specified. This vehicle body region is specified in advance and the blind spot region data is recorded in the memory.
Based on the blind spot area data stored in the memory, the blind spot area b is uniformly applied to the image obtained by the camera 13.
Decide on 1. In this way, the vehicle body region specifying means specifies the vehicle body region (blind spot region) from the captured image based on the preset blind spot region data. In addition, the contour of the vehicle body may be detected from the captured image, and the portion surrounded by the contour line may be specified as the blind spot area. As a method for detecting the contour line, a known image processing method described later can be used.

The blind spot area b1 is shown in the visible area a2 at the position (vehicle 21) before the vehicle retreats. Therefore, the image of the region b1 that is in the blind spot at the current position (vehicle 22) is the image before reaching the current position (taken image in the past), that is, the position (vehicle 21) ahead of the current position. The imaged image at this time (the imaged image at the time of being located in front) is used, and this is generated as the image of the blind spot area portion b1. That is, the blind spot area b1 in the image at the current position is specified, cut out, and the corresponding area is cut out from the images displayed in the visible area a2 in the past captured images, and this is made the current image. To synthesize.

In this way, by replacing (synthesizing) the image of the generated blind spot area b1 with the blind spot area of the currently displayed image, the blind spot area b1 part is reflected as a whole camera image. Display as it is. The installation position of the camera 13 is most preferably the position of the driver's viewpoint, but may be a position close to the position of this viewpoint. For example, in addition to the position of the rearview mirror, the position near the center of the dashboard, the shoulder of the seat, the position near the center of the rear window, etc. may be mentioned.

The operation of the image processing apparatus 1 of the present invention configured as above will be described. 3, 4 and 5
3 is a flowchart showing the operation of the processing device 12.
The power switch is turned on by turning on the ignition switch (step S100). In addition to this, the switch-on may be triggered by setting the shift change lever to the D position, or to the R position when displaying a rear image when moving backward.

Next, the data stored in the storage device 16 is searched so that image data and the like can be written in the storage area, and if there is no storable area, the next data to be acquired can be overwritten. In addition, referring to the distance and time stored together with the image data, the data is prepared for writing such as erasing the data not used in the future (step S101). For example, a variable for monitoring a set value for determining whether to update the storage data of the image stored in the viewpoint conversion image data memory 161 or the like is initialized.

Next, it is determined whether or not the vehicle speed exceeds a preset speed based on the detected value of the vehicle speed sensor 152 which functions as a vehicle speed detecting means (step S102).

Since this embodiment is used for recognizing the periphery of the vehicle, the purpose of use is to align the vehicle, put it in a garage, or the like, and when the speed is high, it is determined that it is not necessary. The set value of the speed when used for such a purpose is the moving speed when the vehicle is moved while frequently correcting the direction in order to correct the parking position of the vehicle. That is, it is the upper limit of the speed at which the rudder can be changed frequently to control the traveling direction of the vehicle.

As described above, when the vehicle is moved by frequently operating the steering wheel (rudder), it is necessary to grasp the position of an obstacle existing near the periphery of the vehicle. It is necessary to recognize the obstacles in the blind spot. Therefore, if the set speed is exceeded,
Since it can be determined that it becomes less necessary to grasp the blind spot that is a shadow of the vehicle, the process returns to step S102 again. The set speed may be changed as needed. If it does not exceed, it is determined that the image processing is necessary (to grasp the blind spot), and the process proceeds to the next step in order to perform the image processing.

FIG. 6 is a flow chart showing the procedure of image processing, and FIGS. 7 to 19 are enlarged views of the respective figures shown in FIG. FIG. 4 is a flowchart showing the processing contents of the viewpoint conversion processing (step 103). This process is a process of correcting an image captured by the camera 13 into an image expected to be displayed when the image is captured by the camera from the viewpoint position of the driver. This processing is performed when the driver's viewpoint position and the image capturing position of the camera 13 are apart from each other by a distance W1 along the moving direction of the vehicle, because the camera 13 has already passed the viewpoint position T1. Is used as the image at the viewpoint position, and the camera image is reduced and corrected based on this.

That is, the current image acquisition means for acquiring the current image, the viewpoint position image acquisition means for acquiring the camera image at the time when the camera has passed the current driver's viewpoint position, and the current image acquisition means. And a correction unit that corrects the obtained current image according to the viewpoint position image. In addition, when the camera 13 is on the traveling direction side of the driver's viewpoint position along the traveling direction of the vehicle, the image is reduced, and image processing is performed as if the camera viewpoint moved backward by the distance W1 ( It is also possible to adopt a method such as (reduction).

Specifically, the following processing is performed.
The images captured by the camera 13 (step S201) are continuously stored in the viewpoint conversion image data memory 161 of the storage device 16 together with the captured position and time (step S202). ROM for the distance W1 from the current camera position to the driver's seat (the position of the viewpoint when the driver looks backward)
The distance W calculated from the vehicle body data stored in 122 and calculated based on the position information stored together with the image.
The past image BX (FIG. 7) captured at the position corresponding to 1 (the position where the moving distance is before the distance W1) is read from the viewpoint conversion image data memory 161 (step 203).

An image BY2 (FIG. 9) obtained by detecting the vehicle body part of the currently obtained image BY1 (FIG. 8) and removing the vehicle body part
Is generated (step S204). The step S204 constitutes a current image acquisition means. At this time, the region of the vehicle body portion is judged from the data of the vehicle body in which the device of the present invention is installed, which is stored in the ROM 122. The vehicle body data is three-dimensional data of the vehicle body, the distance to each component of the vehicle, the length of the component, or the installation height (h) of the camera 13, the camera, as shown in FIG. 13 viewpoints,
It is an angle (θ) formed by a vertical line and a straight line m passing through the contour p of the rear trunk. The read image BX becomes the same image as the driver's current visual field, so the image BY2 is reduced based on the image BX (step S205).
The reduced image is set to BY2 ′ (FIG. 10), and the image BX and the image BY2 ′ are combined to generate the image B (FIG. 10) (step S206). A correction unit is configured by these steps S205 and S206.

The above steps S201 to S20
By the processing of 6, the image B viewed from the driver's viewpoint position is generated. Then, the image B combined in step S206 is stored in the image data memory 162 for combining work (step S207).

As in this viewpoint conversion processing step S103, the captured image conversion means is configured by converting the viewpoint of the captured image like the image viewed from the driver's viewpoint. Such viewpoint conversion processing (step S103)
Is continuously performed on the screen imaged by the camera 13, and the image corrected to the viewpoint position is continuously stored in the combined work image data memory 162 at the same position as the image BX and with the time of the image BY1. It It progresses to the following step after step S103.

Whether or not the vehicle has moved is determined from the detection values of various sensors stored in the current position detecting device 15 (step S104). For example, it is determined from the detection value of the vehicle speed sensor 152 that the vehicle has moved when the vehicle has reached a certain speed or higher. Alternatively, the distance sensor 155 determines that the vehicle has moved when the moving distance is equal to or longer than a predetermined distance. If it is determined that the vehicle has not moved, step S104 is repeated. This is because when the vehicle does not move, the past image cannot be obtained and the past image cannot be combined with the blind spot area of the current image.

Then, the blind spot area of the image and the moving distance (predetermined distance) necessary to obtain the past image necessary for image combination are specified as follows. In the image acquired by the camera 13, the blind spot area is generated by being hidden by the vehicle body portion, that is, the shadow of the vehicle. Therefore, the size of the vehicle body, the camera 1
The blind spot area can be uniformly specified by the numerical data recorded as the vehicle body data such as the installation position of No. 3, and if the blind spot area can be specified, an image to be combined with the blind spot area can be obtained. The distance can also be obtained. The specifying method will be described below.

As shown in FIG. 2, for example, the blind spot region b1 on the traveling direction side is determined by the viewpoint position of the camera 13 and the bonnet contour position (rear trunk contour) p, and Once the installation position is determined, the range of the blind spot area is determined. That is, as shown in FIG. 2, the angle h between the vertical line and the straight line m connecting the camera 13 with the height h of the ground surface, the viewpoint of the camera and the position where the field of view is blocked (the contour p of the rear trunk). And camera 13 to camera 1
The horizontal distance 1 to the lower limit position of the field of view of 3 and the horizontal distance x to the camera 13 from the point where the straight line m intersects the ground surface,
It is easily determined by the following formula. These data are recorded in the ROM 122 as body data,
It is read from the ROM 122. Therefore, by moving at least the distance d, it is possible to generate an image of the blind spot region b1 from the first accumulated past image. That is, the predetermined distance is equivalent to the distance d.

When it is determined that the vehicle is moving,
The image synthesized before the movement cannot be used as an image for compensating for the blind spot area on the screen immediately after the movement (until the moving distance of the vehicle reaches the distance d). Therefore, when the vehicle starts moving,
It is necessary to newly generate a (composite) image, and pseudo-transparent image processing is performed (step S105). The pseudo transparent image processing will be described below. FIG. 5 is a flowchart showing the pseudo transparent image processing. As for the current image, the image converted into the image at the viewpoint position of the driver by the viewpoint conversion process similar to steps S201 to S207 is stored in the image data memory 162 for synthesis work. That is, the distance W from the current camera position to the driver's seat
1 is calculated from the vehicle body data stored in the ROM 122, the past image captured at the position corresponding to the calculated distance W1 is taken as the image AX (FIG. 11), and the image AX is read from the viewpoint conversion image data memory 161. read out.

An image AY2 (FIG. 1) obtained by detecting a vehicle body portion of the currently acquired image AY1 (FIG. 12) and removing the vehicle body portion is detected.
3) is generated. Since the read image AX becomes an image that is substantially the same as the driver's current visual field, the image AY2 is reduced based on the image AX, and the image is AY2 '(FIG. 14).
, The image AX and the image AY2 ′ are combined, and the image A (FIG.
4) is generated. This image A is stored in the image data memory 162 for composition work together with the same position and time as the image AX. In this way, the latest image A that has undergone the viewpoint conversion processing (the image that has been finally processed and generated in the viewpoint conversion processing) is read from the image data memory 162 for combining work (step S301).

Next, the moving amount is calculated, and the image B for capturing the blind spot of the image A corresponding to the calculated moving amount is read from the image data memory 162 for synthesizing work (step S3).
02). At this time, the movement amount is determined based on the movement amount input from the distance sensor 155. In addition, the vehicle speed value detected by the vehicle speed sensor 125 may be integrated by the elapsed time. Alternatively, it may be obtained from the movement trajectory of the absolute position of the vehicle obtained from the GPS receiving device 153. The method of selecting the image B may be a method based on the movement amount as described above, or may be selected as an image at a point before the distance d from the current position.

Since the image seen from the driver's viewpoint and the image A are the same image, a part B0 of the read image B is cut out (FIG. 15). The image B0 is enlarged on the basis of the image A (so that the subject has the same size as the image A), and an image B '(FIG. 16) is generated (step S303).

The magnification for enlarging the image can be determined by the following method, for example. One is a method that uses a magnification determined based on experience from experiments. For example, the scaling factor is determined according to the moving distance while keeping the angle of view unchanged. Table 1 shows the relationship between the moving distance from the camera viewpoint and the magnification.

[0047]

[Table 1] Alternatively, the image A (current image) and the image B '
A characteristic area is set for each of (past images). In this area, a change in brightness and color in the image is remarkable, and an area in which the target is easily specified is selected. The image set in the characteristic area needs to have the same object as the image A and the image B ′. And
Image correction is performed so that the sizes of the objects displayed in this characteristic area are the same.

As still another method, based on a fixed value such as a focal length of a camera or an assumed distance to a subject,
It determines the magnification. For example, when the moving distance of the camera is D and the fixed value is L, the magnification is {(D + L) / L}. The vehicle body image area acquisition unit is configured by the above steps S302 and S303.

After step S303, based on the vehicle body data stored in the ROM 122, the portion of the image A in which the vehicle body is a blind spot is specified. In this way, the body region specifying unit of the present invention is configured by specifying the portion where the vehicle body is reflected (the region hidden by the vehicle body) in the captured image.

Then, the image A'excluding the specified part
(FIG. 17) is generated (step S304). Image A '
And the image B ′ are combined, and the pseudo transparent image A ′ + B ′ (see FIG.
8) is generated (step S305). The pseudo-transparent image A ′ + B ′ is in a state in which the corresponding portion of the image B ′ is attached to the vehicle body portion cut out from the image A. The generated pseudo transparent image is temporarily stored in the pseudo transparent image data memory 163 (step S30).
6). As described above, the pseudo transparent image processing step S105
Ends and the process returns to the main routine to proceed to the next step. An image (pseudo-transparent image) is obtained by specifying the vehicle body region portion in the image that has undergone the viewpoint conversion by the captured image conversion device by the vehicle body region specification device and replacing the vehicle body region with the image region (image B ′) that has undergone the viewpoint conversion by the image conversion device. (Image A '+ B')
The image creating means for creating
It is composed of S305.

The pseudo transparent image A '+ B' is read from the pseudo transparent image data memory 163, the contour line 6 of the portion symbolizing the vehicle as seen from the driver's viewpoint is synthesized, and the final synthesized image (FIG. 19) is generated. (Step S106). This contour line may be a contour line of a portion that symbolizes the vehicle body, and includes, for example, at least one of the shapes of the vehicle body, bumper, light, wiper, instrument panel, steering wheel, mirror, tire, seat, and window. It is a contour line.

Particularly, the contour line of the outermost portion of the vehicle body or the contour line of the portion visually recognized at the same time as the external scenery when the driver looks at the outer side of the vehicle body is preferable. The parts located on the outermost side of the vehicle body include the car body, bumpers, lights, tires, side mirrors, and the like.
Instrument panel, steering wheel, rearview mirror,
Examples include seat and window shapes. By synthesizing the contour lines of the outermost portion of the vehicle body, it is possible to easily grasp the distance between the vehicle body and the external obstacle. In addition, by synthesizing the outlines of the parts recognized in the field of view, an image similar to the field of view when recognizing and operating with the naked eye can be obtained, so that the driver can feel the external obstacle with the same sensation as the conventionally learned sensation. It is possible to grasp the positional relationship with the vehicle. The vehicle body shape image synthesizing unit is configured by the above step S106. The contour line data is created based on the vehicle body data in which the device of the present invention is installed, and is stored in advance in the storage device 16 or the ROM 1.
22 and the like.

The final composite image generated in step S106 is output to the image display device 14 (step S10).
7). It is determined whether there is an opportunity to terminate the system (step S108). For example, when the image processing apparatus of the present invention is switched off or the display of the image apparatus 14 is switched, the system is stopped. If the ignition switch is turned off, it means that the vehicle is parked, and the system is similarly stopped.

Further, when the vehicle speed exceeds a predetermined speed, it means that the vehicle is in a normal running state, and it is not necessary to recognize a blind spot area near the vehicle such as when entering a garage. Is stopped. Therefore, in the case of Yes in step S108, the flowchart of this image processing ends (step S111).

When there is no trigger to terminate the system (step S108: No), step S101
It is determined whether or not the variable initialized in step S4 exceeds the set value (step S109). These variables represent distance, time, remaining memory, etc.When these values reach the preset values, the old data is returned to the position where the data writing was started. It is rewritten with new image data (step S110). Here, step S101
The variable for monitoring the set value is initialized. If the variable does not exceed the set value in step S109, the process returns to step S102.

The operation described above is for the case where the camera 13 is provided at a position away from the driver's viewpoint position, but the camera 13 may be provided near the viewpoint position. In this case, in the flowchart shown in FIG. 3, the processing except the viewpoint conversion processing in step S103 is performed. In other words, there is no need to change the viewpoint,
No viewpoint conversion processing is performed. FIG. 20 is a flowchart showing the procedure of image processing when the viewpoint conversion processing is not performed.
The order of processing operations will be described later with reference to FIG.

(A) The image picked up by the camera 13 as the image pickup means is stored in the storage device 16 together with the picked-up position and time.
Are continuously stored in the viewpoint conversion image data memory 161. Then, the current image A is read from the viewpoint conversion image data memory 161.

(B) Next, in image A, there is a blind spot area (vehicle body area) that is blocked by a part of the vehicle, and this blind spot area is shown in the image taken before the vehicle retreats. It was what was there. Therefore, the predetermined distance d at which an image capable of acquiring the blind spot area is obtained is calculated. Predetermined distance d
As shown in FIG. 2, for example, the blind spot region b1 on the traveling direction side is determined by the viewpoint position of the camera 13 and the bonnet contour position (rear trunk contour) p, and the installation position of the camera 13 is determined. Is determined, the range of the blind spot area is determined. That is, as shown in FIG. 2, the angle h between the vertical line and the straight line m connecting the camera 13 with the height h of the ground surface, the viewpoint of the camera and the position where the field of view is blocked (the contour p of the rear trunk). Then, the horizontal distance 1 from the camera 13 to the lower limit position of the field of view of the camera 13 and the horizontal distance x from the point where the straight line m intersects the ground surface to the camera 13 are easily determined by the following formula. These data are recorded in the ROM 122 as vehicle body data.
It is read from 2.

D = h × tan θ−1 Therefore, if the distance d is moved at least, the image of the blind spot area b1 can be generated from the first accumulated past image. That is, the predetermined distance is equivalent to the distance d.

(C) The image B from which the blind spot area of the image A can be obtained is read from the viewpoint conversion image data memory 161.
In the method of selecting the image B, the image corresponding to the moving distance d calculated in (b) is read based on the moving amount input from the distance sensor 155. In addition, the vehicle speed sensor 125
The vehicle speed value detected by may be obtained by integrating with the elapsed time.
Alternatively, it may be obtained from the movement trajectory of the absolute position of the vehicle obtained from the GPS receiving device 153.

In addition to the method based on the amount of movement as described above, an image at a point a distance d before the current position from the position stored with the image may be selected.

(D) For image A, which is the current image,
Based on the vehicle body data stored in ROM122, image A
The body area (blind spot area) in which the vehicle body of is reflected is specified.

As in this process (d), the body region specifying means of the present invention is constituted by specifying the portion where the vehicle body is reflected (the region hidden by the vehicle body) in the captured image.

(E) Then, an image A'excluding the specified vehicle body region is generated. (F) On the other hand, with regard to the read image B, the scenery seen from the driver's viewpoint and the image A are the same image, so the image B0 of the portion corresponding to the image A is cut out from the image B. .

(G) The image B0 is enlarged on the basis of the image A which is the same as the scenery seen from the driver's viewpoint (so that the subject has the same size as the image A), and an image B'is generated. . Similar to the method described in the above embodiment, the magnification when enlarging an image uses the magnification determined based on experience through experiments, etc., and sets the characteristic area of each image and displays it in the characteristic area. The image is corrected so that the size of the object becomes the same, the magnification is determined based on a fixed value such as the focal length of the camera or the assumed distance to the subject.

The enlarged image B'includes an image region that is supposed to be visible if the vehicle body region in the image A is transparent, and is substantially the same as the image of the image A in other regions. . Therefore, extract the B0 part of image B,
Corresponds to the vehicle body region that is assumed to be visible if the specified vehicle body portion is transparent, as in the processing of enlarging the image B0 according to the size of the image A (processing (f) (g)) By acquiring the image area to be processed, the vehicle body image area acquiring means of the present invention is configured.

(H) Then, the image A'and the image B'are combined. The pseudo-transparent image A ′ + B ′ is in a state in which the corresponding portion of the image B ′ is replaced with the vehicle body region cut out from the image A. As in this combination processing (processing (h)), the image forming means of the present invention is configured by replacing the vehicle body area in the captured image with the image area acquired by the vehicle body image area acquiring means to create an image. To be done. The generated pseudo transparent image has the same position and time as the image A and the pseudo transparent image data memory 163.
Will be remembered once.

(I) Next, the pseudo transparent image data memory 1
The pseudo transmission image A ′ + B ′ is read from 63, the contour line 6 of the portion symbolizing the vehicle viewed from the driver's viewpoint is synthesized, and the final synthesized image is generated. This processing (i) constitutes a vehicle body shape image synthesizing means.

The contour line may be a contour line of a portion symbolizing the vehicle body, and examples thereof include a vehicle body, a bumper, a light, a wiper, an instrument panel, a steering wheel, a mirror,
It is a contour line including at least one of a shape of a tire, a seat, and a window. In particular, it may be the contour line of the outermost portion of the vehicle body, or the contour line of the portion that is visually recognized at the same time as the external scenery when the driver looks at the outer side of the vehicle body. The parts located on the outermost side of the vehicle body include the car body, bumpers, lights, tires, side mirrors, and the like.
Instrument panel, steering wheel, rearview mirror,
Examples include seat and window shapes. By synthesizing the contour lines of the outermost portion of the vehicle body, it is possible to easily grasp the distance between the vehicle body and the external obstacle. In addition, by synthesizing the outlines of the parts recognized in the field of view, an image similar to the field of view when recognizing and operating with the naked eye can be obtained, so that the driver can feel the external obstacle with the same sensation as the conventionally learned sensation. It is possible to grasp the positional relationship with the vehicle. This contour line data is created based on vehicle body data in which the device of the present invention is installed, and is stored in advance in the storage device 16, the ROM 122, or the like.

The contour line of the vehicle body to be combined in this case may be detected from the photographed image A. In this case, the actual vehicle condition is displayed in the image,
The relationship between the external image and the vehicle body can be grasped more realistically. For example, when moving backward, a figurine or the like placed on the dashboard of the rear window is displayed in the image, and can be brought closer to the image actually seen by the naked eye.

A known image processing method can be used as the contour line detection method. For example, the Sobel filter process or the Laplacian filter process is used to emphasize the contour, and only the contour line is extracted. You can Further, the configuration is not limited to the configuration in which only the contour line is combined, and an image showing the vehicle body or the inside of the vehicle (an image of a portion that generates a blind spot) may be semitransparent and combined. The process of detecting such a contour line 6 is performed, for example, in steps S105 and S1.
It is performed between 06 and functions as a contour line detection means.

Further, the color of the contour line to be combined in this embodiment can be automatically adjusted so that the driver can easily discriminate it. For example, if the created image has a light color, a black outline, and if the image has a dark color,
Synthesize the white outline. Also, the pixel 1 of the created image
For each one, the color is inverted (complementary color) and the outline is combined.

Next, the camera 13 can use a wide-angle lens. FIG. 21 shows the range of the field of view of the camera when a wide-angle lens is used. When a wide-angle lens is used, a large distortion occurs in the image due to the lens characteristic. Therefore, the step of performing the normal image conversion process for correcting the distortion based on the lens characteristic is the above step S206.
And step S207.

Further, the area to be cut out as an external image can also be configured to be changed according to the steering angle detected by the steering angle sensor 151. For example, when a steering angle of α degrees is detected to the right, the position where the image is cut is moved from E1 to E2 (to the right) according to the angle.

The operation described above is an example in the case of moving backward, but the same effect is obtained even when moving forward. When moving forward, it is preferable that the camera is located at the substantially central portion of the dashboard or the position of the base of the rearview mirror. If a distance measuring sensor that measures the distance between the external obstacle and the vehicle body by ultrasonic waves or the like is provided on the vehicle body and the distance between the vehicle and the external obstacle is measured together with the display of the composite image, a further sense of distance can be obtained. It is easy to grasp and is preferable. For example, if the distance to the obstacle detected by the ultrasonic sensor is less than or equal to a predetermined distance, an alarm is issued as an auditory signal,
For example, the screen may be displayed in a red silhouette as a visual signal. Further, when the camera 13 is provided closer to the driver's viewpoint position, the display image becomes an image closer to the driver's field of view, and the positional relationship between the surrounding obstacles and the vehicle can be further grasped from the image. It will be easier.

[0076]

According to the first aspect of the present invention, a blind spot region that cannot be seen by the driver is generated, so that anxiety about the blind spot can be suppressed. According to the invention described in claim 2, the display image is the same image as the scenery seen from the viewpoint position of the driver, and the positional relationship between the obstacles around the vehicle and the vehicle is sensuous as in the case of being visually recognized. It will be easier to understand.

According to the third aspect of the invention, when the distance between the image pickup means and the viewpoint position of the driver becomes the same value as the moving distance of the vehicle, the viewpoint conversion is performed using the image at the driver position. Therefore, it is possible to generate the same image as the scenery seen from the driver's viewpoint position. According to the invention described in claim 4, since the blind spot area is synthesized using the past image, it is not necessary to provide a dedicated camera for photographing the blind spot.

According to the fifth aspect of the present invention, the vehicle speed can be detected by the vehicle speed detecting means and the operation state in which the blind spot area around the vehicle needs to be grasped can be detected. Therefore, the necessary image processing is performed only in the necessary state. It can be performed. Claim 6
According to the invention described in (1), since the moving distance detecting means for detecting that the past image has moved the distance that can be acquired is provided, the process of acquiring the image area corresponding to the vehicle body area (blind spot area) is accurately performed. It becomes possible.

According to the seventh aspect of the present invention, the distance that allows the past image to be acquired is calculated based on the body data recorded in advance. Therefore, the distance calculation can be facilitated and the distance calculation process can be speeded up. it can. According to the invention described in claim 8, since the vehicle body region is specified by the previously recorded vehicle body data, the processing time of the image processing for specifying the vehicle body region can be shortened. According to the invention described in claim 9, by synthesizing the contour lines of the shape of each part of the vehicle body with the image, it is possible to further easily grasp the positional relationship between the external environment displayed on the screen and the vehicle body.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an image processing apparatus of the present invention.

FIG. 2 is a side view of a vehicle equipped with the image processing apparatus of the present invention.

FIG. 3 is a flowchart showing the operation of the image processing apparatus of the present invention.

FIG. 4 is a flowchart showing the operation of the image processing apparatus of the present invention.

FIG. 5 is a flowchart showing the operation of the image processing apparatus of the present invention.

FIG. 6 is a flowchart showing a procedure of image processing.

7 is an enlarged view of the processed image shown in FIG.

FIG. 8 is an enlarged view of the processed image shown in FIG.

9 is an enlarged view of the processed image shown in FIG.

10 is an enlarged view of the processed image shown in FIG.

11 is an enlarged view of the processed image shown in FIG.

12 is an enlarged view of the processed image shown in FIG.

FIG. 13 is an enlarged view of the processed image shown in FIG.

14 is an enlarged view of the processed image shown in FIG.

FIG. 15 is an enlarged view of the processed image shown in FIG.

16 is an enlarged view of the processed image shown in FIG.

17 is an enlarged view of the processed image shown in FIG.

18 is an enlarged view of the processed image shown in FIG.

19 is an enlarged view of the processed image shown in FIG.

FIG. 20 is a flowchart showing another image processing procedure.

FIG. 21 is a plan view showing the field of view of a camera using a wide-angle lens camera.

[Explanation of symbols]

1 Image processing device 12 Processor 13 cameras 14 Image display device 15 Current position detector 16 storage

Claims (9)

[Claims] 1. An image pickup means for picking up an image in a traveling direction of a vehicle, a vehicle body area specifying means for specifying a vehicle body area hidden by a vehicle body portion in the image picked up by the image pickup means, and the vehicle body area specifying means. A vehicle body image area acquiring unit that acquires an image region corresponding to the vehicle body region that is assumed to be visible if the vehicle body portion specified by the means is transparent; and the vehicle body region in the image captured by the image capturing unit. An image processing apparatus for a vehicle, comprising: an image creating means for creating an image by replacing the image area acquired by the vehicle body image area acquiring means; and a display means for displaying the image created by the image creating means. 2. An image pickup means for picking up an image of a traveling direction of a vehicle, a picked-up image conversion means for changing a viewpoint of a picked-up image such as an image seen from a driver's viewpoint, and an image picked up by the image pickup means. Body image specifying means for specifying a vehicle body area hidden by the vehicle body portion, and a vehicle body image for acquiring an image area corresponding to the vehicle body area which is supposed to be visible if the vehicle body portion is transparent. In the area acquisition means and the image converted in viewpoint by the captured image conversion means,
An image creating means for creating an image by replacing the vehicle body area part specified by the vehicle body area specifying means with the image area acquired by the vehicle body image area acquiring means, and displaying the image created by the image creating means. An image processing apparatus for a vehicle, comprising: a display unit. 3. A moving distance detecting means for detecting a moving distance of a vehicle, wherein the captured image converting means detects a distance from a position of a driver to a position where the image capturing means is installed, and the moving distance detecting means. The image processing device for a vehicle according to claim 2, wherein the viewpoint of the picked-up image is converted when the vehicle movement distance from the position detected by the image pickup means is equal to that detected by the means. 4. The vehicle according to claim 1, wherein the image area assumed to be visible if the vehicle body is transparent uses an image area captured in the past by the imaging means. Image processing device. 5. A vehicle speed detecting means for detecting the speed of the vehicle, wherein the image created by the image creating means is displayed when the speed detected by the vehicle speed detecting means is less than or equal to a predetermined speed. The image processing device for a vehicle according to claim 1 or 2. 6. A moving distance detecting means for detecting a moving distance of the vehicle, wherein the vehicle body image area acquiring means is an image corresponding to the vehicle body area when the detected distance is a predetermined distance or more. The image processing apparatus for a vehicle according to claim 1, wherein the area is acquired. 7. Vehicle body data storage means for storing vehicle body data of the vehicle, wherein the predetermined distance is determined based on data stored in the vehicle body data storage means. The image processing device for a vehicle according to. 8. Vehicle body data storage means for storing vehicle body data, said vehicle body area specifying means, based on said vehicle body data,
The vehicle image processing apparatus according to claim 1, wherein a vehicle body region hidden by the vehicle body portion is specified. 9. A vehicle body, a bumper, a light, a wiper, an instrument panel, a handle, a mirror, a tire,
A vehicle body shape image synthesizing unit configured to further add a contour line of a vehicle body shape including at least one of seat and window shapes to the image created by the image creating unit to create a composite image. The image processing device for a vehicle according to 1 or 2.