JP3372944B2 - Monitoring system - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing technique for generating a composite image by using captured images of a plurality of cameras installed in a vehicle, and particularly to assist in safety confirmation when driving a vehicle. It belongs to the technology effective for the monitoring system used.

[0002]

2. Description of the Related Art Conventionally, as a device for monitoring the surroundings of a vehicle using a camera, there is known a system in which the camera is installed rearward in a rear trunk portion of the vehicle and the image obtained from the camera is presented to a driver. Has been. by this,
The driver can see what is behind the vehicle without looking at the mirror. In addition, recently, a system that not only simply presents a camera image but also presents a tire movement track on the camera image (this is called "conventional method 1").
Is also becoming known. As a result, the driver can grasp the state behind the vehicle and also predict the destination of the vehicle.

However, in the conventional method 1, although the rear side of the vehicle can be seen, it is necessary to rely on the visual check and the mirror for other portions. On the other hand, as a system for informing the driver of the positional relationship between the own vehicle and the entire surroundings in an objective and easy-to-understand manner, Reference 1: Japanese Patent Laid-Open No. 11-78.
The device described in Japanese Patent No. 692 is available. In this document 1, a plurality of (eight) cameras for providing video for a vehicle are attached around the vehicle, and images obtained by transforming the images captured by the cameras (hereinafter referred to as “partial images”) are combined. Then, a method of obtaining a new image capable of looking around the surroundings of the vehicle at once (this is referred to as "conventional method 2") is disclosed. In particular, when deforming and arranging images captured by a plurality of cameras, it is easy to grasp the approximate positional relationship between the vehicle and surrounding objects by maintaining continuity at the boundary between adjacent partial images. The image can be provided to the driver.

[0004]

By the way, in various driving situations, when the driver is presented with an image synthesized by the above-mentioned conventional method, the following problems occur.

1. Problems of Conventional Method 1 and Problems to be Solved In Conventional Method 1, since the camera image shows only a limited angle region of the rear portion of the vehicle, the front portion of the rear portion of the vehicle is The degree of contact between a part, for example, the left and right bodies of the vehicle and surrounding objects can be determined only by the door mirror. Therefore, in order to grasp both the degree of contact and the state of the rear of the portion, the driver is required to use the display image and the door mirror together, which imposes a heavy burden on the driver.

Further, the conventional method 1 displays the state of a limited field of view behind the vehicle. Therefore, it is not suitable for use in a driving scene where it is necessary to know a wide range of situations. For example, when the vehicle is guided to the road by turning the steering wheel largely while moving backward as shown in FIG. 4, in order to confirm the approaching vehicle from the left and right of the road, the image including the rear view of substantially 180 degrees is driven. Although it is necessary to present the image to the person, the image presented by the conventional method 1 has a narrow field of view and cannot be used for such an application at all.

Therefore, the first problem to be solved is to provide the driver with a composite image in which the driver can drive the car with a sense of security just by looking at the case as shown in FIG.

2. Problems of Conventional Method 2 and Problems to be Solved-First Problem: Display Range In Conventional Method 2, the surroundings are looked down from above as if they were in order to present the positional relationship in the immediate surroundings of the vehicle in an easy-to-understand manner. It only discloses presenting such images. However, regarding the method of presenting images so that both the vicinity of the vehicle and other parts can be grasped at the same time,
No disclosure is made. Therefore, the following example (a),
In the driving situation as shown in (b), the presentation of the composite image according to the conventional method 2 cannot be said to be sufficient to support driving.

(A) Width adjustment to the road shoulder, etc. When performing width adjustment to the road shoulder or the like, the following two points are required for the image displayed to the driver. In other words, how close the road is to the shoulder, and whether there is an obstacle, such as a telephone pole or ticket machine, ahead of the direction of travel when moving forward or backward as it is, and whether an oncoming vehicle is coming. The question is whether or not they should be contacted when they are confirmed.

However, since the viewable area is limited in the view-down image, it is necessary to obtain an image of only the immediate surroundings in order to look closely at the condition of approaching to the road shoulder. In order to see, you need an image that can be seen at least several meters to a dozen meters ahead. In any case, one image cannot satisfy both requirements.

Therefore, the second problem to be solved is not only how the vehicle approaches the shoulder of the road in the case of approaching the shoulder of the road, but also the state of the forward direction when the vehicle moves forward or backward. Is to provide the driver with a composite image that can be seen at the same time.

(B) In the case of presenting the looking-down image for parking including the turning back,
It is necessary to confirm the state of the rear when the vehicle slowly moves backward in the first stage of parking operation. However, since the viewable area is limited in the looking-down image, the mirror must be used for rearward confirmation when the vehicle moves backward, and the driver's burden increases. In addition, in the case of frequently performing a turning operation (an operation of guiding the vehicle to a desired place by a fine movement forward and backward) during parking, not only backward confirmation but also forward confirmation is required.

Therefore, the third problem to be solved is to drive a composite image so that not only the surrounding conditions but also the forward direction when the vehicle moves forward or backward can be understood at the same time when the vehicle is turned back. It is to provide to the person.

Second problem: Disappearance of an object Disappearance of an object is a three-dimensional object which is deformed in accordance with the deformation in order to deform the image so that at least the road surface portion is continuous at the boundary portion. Is deleted due to the layout of the partial image.

FIG. 17 (a) shows a pole P1 standing at the rear right corner of the vehicle. Assuming that the pole P1 is photographed by both the camera 2 and the camera 3 shown in FIG. 2, the pole P1A in the photographed image of the camera 2 is distorted by image deformation so as to extend downward with respect to the grounding point of the pole P1. On the other hand, the pole P1 in the image captured by the camera 3
B is distorted so as to extend rightward with respect to the grounding point of the pole P1 due to image deformation. By the way, each pole P1
Since both A and P1B extend to the area not used as a partial image, the extended portion is deleted, and as a result, only the portion where the pole P1 contacts the ground remains on the composite image (FIG. 17 (b)). . Therefore, in the composite image, it is difficult to recognize the pole that actually exists. This is the cause of the disappearance problem.

Therefore, a fourth problem to be solved is to avoid the above-mentioned problem, and to obtain an approximate position relationship between the vehicle and surrounding objects and a distance between the surrounding objects at the same time so that the composite image can be grasped. Is provided to the driver.

Third problem: boundary discontinuity In the conventional method 2, each camera image is deformed so as to maintain continuity at the boundary between adjacent partial images, but all parts of the boundary are distorted. It is theoretically difficult to maintain continuity at. Therefore, the present method discloses that at least the road surface portion is deformed so as to maintain continuity on the road surface portion so that it can be grasped without a feeling of strangeness. However, this causes the object other than the road surface to be greatly distorted so as to be stretched, and this causes the object other than the road surface located at the boundary portion of the partial image to be discontinuous at the boundary,
Causes the problem. For example, in FIG. 18, the parked car located at the boundary between the partial images obtained by transforming the images of the cameras 2 and 3 becomes discontinuous at the boundary, and as a result An example is shown in which the image is clear. Therefore, in such an unclear image, the distance to the surrounding objects cannot be grasped, and the driver is eventually forced to look at the mirror or the like to confirm it, which increases the burden.

Therefore, a fifth problem to be solved is to avoid the above problem and provide the driver with a synthetic image which at least includes information necessary for safe driving in the camera image.

Further, in view of the above-mentioned problem of discontinuity at the boundary between the partial images, where to bring the boundary in the composite image in various driving scenes is greatly related to the usefulness of the composite image. come. Generally, in many complicated driving operations, such as parking (parallel or parallel) or side-by-side driving, the place where the driver wants to know the detailed situation is 180 degrees to the left and right of the vehicle rather than the front and rear. It is known from the experiments conducted by the inventors about 10 subjects that the present invention is concentrated.

Therefore, it is considered that the sixth problem is to present a combined image in which the left and right sides of the vehicle are more continuous, but the conventional method 2 discloses nothing about the solution to such problem. Not not.

[0021]

[Means for Solving the Problems] In order to solve the above-mentioned problems, a solution means provided by the invention of claim 1 is, as a surveillance system, at least one camera for photographing the surroundings of a vehicle, and the photographing of said camera. An image processing unit that receives an image, generates a composite image from the camera image, and causes a display device to display the composite image, the image processing unit including scaling of a portion near a vehicle including a grounding portion of at least one tire of the vehicle. It has a mode for displaying a combined image so that the ratio is relatively higher than the expansion / contraction ratio of the peripheral portion of the vehicle and the portion along the side surface of the vehicle has linearity. .

According to the first aspect of the present invention, the portion of the vehicle in the immediate vicinity of the vehicle including the ground contact portion of at least one tire is
A composite image with a relatively high scaling ratio is displayed as compared with the surrounding area of the vehicle. Therefore, the user can simultaneously check the detailed situation immediately under the vehicle and the state ahead of the traveling direction from the same image.

Further, in order to solve the above-mentioned third problem, the solution means taken by the invention of claim 2 is, as a monitoring system, at least one camera for photographing the surroundings of a vehicle, and a photographed image of said camera. And an image processing unit for generating an image viewed from a virtual viewpoint from this camera image and displaying the image on a display device, wherein the image processing unit is an image in a moving direction of a vehicle or a movable direction. It has a mode of displaying side by side with the virtual viewpoint image while maintaining the positional relationship with the virtual viewpoint image.

According to the second aspect of the present invention, the images in the moving direction or the movable direction of the vehicle are displayed side by side while maintaining the positional relationship with the virtual viewpoint image. Therefore, the user can simultaneously check not only the situation around the vehicle but also the state in the traveling direction when moving forward or backward with the same image.

Further, in order to solve the above-mentioned fourth problem, the solution means taken by the invention of claim 3 is, as a monitoring system, a plurality of cameras for photographing the surroundings of a vehicle, and images taken by the respective cameras. And an image processing unit that generates an image viewed from a virtual viewpoint from these camera images and causes the display device to display the image, and the plurality of cameras include first and second cameras having overlapping shooting ranges. Including the first virtual viewpoint image created using the first camera and not using the second camera; and the second virtual viewpoint image.
And a second virtual viewpoint image created without using the first camera.

According to the third aspect of the present invention, for the first and second cameras whose photographing ranges overlap, a plurality of virtual viewpoint images created using one camera and not the other camera are generated. It will be possible. That is, since each virtual viewpoint image includes both the overlapping portion and the non-overlapping portion with the other camera in the shooting range of the one camera used for the creation, the object existing in the overlapping portion is the virtual viewpoint image. It never disappears above. Therefore, it is possible to avoid the problem of object disappearance in the overlapping portion of the shooting ranges, which occurs due to the processing of the camera images performed to maintain the continuity of the virtual viewpoint images. Therefore, it becomes possible to provide the driver with an image including information necessary for safe driving.

According to the invention of claim 4 , the image processing unit in the surveillance system of claim 3 has a mode for displaying the first and second virtual viewpoint images side by side on the same screen.

In the invention of claim 5 , the first virtual viewpoint image in the monitoring system of claim 3 is a virtual viewpoint image using only images captured by a camera installed on the side of the vehicle, It is assumed that the second virtual viewpoint image is a virtual viewpoint image using only images captured by cameras installed in front of and behind the vehicle.

[0029]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings. First, the overall configuration of the monitoring system according to the present invention, which is necessary to implement each embodiment, will be described, and then various examples of various display screens according to the present invention will be described in detail.

FIG. 1 is a block diagram showing the structure of a monitoring system according to the present invention. In the monitoring system shown in FIG. 1, the image processing unit 20 receives a plurality of camera images output from the camera 11, transforms and combines these, and generates a new image. This composite image is displayed by the display device 30. The image processing unit 20 constitutes an image processing apparatus according to the present invention.

In the present specification, the "composite image" is not limited to an image generated from a plurality of camera images,
An image generated by transforming and combining one camera image is also included.

The display device 30 of the present invention is typically a liquid crystal display, but other display devices such as a plasma display can also be used. The display of the present invention may be shared with a vehicle-mounted GPS terminal display (so-called car navigation system display), or may be prepared separately.

The image pickup section 10 is typically a CCD or CMO.
It is a color or monochrome digital camera having a solid-state image sensor such as an S device. Further, the photographing means is composed of, for example, a combination of a lens and a prism or a mirror, and the incident light on the lens, the prism or the mirror is transmitted to the image pickup element located at a position distant from the photographing means through a predetermined optical path. It may be configured as follows.

FIG. 2 shows an example of the arrangement of the cameras, and FIG. 3 shows an example of the image picked up by each camera shown in FIG. In FIG. 2, WL
1 to WL5 are white lines, and P1 to P4 are poles. The example of FIG. 2 shows a case where two pair cameras, that is, a front grill, left and right door mirrors, and a rear trunk, are used at a total of four locations, that is, eight cameras. The camera arrangement shown in FIG. 2 is for the purpose of obtaining an image of 180-degree field of view on each of the four front, rear, left, and right sides of the car. The two cameras are paired to obtain a field of view of substantially 180 degrees.

The image processing unit 20 transforms and synthesizes eight camera images (four camera images when one pair camera is used) as shown in FIG. 3, and looks down from above the vehicle vertically downward, for example. To generate a composite image like this. In order to generate a composite image, image modification processing and composition processing of partial images obtained by cutting necessary parts of the modified image (including processing such as boundary smoothing (hereinafter, boundary processing)) are required. However, in the configuration of FIG. 1, the mapping table reference unit 21 is provided, and the mapping table MPT is used in order to process the captured image in one step.

The image processing section 20 inputs photographed images from the cameras 1 to N and processes these photographed images. The processing here is image transformation cutout processing, and combining processing of cutout partial images (including boundary processing). The treatments with these may be performed separately, or all or part of them may be performed in one step. In FIG. 1, a mapping table is provided in order to process a captured image in one step.

The "mapping table" is a table in which the correspondence between the pixels of the composite image and the pixel data of each camera image is described, and is used to perform the process of generating the composite image at high speed. A desired composite image can be generated at high speed by creating such a mapping table in advance by calculation using geometric transformation or the like or by manual work. Then, the screen display mode can be switched by changing the mapping table.

More specifically, the mapping table is, for example, a ROM (such as an EEPROM, which is a writable / erasable RO).
(Including M) or stored in RAM. The mapping table is stored, for example, by storing the mapping data calculated by the processor in the image processing unit in the ROM or RA.
It may be realized by writing in M, or mapping table data provided as firmware,
The data may be written in the RAM or the ROM by using a data transfer means such as a communication line or a disk drive.

Next, examples of various display screen modes according to the present invention will be described in detail.

(First Embodiment) The first embodiment is an aspect of the invention made to solve the above-mentioned first problem. For example, as shown in FIG. 4, the handle is turned largely while retreating. When a vehicle is guided to a road by means of this, it is used to confirm approaching vehicles from the left and right of the road.

Next, an example of the display screen mode will be described.

FIG. 5A is an example of a screen mode according to the present embodiment for use in confirmation. Specifically, the images taken by the cameras 3 and 7 shown in FIG. This is an example in which an image having a field of view of 180 degrees to the left and right is inverted and displayed, and a traveling locus 41 when the own vehicle retreats is displayed in an overlapping manner (the photographed image that is the basis of the synthesis is not shown). As can be seen from FIG. 5 (a), in a situation where the rear end of the vehicle is barely facing the road passing vertically through the vehicle, from one end of the road to the other end, that is, 180 degrees left and right. Can be seen. Therefore, the driver can understand the state of the other vehicle 42 approaching from the left side, for example, while staying in the driver's seat by looking at this image.

FIG. 5B is another example of the screen mode according to the present embodiment. This mode may be used when the distance from a distant vehicle to an approaching vehicle is difficult to understand due to a low resolution of a camera or a screen. Specifically, the distant area (broken line rectangular area 43) is enlarged and displayed as a sub-window (solid line rectangular area 44) in another place on the screen. As a result, it is possible to grasp the distance and the vicinity in the entire image at the same time, and it is possible to confirm the state of the distance from the sub window. Since the sub-window is pasted on the whole image, it is advisable to decide the pasting place in advance in order to prevent information important for driving from being hidden by this as much as possible.

FIG. 5C is another example of the screen mode according to this embodiment. In this mode, when it is difficult to understand the distance from a distant vehicle to an approaching vehicle, an auxiliary line indicating the distance is displayed overlapping the image. As a result, the approximate distance of the approaching vehicle can be known. In FIG. 5C, a solid line 45, a fine broken line 46, and a rough broken line 47 indicate positions of about 2 m, about 5 m, and about 10 m to the left and right of the own vehicle, respectively.

Next, a method of creating an image having such a field of view of 180 degrees will be briefly described.

FIG. 6 shows a method of obtaining a panoramic image with a field of view of 180 degrees by projecting the images of the pair cameras on a cylindrical model and arranging the projected images as they are on the rectangular image plane from the end. In this method, since the vertical width corresponds to the angle in the visual field range of the camera, an image of the same magnification can be obtained in any direction of the camera.

FIG. 7 shows a method of obtaining an image with a field of view of 180 degrees by using a bowl-shaped model as a model other than a cylinder. The point of projecting the camera image on the model is the same as that of FIG. 6, except that a virtual viewpoint for viewing the projected image is provided and the image is created by perspective projection conversion. In this model, by deforming the shape of the bowl or moving the position of the virtual viewpoint, you can create an image in which the direction directly behind the vehicle is enlarged and the peripheral direction is reduced, or conversely, the direction directly behind the vehicle is reduced. It is also possible to create an image in which the peripheral direction is enlarged.

In the present embodiment, an example of displaying a field of view of 180 degrees rearward is shown. However, if there is practically no problem in having a field of view of about 180 degrees, the camera should always include 180 degrees. Need not be installed, and a slightly smaller viewing angle does not cause any problem. And of course,
Parts that are unnecessary for rearward safety confirmation, such as the part where the sky is visible, may be cut off and displayed.

(Second Embodiment) The second embodiment is an aspect of the invention made to solve the above-mentioned second problem. For example, in the case of width adjustment to a road shoulder, The present invention provides the driver with a composite image that shows not only how close the vehicle is to the road shoulder but also the state of the forward direction when the vehicle moves forward or backward.

Next, an example of the display screen mode will be described.

FIGS. 8A and 8B show screen modes according to the present embodiment, which are created by using the four cameras 1, 2, 5, 6 shown in FIG. 2 (composite). The image taken as the source is not shown). The image on the right side of the host vehicle is made by using the cameras 1 and 2, and the image on the left side is made by using the images of the cameras 5 and 6, respectively. The respective synthetic images are synthesized such that the enlargement / reduction rate of the vehicle immediate portion including the ground contact portion of the tire is relatively higher than the enlargement / reduction rate of the vehicle peripheral portion. Further, the scaling ratio decreases as the distance from the vehicle nearest portion to the vehicle peripheral portion increases.

An example of such an image synthesizing method will be briefly described. One effective method for easily creating such an image composition is a method of composition using a model, similar to the image composition in the first embodiment. For example, by using the bowl model shown in FIG. 7, the image on the left side of FIG. 8A can be synthesized as follows. 1. The bowl model is created so that the bottom of the bowl model is in contact with the road surface at a position between the camera 5 and the camera 6 of the car. At this time, the size of the bowl model should be such that the car is completely surrounded by the inner surface of the bowl. 2. The images taken by the cameras 5 and 6 are projected on the inner surface of the bowl, respectively. 3. 1. The virtual viewpoint is installed downward (upwardly, for example, 4 m) between the cameras 5 and 6, and 2. The images of the cameras 5 and 6 projected on the bowl by the processing of 1 are viewed from the virtual viewpoint.

By the processing of these three steps, the enlargement / reduction rate of the portion near the vehicle including the ground contact portion of the tire as shown on the left side of FIG. 8A is relatively higher than the enlargement / reduction rate of the peripheral portion of the vehicle. It is possible to obtain an image synthesized as

Here, the reason why the peripheral portion is further reduced when the bowl model is used is that the model plane on which the camera image is projected (the inner surface of the bowl in this example) is the direction of the virtual viewpoint (directly downward in this example). This is because it is based on the geometrical property that the image projected on the model surface is reduced as it approaches in parallel with.

A major advantage of the above method is that the bowl model can be modified to freely determine the expansion / contraction ratio of the vehicle peripheral portion with respect to the expansion / contraction ratio of the portion in the immediate vicinity of the vehicle. For example, when the bowl model is circular, an image is obtained that is centered on the portion corresponding to the bottom of the bowl and is reduced as it moves away from it regardless of the direction. Also, leave the bowl model at the same place at the bottom and make the shape oval. If the major axis of this ellipse is aligned with the side of the car, the degree of reduction in the direction of travel of the car is the degree of reduction in the lateral direction of the car, centered on the part corresponding to the bottom of the bowl. Can be smaller than. Of course, the reverse is also possible.

Note that, here, an image is synthesized using a bowl model so that the scaling ratio of the vehicle nearest portion including the ground contact portion of the tire is relatively higher than the scaling ratio of the vehicle peripheral portion. Although the method has been described, it goes without saying that the synthesis using a method other than the bowl model is possible.

Further, a camera is attached to the body, a virtual viewpoint is provided substantially above and downward of the camera, and images are combined using the bowl model. That is, even if the above-described reduction deformation is performed centering on the mounting position of the camera, the linearity of the side surface of the body and the linearity in the traveling direction beyond that are maintained. The image in FIG. 8 (b) is
Using an image synthesized by using a bowl model with a virtual viewpoint provided substantially above and downward of the cameras 1 and 2, and an image synthesized using a bowl model provided with a virtual viewpoint provided substantially above and downward of the cameras 5 and 6. , Created.

Next, the manner of width adjustment to the groove on the road shoulder will be specifically described with reference to FIG. In FIG. 8 (a), the nearest part of the vehicle is enlarged so that at least about 5 cm can be discriminated in order to understand how much room there is from the passenger side of the vehicle to the groove on the road shoulder. The resolution is maintained. Moreover, the tires of the own vehicle are also displayed so that the user can judge by just seeing how close the own vehicle is to the roadside (solid circled area 51). Experiments showed that by presenting such an image, a driver, who had been difficult to cut the width of the roadside belt within 10 cm, could easily reach within 5 cm. There is. That is, it has been confirmed that this display mode has a great effect on the side-by-side driving.

On the other hand, from this display mode, it is possible to confirm how much space is left between the host vehicle and the oncoming vehicle passing the right side of the host vehicle (circled area 52 of broken line). Figure 8
FIG. 8B shows the same display mode as FIG. 8A, and shows an example of a situation in which such confirmation is performed. In this display mode, since the front side is reduced while maintaining the linearity, a line drawn so as to contact the front and rear tires (a line 53 passing through TIRE1 and TIRE3)
And the line 54) passing through TYRE2 and TYRE4 becomes the traveling line of the vehicle. For this reason, when an oncoming vehicle approaches from the front right, it can be easily determined whether or not the oncoming vehicle will contact the oncoming vehicle by observing whether the oncoming vehicle is on the right side of the right side traveling line or crossing the traveling line. is there. Of course, the screen becomes smaller toward the front, so even if an oncoming vehicle is approaching from a distant place, the above judgment can be made on the screen in advance so that the driver can operate the vehicle with a margin. In the example of FIG. 8B, it can be seen that the oncoming vehicle on the right of the traveling line on the right passes safely without touching the right side of the vehicle. It has been confirmed by experiments that it is easy to drive when 5 m ahead is visible.

However, in the case of actually passing an oncoming vehicle,
In some cases, this is not enough. That is, when there is an object protruding further outside the body, such as a door mirror, it is necessary to visually confirm the screen as well as the screen.

Although all four tires are displayed in FIG. 8, only some of the tires, for example, only the previous tire may be displayed in the composite image.

Up to this point, the case of using four cameras has been described as an example of the screen mode in the second embodiment. Of course, the present invention is not limited to this, and can be implemented by using only one camera. For example, even if only one front-facing camera can be installed on the side of the vehicle on the passenger side due to cost constraints, the present invention provides the user with an image effective for sideways approaching the shoulder. Is possible.

FIG. 19 is a diagram showing a screen mode according to this embodiment using only one camera, in comparison with a case using a plurality of cameras. In the figure, (a) is an example of a camera image of four cameras RF, RR, LF, LR installed on the left and right of the vehicle, and (b) is a diagram of the four camera images shown in (a).
It is a composite image created in the same manner as. On the contrary,
(C) is an image synthesized from only the camera image of the camera LF in (a). It should be noted that the illustrations of the own vehicle in FIGS. 19B and 19C are attached to show the reference of the own vehicle position with respect to the composite image, and do not necessarily match the own vehicle portion included in the image.

In the example of FIG. 19C, since only one camera is used for image combination, the visual field range shown in the image is more limited than that of the example of FIG. However, by enlarging the immediate portion of the vehicle, it is possible to maintain a resolution at which an object of at least about 5 cm can be discriminated and to know how much room is left from the passenger side of the vehicle to the groove of the road shoulder. Moreover, the tires of the own vehicle are also displayed so that the extent to which the own vehicle is approaching the roadside can be easily recognized only by looking at the image (solid circle area).

Therefore, when the image as shown in (c) is displayed, in a driving environment where there is a groove on the shoulder of the passenger seat on a narrow road, even if an oncoming vehicle is avoided from the front, the groove of the road shoulder is avoided. While making sure that the tires do not fall, it is possible to move forward while moving forward. Of course, in this case, the driver himself needs to check the distance from the oncoming vehicle, but the burden of driving due to the width-shifting is significantly reduced.

Further, by using the fisheye lens, a sufficiently wide field of view can be obtained even with one camera. For example, install a camera with a fisheye lens with a viewing angle of around 180 ° on the side of the passenger seat of the vehicle,
The present invention may be implemented using this camera image. As a result, the image on the side opposite to the driver's seat can be displayed in a wide range, so that the driver can drive more safely.

(Third Embodiment) The third embodiment is an aspect of the invention made to solve the above-mentioned third problem. For example, in parking involving turning back, only the surrounding situation is present. Instead, it provides the driver with a composite image so that the driver can see the state ahead of the vehicle when moving forward or backward.

Next, regarding the example of the display screen mode, FIG. 9 is shown according to the three situations of the vehicle being stopped, moving forward, and moving backward.
A detailed description will be given with reference to FIG. 10 and FIG.

1. Example of display mode when vehicle is stopped: FIG. 9A shows a virtual viewpoint image viewed from above, which is necessary for grasping distances in front and rear of the vehicle in respective directions. This is an example in which an image is pasted. Not only the surroundings of the vehicle but also the direction ahead of the vehicle can be seen at a glance, so it can be used to check the surrounding conditions before and immediately after starting at the stop of traffic lights, and also for parallel parking and parallel parking. Applicable.

2. Example of display mode when vehicle is moving forward at low speed: In each of FIGS. 9B and 9C, with respect to the virtual viewpoint image viewed from above, a portion corresponding to the front of the own vehicle is grasped as a distant front. An example in which an image necessary for this is pasted will be shown. At a glance, you can see not only the area around your vehicle but also the way ahead.

The difference between FIGS. 9 (b) and 9 (c) is the difference in the image for distant grasp in the front. First, FIG. 9B shows a case where the boundary between the looking-down image and the front image is connected at the road surface portion. Since the road surface is connected, it is easy to see where the front obstacle is in front of the vehicle. This is effective not only when starting at low speeds, but also when driving on narrow roads and when narrowing toll gates. On the other hand, FIG. 9C shows a case where the field of view of the front image is set to 180 degrees in order to compensate for a blind spot in the looking-down image. Although there is a problem that the boundaries between images are not connected, there is a merit that information on a wide field of view regarding the front can be seen at once. It is especially suitable for checking the surroundings of 180 degrees ahead before starting.

Regarding the virtual viewpoint image, the area in front of the own vehicle becomes wider as the speed of forward movement increases, and conversely, the position of the own vehicle comes to the center of the screen as the speed of forward movement decreases. It may be continuously switched to. Furthermore, when the speed exceeds a certain speed, the screen mode may be limited to the rear panoramic image.

3. Example of display mode when the vehicle is moving backward: In FIGS. 10A and 10B, a virtual viewpoint image viewed from above is shown in a portion corresponding to the rear of the own vehicle.
An example in which an image necessary for grasping the distance behind is pasted will be shown. At a glance, you can see not only the area around your vehicle but also the way it looks in the backward direction.

The difference between FIGS. 10 (a) and 10 (b) is the difference in the image for rearward grasping. First, FIG. 10A shows a case where the boundary between the looking-down image and the front image is connected at the road surface portion. Since the road surface is connected, it is easy to see where the front obstacle is in front of the vehicle. Since the road surface has continuity at the boundary portion, it is suitable for an image displayed at the start of parallel parking and parallel parking. On the other hand, FIG. 10B shows a case where the field of view of the rear image is set to 180 degrees in order to supplement a blind spot in the looking-down image. Although there is a problem that the boundaries between the images are not connected, there is an advantage that the blind spot disappears in the rear of the vehicle with many blind spots. It is especially suitable for checking the surroundings of 180 degrees behind before retreating.

Regarding the virtual viewpoint image, the area behind the vehicle becomes wider as the backward moving speed increases.
On the contrary, as the reverse speed decreases, the position of the vehicle may be continuously switched so that the position of the vehicle is in the center of the screen.

The display mode as shown in FIG. 9A is not only displayed when the vehicle is not moving at all, but may be displayed when forward and backward movements such as turning back are frequently repeated. . For example, when switching back,
Switching between (b) and FIG. 10 (a) for display may be bothersome to see. In such a case, the image as shown in FIG. 9A may be always displayed during the period when the absolute value of the speed is equal to or less than the predetermined value.

Up to this point, the case of using eight cameras has been described as an example of the screen mode in the third embodiment. Of course, the present invention is not limited to this, and even if only two or only one camera can be installed in the central portion of the rear surface of the vehicle due to constraints such as cost, it is possible to retreat according to the present invention. Sometimes it is possible to provide the user with a valid image.

FIG. 20 is a diagram showing a screen mode according to this embodiment using two cameras. In the figure, (a) is an example of a camera image of two cameras BR, BL installed on the rear surface of the vehicle, and (b) is the position of the virtual viewpoint when creating a virtual viewpoint image (the center of the rear end of the vehicle). FIG. 3C is a view showing the image from above and directly below, and FIG. 3C is a composite image generated using the two camera images shown in FIG. In the composite image of FIG. 20 (c),
A virtual viewpoint image looking down from above is arranged below, and an image necessary for grasping the far distance behind is arranged above it.

When only two cameras installed in the rear of the vehicle are used as in this example, an image of the side of the vehicle cannot be obtained, but an image of a field of view of approximately 180 degrees is obtained in the rear of the vehicle. Therefore, not only the surroundings of the vehicle but also the state in the backward direction can be seen at a glance from the image. Therefore, at the time of backward movement in parallel parking or parallel parking, even a composite image generated using such two cameras is sufficiently practical.

FIG. 21 is a diagram showing a screen mode according to this embodiment using one camera. In the figure, (a) is the image of one camera installed on the rear surface of the vehicle, (b) is the position of the virtual viewpoint when creating the virtual viewpoint image (looking down from above the central portion of the rear end of the vehicle directly below) And (c) is a composite image generated using the camera image shown in (a). Figure 21
In the composite image in (c), the virtual viewpoint image viewed from above is arranged below, and the image necessary for grasping the far distance behind is arranged above it.

In this example, since there is only one camera, not only the side of the vehicle but also diagonally behind the vehicle is out of the field of view. However, as can be seen from FIG. 21 (c), if the angle of view of the camera is about 130 degrees, a sufficient field of view for rearward confirmation can be obtained, and not only the surroundings of the vehicle but also the state in the backward direction can be seen. You can see at a glance. Therefore, even a composite image generated using such a single camera is sufficiently practical when the vehicle moves backward in parallel parking or parallel parking. Further, by using the fisheye lens, it is possible to obtain a sufficiently wide field of view even with one camera.

The illustrations of the own vehicle in FIGS. 20 (c) and 21 (c) are attached to show the position of the own vehicle with respect to the composite image. Needless to say, driving can be further facilitated by accurately matching the size of the illustration with the composite image.

(Fourth Embodiment) The fourth embodiment is an aspect of the invention made to solve the above-mentioned fourth problem, and also in a virtual viewpoint image obtained by transforming / cutting out a camera image and synthesizing it. It provides the driver with a composite image so that objects other than the road surface do not disappear.

Next, an example of a display screen mode will be specifically described with reference to the drawings.

FIG. 11A shows, as an example of an image according to this embodiment, a camera (cameras 2 and 6) facing rearward on the side of the vehicle.
An example is shown in which a first virtual viewpoint image using only the above and a second virtual viewpoint image using only the paired cameras (cameras 3 and 7) after that are displayed in parallel on the screen. These virtual viewpoint images are created by using one of the cameras (camera 2 and camera 3, or camera 6 and camera 7) whose shooting ranges overlap, and not using the other camera. It is an image. In each virtual viewpoint image, the shooting range of one camera used for its creation includes both an overlapping portion and a non-overlapping portion with the other camera. As a result, FIG.
The pole P1 standing in the right rear corner of the own vehicle, which has disappeared in the virtual viewpoint image of (b), is reflected in both of the two virtual viewpoint images without disappearing. Of course, instead of displaying them side by side on the screen, a plurality of virtual viewpoint images may be displayed on another screen to switch the screens.

In general, such a phenomenon that an object outside the road surface disappears occurs when both of the following two conditions are satisfied.

Condition 1: The directions in which the respective cameras, which are the basis of the two partial images constituting the boundary, are shooting the object are different. This is because the directions in which the object is viewed are different between the two cameras. And, when the respective camera images are transformed into virtual viewpoint images, the directions in which the object is distorted are different. This is also apparent from the modification of the pole P1 shown in FIG.

Condition 2: The object is in the vicinity of the boundary between the partial images. This means that when the object distorted by deformation is in the vicinity of the boundary, the distorted and extended part is generated by the cutting process for creating the partial image. It means that it is almost erased. This is also apparent from the modification of the pole P1 shown in FIG.

Therefore, the object does not disappear unless the condition 1 is satisfied. FIG. 12A is a conceptual diagram showing an example in which an object does not disappear even though Condition 2 is satisfied. Figure 1
In 2 (a), the pole stands immediately behind the paired camera behind the vehicle, but since Condition 1 is not satisfied, that is, both camera 3 and camera 7 face in substantially the same direction with respect to the pole, No matter which camera image is transformed, the pole will be distorted to extend to the bottom of the screen as well. Therefore, at the boundary, the partial image of the camera 3 and the camera 7
It is possible to prevent the pole from disappearing by performing processing such as mixing with the partial image of.

Similarly, the object hardly disappears unless the condition 2 is satisfied. FIG. 12B is a conceptual diagram showing an example in which an object does not disappear even though Condition 1 is satisfied. 12
In (b), Condition 1 is satisfied because the pole stands diagonally to the right rear of the vehicle and is slightly away from it and is in the visual field range of the cameras 2 and 3 located at different positions. However, because the condition 2 is not satisfied, that is, the pole is distorted so as to extend from the root to the right of the screen in the partial image of the camera 3 because it is far from the boundary between the partial image of the camera 2 and the partial image of the camera 3, At least until the boundaries are visible and remain.

From this, as a guideline when designing a screen configuration plan, in order to avoid the above condition 1, a boundary between partial images created from two different camera images is displayed on one screen. It does not make a. The above-mentioned figure shows just one example, and is divided into two screens so that the boundary between the camera 2 and the camera 3 and the boundary between the camera 6 and the camera 7 are not formed on the same screen.

In the case of using the 8-camera image as shown in FIG. 2, the image can be composed of two screens as shown in the example of FIG. do not do). In FIG. 11B, in the virtual viewpoint image, a combined image using only the cameras on both sides and a combined image using only the front and rear cameras are arranged in parallel and displayed on the screen. In any case, any screen configuration other than that shown in this example may be used as long as it is created based on the above guideline.

(Fifth Embodiment) The fifth embodiment is an aspect of the invention made in order to solve the above-mentioned fifth problem. In the virtual viewpoint image, an image is formed at a boundary between adjacent partial images. In order to prevent the driving from being disturbed due to discontinuity and blur, another image showing the discontinuity is embedded in the virtual viewpoint image and presented to the driver.

Next, an example of the display screen mode will be specifically described with reference to the drawings.

FIG. 13A shows the cameras 2, 3 shown in FIG.
It is a figure which shows the screen mode produced | generated using four cameras of 6,7. As a whole, a virtual viewpoint image looking down from above, which is synthesized by using the above four camera images, is used, and a right-and-left inverted image of the camera 2 that photographs the right rear part of the vehicle is displayed on the right of the upper half of the image. On the left of the upper half of the image, a left-right inverted image of the camera 6 capturing the left rear part of the vehicle is displayed with a sub-window. In the virtual viewpoint image, the degree of approach to the parked vehicle can be roughly grasped in the elliptical portion surrounded by the broken line. However, the degree of approach of the parked vehicle can be grasped at a glance by observing the part of the circle surrounded by the solid line in the horizontally inverted image of the camera 2 displayed in the sub window.

Further, FIG. 13B is a diagram showing another screen mode generated by using the four cameras 2, 3, 6, and 7 shown in FIG. Is not shown). As a whole, a virtual viewpoint image which is synthesized by using the images of the cameras 3 and 7 and which is diagonally rearwardly looked down from above and whose left and right are inverted is used, and the inverted image of the camera 2 is displayed on the right of the lower half of the image. A reverse image of the camera 6 is displayed on the left side of the lower half of the image by providing a sub window. FIG.
In the example of (b), since it is assumed that the gazing point of the virtual viewpoint image comes to the upper half of the entire screen, the sub-windows are arranged on the right and left sides of the lower half of the screen. The effect is similar to that of FIG.

In any of the examples, the sub-window may be fixedly arranged in a portion of the virtual viewpoint image as the background image that does not hinder driving even if it is hidden, or may be arranged according to the surrounding conditions. You may move the place appropriately.

With such a screen configuration, the virtual viewpoint image can be used to ascertain the positional relationship between the vehicle and surrounding objects, and the portion discontinuous at the boundary can be displayed on the sub-window. By looking at the image, it is possible to grasp the details of the degree of approach to the object around the vehicle without moving the line of sight.

(Sixth Embodiment) The sixth embodiment is an aspect of the invention made to solve the above-mentioned sixth problem, and is used at a place where a detailed situation is desired to be grasped during a complicated driving operation. The side area of a certain vehicle is presented to the driver in preference to other areas.

FIGS. 14 (a) and 14 (b) are diagrams showing a composite image at the start of the parking operation and during the parking operation in parallel parking, which is one of the complicated driving operations. In any of the virtual viewpoint images, the parked vehicle P_ is set by preferentially using the camera installed on the side of the vehicle.
Since the CAR is reproduced in the image only by the cameras 4 and 5, the problem of discontinuity is solved.

Of course, when the layout of the partial images is set in such a mode, discontinuous portions are concentrated in the front and rear of the vehicle, which may give the user a feeling of strangeness. However, regarding this point, for example, as described in the third embodiment, an image necessary for grasping the destination of the traveling direction of the vehicle is displayed side by side on the virtual viewpoint image, and the discomfort of the discontinuous portion is displayed. It can be alleviated.

In the above example, the camera on the side of the vehicle is preferentially used so that the boundary in the virtual viewpoint image is configured so that the portion to be watched during parallel parking can be easily seen. The boundary between the partial images may be appropriately switched depending on the application and the driving situation. This allows
It is possible to provide an image that is easier for the driver to see.

FIG. 15 is a diagram showing an example of switching the boundary between partial images during parallel parking. When parallel parking is performed, from the beginning of the operation (FIG. 15A) to the middle (FIG. 15A).
Up to (b)), an object on the side of the vehicle (for example, a parked car P)
_CAR_S) should be checked mainly. Therefore, a camera installed on the side of the vehicle is preferentially used, and a virtual viewpoint image in which the captured image of the camera on the side of the vehicle is used as widely as possible is presented. On the other hand, at the final stage when about half of the vehicles have already entered the parking space, an object behind the vehicle (for example, a parked car P_CAR_
It is mainly necessary to confirm whether or not it does not hit B). Therefore, a camera installed at the rear of the vehicle is preferentially used, and a virtual viewpoint image in which an image captured by the camera at the rear of the vehicle is used as widely as possible is presented.

Here, in order to switch the mode in which the camera on the side of the vehicle is prioritized to the mode in which the camera on the rear of the vehicle is prioritized, for example, a detection signal of a sensor provided in the vehicle for detecting a rear object is used as a trigger. You can use it. If the sensor detects that there is any object behind the vehicle, this may be used as a trigger to switch to a screen mode in which the camera at the rear of the vehicle is prioritized.

As the switching trigger, any operation such as handle operation or gear operation may be used instead of the object detection sensor. Of course, the driver may manually switch the screen mode.

FIG. 16 is a block diagram showing a system configuration that enables switching of the above-mentioned screen modes. In the configuration of FIG. 16, the mapping table selection unit 25 receives the screen mode switching signal (for example, the signal of the object detection sensor),
The mapping table MPT is switched according to the signal content. For example, in order to realize the automatic switching as described above, a screen mode mapping table that prioritizes the cameras on the side of the vehicle and a screen mode mapping table that prioritizes the cameras on the rear of the vehicle may be prepared in advance. .

The display mode of the composite image for realizing the object of the present invention has been described above. As a device configuration for realizing this composite image, a mapping table corresponding to the number of images to be displayed may be provided, or a mapping table may be automatically created according to the situation. Absent.

In the present invention, the vehicle includes ordinary cars, light cars, freight cars, buses and the like. Further, a special vehicle such as a crane truck or a shovel car can be the vehicle of the present invention as long as the technical idea of the present invention can be applied.

In the above description, the monitoring system according to the present invention is applied to a vehicle, but it can be similarly applied to a moving body other than the vehicle, such as an airplane or a ship. . Further, the camera may be installed in a monitoring target other than the moving body, such as a store, a residence, or a showroom.

The installation positions and the number of cameras are as follows.
It is not limited to those shown here. For example, each paired camera may be arranged at each of the four corners of the vehicle.

The function of the image processing unit according to the present invention is as follows.
All or part of it may be realized by using dedicated hardware or may be realized by software. It is also possible to use a recording medium or a transmission medium that stores a program for causing a computer to execute all or some of the functions of the image processing unit according to the present invention. For example, in a configuration using a computer, each processing unit such as the image combining unit may be realized by software executed by the CPU and stored in the ROM or the RAM.

[0112]

According to the first aspect of the present invention, a composite image is displayed in which the enlargement / reduction ratio is relatively high in the vehicle proximate portion including the ground contact portion of at least one tire of the vehicle, compared to the vehicle peripheral portion. The user can simultaneously check the detailed situation immediately under the vehicle and the state ahead of the vehicle from the same image. Therefore, in the case of approaching the shoulder of the road, etc., not only the condition of the own vehicle approaching the shoulder of the road but also the state of the forward direction when moving forward or backward can be understood, and the driver is unfamiliar with driving. Also, just by looking at the image, it becomes extremely easy to pass the vehicle on the way to the other side of the canyon and to adjust the width.

According to the second aspect of the present invention, the images of the moving direction or the movable direction of the vehicle are displayed side by side while maintaining the positional relationship with the virtual viewpoint image.
Not only the situation around the vehicle but also the state in the traveling direction when the vehicle moves forward or backward can be checked at the same image at the same time. Therefore, when the vehicle moves forward or backward in a parking involving turning back, it is not necessary to check the state of the traveling direction by visual observation or a mirror, and it becomes possible to concentrate on driving.

According to the third aspect of the present invention, in each virtual viewpoint image, one of the camera images is used without being deleted in the overlapping portion of the shooting range, so that the object existing in the overlapping portion is not deleted in the virtual viewpoint image. It never disappears. For this reason, it becomes possible to provide the driver with an image that always includes information necessary for safe driving, and the driver can check the surrounding situation with ease only by the image.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a monitoring system according to each embodiment of the present invention.

FIG. 2 is a diagram showing an example of a camera arrangement.

FIG. 3 is a diagram showing an example of a captured image of each camera shown in FIG.

FIG. 4 is an example of a situation in which the first embodiment of the present invention is suitable, and is a diagram showing a case where the vehicle is moved backwards and the steering wheel is largely turned to guide the vehicle to the road.

FIG. 5 is a diagram showing an example of a screen aspect according to the first embodiment of the present invention.

FIG. 6 is a diagram showing image generation using a cylindrical model.

FIG. 7 is a diagram showing image generation using a bowl model.

FIG. 8 is a diagram showing an example of a screen mode according to a second embodiment of the present invention.

FIG. 9 is a diagram showing an example of a screen mode according to a third embodiment of the present invention.

FIG. 10 is a diagram showing an example of a screen aspect according to a third embodiment of the present invention.

FIG. 11 is a diagram showing an example of an image according to the fourth embodiment of the present invention.

FIG. 12 is a diagram for explaining a condition in which an object near a boundary disappears in a virtual viewpoint image.

FIG. 13 is a diagram showing an example of a screen aspect according to a fifth embodiment of the present invention.

FIG. 14 is a diagram showing an example of a screen mode according to a sixth embodiment of the present invention.

FIG. 15 is a diagram showing an example of screen mode switching according to a sixth embodiment of the present invention.

16 is a block diagram showing a system configuration that enables switching of the screen modes shown in FIG.

FIG. 17 is a diagram for explaining the fourth problem of the present invention.

FIG. 18 is a diagram for explaining the fifth problem of the present invention.

FIG. 19 is another example of a screen mode according to the second embodiment of the present invention.

FIG. 20 is another example of a screen mode according to the third embodiment of the present invention.

FIG. 21 is another example of a screen mode according to the third embodiment of the present invention.

[Explanation of symbols]

11 cameras 20 Image processing unit 30 display

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI G06T 1/00 330 G06T 1/00 330A (72) Inventor Jun Morimura 1006 Kadoma, Kadoma City, Osaka Matsushita Electric Industrial Co., Ltd. (72) Inventor Nobuhiko Yasui 1006 Kadoma, Kadoma City, Osaka Prefecture Matsushita Electric Industrial Co., Ltd. (72) Inventor Akira Ishida 1006 Kadoma, Kadoma City Osaka Prefecture Matsushita Electric Industrial Co., Ltd. (72) Inventor Ii Atsushi Osaka 1006 Kadoma, Kadoma, Osaka Prefecture Matsushita Electric Industrial Co., Ltd. (72) Inventor Takashi Yoshida 1006 Kadoma, Kadoma City, Osaka Prefecture (56) Reference: JP-A-9-37235 (JP, A) JP-A-1-123587 (JP, A) International publication 00/7373 (WO, A1) (58) Fields investigated (Int.Cl. ⁷ , DB name) B60R 21/00 621 B60R 21 / 00 628 G06T 1/00 330

Claims (5)

(57) [Claims] 1. At least one camera for photographing the surroundings of a vehicle, and an image processing unit for inputting a photographed image of the camera, generating a composite image from the camera image, and displaying the synthesized image on a display device, The image processing unit is arranged so that the expansion / contraction ratio of the vehicle immediate vicinity part including the ground contact part of at least one tire of the vehicle is relatively higher than the expansion / contraction ratio of the vehicle peripheral part and along the side surface of the vehicle. A surveillance system having a mode for displaying a synthesized image so that the portion has linearity. 2. At least one camera for photographing the surroundings of a vehicle, and an image processing unit for inputting a photographed image of the camera, generating an image viewed from a virtual viewpoint from the camera image, and displaying the image on a display device. Wherein the image processing unit has a mode of displaying an image of a moving direction or a movable direction of the vehicle side by side with the virtual viewpoint image while maintaining a positional relationship with the virtual viewpoint image. Surveillance system. 3. An image processing unit for inputting a plurality of cameras for photographing the surroundings of a vehicle and images taken by the respective cameras, generating an image viewed from a virtual viewpoint from these camera images, and displaying the image on a display device. And the plurality of cameras include first and second cameras whose shooting ranges overlap, and the image processing unit uses the first camera and does not use the second camera. A first virtual viewpoint image which is created and includes an overlapping portion and a non-overlapping portion with the second camera in a shooting range of the first camera, and the second camera, and
A second virtual viewpoint image created without using the first camera and capable of generating a second virtual viewpoint image including an overlapping portion and a non-overlapping portion with the first camera in the shooting range of the second camera Surveillance system characterized by being. 4. The monitoring system according to claim 3 , wherein the image processing unit has a mode in which the first and second virtual viewpoint images are displayed side by side on the same screen. 5. The monitoring system according to claim 3 , wherein the first virtual viewpoint image is a virtual viewpoint image using only an image captured by a camera installed laterally of the vehicle, The virtual viewpoint image of is a virtual viewpoint image using only images captured by cameras installed in front of and behind the vehicle.