JP2002019556A - Monitoring system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a monitoring system around a vehicle, capable of certainly grasping the kind of an article around the vehicle. SOLUTION: In a normal mode, a composite image indicating a relationship between the vehicle and the surroundings is produced by plural camera images. In a range OLa to OLd superimposed by a camera view range, as a result of composing the image, the image of an article existing in the range OLa to OLd (for example, pole P3) has a possibility to be unclear. In an exceptional mode, the camera images related to the superposing range OLa to OLd are individually indicated without blending.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing technique for generating a composite image using images picked up by a plurality of cameras installed in a vehicle, and more particularly to assisting safety confirmation when driving a vehicle. It belongs to the technology effective for the monitoring system used.

[0002]

2. Description of the Related Art Conventionally, as a device for monitoring the periphery of a vehicle by using a camera, a vehicle image providing device described in Japanese Patent Application Laid-Open No. H11-78692 (Document 1) and a US Pat.
No. 49331 (Reference 2), “Display Enhanc
ements for Vehicle Vision System ”is known.

[0003] In Document 1, a plurality of cameras for providing vehicle images are mounted around the vehicle, and images taken by each camera are deformed and combined to create a new image and display it on a monitor. It has been disclosed. In particular, when deforming images arranged by a plurality of cameras and arranging the images, it is possible to provide a more accurate image to the driver by maintaining continuity between the deformed images at a boundary portion adjacent to each other. Have been.

[0004] Document 2 discloses a configuration in which a wide-view rear-view image is presented to a driver using a camera mounted rearward, one at the center of the rear part of the car and one near each of the left and right door mirrors. ing. Further, it is described that each camera image is seamlessly connected.

[0005]

However, in the prior art,
There are the following problems.

In image synthesis, in order to accurately connect the original camera images with each other with high accuracy, in principle, the shapes and distances of all objects shown in the camera images are accurately obtained, and the results are used to calculate the image. It is necessary to perform the transformation / synthesis processing. However, in order to accurately obtain the shapes and distances of all the objects shown in the camera image, a complicated measuring device is required, and the cost becomes extremely high. Further, in this case, the calculation process for image synthesis becomes considerably complicated, and it is difficult to execute the calculation in real time.

Therefore, usually, a method is employed in which attention is paid to a specific portion within the camera field of view, and continuity of an image is guaranteed for the specific portion. For example, when generating a composite image that looks down on the situation around the vehicle from above the vehicle, image synthesis is performed so that at least a portion corresponding to the road surface is continuously connected. By using such a method, it is possible to make the user accurately recognize the relationship between the vehicle and the surrounding situation at low cost.

However, in this case, naturally, the continuity of the image is not guaranteed except for the specific portion where the continuity is guaranteed. This causes problems that are sometimes less favorable.

In the case of the above-mentioned overlooking image, for example, continuity is guaranteed for the road surface on the composite image, but continuity is not guaranteed for other parts, for example, objects on the road surface. Therefore, when there are a plurality of cameras having overlapping shooting ranges, the images of the objects existing in the overlapping ranges usually do not match in each of the camera images after the deformation. For this reason, blend each camera image,
That is, when weighted and mixed, the shape of the object is not always clearly displayed on the composite image. For this reason, it becomes difficult for the driver to grasp what the object is.

Certainly, the driver can accurately grasp the situation around the vehicle on the road surface by using a composite image such as a top-down image. However, there is a case where priority is given to being able to reliably grasp the objects existing in the surroundings.

In view of the above problems, the present invention provides a surveillance system including a plurality of cameras and an image processing unit that generates a composite image from camera images, so that the type of a surrounding object can be reliably grasped. That is the task.

[0012]

Means for Solving the Problems To solve the above-mentioned problems, a solution taken by the invention of claim 1 is a monitoring system, comprising: a plurality of cameras for photographing the periphery of a vehicle; An image processing unit that generates a composite image viewed from a virtual viewpoint from these camera images and displays the synthesized image on a display device, wherein the image processing unit includes a plurality of cameras having overlapping shooting ranges. This has an exception mode in which at least a part of the images in the overlapping range are individually displayed without blending.

According to the first aspect of the present invention, in the exceptional mode, images in the overlapping range of a plurality of cameras having overlapping shooting ranges are individually displayed without blending. For this reason, the user can surely grasp the objects existing in the overlapping range from the individually displayed images.

According to a second aspect of the present invention, the image processing unit in the monitoring system according to the first aspect switches between a normal mode for displaying a composite image and an exceptional mode according to the traveling state of the vehicle.

According to a third aspect of the present invention, the image processing unit in the monitoring system according to the first aspect displays a vehicle image representing the vehicle and displays a plurality of partial images showing a situation around the vehicle in the exceptional mode. Are displayed so as to maintain the positional relationship.

According to a fourth aspect of the present invention, the image processing section in the monitoring system according to the third aspect switches at least one of the sizes of the plurality of partial images in accordance with a running state of the vehicle.

According to a fifth aspect of the present invention, the plurality of partial images in the monitoring system according to the third aspect are a front image, a rear image, a right image, and a left image of the vehicle.

According to a sixth aspect of the present invention, in the monitoring system according to the first aspect, the vehicle includes an object detection sensor for detecting the presence or absence of an object in the overlapping range, and the image processing unit includes the object detection sensor. When the sensor detects an object, the mode is switched from the normal mode to the exception mode, and the camera images in the overlapping range are individually displayed without blending.

[0019]

Embodiments of the present invention will be described below with reference to the drawings.

(First Embodiment) FIG. 1 is a block diagram showing a configuration of a monitoring system according to a first embodiment of the present invention. In FIG. 1, an image processing unit 2 receives a plurality of camera images output from an imaging unit 1 and combines them to generate a new image. This composite image is displayed on the display 3
Displayed by

The image pickup section 1 has a plurality of cameras 11, and each camera 11 is provided with a pair of frame memories 12. Here, each camera 11 is assumed to be a CCD type. When the camera is a CMOS type, the camera can have a function of a frame memory. In this case, the frame memory can be omitted.

The pixel synthesizing unit 21 generates a synthesized image using the camera image output from the imaging unit 1 according to the mapping table MPT set in the mapping table reference unit 22. The timing generation unit 24 generates a timing signal for generating a moving image sequence of a composite image.

FIG. 2 is a diagram showing an example of the camera arrangement.
FIG. 3 is a diagram showing an example of an image of each camera arranged as shown in FIG. In the example shown in FIG. 2, two paired cameras are installed at a total of four locations: a front grill, left and right door mirrors, and a rear trunk of the vehicle. With this camera arrangement, images in a 180-degree field of view can be obtained for the front, rear, left and right of the vehicle, and it is possible to show the situation around the vehicle without fail in order to enhance driving safety. WL1 to WL5 are white lines drawn on the road surface,
P1 to P4 are poles.

The image processing unit 2 transforms and synthesizes the eight camera images as shown in FIG. 3 to generate a synthesized image as if looking down vertically from above the vehicle. In order to generate a synthesized image, image deformation processing and synthesis processing (including boundary processing) of the deformed partial image are required.
Has a mapping table reference unit 22, and uses a mapping table MPT in order to process a captured image in one step.

The "mapping table" refers to a table in which the correspondence between the pixels of the composite image and the pixel data of each camera image is described, and is used for performing the process of generating the composite image at high speed. By creating such a mapping table in advance by calculation using a geometric transformation or the like or by manual operation, a desired composite image can be generated at high speed.

Specifically, the mapping table is, for example, a ROM (writable / erasable ROM such as an EEPROM).
) Or stored in RAM. The storage of the mapping table may be realized, for example, by writing mapping data calculated by a processor in the image processing unit into a ROM or a RAM, or storing mapping table data provided as firmware in a communication line or a disk drive. Using data transfer means such as
You may make it write in RAM or ROM.

By switching the mapping table used to generate the composite image, the display mode of the composite image can be easily switched. The display mode can be switched manually or automatically according to the running state of the vehicle. The mapping table selection unit 41 selects a display mode of the composite image, and sets a mapping table M according to the selected display mode.
It instructs the mapping table reference unit 22 to select a PT.

FIG. 4 is a view showing the photographing range of each camera in the camera arrangement shown in FIG. As shown in FIG. 4, the shooting ranges of the paired cameras overlap each other. And
At the time of image synthesis, a composite image is generated for the overlapping ranges OLa to OLd by blending, that is, weighting and mixing pixel data of each camera image.

In this case, however, a certain object has an overlapping range O
When the object exists in La to OLd, the object may not always be clearly displayed on the composite image. For example, when generating a combined image looking down from a virtual viewpoint above the vehicle, the camera images are processed and combined so that portions corresponding to the road surface are continuously connected. In other words,
Under the assumption that the road surface is reflected in the camera image, the correspondence between the camera image and the composite image is obtained. At this time, an object existing on the road surface is not clearly displayed on the composite image.

A description will be given by taking the pole P3 in the overlapping range OLa as an example. As can be seen from the example of the camera image in FIG. 3, in the camera 4, the pole P3 is photographed so as to straddle the white lines WL1 and WL2. Therefore, when the image of the camera 4 is converted, the image of the pole P3 is projected as shown by P3A in FIG. On the other hand, in the camera 5, the pole P3 is photographed along the white line WL2. For this reason, the camera 5
When the image of FIG. 4 is converted, the image of the pole P3 is represented by P in FIG.
It is projected like 3B. Therefore, the overlapping range OLa
When the pixel data of the images of the cameras 4 and 5 are blended in the above, on the composite image, the pole P3 is not entirely displayed clearly, but the images P3A and P3 are displayed clearly.
3B is only near the root portion where the 3B overlaps.

Of course, since the root portion of the object is displayed on the composite image, it is possible to recognize the presence of the object and confirm the position of the object by looking at the composite image. However, since the shape of the actual object is not clearly displayed in the composite image, it is difficult for the user to surely grasp that the object is a pole. In addition, it is undeniable that the driver who sees the composite image feels strange.

Therefore, in the present embodiment, an exception mode is provided in which images in the overlapping range of a plurality of cameras having overlapping shooting ranges are individually displayed without blending. By displaying each camera image individually without blending, the objects in the overlapping range will be displayed twice, but will not be unclear on the image, and the type thereof can be clearly recognized.

FIG. 5 is a diagram showing a 180-degree field-of-view panoramic image of each paired camera. For example, in a panoramic image (cameras 0 and 4) showing the front of the vehicle, the poles P1 and P3 placed at both front corners of the vehicle are reflected at both ends of the image, so that a field of view of approximately 180 degrees is obtained from the front end of the vehicle. You can see that it is done. The same applies to the rear, left and right images of the vehicle. That is, the surroundings of the vehicle are photographed all over or overlapping with one of the four panoramic images as the partial images shown in FIG.

In this embodiment, in the exception mode, panoramic images as shown in FIG. 5 are individually displayed. The panoramic image as shown in FIG. 5 projects the image of the paired camera onto the cylindrical model as shown in FIG.
The projection image can be obtained by viewing the projection image from a direction perpendicular to the projection plane.

Here, the panoramic image as shown in FIG. 5 is arranged on the screen so as to maintain the positional relationship between the vehicle and the surroundings without impairing the positional relationship. That is, in the exception mode, a vehicle image representing the vehicle is displayed, and panoramic images ahead, behind, right, and left of the vehicle are displayed before, after, right, and left of the vehicle image. Further, at the time of display, images of rectangular areas I1 to I4 indicated by broken lines in FIG. 5 are cut out. Region I
Portions other than 1 to I4 are portions that are unnecessary for checking the surroundings of the vehicle, for example, portions where the sky or the like is reflected. Of course, the region to be cut out need not be a rectangle.

FIG. 7 shows an example of the arrangement of each panoramic image in the exception mode. In the arrangement shown in FIG. 7, the area I1 of the front image is pasted on the square area (D2, D3, D7, D6), and similarly, the area I2 of the rear image is square area (D1, D4, D8, D5). ), The region I of the right image
3 is a square area (D3, D4, D8, D7), and the area I4 of the left image is a square area (D1, D2, D6).
D5). In this screen display, it can be said that the camera images are displayed without blending in all of the four overlapping regions OLa to OLd.

In FIG. 7, the line segments (D1-D5), (D
In (2-D6), (D3-D7), and (D4-D8), there is no continuity between panoramic images adjacent to each other at the boundary. However, the shapes of the poles P1 to P4 are clearly displayed.

Further, each panoramic image may be prepared as required. For example, make sure that the pole is displayed in a reasonably accurate position with respect to the vehicle image,
The position adjacent to the vehicle image may be matched with the position of each part of the vehicle, or may be deformed according to the size of the screen. However, the transformation of the panorama image is
It is necessary to keep the distortion caused by this inconspicuous.

FIGS. 8, 9 and 10 show other examples of the arrangement of each panoramic image. In FIG. 8, each of the extracted regions I
1 to I4 are simply enlarged and reduced horizontally and vertically. In FIG. 9, a blank portion SP is intentionally provided at a boundary having no continuity. By displaying the image as shown in FIG. 9, the discontinuity of the image at the boundary of each panoramic image can be understood at a glance.

Further, instead of transforming each of the regions I1 to I4 into a square region, as shown in FIG. 10, it may be transformed into a region having an arbitrary shape. In this case, some correspondence between the coordinates of the pixels of the original panoramic image and the pixels of the transformed image may be determined, and the other pixels may be determined by appropriate interpolation. For example, in the arrangement of FIG. 10, each vertex of the area I4 shown in FIG.
6 and D5, respectively, and each vertex of the area I3 is set to the point D
3, D4, D8, and D7.

In the above description, the panoramic image is created using a cylindrical model. However, the panoramic image may be created using a model other than the cylindrical model, for example, a bowl model as shown in FIG. Further, the direction in which the image projected on the model is viewed may be changed. In this example, two pairs of cameras are used to obtain a field of view of 180 degrees. However, one camera, such as a camera with a fisheye lens, has a 180-degree view.
A camera that can obtain a good degree of view may be installed.

A mapping table may be provided according to all types of images to be displayed, or a mapping table may be provided in advance only for basic display, and the mapping table may be appropriately processed. Image composition may be performed.

(Second Embodiment) FIG. 12 shows a second embodiment of the present invention.
It is a block diagram showing the composition of the monitoring system concerning an embodiment. In FIG. 12, the mapping table selection unit 4
1, a state signal S1 indicating the driving state of the vehicle is input. The mapping table selection unit 41 causes the mapping table reference unit 22 to select a mapping table MPT corresponding to an appropriate display mode according to the driving condition of the vehicle indicated by the state signal S1. In the configuration of FIG. 12, the image processing unit 2 can appropriately change the size of each panoramic image according to the running state of the vehicle indicated by the state signal S1 in the exception mode.

FIG. 13 is an example of a screen display in the exception mode according to the present embodiment. In the figure, the size of each panoramic image is different between (a) and (b). In FIG. 13A, since the front-rear direction of the vehicle can be seen better, for example, it is displayed when the vehicle is traveling normally or at high speed.
In FIG. 13B, since the surroundings of the vehicle can be seen evenly, it is displayed, for example, when the vehicle is running at low speed. Such switching of the size of the panoramic image can be realized by switching the mapping table.

The switching between the normal mode for displaying the composite image and the exceptional mode may be performed according to the running state of the vehicle indicated by the state signal S1. For example, when the vehicle is traveling normally or at high speed, a composite image viewed from a virtual viewpoint corresponding to the situation is displayed in the normal mode, while an object around the vehicle is displayed when the vehicle is traveling at low speed. May be displayed in the exception mode so that can be clearly recognized.

Of course, the switching of the size of each panoramic image and the switching between the normal mode and the exception mode may be performed using information other than the traveling speed of the vehicle. Other information may include state information on gears, steering angles, turn signals, brakes, etc., and external switch inputs.

(Third Embodiment) FIG. 14 shows a third embodiment of the present invention.
It is a block diagram showing the composition of the monitoring system concerning an embodiment. In FIG. 14, the mapping table selection unit 4
1, a detection signal output from an object detection sensor 50 provided in the vehicle and detecting the presence or absence of an object around the vehicle is input. The mapping table selection unit 41 causes the mapping table reference unit 22 to select a mapping table MPT corresponding to an appropriate display mode according to the detection signal.

FIG. 15 is a diagram showing an example of object detection by the object detection sensor. In FIG. 15, it is assumed that a camera is installed in the vehicle as shown in FIG. Four object detection sensors 50a, 50b, 50c, and 50d are arranged at the four corners of the vehicle, and the overlapping range OLa shown in FIG.
The presence / absence of an object at ~ OLd can be detected.

In the case of FIG. 15A, there are no obstacles around the vehicle, and each of the object detection sensors 50a, 50b, 50c,
50d does not detect any object. In this case, in the normal mode, a composite image viewed from above is displayed. When the vehicle retreats a little, as shown in FIG. 15B, an obstacle (pole PA) at the rear left is detected by the object detection sensor 50c installed at the rear left corner of the vehicle. At this time, the mapping table selection unit 41 receives the detection signal and instructs to change the mapping table so that the display mode is switched to the exception mode.

FIG. 16 shows an example of a screen display in the exception mode according to this embodiment. In FIG. 16, the images of the cameras 6 and 7 related to the overlapping range OLc corresponding to the installation position of the object detection sensor 50c are individually displayed without blending. That is, camera images are individually displayed without blending camera images in one of the four overlapping ranges OLa to OLd. In (a), a top-down image near the periphery of the vehicle is arranged at the center of the screen,
In (b), an overlooked image is arranged on the entire screen, and the images of the cameras 6 and 7 are pasted thereon.

As described above, in the present embodiment, in the exception mode, the camera image relating to the overlapping range where an object is detected is individually displayed, while the overlapping image range where no object is detected is displayed in the camera image. Is displayed as it is. Thus, the driver can accurately recognize the relationship between the vehicle and the surrounding situation, and can also reliably recognize the type of the object existing around.

In this embodiment, the object detection sensor 5
When 0 detects an object, the camera image is displayed individually without blending, but instead, the area where the object is assumed to be on the composite image is changed, for example, by changing the color or brightness. Graphics may be superimposed or displayed.

The installation position of the object detection sensor is not limited to the position shown in the present embodiment, but may be determined appropriately according to the installation position of the camera.

In each of the above embodiments, the display device 3 is typically a liquid crystal display, but other display devices such as a plasma display may be used. In addition, a vehicle-mounted GPS terminal display,
It may be shared with the display of a so-called car navigation system. In addition, the camera 11 used for the imaging unit 1 is typically a color or monochrome digital camera having a solid-state imaging device such as a CCD or a CMOS device, but other imaging devices may be used.

In the above description, the monitoring system according to the present invention is applied to a vehicle. However, the monitoring system can be similarly applied to a moving object other than a vehicle, such as an airplane or a ship. . In addition, a camera may be installed in a monitoring target other than a mobile object, for example, a store, a residence, a show room, and the like.

The installation positions and the number of the cameras are as follows.
It is not limited to the one shown here. For example, each paired camera may be arranged at each of the four corners of the vehicle. In this case, as the partial image as shown in FIG. 5, an image indicating the diagonally right front, the diagonally left front, the diagonally right rear, and the diagonally left rear of the vehicle is obtained. Therefore, as the screen display in the exception mode, it is preferable that the partial images are pasted diagonally to the right, diagonally to the left, diagonally to the right and diagonally to the left around the vehicle image.

The function of the image processing unit according to the present invention is as follows.
All or a part thereof may be realized using dedicated hardware, or may be realized by software. It is also possible to use a recording medium or a transmission medium storing a program for causing a computer to execute all or a part of the functions of the image processing unit according to the present invention. For example, in a configuration using a computer, each processing unit such as an image combining unit may be realized by software executed by a CPU and stored in a ROM or a RAM.

[0058]

As described above, according to the present invention, in the exception mode, the images in the overlapping range of a plurality of cameras having overlapping shooting ranges are individually displayed without blending, so that the user can The type of the object existing in the overlapping range can be surely grasped.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a monitoring system according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of a camera arrangement.

FIG. 3 is a diagram showing an example of an image of each camera arranged as shown in FIG. 2;

FIG. 4 is a diagram showing a shooting range of each camera arranged as shown in FIG. 2;

FIG. 5 is a diagram showing a panoramic image of each paired camera.

FIG. 6 is a diagram illustrating generation of a panoramic image using a cylindrical model.

FIG. 7 is an example of an arrangement of each panoramic image in an exception mode.

FIG. 8 is an example of an arrangement of each panoramic image in an exception mode.

FIG. 9 is an example of an arrangement of each panoramic image in an exception mode.

FIG. 10 is an example of an arrangement of each panoramic image in an exception mode.

FIG. 11 is a diagram illustrating generation of a panoramic image using a bowl model.

FIG. 12 is a block diagram illustrating a configuration of a monitoring system according to a second embodiment of the present invention.

FIG. 13 is an example of a screen display in an exception mode according to the second embodiment of the present invention.

FIG. 14 is a block diagram illustrating a configuration of a monitoring system according to a third embodiment of the present invention.

FIG. 15 is a diagram showing object detection by an object detection sensor.

FIG. 16 is an example of a screen display in an exception mode according to the third embodiment of the present invention.

[Explanation of symbols]

 2 Image processing unit 3 Display device 11 Camera 50, 50a to 50b Object detection sensor

──────────────────────────────────────────────────続 き Continuation of the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) B60R 21/00 621 B60R 21/00 621N 621P 622 622F 622J 622K 622L 622N 622Q 622T 624 624C 624E 1/00 1 / 00 A 1/08 1/08 BC H04N 7/18 H04N 7/18 JV (72) Inventor Hiroshi Ishii 1006 Odakadoma, Kadoma City, Osaka Prefecture Matsushita Electric Industrial Co., Ltd. (72) Inventor Issei Noboru Osaka 1006, Kadoma, Kadoma Matsushita Electric Industrial Co., Ltd.

Claims (6)

[Claims] 1. A plurality of cameras for photographing the periphery of a vehicle, and image processing in which a photographed image of each camera is input, a synthetic image viewed from a virtual viewpoint is generated from these camera images, and displayed on a display device. Wherein the image processing unit has an exceptional mode of individually displaying, without blending, at least a part of the images of the plurality of cameras having overlapping shooting ranges without blending. Characteristic monitoring system. 2. The monitoring system according to claim 1, wherein the image processing unit switches between a normal mode for displaying a composite image and an exceptional mode according to a traveling state of the vehicle. system. 3. The monitoring system according to claim 1, wherein the image processing unit displays a vehicle image representing the vehicle in an exceptional mode and displays a plurality of partial images indicating a situation around the vehicle. A surveillance system that displays information so as to maintain the positional relationship. 4. The monitoring system according to claim 3, wherein the image processing unit determines a size of at least one of the plurality of partial images,
A monitoring system that switches according to the running state of the vehicle. 5. The monitoring system according to claim 3, wherein the plurality of partial images are a front image, a rear image, a right image, and a left image of the vehicle. . 6. The monitoring system according to claim 1, wherein the vehicle includes an object detection sensor that detects the presence or absence of an object in the overlapping range, and wherein the image processing unit detects the object by the object detection sensor. A surveillance system characterized by switching from a normal mode to an exceptional mode and individually displaying camera images in the overlapping range without blending.