JP3300334B2 - Image processing device and monitoring system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing technique for generating a composite image using images captured by a plurality of cameras, and more particularly to a monitoring method used for assisting safety confirmation when driving a vehicle. Belongs to the technology effective for the system

2. Description of the Related Art In a conventional general monitoring system, a part to be monitored is photographed by one or a plurality of monitoring cameras, and the image is displayed on a monitor. In this case, usually, monitors are prepared according to the number of installed cameras.

As a conventional safety monitoring system for a vehicle, for example, there is a system in which a camera for photographing the periphery of the vehicle is installed and an image photographed by the camera is displayed on a monitor installed near a driver's seat. With this system, the driver can confirm on the monitor a place, such as the rear of the vehicle, where it is difficult to visually confirm the safety with a mirror or a mirror.

Further, Japanese Patent Publication No. 2696516
Discloses a configuration in which a monitor screen is divided and displayed according to a gear and a vehicle speed. Specifically, the vehicle stops
When it is determined that the vehicle is in the low-speed state, the monitor screen is divided into three, images captured by the three cameras arranged on the left, right, and below the vehicle are synthesized and reproduced, and it is determined that the vehicle is in a forward running state. In this case, the monitor screen is divided into two, and the images captured by the two cameras arranged on the left and right sides of the vehicle are synthesized and reproduced.

Further, Japanese Patent Laid-Open Publication No.
No. 78692 discloses an arrangement for synthesizing and displaying an accurate image for each scene encountered as a vehicle image presentation device. Specifically, the camera image is deformed and synthesized according to the driving state of the vehicle such as a backward garage state, a forward garage state, a parallel parking state, or a state of entering an intersection with poor visibility. .

However, such a conventional configuration is
The system is not always easy to use for a user such as a vehicle driver.

DISCLOSURE OF THE INVENTION An object of the present invention is to improve the convenience of a user such as a driver of a vehicle as an image processing apparatus or a monitoring system more than before.

[0008] More specifically, the present invention provides an image processing apparatus which receives, as input, images taken by a plurality of cameras for photographing the periphery of a vehicle, and uses these camera images to generate a composite image viewed from a virtual viewpoint. A processing unit, wherein the image processing unit changes at least one of a position of the virtual viewpoint, a direction of a line of sight, and a focal length according to a traveling state of the vehicle.

It is preferable that the image processing section changes at least one of the position of the virtual viewpoint, the direction of the line of sight, and the focal length according to the running speed of the vehicle.

The image processing unit controls at least one of the position of the virtual viewpoint, the direction of the line of sight, and the focal length, and controls the capture of the image outside the visual field of the changed virtual viewpoint. It is preferred to do so. Furthermore, it is preferable to control the capture of the image outside the visual field range of the virtual viewpoint after the change by changing a model for image synthesis.

It is preferable that the image processing section changes at least one of the position of the virtual viewpoint, the direction of the line of sight, and the focal length in accordance with the steering angle of the vehicle.

The image processing unit changes at least one of the position of the virtual viewpoint, the direction of the line of sight, and the focal length in accordance with the detection result of the object detection sensor provided in the vehicle. Is preferred.

The image processing section has an original mapping table, generates a composite image using the mapping table cut out from the original mapping table, and generates a composite image using the mapping table cut out from the original mapping table. Is preferably changed at least one of the position of the virtual viewpoint, the direction of the line of sight, and the focal length.

According to the present invention, there is provided an image processing unit as an image processing device, which receives, as input, images taken by a plurality of cameras for photographing the periphery of a vehicle, and generates a composite image viewed from a virtual viewpoint from these camera images. The image processing unit controls acquisition of an image outside the visual field of view of the virtual viewpoint according to a traveling state of the vehicle.

According to the present invention, as a surveillance system, a plurality of cameras for photographing the periphery of a vehicle and images taken by the plurality of cameras are input, and a synthetic image viewed from a virtual viewpoint is generated from these camera images. An image processing unit, and a display unit for displaying the composite image, the image processing unit,
At least one of the position of the virtual viewpoint, the direction of the line of sight, and the focal length is changed according to the running state of the vehicle.

More specifically, the present invention includes, as an image processing device, an image processing unit that receives images captured by a plurality of cameras that capture the surroundings of a vehicle and generates a composite image from these camera images. The image processing unit includes: a first image viewed from a virtual viewpoint; a virtual viewpoint and a position of the first image;
An image viewed from a viewpoint in which at least one of the line-of-sight direction and the focal length is different, or an image including the first image and a second image having a different model is generated as the composite image.

[0017] Preferably, the second image is at least one of the camera images.

Further, the first image is a nearby image showing the own vehicle and its surroundings, and the second image is a far image showing a region farther than the surrounding area of the own vehicle shown by the nearby image. It is preferred that And it is preferable that the said image processing part arrange | positions the said distant image around the said vicinity image in the said composite image. Furthermore, it is preferable that the far image is an image having continuity with the near image.

Further, the first image is an image showing at least a part of the own vehicle and at least a part around the own vehicle, and the second image is an image of an area indicated by the first image. It is preferable that the image is at least partially enlarged.

The present invention also provides, as a monitoring system, a plurality of cameras for photographing the periphery of a vehicle, an image processing unit which receives images taken by the plurality of cameras, and generates a composite image from these camera images. A display unit that displays the composite image, wherein the image processing unit includes a first image viewed from a virtual viewpoint, and at least one of a virtual viewpoint and a position of the first image, a line-of-sight direction, and a focal length. Any one of which is viewed from a different viewpoint or an image including the first image and a second image having a different model is generated as the composite image.

More specifically, the present invention includes an image processing unit which receives images taken by a plurality of cameras for photographing the periphery of a vehicle and generates a composite image from these camera images.
The image processing unit indicates at least a part of a vehicle area where the own vehicle is present and at least a part of the surrounding area of the own vehicle in the composite image, and includes at least one of a blind spot area around the vehicle that is not photographed by any camera. A warning area, which is a part, is displayed.

Preferably, the composite image is an image viewed from a virtual viewpoint set above the vehicle.

Further, it is preferable that the image processing section displays an illustration image or an actual image of the own vehicle in the vehicle area.

It is preferable that the image processing section determines a range of an area in which the blind spot area and the vehicle area are combined, using area data indicating a reflection area of the own vehicle in each camera image.

The present invention also provides, as a monitoring system, a plurality of cameras for photographing the periphery of a vehicle, an image processing unit which receives images taken by the plurality of cameras and generates a composite image from these camera images, A display unit that displays the composite image, wherein the image processing unit indicates at least a part of a vehicle area where the host vehicle is present in the composite image, and at least a part of a surrounding area of the host vehicle, from any camera. And a warning area which is at least a part of a blind spot area around the vehicle which is not photographed.

More specifically, the present invention includes, as an image processing apparatus, an image processing unit which receives images taken by a plurality of cameras for photographing the periphery of a vehicle and generates a composite image from these camera images. The image processing unit describes first mapping data describing a correspondence relationship between pixels of a composite image and pixels of a camera image, and describes an identifier indicating that the pixels of the composite image correspond to pixel data other than the camera image. The composite image is generated using a mapping table having second mapping data.

Preferably, the pixel data other than the camera image indicates the blind spot area in the vehicle or at least a part of the periphery of the vehicle.

Further, the image processing section stores a predetermined image other than a camera image, and the second mapping data stores, for a pixel of the composite image, the stored predetermined predetermined image corresponding to the pixel. It is preferable to describe coordinate values in an image.

[0029] Preferably, the second mapping data describes pixel data corresponding to pixels of a composite image.

According to the present invention, there is provided an image processing device, comprising: an image processing unit that receives captured images of a plurality of cameras that capture the surroundings of a vehicle and generates a composite image from these camera images; Describes the correspondence between the pixels of the composite image and a plurality of pixel data consisting of one or both of the pixel data of the camera image and the pixel data other than the camera image. Using the mapping data describing the degrees, weighting is performed on each pixel data according to the degree of necessity, and pixel data of the pixels of the composite image is generated.

According to the present invention, there is provided an image processing apparatus, comprising: an image processing unit which receives images taken by a plurality of cameras for photographing the periphery of a vehicle and generates a composite image from these camera images; Has an original mapping table, cuts out a mapping table describing the correspondence between pixels of the composite image and pixels of the camera image from the original mapping table, and generates a composite image using the cut-out mapping table. It is.

BEST MODE FOR CARRYING OUT THE INVENTION FIG. 1 is a conceptual diagram of a monitoring system according to the present invention. FIG.
In the monitoring system shown in (1), the image processing unit 2 receives a plurality of camera images output from the imaging unit 1 and combines them to generate a combined image when viewed from a virtual viewpoint. The composite image is displayed by the display unit 3 such as a liquid crystal display. By the image processing unit 2,
An image processing apparatus according to the present invention is configured.

In this description, it is assumed that the monitoring system is mounted on a vehicle and is used for assisting parking and the like.

FIG. 2 shows an example of camera arrangement, and FIG. 3 shows an example of an image picked up by each camera arranged as shown in FIG. As shown in FIG. 2, here, five cameras are arranged near the vehicle door mirror, near the rear pillar, and on the rear trunk. The image processing unit 2 generates a composite image when viewed from a virtual viewpoint from the five camera images illustrated in FIG.

The virtual viewpoint can be set at an arbitrary position in a three-dimensional space and in an arbitrary direction, similarly to the case of the camera arrangement in the image generation of computer graphics. The parameters for determining the virtual viewpoint are coordinates (X coordinate, Y coordinate,
Z coordinate), three angles representing its orientation, namely, azimuth (horizontal rotation), elevation (tilt) and Twist (rotation around the optical axis), and a focal length that determines the field of view. The focal length is the distance between the virtual viewpoint and the projection plane for generating the composite image. When the focal length is small, the image becomes a wide-angle image, and when it is large, the image becomes a telephoto image. In an actual camera or the like, the distance (m) when converted to the size of the film on the projection surface (35 mm)
m), but in this specification, since the size of the composite image is represented by pixels, the focal length is also considered by pixels.

By selecting the position, orientation, and focal length of the virtual viewpoint according to the situation, it is possible to generate an appropriate composite image.

FIG. 4 is a diagram conceptually showing the relationship between the virtual viewpoint and the real camera. In FIG. 4, the virtual viewpoint is set above the vehicle. FIG. 5 is an example of the generated composite image, and an image showing the own vehicle and its vicinity viewed from the virtual viewpoint shown in FIG. 4 is generated.

In the present embodiment, the image processing section 2 uses the mapping table 4 to generate a composite image from a plurality of camera images. Here, "mapping table"
Is a table in which the correspondence between the pixels of the composite image and the pixel data of each camera image is described. In addition,
As will be described later, the mapping table can also describe the correspondence between pixels of the composite image and pixel data other than the camera image.

<Description of Principle of Mapping Table> The operation of generating a composite image from a plurality of camera images using the mapping table will be described below.

FIG. 6 is a diagram for explaining the outline of this operation. FIGS. 6A to 6C show captured images of three cameras (corresponding to camera 3, camera 1 and camera 2 in FIG. 2, respectively) attached to different positions. FIG. 6D is a mapping table used to generate a composite image from each camera image of FIGS. 6A to 6C. The image processing unit 2 converts one camera image shown in FIG. 6E from the three camera images shown in FIGS. 6A to 6C based on the information in the mapping table shown in FIG. 6D. Is generated.

The mapping table shown in FIG.
Each pixel of the composite image shown in FIG. 6E has mapping data. Each mapping data describes information indicating which camera image pixel generates a corresponding composite image pixel.

Generating a composite image means determining the values of all pixels in the composite image. Here, it is assumed that the image processing unit 2 determines the values of the pixels of the composite image sequentially in the raster order from the upper left, and the operation will be described with an example of determining the value of the pixel P1 in the middle.

First, reference is made to the mapping table MP1 corresponding to the pixel P1 of the composite image. Mapping data M
In P1, the corresponding camera number and the coordinates of the corresponding pixel of the camera image are described. Now, in the mapping data MP1, "1" is described as the camera number, "340" as the X coordinate, and "121" as the Y coordinate.

The image processing section 2 has the mapping data MP1
The pixel data C1 at the coordinates (340, 121) of the image captured by the camera 1 shown in FIG. Here, as the simplest determination method, the value of the pixel data C1 is directly used as the value of the pixel P1. By determining a value for each pixel of the composite image in a similar manner, a composite image as shown in FIG. 6E is generated.

For example, the mapping data MP2 corresponding to the pixel P2 of the composite image indicates the pixel data C2 of the image captured by the camera 2 shown in FIG.
Is given the value of the pixel data C2. Similarly,
Mapping data MP3 corresponding to pixel P3 of the composite image
Indicates the pixel data C3 of the image captured by the camera 3 shown in FIG. 6A, so that the value of the pixel data C3 is given as the value of the pixel P3. In the mapping table of FIG. 6D, the region R1 corresponds to the camera 1, the region R2 corresponds to the camera 2, and the region R3 corresponds to the camera 3.

Three regions R1 to R3 corresponding to each camera
The other region R4 is a region where there is no corresponding camera image due to a region other than the photographing region of each camera or a blind spot hidden by the own vehicle. For example, the pixel P4 of the composite image has such a region R4
Pixel. In this case, in the camera number of the mapping data MP4 corresponding to the pixel P4, a specific camera number (here, “−1”) indicating that there is no corresponding camera image is described. When the camera number is “−1”, the corresponding pixel P4 is set with predetermined pixel data indicating that the pixel is outside the shooting area or a blind spot. Here, black is set as the predetermined pixel data.

In the example of FIG. 6, the mapping table is configured to generate a composite image that looks down around the vehicle from a virtual viewpoint above the vehicle. As will be described later, such a mapping table can be created by using a geometric transformation on the premise of a so-called road surface plane model. Alternatively, it may be created by trial and error while viewing the composite image.

Actually, by using the mapping table, the correspondence between the composite image and the pixel data of the camera image can be freely set according to the application. For example, an arbitrary synthesized image can be generated by enlarging or reducing an arbitrary region of the camera image and synthesizing it with a part of the camera image, or by arranging and synthesizing a plurality of camera images.

Regardless of the mapping table for generating any composite image, the image processing unit 2 refers to the mapping data, refers to the pixel data of the designated camera image, Since it is only necessary to execute the step of setting the value, the processing amount is much smaller than in the case where the image synthesis is executed by calculation each time. For this reason, this system has a constant processing amount regardless of the type of synthesized image and can perform high-speed processing, so it is used for monitoring and driving assistance, etc., which need to finish processing in a fixed time in real time. In particular, they are particularly suitable.

(First Example of Synthesized Image Generation Operation) FIG. 7 is a diagram showing a configuration example of a monitoring system according to the present embodiment. In FIG. 7, an image processing unit 2 includes an image synthesizing unit 2
00, a display pattern storage unit 210 and a display pattern setting unit 220. Display pattern storage unit 210
Has a mapping table storage unit 211 for storing a plurality of mapping tables as described above. The display pattern setting unit 220 stores the mapping table storage unit 2
One mapping table MPT is selected from the mapping tables stored in 11 according to the display pattern of the synthesized image to be generated, and set in the mapping table reference unit 202 of the image synthesis unit 200.

The display pattern storage section 210 has an illustration storage section 212 for storing various illustration images. The display pattern setting unit 220 stores an illustration image necessary for generating a composite image in the illustration image storage unit 212.
From the image synthesizing unit 200.
Set to 3. Here, it is assumed that an illustration image indicating the own vehicle is set.

The image synthesizing unit 200 uses the camera image output from the imaging unit 1 in accordance with the mapping table MPT set in the mapping table reference unit 202.
Generate a composite image. The timing generation unit 205 generates a timing signal for generating a moving image sequence of a composite image.

Each camera 101 is provided with a pair of frame memories 102a and 102b. Here, it is assumed that each camera 101 is a CCD type. When the camera is a CMOS type, the camera can have a function of a frame memory. In this case, the frame memory can be omitted.

FIG. 8 is a flowchart showing the operation of the image synthesizing unit 200 when generating one frame of the synthesized image.

First, in response to the frame start timing signal output from the timing generation unit 205, the imaging unit 1
Switches between a frame memory 102a for writing an image captured by each camera 101 and a frame memory 102b referred to by the image synthesis unit 200 (step S1).
1). Here, the reason why two frame memories are provided for each camera 101 to perform the switching is that the image synthesizing unit 200 writes the pixel data of the camera image according to the order of writing from the camera 101 as described later. Does not matter
This is because writing and reference do not interfere with each other because the reference is made in a discrete manner according to the mapping table MPT.

Next, the timing generation section 205 generates a timing signal for designating a pixel to be subjected to the synthesis processing to the mapping table reference section 202 (step S1).
2). The mapping table reference unit 202 reads mapping data corresponding to the designated pixel from the mapping table MPT, and outputs the data to the pixel synthesizing unit 201 (step S13).

The pixel synthesizing unit 201 outputs the pixel data of each camera image stored in the frame memory 102 and the illustration referring unit 2 according to the contents of the input mapping data.
The pixel value of the designated composite image is generated using the pixel data of the illustration image stored in the image data 03, and is output to the video signal generation unit 204 (step S14). The processing in step S14 will be described later in detail.

The video signal generation unit 204 converts the pixel value of the input composite image into a video signal according to the timing signal output from the timing generation unit 205, and
(Step S15).

The image synthesizing section 200 executes the processing of steps S12 to S15 for all the pixels of the frame (steps S16 and S17). When the processing of the last pixel of the frame ends, the timing generation unit 205 starts the processing of the next frame.

The same processing can be executed on a field basis.

FIG. 9 is a diagram showing an example of the configuration of the mapping table. The mapping table shown in FIG. 9 has four types of mapping data MP11 to MP14.
The configuration of each mapping data is basically the same as the mapping data shown in FIG. 6, but the camera number and x, y
In addition to the coordinates, the necessity of the pixel value is described.
Here, the “necessity” is represented by a value from 0 to 1, and the larger the value, the higher the necessity of the pixel value.

The mapping data MP11 describes the correspondence between the pixels of the composite image and the pixel data of one camera image. In this case, the necessity is “1”. The mapping data MP12 describes the correspondence between pixels of the composite image and pixel data of a plurality of camera images. In this case, the pixel value of the composite image is
The pixel data of the camera image is generated after being weighted according to the necessity.

The mapping data MP13 is used for pasting pixel data of an illustration image to pixels of a composite image. That is, the illustration reference unit 20
A number (here, “99”) that does not correspond to any real camera is assigned to the illustration reference unit 203 so that the illustration image stored in No. 3 can be recognized.

The mapping data MP14 indicates that the pixels of the composite image correspond to a so-called blind spot area. That is, when generating the mapping table, as a result of calculating the coordinate value of the pixel of the camera image to be referred to generate the pixel of the composite image, if the pixel of the coordinate value is, for example, taking a picture of the vehicle itself Means that the pixel of the composite image corresponds to the combined area of the vehicle area and the blind spot area. The area excluding the vehicle area from this area is a blind spot area.
For this reason, for example, a non-existent value is given to the camera number or the x, y coordinates to indicate that the area is a blind spot area.
Here, “−1” is given as the camera number.

FIG. 10 is a flowchart showing the detailed processing flow of the pixel synthesizing step S14.

First, the value of the pixel to be synthesized is initialized to "0" (step S21).

Next, the camera number, the x and y coordinates and the necessity are read from the mapping data corresponding to the pixel to be synthesized (step S22).

In step S22, when the read camera number is the identification number “99” indicating the illustration image (that is, in the case of the mapping data MP13), the process proceeds to step S24, where the illustration image stored in the illustration reference unit 203 is deleted. The pixel data of the designated x, y coordinates is read and held. Then, the process proceeds to step S28. On the other hand, if the camera number is not "99", the flow proceeds to step S25.

In step S25, when the read camera number is the identification number "-1" indicating the blind spot area (that is, in the case of the mapping data MP14), the process proceeds to step S26, where the pixel data representing the blind spot area set in advance is stored. Hold. Then, the process proceeds to step S28.
On the other hand, if the camera number is not “−1”, the step S
Proceed to 27.

In step S27, if the camera number is other than "99" or "-1", that is, if it is determined that the area is not the area where the illustration image is to be pasted or the blind spot area, this camera number is the number indicating the actual camera. As there are
From the camera image stored in the frame memory 102 of the corresponding camera number, the pixel data of the designated x and y coordinates is read and held.

In step S28, the held pixel data is weighted according to its necessity, and added to the pixel value of the composite image. The processing of steps S22 to S28 is repeated for all camera numbers described in the mapping data (steps S29 and S30). When the processing is completed for all camera numbers, the pixel synthesizing unit 201 outputs the values of the pixels to be synthesized (step S31).

For example, the mapping data MP1 shown in FIG.
In the case of 2, the pixel value of the composite image is obtained by the following equation.

Pixel value of composite image = (pixel value of coordinates (10,10) of camera 2 × 0.3 + pixel value of coordinates (56,80) of camera 3 × 0.5) / (0.3 + 0. 5) Here, the reason for dividing by the sum of the necessity (0.3 + 0.5) is to normalize the pixel value.

By the above operations, a composite image obtained by mixing a plurality of camera images and a composite image including an illustration image can be easily generated. By weighting the camera image and the illustration image according to the degree of necessity, it is also possible to generate a composite image that displays the illustration image translucently on the real image. Alternatively, it is also possible to generate a composite image in which the illustration images are displayed translucently by weighting pixel data other than the camera image according to necessity.

As the illustration image, in addition to the illustration and the image of the own vehicle, for example, an image having a fixed shape on the screen, such as an image scale or an index, can be given.

In this description, the camera number (“99”) for referring to the illustration image and the camera number (“−1”) indicating the blind spot area are individually set. However, if an image representing the blind spot area is stored in the illustration reference unit 203, the same camera number (“9
9 "). In this case, for example, it is also possible to store the illustration image representing the vehicle and the image representing the blind spot area together in the illustration reference section 203.

When changing the display pattern in accordance with the designation of the display mode, the display pattern setting section 220 reads a mapping table corresponding to the new display pattern from the mapping table storage section 211, and
What is necessary is just to set it in the mapping table reference part 202. Alternatively, a plurality of mapping tables may be combined to generate a new mapping table.

In this case, the X and Y coordinate values of the illustration reference unit 203 are described to display the illustration image, as in the mapping data MP13 shown in FIG. The pixel data itself can be described in the mapping data.

FIG. 11 shows an example of the mapping data in such a case. Mapping data MP15 in FIG.
When the camera number is “99”, that is, when an illustration image is displayed, the pixel data itself of the illustration image is directly stored in the area for storing the X coordinate and the Y coordinate in red (R), green (G), and blue ( B) is stored.

For example, if the X coordinate and the Y coordinate are each represented by 16 bits, the area has a size of 32 bits. On the other hand, the image data of the illustration image is R,
Assuming that each color of G and B is represented by 8 bits, a total of 24
Therefore, "0" is added to the upper 8 bits, and the data is stored in the coordinate area as 32-bit data. In this case, in step S24 of FIG.
Instead, R, G, and B values described in the mapping data MP15 are read out and held.

In this case, the mapping table and the illustration image are integrated, and the illustration reference unit 203
Becomes unnecessary. Also, instead of reading out the X and Y coordinate values from the mapping data and reading out the pixel data of the corresponding illustration image, R, G,
Since the B value is read, one processing procedure can be omitted.

In the case of displaying an illustration image of the own vehicle, the necessity is usually set to 1.0, but as shown in the example of FIG.
By making the value smaller than 0 and combining with another camera image, it becomes possible to display the scale and the index in a translucent manner on the combined image.

<Example of Basic Synthetic Image> The mapping table is large and can be divided into a single map and a composite map. "Single map"
The camera image and the virtual viewpoint image are associated at a pixel level using a predetermined space model (details will be described later). The “composite map” will be described later.

The display mode of the image processing unit 2 is determined depending on which mapping table is set in the image synthesizing unit 200. This display mode can be switched manually or automatically.

A typical single map and an example of a composite image using the same are shown below.

Vertical looking down (FIGS. 12 to 14) As a spatial model, a road plane model is used. The position of the virtual viewpoint is above the own vehicle, and the direction of the line of sight is directly below. The feature is that the sense of distance can be understood at a glance, and parking (right angle, parallel) or the like is considered as a main use. FIG. 12 shows an example using eight camera images, FIG. 13 shows an example using four camera images, and FIG. 14 shows an example using two camera images.

Looking down obliquely (FIG. 15) As a space model, a road surface plane model is used. The position of the virtual viewpoint is above the own vehicle, and the direction of the line of sight is obliquely rearward. The field of view can be freely changed by adjusting the angle of the line of sight. In this mode, it is possible to overlook the rear of the vehicle with a correct positional relationship. As a main use, it can be considered as a rear-view monitor during normal running, or confirmation at low-speed retreat such as at the start of a parking operation. When traveling at low speed, the field of view and direction may be switched according to the speed.

Panorama (FIGS. 16 and 17) A cylindrical model is used as a space model. FIG. 16 shows the front panorama mode, in which the position of the virtual viewpoint is the front end of the vehicle and the direction of the line of sight is directly in front. In this mode, it is possible to know the state in front of the traveling direction, so that it can be used as a blind corner view monitor. That is, when the vehicle goes out from a narrow alley to a wide street, the mode of the wide street can be recognized at a glance by this mode.

FIG. 17 shows the rear panorama mode, in which the position of the virtual viewpoint is the rear end of the vehicle and the direction of the line of sight is right behind. In this mode, the panorama 1
You can overlook it as an 80-degree image. As a main use, it can be used as a rear view monitor during traveling (particularly at high speed traveling).

Obliquely looking down + panorama (FIG. 18) As the space model, a road surface plane model and a cylindrical model are used. In other words, as shown in FIG. 18, the area immediately behind the host vehicle is displayed in an oblique looking down mode using a road surface plane model, and the area far behind the host vehicle is displayed in a panoramic mode using a cylindrical model. Therefore, this mode is compatible with both the oblique looking down mode and the panoramic mode.

<Automatic Display Switching> The present monitoring system can support safer and more comfortable driving by appropriately switching the mapping tables exemplified so far according to various driving situations. In addition, even in one driving operation, switching the mapping table according to the situation of the surroundings of the own vehicle that changes sequentially is an essential function for the driver.

That is, at least one of the position of the virtual viewpoint, the direction of the line of sight, and the focal length of the composite image is
By changing according to the traveling state of the vehicle, the convenience for the driver can be improved.

FIG. 19 is a diagram showing an example in which the height of the virtual viewpoint of the composite image is changed according to the running state of the vehicle. In each of the images shown in FIGS. 19A to 19F, the height of the virtual viewpoint gradually increases toward the bottom, and the display area including the vehicle gradually increases. That is, zoom-in is realized by lowering the height of the virtual viewpoint, and zoom-down is realized by raising the height of the virtual viewpoint. Even when the focal length of the virtual viewpoint is changed, the composite image can be changed in the same manner as in FIG.

As a trigger for switching the height of the virtual viewpoint, first, the traveling speed of the vehicle can be considered. For example, as the traveling speed of the vehicle increases, the virtual viewpoint may be raised so that a wider area is displayed, and as the vehicle speed decreases, the virtual viewpoint may be lowered so that a narrower area is displayed. Alternatively, when the vehicle includes an object detection sensor that measures the distance between the vehicle and the obstacle, the output signal of the object detection sensor may be used as a switching trigger. For example, as the distance to the detected obstacle decreases,
The virtual viewpoint may be lowered so that a narrow area is displayed, and the virtual viewpoint may be raised so that a wider area is displayed as the distance increases. Furthermore, a changeover switch may be provided so that the driver or another occupant can specify enlargement / reduction via the switch.

FIG. 20 is a diagram showing an example in which the direction of the line of sight of the virtual viewpoint of the composite image is changed according to the running state of the vehicle. In each of the images shown in FIGS. 20A to 20D, the direction of the line of sight of the virtual viewpoint gradually changes from diagonally backward to vertically downward as going downward. Also, the model for generating the composite image changes along with the direction of the line of sight.
That is, as the direction of the line of sight is inclined obliquely, the area of the composite image of the pseudo cylinder model becomes larger, and it becomes easier to see far away.

In FIG. 20, it can be said that the change of the direction of the line of sight of the virtual viewpoint and the capture of the image outside the visual field range of the changed virtual viewpoint are controlled. That is, by using the pseudo-cylindrical model, an image outside the visual field range of the virtual viewpoint is captured in the composite image. The presence / absence of an image outside the visual field of view of the virtual viewpoint, its size, and its imaging range are controlled in accordance with the traveling state of the vehicle.

As the trigger for switching the direction of the line of sight of the virtual viewpoint, the traveling speed of the vehicle, switch input, and the like are assumed as in the case of switching the height of the virtual viewpoint.
For example, when the traveling speed of the vehicle is low, the direction of the line of sight may be set directly below, and as the traveling speed increases, the direction of the line of sight may be tilted so as to display a more rearward direction.

FIG. 21 is a diagram showing an example in which the direction of the line of sight of the virtual viewpoint of the composite image is changed according to the steering angle of the vehicle. In the example of FIG. 21, the virtual viewpoint is rotated around the optical axis according to the steering angle. (A) is a composite image in the case where the gear is reverse and the steering wheel is straight. In this case, since the vehicle goes straight backward, the virtual viewpoint is set so that the own vehicle in the composite image is straight and the left and right regions of the own vehicle are displayed evenly so that the rear area is easy to see. I do. On the other hand, (b) is a composite image when the steering wheel is turned to the left with the gear in reverse. In this case, since the vehicle travels to the rear left of the vehicle, the virtual viewpoint is rotated around the optical axis so that the region where the vehicle moves at that time, that is, the right side, the rear right side, and the rear side of the own vehicle becomes easy to see. This increases safety.

FIG. 22 is a diagram showing an example of image switching according to the output signal of the object detection sensor. In the figure, (a) is a screen before the object detection sensor detects an obstacle approaching the own vehicle. When the object detection sensor detects another vehicle close to the own vehicle, the screen switches as shown in FIG. That is, the image is enlarged and displayed so that the distance to the detected obstacle can be more recognized.

As described above, when the object detection sensor detects an obstacle, the image is displayed in an enlarged or reduced / rotated manner in a stepwise or continuous manner according to the distance to the obstacle, thereby displaying the obstacle. The occupants can be alerted to things. Further, the position of the virtual viewpoint may be changed according to the position where the obstacle exists.

Further, as shown in FIG. 23, an area where an obstacle is present may be enlarged and displayed, and the area may be surrounded and displayed. Alternatively, the frame may blink, or the color in the frame may be inverted. In this way, the occupant's attention can be more reliably raised.
Of course, it is needless to say that the occupant can be alerted even if the image is simply displayed in a frame without being enlarged. In addition, the presence of an obstacle may be notified by a warning sound along with the change of the screen display.

FIG. 24 is a diagram showing another example of image switching according to the detection result of the object detection sensor. In FIG. 24, when an obstacle is detected behind the vehicle, the direction of the line of sight of the virtual viewpoint is changed from directly below to slightly backward. This makes it easier to see obstacles behind the vehicle. Further, the virtual viewpoint may be translated so that the detected obstacle is at the center of the composite image.

Although FIGS. 19 to 24 show an example of a composite image in the case where the virtual viewpoint is above the vehicle, FIGS.
Even when the virtual viewpoint is at another position, the convenience of the driver is improved by changing at least one of the position of the virtual viewpoint, the direction of the line of sight, and the focal length according to the traveling state of the vehicle. Can be improved. Other examples of the position of the virtual viewpoint include the position of the rear trunk and the position of the driver's head.

In FIG. 20, the capture of the image outside the visual field range is controlled together with the change in the direction of the line of sight of the virtual viewpoint. May be controlled. Further, an image outside the visual field of the virtual viewpoint may be captured without using a model. Furthermore, without changing the virtual viewpoint, only the control of capturing the image outside the visual field range of the virtual viewpoint may be performed according to the traveling state of the vehicle.

(Second Example of Synthetic Image Generation Operation) Switching of the display image can be easily realized by appropriately changing the mapping table used. However, in this case, in order to continuously switch the display images, it is necessary to prepare a large number of mapping tables. For this reason, it is necessary to provide a storage unit having a huge storage capacity in the apparatus, which is not preferable.

Here, it is assumed that an original mapping table larger than the synthesized image is provided, and a synthesized image is generated using the mapping table cut out from the original mapping table. In this case, by appropriately changing the mapping table to be cut out, continuous switching of display images can be easily realized.

FIG. 25 is a diagram showing the configuration of a monitoring system according to this example. 7 is different from the configuration illustrated in FIG. 7 in that the map area setting unit 302 determines the coordinates of the pixels of the composite image output from the timing generation unit 205 and the display pattern setting unit 22.
The point is that a corresponding mapping table element is generated based on the mapping table area designation information set by 0A and output to the mapping table reference unit 202. In the mapping table reference section 202, the original mapping table stored in the original mapping table storage section 301 of the display pattern storage section 210A is set by the display pattern setting section 220A.

That is, in the first example described above, in step S13 shown in FIG. 8, for the pixel of the composite image to be currently output set by the timing generation unit 205, the mapping data at the same position as this pixel is read out. However, in this example, after setting the area of the mapping table to be used on the original mapping table, the mapping data corresponding to the pixels of the composite image is read.

A method of reading mapping data in this example will be described. FIG. 26 and FIG. 27 schematically show the relationship between the composite image and the original mapping table in this example. In the example of FIG. 26, the size of the region to be cut out from the original mapping table is fixed to the same size as the composite image, and the offset value is changed to realize the parallel movement of the composite image. In the example of FIG. 27, enlargement / reduction of the synthesized image is realized by changing the size of the region cut out from the original mapping table and the step of reading the elements.

First, the case of performing parallel movement of a composite image will be described with reference to FIG.

The composite image shown in FIG. 26A has a width W_DI
It has the size of SP and height H_DISP, and FIG. 26 (b)
Is a width W_MAP, a height H_
It has the size of DISP. Here, as an original mapping table, a table that creates an image as if looking down a wide area around the vehicle from a virtual viewpoint above the vehicle. This is because, in the first example described above, if the focal length is constant, the size of the projection plane is (W_DISP, H_DISP) while the distance between the virtual viewpoint and the projection plane is constant.
From (W_MAP, H_MAP)
An original mapping table may be configured. Due to the enlargement of the projection plane, the original mapping table contains information on a wider area. FIG. 26D is an image in which each element of such a mapping table is replaced with a corresponding camera image and illustration image.

Next, the display pattern setting section 220A specifies, on the original mapping table, an area to be used as a mapping table for generating a composite image by using an offset (off_x, off_y) from the origin (0, 0). An area MPT of a size (W_DISP, H_DISP) starting from the point (off_x, off_y) is
This is a mapping table used for image synthesis. X, Y of pixels obtained by raster-scanning the synthesized image
The mapping data at the position obtained by adding the offset (off_x, off_y) to the coordinates is read from the mapping table reference unit 202 and input to the pixel synthesizing unit 201. The pixel synthesizing process is performed according to the flowchart of FIG. 10 as in the first example. FIG. 26 (c) is a composite image obtained using the mapping table MPT shown in FIG. 26 (b).

When the coordinates to which the offset has been added exceed the range of the original mapping table, the pixel synthesizing unit 201 is notified and a display such that a predetermined color (for example, black) or the like is used as a pixel value so that the area can be recognized. I do.

According to this example, the virtual viewpoint is arranged so as to vertically look down on the road surface from above the vehicle, and when a road surface plane model is used, the composite image obtained when the virtual viewpoint is moved in parallel with the road surface is mapped. Generation can be performed without increasing the number of tables. That is, in the above-described first example, a mapping table corresponding to each of the virtual viewpoints that have been translated is required, but in this example, only one original mapping table is required. For example, when the virtual viewpoint is moved one pixel at a time in order to perform a smooth viewpoint movement, a large number of mapping tables corresponding to all the movement steps are required in the first example described above. Start position (off_x, o) for cutting out the mapping table in one original mapping table
ff_y) can be realized. In this example, for example, to change the position of the virtual viewpoint by turning a dial, or to change the position of the virtual viewpoint in the direction of the obstacle in proportion to the magnitude of the output of the obstacle detection sensor, Since the position of the virtual viewpoint can be changed in fine steps, a composite image according to the situation can be easily generated.

In this example, since it is not necessary to use a large number of mapping tables to translate the virtual viewpoint in fine steps in parallel, the memory for storing the mapping tables has a capacity capable of storing one original mapping table. If there is. Further, in the first example, it takes time to switch and set the mapping table, whereas in the present example, only the setting of the offset is required, so that the processing can be speeded up.

Next, a case of enlarging / reducing a composite image using the same method will be described with reference to FIG.

In FIG. 27, timing generation section 205
Is assumed to designate the coordinates (x, y) of the pixel P1 of the composite image shown in FIG. The display pattern setting unit 220A specifies the element reading step (step_x, step_y) together with the area start position (off_x, off_y). At this time, if the coordinates of the mapping data MD corresponding to the pixel P1 are (u, v), the coordinates (u, v)
Is determined by the following equation.

U = step_x * x + off_x v = step_y * y + off_y By changing the value of the reading step (step_x, step_y), it is possible to smoothly enlarge / reduce the synthesized image. The area of the mapping table cut out from the original mapping table is set to MP
When it changes to T1, MPT2, MPT3, MPT4,
The composite image changes as shown in (d), (e), (f), and (g).

When the coordinates (u, v) exceed the range of the original mapping table, a designated color or the like is output as in the case of the above-described parallel movement. In addition, a value other than an integer can be used as the value of the reading step (step_x, step_y). In this case, (u, v) of the calculation result is used.
Converts the value of to an integer.

According to the enlargement / reduction according to the present example, the virtual viewpoint is arranged so as to vertically look down on the road surface from above the vehicle, and when the road surface plane model is used, the view range is changed by changing the height or the focal length of the virtual viewpoint. A composite image can be generated in the same manner as when the image is enlarged or reduced.

For example, when a part of the original image is enlarged and displayed in response to an instruction from a driver or the like or a sensor input, the enlarged image is not output abruptly but the method according to the present embodiment is used to smoothly display the enlarged image. By displaying the enlarged image, it is easy to grasp which region of the original image is enlarged by the enlarged image.

Further, by changing the offset value along with the change of the readout step, it is possible to realize not only enlargement / reduction of the synthesized image but also parallel movement.

In general, it is not necessary to limit the area of the mapping table to a rectangle. That is, the display pattern setting unit 220A may specify the area of the mapping table by the coordinates of the four vertices of the quadrilateral.

In FIG. 28, the area of the mapping table is a convex quadrilateral and designated by four points n1, n2, n3, and n4. At this time, the coordinates (u, v) of the mapping data MD corresponding to the pixel P1 at the coordinates (x, y) of the composite image
Can be obtained as follows.

First, the vertices n1 and n2 are set to (y / H_DIS
P: 1−y / H_DISP), and the coordinates of a point na that is internally divided. Similarly, vertices n3 and n4 are set to (y / H_DIS
P: 1−y / H_DISP), and obtain a point nb that is internally divided. Next, the points na and nb are set to (x / W_DISP: 1-y
/ W_DISP), the coordinates (u, v) of the corresponding mapping data MD are obtained when the coordinates of a point internally divided are obtained.

With such designation, it is possible to designate an area of an arbitrary mapping table including translation, enlargement, reduction, and rotation.

According to the present method, when the virtual viewpoint looks down the road surface vertically from above using the road surface plane model, the virtual viewpoint can be translated and the visual field range can be expanded or reduced.
A composite image can be generated that exactly matches a change in virtual viewpoint, such as a reduction or rotation parallel to the road surface. However, if the orientation of the model or virtual viewpoint is different, generating a composite image that exactly matches the change of the virtual viewpoint is
Not always. However, even in such a case, it is possible to generate a composite image that approximates a change in the virtual viewpoint, and thus this method is extremely effective.

<Plural Types of Image Display> By displaying a plurality of types of composite images together on one screen, or simultaneously displaying a composite image and a camera image on one screen, the convenience of the driver is improved. Can be enhanced. For example, the near and far directions of the vehicle can be displayed at the same time,
By displaying images in different directions at the same time or displaying the entire vehicle and a part of the enlarged vehicle at the same time, the driver can view the surroundings of the vehicle without switching the screen display. Can be accurately understood.

Such a plurality of types of image display can be easily realized by using a composite map. The “composite map” refers to a mapping table in which a necessary portion of a single map is cut out, appropriately deformed, and pasted together, or a camera image is pasted onto the single map. The image display as described below can be easily realized using a composite map. However, other image displays can be performed in various driving modes by combining single maps or combining a single map and a camera image. It is possible to create a composite map according to the scene.

Here, a top-down image using a road surface plane model as a nearby image showing the own vehicle and its surroundings viewed from a virtual viewpoint set above the vehicle, and a surrounding area of the own vehicle showing this top-down image An example in which an image including a distant image indicating an area farther than that is displayed will be described. FIG. 29 is a diagram showing a camera image which is a premise of the description here.

FIG. 30 shows an example in which camera images of the second image are photographed obliquely rearward on the left and right sides of the top-down image as the first image. In this example, it can be said that the viewpoint of the second image is different from the virtual viewpoint of the first image in the position, the direction of the line of sight, and the focal length. In the image of FIG. 30, the image of the camera 2 that captures the landscape obliquely to the right and left is reversed and arranged on the right side of the top-down image.
Are arranged left and right inverted. That is, the cameras 2 and 6 installed near the left and right door mirrors are arranged left and right inverted so that the images can be actually seen through the door mirrors. Therefore, the driver can intuitively understand the situation around the vehicle with the same feeling as looking at the door mirror. Of course, the image may be displayed without being reversed left and right.

FIG. 31 shows an example in which a top-down image as a first image showing a narrow range and a top-down image as a second image showing a wide range are displayed side by side. In this example, it can be said that the virtual viewpoint of the second image is different in height or focal length from the virtual viewpoint of the first image. Looking at the image of FIG. 31, the vicinity of the own vehicle can be viewed in detail, and the wide range can be viewed at once as thumbnails.

FIG. 32 shows an example in which a front, rear, left and right oblique view image as a second image is pasted around a view image as a first image. In this example, the virtual viewpoint of the second image has a different line of sight from the virtual viewpoint of the first image. FIG. 33 shows an example in which an image as seen through a fisheye lens using a pseudo-cylindrical model as a second image is pasted around a top-down image as a first image. In this example, the second image is different in model from the first image. FIG. 34 is an example in which a front, rear, left, and right panoramic image as a second image is pasted around a top-down image as a first image. In this example, the position of the virtual viewpoint and the model of the second image are different from those of the first image. Looking at the images of FIGS. 32 to 34, it is possible to maintain a sense of distance in the vicinity of the own vehicle and to overlook the distant surroundings of the own vehicle.

It is not always necessary to paste the distant image on all front, rear, left and right of the top-down image, and the distant image may be displayed only in a desired direction, for example, only on the right side or only behind.

As shown in FIG. 35, when pasting the obliquely looking down image around the top down image in front, rear, left and right directions, blank portions are provided at the four corners of the display area to confirm that the oblique looking down image has no continuity with each other. It may be emphasized.

Further, when displaying the images as shown in FIGS. 32 to 35, the surrounding distant images may be blurred and displayed by performing a filtering process. Thereby, the driver's gaze area can be concentrated around the own vehicle. Further, it is possible to make the distortion of the surrounding distant image inconspicuous.

Such partial blurring processing is effective even in the case of displaying an image using a single map. For example, even in the case of a top-down image as shown in FIG. 5, the blur processing is not performed on the own vehicle and the vicinity thereof, and the blur processing is performed on a peripheral area where another parked vehicle is reflected. Similar effects can be obtained. In the case of parking or the like, it is appropriate to set the area where the blurring processing is not performed to an area within 10 m from the host vehicle. Further, the blur intensity may be increased as the distance from the host vehicle increases.

The image shown in FIG. 31 is composed of a self-vehicle, a first image showing the periphery of the self-vehicle, and a second image obtained by enlarging at least a part of the area shown by the first image. It can be said that it is an image. Of course, the first image may indicate a part of the host vehicle, not the entire host vehicle, or a part around the host vehicle.

Further, the display screen is made multi-window,
The composite image, the camera image, the image indicating the position of the virtual viewpoint, the character information, and the like may be displayed in the sub window. As a result, the situation around the host vehicle can be more easily understood, and the convenience of the driver is improved.

Various display methods for the multi-screen are conceivable. For example, as shown in FIG. 36, an image indicating the position of the virtual viewpoint may be displayed in the subwindow together with the camera image and the composite image. Thereby, the position of the virtual viewpoint can be easily grasped.

For example, when the object detection sensor detects an obstacle, a warning mark indicating the position of the obstacle is displayed on the camera image or the composite image of each sub-window, and characters are displayed in another sub-window by characters. The position of the obstacle may be displayed.

<Display of Blind Spot Area and Own Vehicle Area> (Part 1) FIG. 37 is a view showing an example of a camera image and mask data as area data indicating a reflection area of the own vehicle in the camera image. As shown in FIG. 37, a portion where the own vehicle is reflected in each camera image is recognized, and mask data indicating a reflected region of the recognized vehicle is stored in the image processing unit 2 in advance. The black part in the mask data shown in FIG. 37 is the reflection area of the vehicle. The reflection area of the vehicle can be uniquely determined in advance once the specifications and orientation of the camera and the shape of the own vehicle are determined.

FIG. 38A is a vertical looking down image synthesized using the camera image of FIG. In the composite image shown in FIG. 38 (a), the host vehicle appears in the center of the image in white, making it somewhat hard to see.

Therefore, the mask data of each camera image is
The image is converted into an image viewed from a virtual viewpoint, and the reflection area of the vehicle is painted out in the composite image. As a result,
An image as shown in FIG. 38B is obtained, and the visibility is improved.

In FIG. 38 (a), the own vehicle is reflected in the center of the image, which makes it difficult to see, and it is not clear where the own vehicle is actually located.
It is difficult to drive. Therefore, the illustration image or the actual image of the own vehicle is superinvoked at a position where the own vehicle actually exists. For example, the image processing unit 2 previously stores an illustration image or a real image of the own vehicle on the assumption that the image is viewed from the standard virtual viewpoint, and displays the image as viewed from the virtual viewpoint of the composite image to be displayed. And then superimpose it. As a result, an image as shown in FIG. 38C was obtained, and the size and the relative positional relationship of the host vehicle with respect to the surrounding objects could be understood at a glance. The conversion of the own vehicle image can be easily realized by using the above-described mapping table.

In this case, as the standard virtual viewpoint, for example, a virtual viewpoint provided above the own vehicle, a virtual viewpoint located on the side, or a virtual viewpoint provided in front of or behind the own vehicle can be used. it can. Alternatively, a plurality of standard virtual viewpoints may be provided, and an image may be prepared for each of the plurality of standard virtual viewpoints. When the position of the virtual viewpoint for generating the composite image is determined, a standard virtual viewpoint that minimizes image distortion during image conversion is selected from the positional relationship between the virtual viewpoint and each standard virtual viewpoint. It doesn't matter.

Instead of pasting the illustration or actual image of the vehicle, a 3D model of the vehicle using a surface model or solid model known in the field of CAD / CAM or CG is pasted on the composite image. It is also possible to Also in this case, 3
It is sufficient to prepare an image representing the dimensional model in advance, convert this image into an image viewed from a virtual viewpoint of the composite image to be displayed, and superimpose it. This conversion can also be easily realized by using the above-described mapping table.

Further, a composite image itself generated from a camera image may be used as an image indicating an area where the host vehicle actually exists. In addition, part of own vehicle,
For example, only the bumper portion may be displayed as a composite image, and the other portions may be pasted with illustrations or actual images.

(Part 2) A composite image viewed from a virtual viewpoint may include an area that does not belong to the imaging range of any camera. Further, there may be a case where an area that cannot be imaged by each camera is included because the own vehicle is obstructed.

In order to solve this problem, in the above (Part 1), the composite image is made easier to see by giving predetermined pixel data to the reflection area of the vehicle and filling it.

Here, as shown in FIG. 39, when the camera image showing the own vehicle is directly converted into a virtual viewpoint image,
The converted own-vehicle image is included in the composite image, making it difficult to see (A1). In this case, the visibility is improved by filling the area where the own vehicle is reflected.

However, in the virtual viewpoint image, a part of the host vehicle may be reflected in a portion where the host vehicle does not originally exist (A2). In such a case, it is not preferable to paint the portion of the own vehicle in the virtual viewpoint image because a portion that should be originally visible disappears. This problem can occur when the spatial model used when converting the original camera image into the virtual viewpoint image is different from the real world shown in the camera (mostly simplified). Therefore, in order to solve this problem in principle, it is necessary to calculate the depth information of the real world in real time, and an extremely high-speed computing capability is required for the device. Therefore, implementation is difficult.

Therefore, this problem is solved by the following method.

That is, it is determined whether or not the pixels of the composite image are included in the reflection area of the vehicle with reference to the mask data paired with the original camera image. Is switched between the case in which it is included and the case in which it is not. That is, instead of converting the mask data into a virtual viewpoint image, the mask data is used in the form of the original camera image.

FIG. 40 shows an example of mask data. As shown in FIG. 40, for each camera image, mask data indicating an area not used for virtual viewpoint image synthesis such as a reflection part of the own vehicle is provided.

In this case, for example, when the pixel data of the camera image is referred to for generating the composite image, the mask data corresponding to the camera image is also referred to and the pixel is included in the reflection area of the vehicle. In some cases, processing such as painting with a color that can be identified as a blind spot area may be performed without generating a pixel.

As in the case of the above-described mapping data MP14, information as to whether or not the pixels of the composite image are included in the blind spot area may be provided in the mapping table itself. In this case, it is not necessary to refer to the max data for each frame when synthesizing the virtual viewpoint image, and the processing can be speeded up. Further, there is an advantage that it is not necessary to provide a storage unit for storing mask data.

FIG. 41 is a diagram showing an example of the result of image synthesis. In the composite image shown in FIG. 41, a vehicle area where the own vehicle exists and a blind spot area around the own vehicle are shown. By indicating the vehicle area, the driver can easily understand the positional relationship, distance, and the like between the host vehicle and the surrounding situation. In addition, areas that are hard to see directly from the driver's seat can be confirmed by the composite image. However, by indicating the blind spot area, the driver can recognize an area that is not photographed by any camera. Is improved.

Note that it is not always necessary to indicate the entire vehicle area where the host vehicle is present, and it is possible to indicate a part of the vehicle area. For example, the number of cameras installed is limited,
If only part of the vehicle and its surroundings are captured,
Only a part of the photographed vehicle area and the surrounding blind spot area are shown. Alternatively, in a case where a specific part around the vehicle is enlarged and displayed, a part of the vehicle area to be displayed and
The blind spot area around it is shown.

[0160] Instead of the blind spot area, a warning area for calling the occupant of the vehicle may be shown. The alert area may be an area including a blind spot area around the vehicle that is not photographed by any camera, or may be an area corresponding to the blind spot area itself. The alert area may indicate a part around the vehicle.

The alerting area can be set in advance only by the positional relationship with the vehicle area, regardless of the photographing range of the camera, so that the alerting area can be synthesized by simple processing. Also, when the alert area includes a blind spot area,
There is a possibility that the part photographed from the camera that is not the blind spot area may be hidden as a warning area, for example,
The alert area may be displayed translucently and displayed together with the blind spot area.

FIG. 42 is a diagram showing a procedure for generating mapping data when a blind spot area is obtained by using mask data and an illustration image is pasted. In FIG.
<PT1> refers to a pixel of a composite image generated from one camera image, and <PT2> refers to two camera images.
One of the camera images is removed due to the reflection of the vehicle,
A pixel of the composite image generated from the remaining one camera image, <PT3>, refers to one camera image, but the pixel of the composite image from which the camera image is removed due to reflection of the vehicle, <PT3>PT4> refers to one camera image, and is a pixel of the composite image to which the camera image is removed due to reflection of the vehicle and an illustration image of the vehicle is pasted.

First, in step 1, reference coordinates and necessity are calculated. For the pixels of the composite image, reference coordinates are calculated for all camera images. The calculation of the reference coordinates is performed by a geometric operation described later. At this time,
When the obtained reference coordinates are outside the photographing range of the camera image (that is, the blind spot), (-1, -1) is described as a coordinate value. Here, it is assumed that the number of cameras to be referred to is obtained, and the reciprocal thereof is given as the necessity. The method of giving the degree of necessity may be given by another method or may be given by manual input. In step 1, <PT1> refers to camera 6, <PT2> refers to cameras 4 and 5, <PT3> refers to camera 6, <PT4>.
It is obtained that the reference to> is the camera 2.

Next, in step 2, blind spot processing is performed with reference to the mask data indicating the reflection area of the vehicle. In the mask data corresponding to the camera whose coordinate value is other than (-1, -1), it is determined whether or not the point indicated by the coordinate value belongs to the reflection area of the vehicle. If it belongs to the reflection area of the vehicle, its coordinate value is converted into (-1, -1). When the number of referenced cameras is reduced by this conversion, the value of the necessity is calculated again.

As for <PT1>, as shown in FIG.
Since (0, 200) does not belong to the reflection area of the vehicle, the coordinate value is not converted. Regarding <PT2>, FIG.
As shown in (b), since the point (150, 280) in the mask data of the camera 4 belongs to the reflection area of the vehicle, the coordinate value of the camera 4 is converted into (-1, -1). On the other hand, as shown in FIG. 43 (c), since the point (490, 280) in the mask data of the camera 5 does not belong to the reflection area of the vehicle, the coordinate value of the camera 5 is not converted. Then, the necessity is converted into “1”. Regarding <PT3>, as shown in FIG. 43D, the point (110, 250) in the mask data of the camera 6 belongs to the reflection area of the vehicle, so the coordinate value of the camera 6 is set to (-1, -1). ). Regarding <PT4>, since the point (600, 290) in the mask data of the camera 2 belongs to the reflection area of the vehicle, the coordinate value of the camera 4 is converted into (-1, -1).

Next, in step 3, redundant data is arranged in order to make the data minimum necessary for image synthesis. First, all camera numbers whose coordinate values are (-1, -1) are changed to "-1". If all the camera numbers are "-1", these data are integrated into one (<PT3>, <PT4>). If there is a camera number other than "-1", the data whose camera number is "-1" is removed (<PT1>, <PT2>).

Then, in step 4, an illustration image of the own vehicle is pasted. For the pixels of the composite image to which the illustration image of the vehicle is pasted, the camera number of the mapping data is converted to “99”, and the reference coordinates of the illustration image are set. Here, since <PT4> is a pixel to which an illustration image is pasted, the camera number is set to “99”.
And the coordinate values are converted into reference coordinate values (360, 44) of the illustration image.

FIG. 44A shows an example of a composite image when the mapping data at the end of step 1 is used.
FIG. 44B shows an example of a composite image when the mapping data at the end of step 3 is used, and FIG.
It is an example of a synthetic image when the mapping data at the end is used.

FIG. 45 is a diagram showing a procedure for generating mapping data in the case where a warning area is designated without using mask data. In FIG. 45, <PT5> is a pixel of a composite image generated from one camera image, and <P5>
T6> is a pixel of the composite image designated as the alert area, <
PT7> is a pixel of the composite image to which the illustration image of the vehicle is pasted, which is designated as the alert area.

First, in step 1 ', an initial mapping table is generated. This is executed by the processing of step 1 and step 3 shown in FIG.

Next, in step 2 ', an attention area is designated. A composite image is created using the initial mapping table generated in step 1 ', and a warning area is designated in the composite image.

Next, in step 3 ', the mapping data corresponding to the alert area is rewritten. The mapping data corresponding to the alert area specified in step 2 ′ is stored in a camera number “−1” and coordinates (−1, −1).
Convert to Then, when the number of referenced cameras decreases, the value of the necessity is calculated again.

Then, in step 4 ', an illustration image of the vehicle is pasted. Here, <PT7>
Is a pixel to which the illustration image is to be pasted, so that the camera number is converted to “99” and the coordinate value is converted to the reference coordinate value (360, 44) of the illustration image.

FIG. 46A shows an example of a composite image when the mapping data at the end of step 1 'is used.
FIG. 6B shows an example of a composite image when using the mapping data at the end of Step 3 ′, and FIG. 46C shows an example of a composite image when using the mapping data at the end of Step 4 ′.

In this way, it is possible to easily perform processing such as superimposing the own vehicle image on the composite image or filling the blind spot area or the alert area with a specific color. When setting the alert area, a synthesized image can be more easily generated by combining a predetermined alert area without referring to mask data for the camera. On the other hand, when setting a blind spot area,
By referring to the mask data for the camera,
This makes it possible to reflect the situation around the vehicle reflected on the camera in the composite image without leaving it.

In the above description, the surveillance system and the image processing apparatus according to the present invention are applied to vehicles. However, the present invention is similarly applicable to moving objects other than vehicles, such as airplanes and ships. can do. In addition, a camera may be installed in a monitoring target other than a mobile object, for example, a store, a residence, a show room, and the like.

[0177] The installation positions and the number of cameras are
It is not limited to the one shown here.

Further, all or a part of the functions of the image processing apparatus according to the present invention may be realized by using dedicated hardware, or may be realized by software. It is also possible to use a recording medium or a transmission medium storing a program for causing a computer to execute all or a part of the functions of the image processing apparatus according to the present invention.

<Geometric Transformation> In order to create a mapping table for a composite image, it is necessary to determine pixel coordinates of each camera image corresponding to each pixel of the composite image viewed from a virtual viewpoint.

For this purpose, first, the world coordinate system (Xw, Yw, Zw) corresponding to each pixel of the composite image from the virtual viewpoint is obtained, and the pixels of the camera image corresponding to the three-dimensional coordinates of the world coordinate system are obtained. It is easy to understand if you consider two steps to find the coordinates.

The final necessary relationship is only the relationship between each pixel of the composite image of the virtual viewpoint and the pixel of each camera image.
The mapping table is not limited to the mapping table passing through the world coordinate system. However, since the mapping table via this world coordinate system clarifies the meaning in the world coordinate system, which is the coordinate system of the real world of the composite image, the surrounding situation corresponds to the actual distance and positional relationship. This is important in generating a composite image that can be easily attached.

As parameters representing the position and orientation of the virtual viewpoint, the coordinates of the viewpoint in the world coordinate system are represented by the position vector Tv.
= (Txv, Tyv, Tzv), if the direction of the line of sight is represented by a rotation matrix Rv of 3 rows and 3 columns representing a rotation for matching the viewing plane coordinate system to the direction of the world coordinate system, the viewpoint coordinates (Vx
e, Vye, Vze) corresponding world coordinates (Xw, Yw, Zw)
Is obtained by Expression (1).

(Equation 1)

FIG. 47 is a schematic diagram for explaining the relationship between the viewpoint coordinate system and the world coordinate system.

As shown in FIG. 48, the direction of the virtual viewpoint is
Let αv be the angle of rotation (azimuth) of the line of sight with respect to the world coordinate system YZ plane, and βv be the angle of inclination (elevation angle) with respect to the XZ plane, and γ be the rotation around the optical axis of the camera (Twist).
Given v, the rotation matrix Rv is

(Equation 2) Becomes

On the other hand, the viewpoint coordinate system (Vxe, Vye,
The relationship between Vxe and Vye of (Vze) and the two-dimensional coordinates Uv and Vv on the projection plane is expressed by the following equation (3) using the focal length fv from perspective projection transformation.

(Equation 3)

As the unit of the focal length, there are a case where the projection plane is expressed as a film or a CCD and the size is expressed in mm or inch corresponding to the size or a case where the projection surface is expressed as a pixel according to the size of the composite image. Let the projection plane be the width 2 and height 2 normalized about the projection center, and consider the focal length for that.

Therefore, the relationship between the coordinates on the projection plane and the pixels of the composite image is such that the coordinates (Uv, Vv) on the projection plane corresponding to the pixel located at (Sv, Tv) from the upper right of the image are Width
If Wv pixels and Hv pixels are Hv pixels, Uv = 2 × Sv / Wv−1 Vv = 2 × Tv / Hv−1 (4)

As described above, any pixel (Sv, T
v) 3D coordinates (Xw, Yw, Z) of the world coordinate system corresponding to
w) is the camera position Txv, Ty from equations (1) to (4).
v, Tzv, camera orientations αv, βv, γv, and focal length fv can be obtained by the following equation (5).

(Equation 4)

However, in equation (5), the depth Vze corresponding to the coordinates (Sv, Tv) of the composite image is undecided. In other words, it is necessary to determine the depth value from each pixel up to the object shown in the composite image.

If the three-dimensional shape of the object viewed from the virtual viewpoint can be known, the depth of each pixel can be obtained, but it is generally difficult. Therefore, by assuming some model for the shape of the virtual viewpoint or the visible object,
The above Vze is obtained, and the coordinates of the composite image and 3 of the world coordinate system are obtained.
The relation with the dimensional coordinates is obtained.

-Road Plane Model- As an example, a case where an object is limited to a road plane in contact with a car will be described.

Assuming that all objects exist on the plane (road surface) of the world coordinate system,
The dimensional coordinates (Xw, Yw, Zw) satisfy the following plane equation.

Ax _w + by _w + cz _w = 0 (6) Therefore, substituting equation (6) into equation (5) and obtaining Vze gives

(Equation 6) Becomes

Therefore, by substituting equation (7) into equation (5), the coordinates (Sv, Tv) of the pixel of the composite image of the virtual viewpoint can be obtained.
From the corresponding three-dimensional coordinates (X
w, Yw, Zw).

The three-dimensional coordinates (Xw,
Yw, Zw), the coordinates of each image of each camera image are expressed by the same relational expression as Expression (1), and the parameters of Tx, Ty, Tz, α, β, γ corresponding to the position and orientation of each camera. Can be calculated by substituting

For example, when the position of the camera 1 is Tx1, Ty1, Tz
1, if the directions are α1, β1, and γ1, the pixels (S
v, Tv), the camera coordinate system Xe1, Ye1, Ze of camera 1
1 can be calculated from the following equation (8).

(Equation 7) The relationship between the camera coordinate system and the coordinate system (U1, V1) of the camera image is expressed as follows: U1 = f1 / Ze1 × Xe1 V1 = f1 / Ze1 × Ye1 (Equation (3)) where f1 is the focal length of the camera 1. 9) can be calculated as The pixel of the corresponding camera image can be calculated by the following equation (10), assuming that the size of the camera image is H1 pixels vertically and W1 pixels horizontally, the aspect ratio is 1: 1, and the center of the camera is the center of the image. S1 = W1 / 2 × (Uv + 1) T1 = H1 / 2 × (Vv + 1) (10) By the above procedure, the pixels (S1, T1) of the image of the camera 1 corresponding to the pixels (Sv, Tv) of the virtual viewpoint image. ). Similarly, pixel coordinates (Sn, Tn) corresponding to (Sv, Tv) can be calculated for a general camera n other than the camera 1. Actually, the parameter table contains
Depending on various conditions such as whether (Sn, Tn) is within the range of the actual camera image, whether the pixel enlargement and reduction ratio are not large, one or more of the optimal ones are selected, and the camera number n and Write the coordinates (Sn, Tn).

-Cylindrical model-In the above-mentioned road plane model, an object reflected above the horizon in the camera image does not rest on the road plane even if the road plane is extended to infinity. Can not.

In order to reflect these objects in a composite image from a virtual viewpoint, a cylindrical model as shown in FIG. 49 is considered as a target three-dimensional shape. This model is effective, for example, when the direction of the virtual viewpoint becomes nearly parallel to the road surface.

For the sake of simplicity, consider a cylindrical model having axes on the X axis and the Z axis. If the center of the elliptical cylinder is (Xc, Zc) and the ellipse parameters (a, c) are used, the model Is expressed as the following equation (11). Note that a model having axes other than the X axis and the Z axis can be easily extended by considering rotation on the XZ plane.

(Equation 8)

By eliminating Vze from equation (5) using equation (11), the coordinates (Sv, T
v) 3D coordinates (Xw, Yw, Z) of the world coordinate system corresponding to
w) can be obtained. From the coordinates, the corresponding pixel of each camera image is calculated in the same manner as in the road surface plane model to obtain the relationship between the pixel (Sv, Tv) of the virtual viewpoint image and the pixel (Sn, Tn) of the camera image. , Create a mapping table.

A combination of a road surface plane model and a cylindrical model is also possible. First, the three-dimensional coordinates of the world coordinate system by the road surface plane model are obtained. If the three-dimensional coordinates do not produce a solution outside the cylinder model or have no intersection with the plane, then the three-dimensional coordinates by the cylinder model are used. Find dimensional coordinates. This makes it possible to combine by combining the road surface plane model and the cylindrical model.

-Pseudo-Cylinder Model- In order to make it easier to grasp the distant situation around the road surface plane model, a bowl-shaped pseudo-cylinder model is introduced around the periphery. FIG. 50 shows the shape of the model. The closer to the periphery of the virtual viewpoint image,
The distant part is compressed and synthesized, so that a wider range can be displayed. The shape of this pseudo cylinder is represented by equation (12).

(Equation 9)

The center of the bowl is (Xc, Yc, Zc), the X axis, the Y axis,
It has a length of (a, b, c) in the Z-axis direction. Similarly to the cylindrical model, three-dimensional coordinates (Xw, Yw, Zw) in the world coordinate system corresponding to the coordinates of the composite image from the virtual viewpoint are calculated from Expressions (12) and (5), and Each pixel,
It is possible to obtain the correspondence between pixels of each camera image.

It is to be noted that, similarly to the cylindrical model, it is also possible to combine a composite model by combination with a road surface plane model.

-Lens Distortion Correction Processing- Next, a method for correcting lens aberration using a mapping table will be described. If the actual camera image has distortion due to lens aberration, this distortion is calculated from the composite image by calculating the pixel coordinates corrected for the distortion when calculating the actual pixels S1 and T1 from the above Uv and Vv. It is possible to eliminate the influence of lens aberration. This distortion correction is performed by combining the composite image (Sv, Tv) of the mapping table and the camera image (Sv
n, Tn), so if you create a mapping table with distortion correction first,
Calculation for correction is not required. Even if distortion such as a function of distance from the lens center used for conventional lens distortion correction cannot be formulated, a pattern such as a grid is photographed, and even information on how each pixel moves due to lens distortion. If it is, distortion correction is possible. [Brief description of drawings]

FIG. 1 is a conceptual diagram of a monitoring system according to the present invention.

FIG. 2 is a diagram illustrating an example of a camera arrangement.

FIG. 3 is an example of a captured image of each camera in FIG. 2;

FIG. 4 is a diagram conceptually showing a relationship between a virtual viewpoint and a real camera.

FIG. 5 is an example of a generated composite image.

FIG. 6 is a diagram illustrating an image synthesizing operation using a mapping table.

FIG. 7 is a configuration example of a monitoring system according to an embodiment of the present invention.

FIG. 8 is a flowchart illustrating an operation of the image combining unit in FIG. 7;

FIG. 9 is an example of a configuration of a mapping table.

FIG. 10 is a flowchart illustrating details of a pixel synthesizing step S14 in FIG. 8;

FIG. 11 is an example of mapping data describing pixel data other than a camera image.

FIG. 12 is an example of a vertical look-down image using eight camera images.

FIG. 13 is an example of a vertical top-down image using four camera images.

FIG. 14 is an example of a vertical top-down image using two camera images.

FIG. 15 is an example of a diagonal top-down image.

FIG. 16 is an example of a panorama image in a front panorama mode.

FIG. 17 is an example of a panorama image in a rear panorama mode.

FIG. 18 is an example of an image obtained by combining a diagonal top-down image and a panoramic image.

FIG. 19 is an example of a composite image in a case where the height of the virtual viewpoint is changed according to the traveling state of the vehicle.

FIG. 20 is an example of a composite image when the direction of the line of sight of the virtual viewpoint is changed according to the running state of the vehicle.

FIG. 21 is an example of a composite image when the direction of the line of sight of the virtual viewpoint is changed according to the steering angle of the vehicle.

FIG. 22 is a diagram illustrating an example of image switching according to an output signal of an object detection sensor.

FIG. 23 is a diagram illustrating an example of image switching according to an output signal of an object detection sensor.

FIG. 24 is a diagram illustrating an example of image switching according to an output signal of an object detection sensor.

FIG. 25 is a configuration example of a monitoring system according to an embodiment of the present invention.

FIG. 26 is a diagram illustrating cutting out of a mapping table when an image is translated.

FIG. 27 is a diagram illustrating cutting out of a mapping table when an image is enlarged / reduced.

FIG. 28 is a diagram illustrating an example in which a mapping table is cut out in a quadrilateral other than a rectangle.

FIG. 29 is an example of a camera image.

FIG. 30 is an example of a composite image in which camera images of obliquely rearward views are pasted on the left and right of the top-down image.

FIG. 31 is an example in which a top-down image showing a narrow range and a top-down image showing a wide range are displayed side by side.

FIG. 32 is an example in which a diagonal top-down image is pasted around the top-down image.

FIG. 33 is an example in which an image using a pseudo cylinder model is pasted around a top-down image.

FIG. 34 is an example in which a panoramic image is pasted around a top-down image.

FIG. 35 is an example in which a diagonal top-down image is pasted around the top-down image and blank portions are provided at four corners.

FIG. 36 is an example in which an image indicating a position of a virtual viewpoint is displayed together with a camera image.

FIG. 37 is a diagram illustrating an example of a camera image and mask data indicating a reflection area of the own vehicle.

FIG. 38 is a diagram showing an example in which an image of a host vehicle is superimposed.

FIG. 39 is a diagram showing an example of a composite image when the reflection of the own vehicle is converted as it is.

FIG. 40 is a diagram showing a part of mask data.

FIG. 41 is an example of a composite image showing a vehicle area and a blind spot area.

FIG. 42 is a diagram showing a procedure for generating mapping data when a blind spot area is obtained and an image of a vehicle is pasted.

FIG. 43 is a diagram illustrating a relationship between mask data and pixels of a composite image.

FIG. 44 is an example of a composite image when using mapping data obtained as a result of the procedure in FIG. 42;

FIG. 45 is a diagram showing a procedure for generating mapping data when a warning area is specified.

FIG. 46 is an example of a composite image in a case where mapping data obtained as a result of the procedure in FIG. 45 is used.

FIG. 47 is a diagram for explaining geometric transformation.

FIG. 48 is a diagram for describing geometric transformation.

FIG. 49 is a diagram for describing geometric transformation.

FIG. 50 is a diagram for describing geometric transformation.

────────────────────────────────────────────────── ─── Continued on the front page (72) Inventor Atsushi Morimura 4-14-8 Nishitomigaoka, Nara City, Nara, Japan (56) References JP-A-8-48198 (JP, A) JP-A-11-151975 (JP) JP-A-58-110334 (JP, A) JP-A-1-123587 (JP, A) JP-A-9-180088 (JP, A) JP-A-5-238311 (JP, A) (58) Field surveyed (Int.Cl. ⁷ , DB name) H04N 7/18 B60R 1/00 B60R 21/00 G06T 1/00 G08G 1/16

Claims (27)

(57) [Claims] 1. An image processing unit that receives captured images of a plurality of cameras that capture the surroundings of a vehicle and generates a composite image viewed from a virtual viewpoint from these camera images, wherein the image processing unit includes: An image processing apparatus, wherein at least one of a position of a virtual viewpoint, a direction of a line of sight, and a focal length is changed according to a running state of the vehicle. 2. The image processing apparatus according to claim 1, wherein the image processing unit sets at least one of a position of the virtual viewpoint, a direction of a line of sight, and a focal length to a traveling speed of the vehicle. An image processing apparatus characterized in that the image processing apparatus changes according to the conditions. 3. The image processing apparatus according to claim 1, wherein the image processing unit is configured to change at least one of a position of a virtual viewpoint, a direction of a line of sight, and a focal length, and change a virtual viewpoint after the change. An image processing apparatus for controlling capture of an image outside a visual field range. 4. The image processing device according to claim 3, wherein the image processing unit controls capturing of an image outside the visual field range of the changed virtual viewpoint by changing a model for image synthesis. An image processing apparatus characterized by performing the above. 5. The image processing device according to claim 1, wherein the image processing unit sets at least one of a position of the virtual viewpoint, a direction of a line of sight, and a focal length to a steering angle of the vehicle. An image processing apparatus characterized in that the image processing apparatus changes according to the conditions. 6. The image processing apparatus according to claim 1, wherein the vehicle includes an object detection sensor that detects an obstacle, and the image processing unit includes a position of the virtual viewpoint, a direction of a line of sight, and a focus. An image processing apparatus, wherein at least one of the distances is changed according to a detection result of the object detection sensor. 7. The image processing apparatus according to claim 1, wherein the image processing unit has an original mapping table, and generates a composite image using a mapping table cut out from the original mapping table. And changing at least one of a position of a virtual viewpoint, a direction of a line of sight, and a focal length by changing a mapping table cut out from the original mapping table. 8. An image processing unit which receives captured images of a plurality of cameras that capture the surroundings of a vehicle as input and generates a composite image viewed from a virtual viewpoint from these camera images, wherein the image processing unit includes: An image processing apparatus, which performs control of capturing an image outside the visual field range of the virtual viewpoint according to a traveling state of a vehicle. 9. A plurality of cameras for photographing a periphery of a vehicle, an image processing unit which receives captured images of the plurality of cameras and generates a composite image viewed from a virtual viewpoint from these camera images, A display unit for displaying an image, wherein the image processing unit changes at least one of a position of the virtual viewpoint, a direction of a line of sight, and a focal length according to a traveling state of the vehicle. A monitoring system, characterized in that: 10. An image processing unit that receives captured images of a plurality of cameras that capture the surroundings of a vehicle and generates a composite image from these camera images, wherein the image processing unit includes One image and a virtual viewpoint and position of the first image, a direction of a line of sight, and at least one of a focal length viewed from a different viewpoint, or a different model from the first image. An image processing apparatus, comprising: generating an image including the second image and the second image as the composite image. 11. The image processing apparatus according to claim 10, wherein the second image is at least one of the camera images. 12. The image processing apparatus according to claim 10, wherein the first image is a neighboring image showing the own vehicle and its surroundings, and the second image is a surrounding image of the own vehicle shown by the nearby images. An image processing apparatus, wherein the image processing apparatus is a distant image indicating an area farther than the area. 13. The image processing apparatus according to claim 12, wherein the image processing section arranges the distant image around the vicinity image in the composite image. 14. The image processing apparatus according to claim 13, wherein the distant image is an image having continuity with the neighboring image. 15. The image processing apparatus according to claim 10, wherein the first image is an image showing at least a part of the own vehicle and at least a part around the own vehicle, and the second image Is an image obtained by enlarging at least a part of a region indicated by the first image. 16. A plurality of cameras for photographing the periphery of a vehicle, an image processing unit which receives captured images of the plurality of cameras and generates a composite image from these camera images, and a display for displaying the composite image. A first image viewed from a virtual viewpoint, and a viewpoint different from at least one of a virtual viewpoint and a position, a line-of-sight direction, and a focal length of the first image. A monitoring system that generates, as the composite image, an image that is viewed from above or includes a second image that is different in model from the first image. 17. An image processing unit which receives captured images of a plurality of cameras that capture images around the vehicle and generates a composite image from these camera images, wherein the image processing unit includes: To display at least a part of the vehicle area where the vehicle is present and at least a part of the blind spot area around the vehicle which indicates at least a part of the surrounding area of the vehicle and is not photographed by any camera. An image processing apparatus characterized by the above-mentioned. 18. The image processing apparatus according to claim 17, wherein the composite image is an image viewed from a virtual viewpoint set above the vehicle. 19. The image processing apparatus according to claim 17, wherein the image processing unit displays an illustration image or a real image of the own vehicle in the vehicle area. 20. The image processing apparatus according to claim 17, wherein the image processing unit uses an area data indicating a reflection area of the own vehicle in each camera image to generate an area obtained by combining the blind spot area and the vehicle area. An image processing device for determining a range. 21. A plurality of cameras for photographing the periphery of a vehicle, an image processing unit which receives captured images of the plurality of cameras and generates a composite image from the camera images, and a display for displaying the composite image. In the composite image, the image processing unit indicates at least a part of a vehicle area where the own vehicle is present, and at least a part of the surroundings of the own vehicle, around the vehicle that is not photographed by any camera. A monitoring system characterized by displaying a warning area which is at least a part of a blind spot area. 22. An image processing unit that receives captured images of a plurality of cameras that capture images around the vehicle and generates a composite image from these camera images, wherein the image processing unit includes: a pixel of the composite image; First mapping data describing the correspondence relationship with the pixels of the camera image, pixels of the composite image,
An image processing apparatus comprising: generating a composite image by using a mapping table having second mapping data that describes an identifier indicating correspondence with pixel data other than a camera image. 23. The image processing device according to claim 22, wherein the pixel data other than the camera image is the vehicle or
An image processing apparatus for indicating a blind spot area in at least a part of the periphery of the vehicle. 24. The image processing device according to claim 22, wherein the image processing unit stores a predetermined image other than a camera image, and wherein the second mapping data An image processing apparatus for describing a stored coordinate value in a predetermined image corresponding to a pixel. 25. The image processing apparatus according to claim 22, wherein the second mapping data describes pixel data corresponding to pixels of a composite image. 26. An image processing unit that receives captured images of a plurality of cameras that capture the surroundings of a vehicle and generates a composite image from these camera images, wherein the image processing unit includes: a pixel of the composite image; Mapping data that describes a correspondence relationship with a plurality of pixel data, which includes one or both of pixel data of a camera image and pixel data other than a camera image, and describes the necessity of each pixel data. , Weighting each pixel data according to the degree of necessity,
An image processing apparatus, wherein pixel data of a pixel of the composite image is generated. 27. An image processing unit that receives captured images of a plurality of cameras that capture the surroundings of a vehicle and generates a composite image from these camera images, wherein the image processing unit has an original mapping table. An image processing apparatus characterized in that a mapping table describing the correspondence between pixels of a composite image and pixels of a camera image is cut out from the original mapping table, and a synthesized image is generated using the cut-out mapping table.