JP2002369187A - Panoramic image display method and equipment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a panoramic image display method wherein the contents or directional sense in a panoramic image displayed on a monitor are easy to grasp sensually by a viewer without depending on character information or the like, and instantaneous visibility is superior. SOLUTION: In this panoramic image display method, images Ra, Rb picked up by a plurality of cameras 15a, 15b are virtually projected from the position of an axial line L on a cylindrical screen Sc. A composited image obtained by picking up an image Q projected on the cylindrical screen Sc with a virtual camera 14 arranged on a position distant d as prescribed from the position of the axial line L is displayed as a panorama image on a monitor.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for displaying a panoramic image on a monitor by synthesizing images taken by a plurality of cameras.

[0002]

2. Description of the Related Art For example, a video photographed by an on-vehicle camera is a panoramic image (in the present specification, a "panoramic image" mainly refers to a semi-peripheral image having an ultra-wide angle of view of about 180 °. As a method of displaying on a monitor as ()), there is a method already proposed by the applicant in Japanese Patent Application No. 2001-21612. In this panoramic image display method,
As shown in FIG. 17, a panoramic image is displayed on the entire surface of the monitor 1, and the direction (direction) of each part of the panoramic image is represented by superimposing the character direction information 2, the graphic direction information 3, and the like.

[0003]

However, when a large number of cameras are mounted on vehicles and the like in the future,
Whether the image currently displayed on the monitor is a video shot by a normal camera with a not so large angle of view or a panorama video by ultra-wide-angle shooting, and in which direction the video can be shot when multiple directions can be shot Although it is necessary for the viewer of the monitor to instantly recognize whether or not there is, there is a possibility that the above method of displaying a panoramic image on the entire surface of the monitor cannot always sufficiently satisfy such a demand.

When it is difficult to recognize the direction of the object projected on the monitor only by the video, the viewer must read the superimposed character information or the like. The fact that the character information must be read seems to have room for improvement from the viewpoint of instantaneous visibility. On the other hand, even if the display of character information and the like is enlarged in order to improve the visibility, the display screen tends to be troublesome as a whole because the in-vehicle monitor or the like is relatively small, and it is not an appropriate solution. There is a risk.

The present invention has been made in view of the above circumstances, and makes it easy for a viewer to intuitively grasp the content or sense of direction of a panoramic image displayed on a monitor without relying on character information or the like. An object of the present invention is to provide a panoramic image display method and a panoramic image display device that are excellent in instantaneous visibility.

[0006]

In order to solve the above-mentioned problems, the invention according to claim 1 is directed to a method in which an image taken by a plurality of cameras is placed on a concave virtual screen formed by cutting out a part of a cylinder. Virtually projected from the body axis position, a composite image obtained when the image projected on the virtual screen is captured by a virtual camera installed at a position separated from the axis position by a predetermined distance is displayed on a monitor as a panoramic image. A method for displaying a panoramic image to be displayed is characterized.

According to a second aspect of the present invention, in the panoramic image display method according to the first aspect, the sectional shape of the cylindrical body is circular, elliptical, or polygonal.

According to a third aspect of the present invention, in the panoramic image display method according to the first or second aspect, an auxiliary line extending radially from the axis position toward the virtual screen is displayed on the monitor together with the panoramic image. It is characterized by displaying.

According to a fourth aspect of the present invention, in the panoramic image display method according to the third aspect, the plurality of cameras and the monitor are mounted on a vehicle.

According to a fifth aspect of the present invention, in the method for displaying a panoramic image according to any one of the first to third aspects, the plurality of cameras and the monitor are mounted on a vehicle, and an image image of the vehicle is provided. Is displayed on the monitor together with the panoramic image.

According to a sixth aspect of the present invention, in the panoramic image display method according to the fourth or fifth aspect, the display method of the auxiliary line or the image image is changed by operating a switch in the vehicle or a running state of the vehicle. The switching is performed according to

According to a seventh aspect of the present invention, an image photographed by a plurality of cameras is virtually projected from a position of an axis of the cylinder on a concave virtual screen formed by cutting out a part of the cylinder. A panorama image display device that displays a composite image obtained when a video projected on a screen is captured by a virtual camera installed at a position separated by a predetermined distance from the axis position on a monitor as a panorama image,
A / D conversion means connected to the plurality of cameras and converting analog image data of video shot by the plurality of cameras into digital image data; and A / D conversion means connected to the A / D conversion means and An image synthesizing unit that converts digital image data of an image captured by the plurality of cameras into digital image data of the composite image based on a correspondence between the captured image and the composite image; and And D / A conversion means connected to the monitor for converting digital image data of the composite image into analog image data and outputting the analog image data to the monitor.

According to an eighth aspect of the present invention, in the panoramic image display device according to the seventh aspect, the sectional shape of the cylindrical body is circular, elliptical, or polygonal.

According to a ninth aspect of the present invention, in the panoramic image display device according to the seventh or eighth aspect, between the image synthesizing means and the D / A converting means or between the image synthesizing means and the D / A converting means. An auxiliary line superimposing unit between the monitor and the monitor, wherein the auxiliary line superimposing unit displays an auxiliary line radially extending from the axis position toward the virtual screen together with the panoramic image on the monitor, The image data of the auxiliary line is superimposed on the image data of the panoramic image.

According to a tenth aspect of the present invention, in the panoramic image display device according to the ninth aspect, the apparatus is mounted on a vehicle together with the plurality of cameras and the monitor.

According to an eleventh aspect of the present invention, in the panoramic image display device according to any one of the seventh to ninth aspects, the panoramic image display device is mounted on a vehicle together with the plurality of cameras and the monitor, and the image synthesis is performed. Means and the D / A
An image image superimposing means between the D / A converting means and the monitor, between the D / A converting means and the monitor, and the image image superimposing means causes the monitor to display the image image of the vehicle together with the panoramic image; For this purpose, the image data of the image image is superimposed on the image data of the panoramic image.

According to a twelfth aspect of the present invention, in the panoramic image display device according to the tenth aspect, the auxiliary line superimposing means displays the auxiliary line according to a switch operation in the vehicle or a running state of the vehicle. Is switched.

According to a thirteenth aspect of the present invention, in the panoramic image display device according to the eleventh aspect, the image image superimposing means displays the image image according to a switch operation in the vehicle or a traveling state of the vehicle. Is switched.

According to the first or seventh aspect of the present invention, images taken by a plurality of cameras are virtually projected on a concave virtual screen obtained by cutting out a part of the cylinder from the axial position of the cylinder. Then, an image obtained when the image projected on the virtual screen is photographed by a virtual camera installed at a predetermined distance from the axis position is displayed on a monitor as a panoramic image. (Naturally) deforms naturally into a video with a sense of depth, so that the viewer can intuitively understand that it is a super wide-angle panoramic video, and naturally adjust the enlargement / reduction of the panoramic image. Is also possible, and the panoramic image becomes excellent in instantaneous visibility.

According to the second or eighth aspect of the invention, since the cross-sectional shape of the cylindrical body is circular, elliptical or polygonal, the panorama image is displayed in a very natural and very easy-to-view panorama image. You.

According to the third, fourth, ninth, or tenth aspect of the present invention, an auxiliary line extending radially from the axial position toward the virtual screen is displayed on the monitor together with the panoramic image. The sense of direction of the image is emphasized, and the viewer can more easily grasp the sense of direction.

According to the fifth or eleventh aspect of the present invention, a plurality of cameras and monitors are mounted on a vehicle, and an image of the vehicle is displayed on the monitor together with a panoramic image. Is installed, the viewer can easily grasp the contents of the panoramic image, that is, which camera is the captured image.

According to the sixth, twelfth or thirteenth aspect, the display method of the auxiliary line or the image image is switched according to the switch operation in the vehicle or the running state of the vehicle. A panoramic image suitable for the user's sense is displayed, and the visibility of the panoramic image is further improved.

[0024]

Embodiments of the present invention will be described with reference to the drawings.

[Overall Configuration of Panoramic Image Display Apparatus] FIG.
1 shows a schematic configuration of a panoramic image display device according to the present invention. The panoramic image display device 4 is mounted on a vehicle to display an image behind the vehicle 5 on a monitor 6 in the vehicle compartment (FIG. 2).
), An A / D conversion section 7, an image synthesis section 8, a synthesis pattern calculation / memory section 9, a D / A conversion section 10, an image superposition device 11, and a superposition pattern calculation / memory section 12. .

The A / D converter 7 includes two cameras 13a, 13a installed at the rear of the vehicle 5 as schematically shown in FIG.
3b (in the figure, cameras 13a, 13b
These are clearly shown in order to clarify the existence of, but these are actually invisible except from the outside because the body is covered except for the lens portion. In the following, the camera is also referred to as a “real camera” to distinguish the camera from a virtual camera described later. ), Converting the analog image data of the video captured by each camera into digital image data.

The image synthesizing unit 8 is connected to the A / D conversion unit 7 and the synthesizing pattern calculation / memory unit 9, and the cameras 13a, 1
The panorama image is generated by synthesizing the video shot by 3b as described later. The image combining in the image combining unit 8 is performed in accordance with the processing of the combining pattern calculation / memory unit 9. Here, the image combining unit 8 and the combining pattern calculation / memory unit 9 function as an image combining unit.

The D / A converter 10 is connected to the image synthesizing unit 8 and to the monitor 6 via the image superimposing device 11. The D / A converter 10 converts digital image data of a panoramic image generated by the image synthesizer 8 into analog image data.

The image superimposing device 11 superimposes analog image data of a pattern designed to improve the visibility of a screen on analog image data of a panoramic image. The superimposition of images in the image superimposing device 11 is performed in accordance with the processing of the superimposition pattern calculation / memory unit 12. Here, the image superimposition device 11 and the superimposition pattern calculation / memory unit 12 function as auxiliary line superimposing means or image image superimposing means. . In this embodiment, the image superimposing device 11 is provided between the D / A converter 10 and the monitor 6, but is provided between the image synthesizer 8 and the D / A converter 10. Alternatively, the image superimposing device 11 may be provided integrally with the image synthesizing unit 8, and at this time, the image superimposing device 11 superimposes the digital image data of the pattern on the digital image data of the panoramic image.

[Image Synthesizing Process in Image Synthesizing Unit] FIG.
Shows a model for explaining a method of generating a panoramic image in the image synthesizing unit 8 and the synthesizing pattern calculation / memory unit 9. In the figure, reference numeral Sc indicates a concave virtual three-dimensional screen (cylindrical screen) obtained by cutting a part of a cylindrical body (cylinder) having a circular cross section, reference numeral L indicates an axis of the cylindrical body, and reference numeral Ra. , Rb are each a live-action camera 13
3A and 3B show captured images (real camera images and real camera screens).

As a concept of image synthesis in the image synthesizing unit 8 and the synthesizing pattern calculating / memory unit 9, first, the images Ra and Rb photographed by the real cameras 13a and 13b are firstly moved to the position of the axis L (including the vicinity of the axis L). .) Onto the cylindrical screen Sc to obtain projected images Qa and Qb corresponding to the captured images Ra and Rb. Conventionally, a projection image Q composed of the projection images Qa and Qb is displayed on a cylindrical screen Sc.
Was opened to form a panoramic image as shown in FIG. 4, but in the present embodiment, a virtual camera is assumed at a position separated by a predetermined distance from the position of the axis L, and projected by this virtual camera. An image obtained when the video Q is captured is a panoramic image. In FIG.
Reference numeral 4 denotes a lens center of the virtual camera, and reference numeral I denotes a captured image (virtual camera image, virtual camera screen) of the virtual camera.

For example, when a simple pinhole camera model is used for the virtual camera, the lens center 1 of the virtual camera is used.
It is sufficient to look at the cylindrical screen Sc from 4 and imagine projecting the projection image Q on the cylindrical screen Sc onto the virtual camera screen I. According to this method, the distance to the corresponding point on the cylindrical screen Sc differs between a point near the center and a point near the left and right ends on the virtual camera screen I, and as a result, is projected onto the virtual camera screen I. The enlargement / reduction ratio of the image differs depending on the position in the left-right direction on the virtual camera screen I. As can be seen from FIG. 3, since the distance from the point near the center on the virtual camera screen I to the cylindrical screen Sc is the longest, the image is relatively reduced near the center on the virtual camera screen I. As a result, a natural deformation in which the vicinity of the center shrinks in the vertical direction as shown in FIG. 5 is realized on the virtual camera screen I, and the image has a visual effect as if the viewer were looking at a three-dimensional object. A panoramic image for intuitive understanding of the content and sense of direction is generated.

The virtual camera is connected to a cylindrical screen Sc.
, The enlargement / reduction ratio in the vicinity of the center on the virtual camera screen I and in the vicinity of the left and right ends can be naturally adjusted. This is the distance from the point on the cylindrical screen Sc corresponding to the point near the center on the virtual camera screen I to the virtual camera,
This is because the ratio of the distance from the point on the cylindrical screen Sc corresponding to the point near the left or right end on the virtual camera screen I to the virtual camera changes continuously as the position of the virtual camera changes. At this time, the overall size of the panorama image formed on the virtual camera screen I is also enlarged or reduced, but the focal length of the virtual camera (the distance between the lens center 14 of the virtual camera and the virtual camera screen I) f By adjusting the 'together, the size can be kept unchanged.

Here, in order to formulate the above idea, an XYZ orthogonal coordinate system is set so that the axis L coincides with the Y axis, and as shown in FIG.
Are located at a distance r from the origin (observation point) O in the XZ plane. If the radius of the cylindrical screen Sc is P, r usually takes a considerably smaller value than P, but here, the ratio of r to P is shown larger than the actual value. Also, the real camera 13a, 13b
They are arranged symmetrically with respect to the axis and their line of sight (LOS: Line
of Site) and the Z axis are θ and −θ, respectively, and the focal length is f. Further, an X'Y 'coordinate system is set on the cylindrical screen Sc with the intersection point with the Z axis as the origin, and the LOS is set on the real camera screens Ra and Rb.
The xy coordinate system is set with the intersection with the origin as the origin o, and the intersection with the Z axis is set as the origin o 'on the virtual camera screen I and x'
A y 'coordinate system is set (each axis of Y', y, y 'is assumed to be parallel to the Y axis).

Now, paying attention to the camera 13a, the point (x, y) on the real camera screen Ra is changed to the cylindrical screen S
Consider a case where the projection is made to a point K (X ′, Y ′) on c.
Assuming that the angle formed by the line segment OK and the Z axis in the XZ plane is α and the angle formed by the line segment oK and the LOS of the camera 13a is θ ′,

[0036]

(Equation 1)

[0037]

(Equation 2)

Alternatively,

[0039]

(Equation 3)

[0040]

(Equation 4)

At this time,

[0042]

(Equation 5)

Or

[0044]

(Equation 6)

The following relationship holds. By using these relationships, the point (x, y) on the real camera screen Ra and the point K (X ′, Y ′) on the cylindrical screen Sc can be converted into each other, and θ in each equation is − If θ is used, the above relationship can be used for the camera 13b.

Further, a point K on the cylindrical screen Sc
When the relationship between (X ′, Y ′) and the point (x ′, y ′) on the virtual camera screen I is obtained, it is assumed that the lens center 14 of the virtual camera is located at a distance d from the origin O in the XZ plane.

[0047]

(Equation 7)

[0048]

(Equation 8)

The following relationship holds.

By using the above equations (1) to (8), the coordinates (x,
y) and coordinates (x ', y') on the virtual camera screen I
Can be specifically associated with each other, and an example of image synthesis processing actually performed by the image synthesis unit 8 and the synthesis pattern calculation / memory unit 9 using these equations will be described below.

The image synthesizing unit 8 includes an input image memory 8a for storing digital image data for each pixel of the video captured by the cameras 13a and 13b, and stores digital image data for each pixel of the synthetic image (panoramic image). An output image memory 8b is provided (see FIG. 1), and as shown in FIG. 7, an address is generated for an area corresponding to a predetermined pixel of the output image memory 8b (step 1 (referred to as S.1 in FIG. Similar)).

The address is received by the composite pattern calculation / memory section 9 as shown in FIG. 8 (step 2), and is converted into coordinates (x ', y') on the virtual camera screen I (step 3). Corresponding coordinates (X ', Y') on the cylindrical screen Sc are calculated by the equations (7) and (8) (step 4).

Subsequently, a corresponding camera is selected according to the calculated X 'coordinate value and the like (step 5), and the camera 13 is selected.
Expressions (1) and (2) are used when a is selected, and Expressions (1) and (2) where θ is replaced by -θ are used when the camera 13b is selected. The corresponding coordinates (x, y) on the screen are determined (step 6).

Then, the address of the area of the input image memory 8a corresponding to the coordinates (x, y) is obtained by calculation (step 7), and the combined pattern calculation / memory unit 9 outputs the address to the image combining unit 8. (Step 8).

When receiving the address (step 9), the image synthesizing unit 8 transmits the image data stored in the area of the address in the input image memory 8a to the destination of the output image memory 8b (the address transmitted in step 1). ) (Step 10).

By performing the processing of steps 1 to 10 for all the pixels of the composite image, the digital image data of all the pixels are determined and a panoramic image is generated. Instead of moving the image data to the output image memory 8b to generate a panoramic image, the image data is generated by appropriately performing various calculations such as processing of a portion where the images captured by the cameras 13a and 13b overlap and brightness adjustment and the like. You may.

Also, as shown in FIG. 8, the processing of steps 4 to 6 is based on the parameters d,
f, the parameter P for the virtual screen, and the parameters r, f, θ for the real camera, and these parameters change in real time according to the running state of the vehicle 5 (switch information, sensor information). By doing so, a panoramic image matching the driver's feeling at that time is displayed. This will be described later. However, in step 5, as described above, if the virtual screen is cylindrical and the number of real cameras is two, the camera 13a is determined if X ′ obtained in step 4 satisfies X ′ ≧ 0.
Is selected, and if X '<0 is satisfied, the processing is simple if the camera 13b is selected.

Next, a case where the virtual screen is not cylindrical but has another shape will be described.

FIGS. 9 to 11 show models for explaining a method of generating a panoramic image when the shape of the virtual screen is changed. In FIG. 9, reference symbol So denotes a part of a cylindrical body having an elliptical cross section. And a and b indicate the lengths of the major axis and the minor axis of the ellipse, respectively. Other reference numerals are the same as those in FIG. In addition,
Since the relational expression that holds for the elliptic cylindrical screen So is more complicated than that obtained for the cylindrical screen Sc, the coordinates (x ′, y) on the virtual camera screen I that are important in the flow of FIG. ') And the coordinates (X, Y, Z) on the elliptical cylindrical screen So, and
Only the operation procedure for deriving the relationship between the coordinates (X, Y, Z) on the elliptic cylindrical screen So and the coordinates (x, y) on the real camera screen Ra will be described.

The coordinates (x ′,
y ′) is a point K (X, Y,
Z) = (XK, YK, ZK), X
K and ZK are obtained by solving the following simultaneous equations for X and Z.

[0061]

(Equation 9)

[0062]

(Equation 10)

YK is obtained by substituting XK into the following equation.

[0064]

[Equation 11]

Subsequently, when the coordinates on the real camera screen Ra corresponding to the point K are obtained in the XYZ space, the coordinates (XC, YC, ZC) are obtained at the position (XK, YK, Z) of the point K.
K) and the position of the camera 13a (r sin θ, 0, r cos θ) correspond to the coordinates of the intersection of a line segment and the real camera screen Ra. XC and ZC express the following simultaneous equations as X,
It is determined by solving for Z.

[0066]

(Equation 12)

[0067]

(Equation 13)

Further, YC is obtained by substituting XC into the following equation.

[0069]

[Equation 14]

The coordinates (x, y) on the real camera screen Ra corresponding to K are given by the following equations using the XC, YC, and ZC.

[0071]

(Equation 15)

[0072]

(Equation 16)

By using the relation obtained above in Steps 4 and 6 in FIG. 8, image synthesis by the elliptical cylindrical screen So is realized. The enlargement / reduction rate can also be adjusted. At this time, the parameters of the virtual screen in FIG. 8 are a and b.

On the other hand, in FIG. 10 and FIG.
r denotes a virtual screen (rectangular screen) obtained by cutting a part of a cylindrical body having a rectangular cross section. The rectangular screen Sr has an X axis as a normal and an X axis and a coordinate value −XD.
A screen S ₁ intersects with a screen S ₂ intersecting the Z-axis coordinate value ZD the Z-axis as a normal line, which is the screen S ₃ Metropolitan intersecting in X-axis and a coordinate value XD the X-axis as a normal line. The screens S ₁ , S ₂ , and S ₃ have an X ₁ Y ₁ coordinate system having an origin O ₁ at an intersection with the X axis, an X ₂ Y ₂ coordinate system having an origin O _{2 at} an intersection with the Z axis, and an X axis. X ₃ Y with the intersection point with the origin O _3
3 coordinate systems are set (each axis of Y ₁ , Y ₂ , Y ₃ is parallel to the Y axis). It is difficult to handle these screens at the same time. Y
n) (n = 1, 2, 3).

The coordinates on the rectangular parallelepiped screen Sr (X
n, Yn) and coordinates (x,
y)

[0076]

[Equation 17]

[0077]

(Equation 18)

[0078]

[Equation 19]

[0079]

(Equation 20)

[0080]

(Equation 21)

[0081]

(Equation 22)

Holds. Therefore, by using these relational expressions and their inverse functions, the coordinates (x, y) on the real camera screen Ra and the rectangular screen S
The coordinates (Xn, Yn) on r can be mutually converted. Also, the coordinates (Xn,
Yn) and coordinates (x ′,
y ')

[0083]

(Equation 23)

[0084]

(Equation 24)

[0085]

(Equation 25)

[0086]

(Equation 26)

[0087]

[Equation 27]

[0088]

[Equation 28]

Since the above equations are satisfied, the coordinates (Xn, Xn,
Yn), the coordinates (x, y) on the real camera screen Ra and the coordinates (x ′,
y '). Then, a panoramic image of an atmosphere surrounding three sides as shown in FIG. 12 can be generated by image synthesis using the rectangular parallelepiped screen Sr based on the correspondence.

[Image Superposition Processing in Image Superposition Apparatus] The image data of the panoramic image generated by the image synthesizing unit 8 and the synthesizing pattern calculation / memory unit 9 is converted by the D / A conversion unit 1
After passing through 0, the image is superimposed on the image data of the pattern in the image superimposing device 11. As an example, FIG.
An image in the case where an auxiliary line is superimposed as a pattern on a panoramic image generated using the cylindrical screen Sc is shown. That is, an arc 16 conscious of the center of the cylinder and an auxiliary line 17 extending radially from the center to the cylindrical screen Sc are drawn at the lower part of the screen of the monitor 6.
It is designed to make the viewer strongly aware of the stereoscopic panoramic image. The auxiliary line 17 is drawn at regular angular intervals from the center, so that the viewer can more easily grasp the sense of direction or azimuth angle of the panoramic image.

FIG. 14 shows an image in the case where an image of the own vehicle is superimposed as a pattern on a panoramic image generated using the cylindrical screen Sc. That is,
An image 18 of the vehicle is drawn in the center of the lower part of the screen of the monitor 6 so as to make the viewer strongly aware of the position or the direction of the photographing camera. Image 18
In the case where a plurality of live-action cameras are attached to the rear, side, front, etc. of the vehicle 5, and the images captured by those cameras are switched and displayed on the monitor 6, the design is changed in accordance with the switching. You.

The design of these patterns is determined in the superimposition pattern calculation / memory section 12, and the image data of the determined design is transmitted to the image superimposition device 11 to perform the above-described superimposition.

[Change of Image Display Method by Switch Operation, etc.] FIG. 15 shows the switch device 1 at hand of the driver of the vehicle 5.
9 shows an example in which the image display method is changed by operating the switch device 19. For example, when the first switch from the left of the switch device 19 is selected, the composite pattern calculation / memory unit 9 selects the first one of the display patterns stored in the memory from the top in the figure.
The superimposition pattern calculation / memory unit 12 outputs the first one of the display patterns stored in the memory from the top in the figure to the image synthesis unit 8 and the image superimposition device 11, respectively. That is, the image synthesizing unit 8 and the synthesizing pattern calculation / memory unit 9 generate a panoramic image by the cylindrical screen Sc, and the image superimposing device 11 and the superimposing pattern calculation / memory unit 12 superimpose any image on the panorama image. Instead, an image as shown in FIG. 5 is displayed on the monitor 6 after all.

When the second switch from the left of the switch device 19 is selected, the composite pattern calculation / memory section 9 selects the second one from the top among the display patterns stored in the memory. The superimposition pattern calculation / memory unit 12 outputs the second one from the top among the display patterns stored in the memory to the image synthesizing unit 8 and the image superimposition device 11, respectively. That is, the image synthesizing unit 8 and the synthesizing pattern calculation / memory unit 9 generate a panoramic image using the rectangular screen Sr,
Image superposition device 11 and superposition pattern calculation / memory unit 12
Depending on the screen S ₁ , S ₂ and the screen S ₂ , S ₃
Are superimposed, and an image as shown in FIG.

When the third switch from the left of the switch device 19 is selected, the composite pattern calculation / memory section 9 superimposes the third display pattern from the top of the display patterns stored in the memory. The pattern calculation / memory unit 12 outputs the third one from the top of the display patterns stored in the memory to the image synthesizing unit 8 and the image superimposing device 11, respectively. That is, the image synthesizing unit 8 and the synthesizing pattern calculation / memory unit 9 generate a panoramic image by the cylindrical screen Sc, and the image superimposing device 11 and the superimposing pattern calculation / memory unit 12 superimpose the arc 16 and the auxiliary line 17. The monitor 6 displays an image as shown in FIG.

When the fourth switch from the left of the switch device 19 is selected, the composite pattern calculation / memory section 9 superimposes the fourth display pattern from the top among the display patterns stored in the memory. The pattern calculation / memory unit 12 outputs the fourth display pattern from the top among the display patterns stored in the memory to the image synthesizing unit 8 and the image superimposing device 11, respectively. That is, a panoramic image is generated by the cylindrical screen Sc by the image synthesizing unit 8 and the synthesizing pattern operation / memory unit 9, and the arc 16, the auxiliary line 17, and the image image are generated by the image superimposing device 11 and the superimposing pattern operation / memory unit 12. 18 is superimposed, and an image as shown in FIG. 14 is displayed on the monitor 6.

In the case where the display method is switched in a pattern as described above, if the automatic adjustment of the image in real time is unnecessary, the composite pattern calculation / memory section 9 may be used.
And the superimposition pattern calculation / memory unit 12 stores some display methods in advance and mechanically outputs them to the image synthesis unit 8 and the image superimposition device 11, whereby the synthesis pattern calculation / memory unit 9 and the superposition pattern calculation / memory unit 12 do not need to perform calculation processing, and the configuration is simplified.

FIG. 16 shows an example in which the image display method is changed according to the vehicle speed which is one of the parameters indicating the running state of the vehicle 5. When it is desired to enlarge or reduce the image near the center of the monitor screen relative to the image at the peripheral portion in accordance with the vehicle speed, increase or decrease the distance d defining the position of the virtual camera, and make the distance d substantially proportional thereto. In addition, the focal length f 'may be increased or decreased.

More specifically, when the vehicle speed is detected by the speed sensor 20, an appropriate distance d and a focal length f 'are determined in the composite pattern calculation / memory unit 9 according to the vehicle speed, and based on the determined values, image synthesis is performed. Coordinate conversion values (x,
y, etc.) are calculated in real time, and the image combining unit 8 performs image combining. For example, when the vehicle speed is low, such as when parking, the driver often wants to see the surroundings widely, so the distance d and the focal length f 'are increased to relatively increase the vicinity of the center of the panoramic image. On the other hand, when the vehicle speed is higher than a certain value, such as when driving at high speed, the driver's view through the window becomes narrower. f 'is increased (the display magnification at this time may be different from the display magnification when the vehicle speed is low, of course). Further, in the superposition pattern calculation / memory unit 12, the auxiliary line 17 is also determined based on the distance d and the focal length f '.
Are calculated, and the image superimposing device 11 performs image superimposition.

As described above, the display method of the auxiliary line 17 and the image image 18 is switched according to the switch operation on the vehicle 5 or the traveling state of the vehicle 5, so that the panoramic image suitable for the driver's feeling at that time is obtained. Is displayed, and the visibility of the panoramic image is further improved.

The embodiments of the present invention are not limited to those described above. For example, a virtual screen obtained by cutting off a part of a cylindrical body having another cross-sectional shape may be used. The image display method may be changed by judging the state by gear operation, steering angle, or the like.

[0102]

As described above, according to the first or seventh aspect of the present invention, an image captured by a plurality of cameras is formed on a concave virtual screen formed by cutting out a part of a cylindrical body. The image obtained when the image projected on the virtual screen is virtually projected from the axis position of the virtual camera and the virtual camera installed at a position separated from the axis position by a predetermined distance is displayed as a panoramic image on the monitor. ,
The panoramic image is naturally transformed (in a human sense) into an image with a sense of depth, which allows the viewer to intuitively recognize that the image is a super wide-angle panoramic image,
The enlargement / reduction of the panoramic image can be adjusted naturally, and the panoramic image becomes excellent in instantaneous visibility.

According to the second or eighth aspect of the present invention, since the cross-sectional shape of the cylindrical body is circular, elliptical or polygonal, the panorama image is very natural in deformation and a very easy-to-view panorama image is displayed. You.

According to the third, fourth, ninth, or tenth aspect of the present invention, the auxiliary line extending radially from the axial position toward the virtual screen is displayed on the monitor together with the panoramic image. The sense of direction of the image is emphasized, and the viewer can more easily grasp the sense of direction.

According to the fifth or eleventh aspect of the present invention, a plurality of cameras and monitors are mounted on a vehicle, and an image of the vehicle is displayed on the monitor together with a panoramic image. Even if is installed, the viewer can easily grasp the contents of the panoramic image, that is, which camera is the captured video.

According to the sixth, twelfth, or thirteenth aspect, the display method of the auxiliary line or the image image is switched according to the switch operation in the vehicle or the running state of the vehicle. A panoramic image suitable for the user's sense is displayed, and the visibility of the panoramic image is further improved.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram illustrating a schematic configuration of a panoramic image display device according to the present invention.

FIG. 2 is an explanatory diagram schematically showing an arrangement of a live-action camera in a vehicle.

FIG. 3 is a perspective view showing a model of image synthesis using a cylindrical screen.

FIG. 4 is an explanatory diagram showing a panoramic image when a virtual camera is not used in the model of FIG. 3;

FIG. 5 is an explanatory diagram illustrating a panoramic image when a virtual camera is used in the model of FIG. 3;

6 is a plan view showing a model of image synthesis using the cylindrical screen of FIG. 3;

FIG. 7 is an explanatory diagram showing a flow of processing in an image combining unit.

FIG. 8 is an explanatory diagram showing a flow of processing in a combined pattern calculation / memory unit.

FIG. 9 is a plan view showing a model of image synthesis using an elliptical cylindrical screen.

FIG. 10 is a perspective view showing a model of image synthesis using a rectangular screen.

11 is a plan view showing a model of image synthesis using the rectangular parallelepiped screen of FIG.

FIG. 12 is an explanatory diagram showing an example of a panoramic image obtained by the model of FIG. 11;

FIG. 13 is an explanatory diagram showing an example in which auxiliary lines are superimposed on a panoramic image obtained by the model of FIG. 3;

14 is an explanatory diagram showing an example in which an image image of a vehicle is superimposed on a panoramic image obtained by the model of FIG. 3;

FIG. 15 is an explanatory diagram showing a link between the switch device and a combined pattern calculation / memory unit and a superimposed pattern calculation / memory unit;

FIG. 16 is an explanatory diagram showing the linkage between a speed sensor and a combined pattern calculation / memory unit and a superimposed pattern calculation / memory unit;

FIG. 17 is an explanatory diagram showing a panoramic image displayed by a conventional method.

[Explanation of symbols]

 Reference Signs List 4 Panorama image display device 5 Vehicle 6 Monitor 7 A / D conversion unit (A / D conversion unit) 8 Image synthesis unit 9 Synthesis pattern calculation / memory unit 10 D / A conversion unit (D / A conversion unit) 11 Image superposition device 12 Superposition pattern calculation / memory unit 13a, 13b Camera 17 Auxiliary line 18 Image image 19 Switch device 20 Speed sensor d Predetermined distance L Axis Sc Cylindrical screen (virtual screen) So Elliptic cylindrical screen (virtual screen) Sr rectangular parallelepiped screen ( Virtual screen)

────────────────────────────────────────────────── ─── of the front page continued (51) Int.Cl. ⁷ identification mark FI theme Court Bu (reference) G06T 3/00 100 G06T 3/00 100 300 300 G09G 5/377 G09G 5/36 520L (72) inventor Ohki Tohru 5-35-2, Hakusan, Bunkyo-ku, Tokyo Clarion Co., Ltd. (72) Inventor Tokuyuki Sato 5-35-2, Hakusan, Bunkyo-ku, Tokyo Clarion Co., Ltd. F-term (reference) 5B057 BA02 BA13 BA26 CA08 CA13 CA16 CB08 CB12 CB16 CD05 CD11 CD16 CE08 CH18 5C054 AA01 CA04 CC02 CE15 CH01 EA01 EA05 EB05 EB07 EH01 FD02 FE11 FE12 FF05 GA04 GB15 GD01 HA30 5C082 AA00 BA12 BB25 CA32 CA55 CA76 CB03 MM03 CB03

Claims (13)

[Claims] An image photographed by a plurality of cameras is virtually projected on a concave virtual screen obtained by cutting a part of a cylinder from an axial position of the cylinder, and the image projected on the virtual screen is projected. A panoramic image display method, wherein a composite image obtained when photographed by a virtual camera installed at a position separated by a predetermined distance from the axis position is displayed on a monitor as a panoramic image. 2. The panoramic image display method according to claim 1, wherein the cross-sectional shape of the cylinder is a circle, an ellipse, or a polygon. 3. The panoramic image display method according to claim 1, wherein an auxiliary line extending radially from the axis position toward the virtual screen is displayed on the monitor together with the panoramic image. Panorama image display method. 4. The panoramic image display method according to claim 3, wherein the plurality of cameras and the monitor are mounted on a vehicle. 5. The panoramic image display method according to claim 1, wherein the plurality of cameras and the monitor are mounted on a vehicle, and the image of the vehicle is displayed together with the panoramic image. A panoramic image display method characterized by displaying on a monitor. 6. The panoramic image display method according to claim 4, wherein the display method of the auxiliary line or the image image is switched according to a switch operation in the vehicle or a traveling state of the vehicle. Characteristic panoramic image display method. 7. An image photographed by a plurality of cameras is virtually projected on a concave virtual screen formed by cutting out a part of a cylinder from an axial position of the cylinder, and the image projected on the virtual screen is projected. A panorama image display device that displays a composite image obtained when photographed by a virtual camera installed at a position separated by a predetermined distance from the axis position on a monitor as a panorama image, the panorama image display device being connected to the plurality of cameras, A / D conversion means for converting analog image data of video captured by a plurality of cameras into digital image data; and A / D conversion means connected to the A / D conversion means, wherein the video captured by the plurality of cameras and the composite image An image for converting digital image data of an image captured by the plurality of cameras into digital image data of the composite image based on the correspondence relationship And D / A conversion means connected to the image synthesizing means and the monitor for converting digital image data of the synthesized image into analog image data and outputting the analog image data to the monitor. Characteristic panoramic image display device. 8. The panoramic image display device according to claim 7, wherein the cross-sectional shape of the cylinder is a circle, an ellipse, or a polygon. 9. A panoramic image display device according to claim 7, wherein said image synthesizing means and said D / A converting means or said D / A converting means and said monitor are provided. An auxiliary line superimposing unit, the auxiliary line superimposing unit includes: A panoramic image display device, wherein image data on the auxiliary line is superimposed. 10. The panoramic image display device according to claim 9, wherein the panoramic image display device is mounted on a vehicle together with the plurality of cameras and the monitor. 11. The panoramic image display device according to claim 7, wherein said panoramic image display device is mounted on a vehicle together with said plurality of cameras and said monitor, and said image synthesizing means and said D / A. An image superimposing means between the converting means or the D / A converting means and the monitor, wherein the image superimposing means causes the monitor to display the image of the vehicle together with the panoramic image. A panoramic image display device, wherein the image data of the image image is superimposed on the image data of the panoramic image. 12. The panoramic image display device according to claim 10, wherein said auxiliary line superimposing means switches a display method of said auxiliary line according to a switch operation in said vehicle or a traveling state of said vehicle. Panoramic image display device. 13. The panoramic image display device according to claim 11, wherein said image image superimposing means switches a display method of said image image in accordance with a switch operation in said vehicle or a running state of said vehicle. Panoramic image display device.