JP6016662B2 - Image composition apparatus and image composition method - Google Patents

Description

  The present invention relates to an image synthesizing apparatus and an image synthesizing method for synthesizing images taken by a plurality of cameras and generating a panoramic image in a desired direction.

The panoramic image generation processing by the image composition device is processing for generating a single image by superimposing camera images taken by a plurality of cameras.
In addition, panoramic image generation processing is a technology for generating images with an angle of view that cannot be taken with a normal camera and images from a free viewpoint. It is used in.

FIG. 8 is an explanatory diagram showing an example of generating a panoramic image in aircraft driving support.
In the example of FIG. 8, six cameras are attached around the aircraft body, and images of the six cameras are combined to form a panoramic image in a desired direction (hereinafter referred to as “line-of-sight direction”). Generated and output to the display device.
Note that the line-of-sight direction does not necessarily coincide with the direction of someone's line of sight. For example, any direction that the passenger wants to see may be used, or a fixed direction may be used. Of course, it may coincide with someone's line of sight.

In the panorama image generation process, a camera image is projected onto a virtual projection surface of various shapes such as a plane and a curved surface, and a panorama image is generated by cutting out the part corresponding to the display surface in the viewing direction from the virtual projection surface To do.
FIG. 9 is a conceptual diagram showing panoramic image generation processing.

Various algorithms exist for panoramic image generation processing using a virtual projection plane.
The simplest algorithm is to project all the camera images by projective transformation of the camera image from the camera imaging plane to the virtual projection plane, and projectively transform a part of the synthesized image to the plane of the line of sight. A panoramic image in the line-of-sight direction can be obtained.
If the ray tracing method is used, only the image in the line-of-sight direction can be obtained by back calculation, so that a panoramic image in the line-of-sight direction can be obtained with a smaller amount of calculation and memory capacity.

Here, when generating an image on the projection surface from the camera image (or when performing reverse calculation from the projection surface to the camera by the ray tracing method), the relative positional relationship between the camera and the projection surface (hereinafter, “ The coefficient of the arithmetic expression is determined from “camera relative position”.
In conventional panoramic image generation processing, the camera relative position is uniquely set to match the relationship between the virtual projection plane and the actual camera orientation as much as possible, and takes a constant value even if the line-of-sight direction changes. I have to.

In the panorama image generation process, camera-specific distortion exists, so discontinuity (unnaturalness) occurs at the seam of the generated panorama image.
For example, the roughness or brightness of the panoramic image is not uniform, a single object appears to be plural in a part of the panoramic image, or a straight line appears to be bent.
Even if distortion correction is performed on the camera image, it is difficult to completely eliminate this unnaturalness.
That is, there is no distortion in the entire image of all the cameras, and the joint of the A portion is matched, for example, unless ideal distortion correction is performed so that the angle of view of the correction result completely matches the reality. When the joints of the B part are shifted and the joints of the B part are combined, a phenomenon such as the seam of the A part shifting may occur, and unnatural seams may be formed at a plurality of locations on the projection surface.
In particular, when the number of cameras is large, or when the periphery of an object to be imaged is surrounded by four cameras, the number of joints that must be aligned at the same time increases, and thus the above phenomenon occurs remarkably.

  In Patent Document 1 below, whether or not a plurality of camera images are superposed is determined, and if there is a poorly superposed portion, the panning / tilt angle / zoom rate of the camera is adjusted to adjust the seam of the images. An image synthesizing apparatus that generates a suitable panoramic image by eliminating the unnaturalness that occurs in the system is disclosed.

JP 2001-036898 A (paragraph number [0007])

Since the conventional image synthesizing apparatus is configured as described above, it is possible to correct a poorly superimposed portion by adjusting the pan angle / tilt angle / zoom rate of the camera. However, in order to adjust the pan angle, tilt angle, and zoom ratio of the camera, it is necessary to add a mechanism that feeds back a control signal for controlling the camera to the camera. .
Further, since the mechanism for outputting the feedback signal to the camera is a feedback loop through the machine, there is a problem that it is difficult to apply to a device with high real-time characteristics in which the line-of-sight direction always moves.

  The present invention has been made to solve the above-mentioned problems. Even if the line-of-sight direction moves without adding a mechanism for controlling the camera, the discontinuity of the seam of the panoramic image is realized in real time. It is an object of the present invention to obtain an image composition apparatus and an image composition method that can be improved.

  The image composition device according to the present invention provides a parameter indicating a relative positional relationship between the projection planes of each camera in the reference direction for each reference direction that is a direction from the viewpoint with respect to each position of the projection planes projected by a plurality of cameras. Parameter set storage means for storing the set, coincidence degree calculating means for calculating the coincidence degree of the plurality of reference directions with respect to the line-of-sight direction, and the parameters according to the coincidence degrees of the plurality of reference directions calculated by the coincidence degree calculating means Parameter set generating means for generating a parameter set used for generating a panoramic image from parameter sets corresponding to a plurality of reference directions stored in the set storage means, and the panorama image generating means is generated by the parameter set generating means. Combine the images from multiple cameras using Kicking is obtained so as to generate a panoramic image.

  According to the present invention, the degree-of-matching calculating means for calculating the degree of matching of the plurality of reference directions with respect to the line-of-sight direction, and the parameter set storage means store the matching degrees according to the degree of matching of the plurality of reference directions calculated by the matching degree calculating means. Parameter set generating means for generating a parameter set used for generating a panoramic image from parameter sets corresponding to a plurality of reference directions, and the panoramic image generating means uses the parameter set generated by the parameter set generating means. Since the panoramic image in the line-of-sight direction is generated by combining the images of the plurality of cameras, the seam of the panoramic image can be moved even if the line-of-sight direction moves without adding a mechanism for controlling the camera. This has the effect of improving discontinuity in real time.

It is a block diagram which shows the image synthesizing | combining apparatus by Embodiment 1 of this invention. It is a flowchart which shows the processing content (image composition method) of the image composition apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows an example of parameter set ad corresponding to four reference directions ad set when combining a panoramic image using three cameras. It is explanatory drawing which shows an example of four reference directions ad and a gaze direction. It is explanatory drawing which shows the example of the production | generation of the parameter set by a weighted average. It is a conceptual diagram which shows the calculation process of the pixel position by a ray tracing method. It is explanatory drawing which shows an example in the case of setting a reference direction for every fixed interval. It is explanatory drawing which shows the example of a production | generation of the panoramic image in the driving | operation control assistance of an aircraft. It is a conceptual diagram which shows the production | generation process of a panoramic image.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing an image composition apparatus according to Embodiment 1 of the present invention.
In FIG. 1, a parameter set storage unit 1 is composed of a storage device such as a RAM or a hard disk, for example, and the direction from the viewpoint for each position of the projection plane projected by N cameras (N is an integer of 2 or more). For each reference direction, a parameter set indicating the relative positional relationship between the projection planes of the cameras in the reference direction is stored. The parameter set storage unit 1 constitutes parameter set storage means.

The coincidence degree calculation unit 2 is constituted by, for example, a semiconductor integrated circuit on which a CPU is mounted, a one-chip microcomputer, or the like, and performs a process of calculating the coincidence degree of a plurality of reference directions with respect to the line-of-sight direction. The coincidence calculation unit 2 constitutes a coincidence calculation unit.
The parameter set generation unit 3 is composed of, for example, a semiconductor integrated circuit on which a CPU is mounted, or a one-chip microcomputer. The parameter set generation unit 3 is configured according to the degree of coincidence in a plurality of reference directions calculated by the degree of coincidence calculation unit 2. A process of generating a parameter set used for generating a panoramic image from a parameter set corresponding to a plurality of reference directions stored in the storage unit 1 is performed. The parameter set generation unit 3 constitutes parameter set generation means.

  The panorama image generation unit 4 is composed of, for example, a semiconductor integrated circuit on which a CPU is mounted, a one-chip microcomputer, or the like, and uses the parameter set generated by the parameter set generation unit 3, and images of N cameras. To generate a panoramic image in the line-of-sight direction. The panorama image generation unit 4 constitutes a panorama image generation unit.

In the example of FIG. 1, each of the parameter set storage unit 1, the coincidence calculation unit 2, the parameter set generation unit 3, and the panorama image generation unit 4 that is a component of the image composition device is configured by dedicated hardware. However, the image composition device may be configured by a computer.
When the image synthesizing apparatus is configured by a computer, the parameter set storage unit 1 is configured on the memory of the computer, and the processing contents of the coincidence calculation unit 2, the parameter set generation unit 3 and the panoramic image generation unit 4 are described. May be stored in the memory of the computer, and the CPU of the computer may execute the program stored in the memory.
FIG. 2 is a flowchart showing the processing contents (image composition method) of the image composition apparatus according to Embodiment 1 of the present invention.

Next, the operation will be described.
In the parameter set storage unit 1, a plurality of parameter sets are set in advance from the outside.
The parameter set is a relative position between the projection planes of each camera in the reference direction for each reference direction that is a direction from the viewpoint with respect to each position of the projection plane projected by N cameras (N is an integer of 2 or more). This shows the relationship (relative positions and angles of N cameras with respect to the projection plane).
That is, when the panoramic image is generated by synthesizing the images of the N cameras, the parameter set indicates how to superimpose the images of the N cameras when the viewing direction of the panoramic image matches a certain reference direction. Together, it indicates whether a suitable panoramic image can be obtained.
For this reason, in setting the parameter set, for each reference direction, the relative positional relationship between the projection planes of the N cameras in the reference direction is set by investigating in advance.

FIG. 3 is an explanatory diagram showing an example of parameter sets a to d corresponding to the four reference directions a to d set when a panoramic image is synthesized using three cameras.
Here, an example in which the number of cameras is three and the number of reference directions is four is shown, but this is only an example. Accordingly, any number of cameras may be used as long as the number of cameras is two or more and the number of reference directions is two or more.
In setting the parameter set, the seam around the reference direction is particularly important. That is, the region deviating from the reference direction is not so important, and the positions and angles of the cameras (1) to (3) are set so that the periphery of the reference direction can be seen naturally and the direction to be seen can be seen.

However, the reference direction may be given in any coordinate format as long as the reference direction with respect to the projection plane of the camera is uniquely determined. For example, the reference direction may be given by orthogonal coordinates or may be given by polar coordinates.
If it is necessary to reduce the amount of calculation for coordinate transformation, it is desirable that the coordinate format is the same as that of the line-of-sight direction, but this is not necessarily the case. The same can be said for the relative position of each camera.

The number of parameter sets is arbitrary as described above. Increasing the number of parameter sets greatly improves the naturalness of the seam, but increases the amount of computation. On the other hand, if the parameter set is reduced, the naturalness of the seam is reduced, but the calculation amount is suppressed.
After parameter sets for a plurality of reference directions are set in the parameter set storage unit 1, the line-of-sight direction is given from the outside.
The line-of-sight direction may be given in any coordinate format, but as described above, if it is necessary to reduce the amount of calculation, it is desirable to give it in the same coordinate format as the reference direction. is not.

When the line-of-sight direction is given from the outside, the degree-of-match calculation unit 2 matches the four reference directions a to d corresponding to the parameter sets a to d stored in the parameter set storage unit 1 and the line-of-sight direction. Is calculated (step ST1 in FIG. 2).
Here, FIG. 4 is an explanatory diagram illustrating an example of four reference directions a to d and a line-of-sight direction.
In the example of FIG. 4, since the line-of-sight direction points to the vicinity of the reference direction b, the line-of-sight direction best matches the reference direction b among the four reference directions a to d. What is done is the reference direction c.
The degree of coincidence between the line-of-sight direction and the reference direction can be obtained by calculating the inner product of the vector in the line-of-sight direction and the vector in the reference direction. The larger the inner product value, the higher the matching degree.

The calculation of the degree of coincidence of the reference direction with respect to the line-of-sight direction may be performed with respect to the reference directions of all parameter sets stored in the parameter set storage unit 1 or may be performed in the reference directions of some parameter sets. You may make it implement only with respect to it.
If there is no regularity between each reference direction, it is impossible to predict which match direction of the line-of-sight direction will be high, so there is no choice but to calculate the coincidence for all reference directions. For example, if polar coordinates are used as the coordinate format of the line-of-sight direction and the reference direction, and a reference direction is set at regular intervals, a candidate for a reference direction having a large approximate value can be easily predicted from the polar coordinates of the line-of-sight direction. Is possible.
In this case, it is possible to reduce the amount of calculation by calculating the degree of coincidence only with respect to the reference direction around the reference direction candidate having a high degree of approximation.

When the coincidence calculation unit 2 calculates the coincidence of the reference directions a to d with respect to the line-of-sight direction, the parameter set generation unit 3 stores the reference stored in the parameter set storage unit 1 according to the coincidence of the reference directions a to d. A parameter set used for generating a panoramic image is generated from parameter sets corresponding to the directions a to d.
Hereinafter, a process for generating a parameter set used for generating a panoramic image will be specifically described.

First, the parameter set generation unit 3 sets the weighting factors of the parameter sets a to d corresponding to the reference directions a to d based on the matching degrees of the reference directions a to d calculated by the matching degree calculation unit 2 ( Step ST2).
As the weight coefficient of the parameter sets a to d corresponding to the reference directions a to d, a larger weight coefficient is set as the matching degree in the reference direction is higher.
When the parameter set generation unit 3 sets the weight coefficients a to d of the parameter sets a to d corresponding to the reference directions a to d, the parameter set corresponding to the reference directions a to d is used using the weight coefficients a to d. The weighted average of a to d is calculated, and the parameter set after the weighted average is output to the panoramic image generation unit 4 as a parameter set used for generating a panoramic image (step ST3).
In the example of FIG. 4, since the degree of coincidence between the reference directions b and c is high, the parameter set used for generating the panoramic image is a parameter set that largely reflects the parameter sets b and c corresponding to the reference directions b and c. .
FIG. 5 is an explanatory diagram showing an example of generating a parameter set by weighted averaging.

Here, an example is shown in which the weighted average of all the parameter sets a to d stored in the parameter set storage unit 1 is shown, but only the parameter set corresponding to the reference direction having a matching degree higher than a predetermined threshold value. An average or a weighted average may be calculated, and the calculation result may be output to the panorama image generation unit 4 as a parameter set used for generating the panorama image.
For example, when the coincidence degree of the reference directions b, c, d is higher than the threshold value and the coincidence degree of the reference direction a is lower than the threshold value, the average or weight of the parameter sets b, c, d corresponding to the reference directions b, c, d The average will be calculated.

Note that, as described above, the degree of coincidence between the line-of-sight direction and the reference direction increases as the inner product value of the line-of-sight direction vector and the vector in the reference direction increases. When the degree is equal to or greater than the degree (when the degree of coincidence is 0 or less), the inner product value is a negative value. Therefore, in the case of a negative value, it is excluded from the target of the weighted average.
As a result, a parameter set with a low degree of coincidence can be easily excluded and a parameter set with a low degree of coincidence is hardly reflected. A suitable overlay result can be obtained when there are few parameter sets and the relative position of the camera changes gradually depending on the reference direction.

Also, here, an example is shown in which a parameter set used for generating a panoramic image is generated by calculating a weighted average of the parameter sets a to d corresponding to the reference directions a to d. A parameter set used for generating a panoramic image may be generated.
First, the parameter set generation unit 3 selects a reference direction having the highest matching degree calculated by the matching degree calculation unit 2 from the four reference directions a to d.
In the example of FIG. 4, the reference direction b is selected as the reference direction with the highest degree of coincidence. In the following description, it is assumed that the reference direction b is selected.
When the parameter set generation unit 3 selects the reference direction b having the highest degree of coincidence, the reference direction having the highest degree of coincidence among the parameter sets corresponding to the reference directions a to d stored in the parameter set storage unit 1 The parameter set corresponding to b is read, and the parameter set is output to the panoramic image generation unit 4 as a parameter set used for generating a panoramic image.

When the parameter set generation unit 3 generates a parameter set used by the parameter set generation unit 3 to generate a panoramic image, the panorama image generation unit 4 combines the images of N cameras using the parameter set to generate a panoramic image in the line-of-sight direction. (Step ST4).
Any algorithm may be used as long as the panoramic image is synthesized by projecting the camera image onto the projection plane from the relative position of each camera, but in the following, as an example, a rate using a spherical projection plane is used. The processing contents when generating a panoramic image by the racing method will be briefly described.

The ray tracing method calculates the corresponding pixel position on the camera image by performing ray tracing (ray tracing) for all the pixels on the display screen, and the pixel at the pixel position (the pixel of the input camera image). ) As an output pixel.
FIG. 6 is a conceptual diagram showing pixel position calculation processing by the ray tracing method.

First, a straight line is drawn from the viewpoint to the target pixel (pixel A in the figure) on the display screen, and the coordinates of the point where the straight line intersects the projection plane (coordinate B in the figure) are calculated.
The coordinates can be calculated from the line-of-sight direction, the information indicating the viewpoint position, the information indicating the display screen, the information indicating the projection plane, and the position of the target pixel.
Next, a straight line is drawn with respect to each camera position from the intersection coordinates on the projection plane, and the coordinates (the coordinates of C in the figure) of the point where the straight line and the camera imaging plane intersect are calculated.
This coordinate can be calculated from the camera relative position result and information indicating the camera imaging surface.

Further, the output pixel value of the target pixel on the display screen is obtained by superimposing the pixel values at the intersections of the camera imaging surfaces between the cameras. Here, when the straight line does not intersect with the imaging surface, the camera is excluded from superposition (D in the figure). For superposition, an average / weighted average or nearest neighbor method is generally used.
As described above, in the panoramic image synthesis processing using the ray tracing method, the pixel position on the projection plane and the pixel position on the imaging plane are calculated backward from the pixel position to be calculated based on the line of sight (viewpoint) and the camera relative position. By doing so, the pixel value to be displayed is obtained.

  As is apparent from the above, according to the first embodiment, the coincidence degree calculation unit 2 that calculates the degree of coincidence of a plurality of reference directions with respect to the line-of-sight direction, and the plurality of reference directions calculated by the coincidence degree calculation unit 2 According to the degree of coincidence, there is provided a parameter set generation unit 3 for generating a parameter set used for generating a panoramic image from parameter sets corresponding to a plurality of reference directions stored in the parameter set storage unit 1, and a panoramic image generation unit 4 Is configured to generate a panoramic image in the line-of-sight direction by combining the images of a plurality of cameras using the parameter set generated by the parameter set generation unit 3, so a mechanism for controlling the camera is added. Even if the line-of-sight direction moves, the discontinuity of the seam of the panoramic image can be improved in real time. Achieve the results.

That is, since the camera relative position adaptive to the line-of-sight direction is calculated from the camera relative position in the reference direction having a high degree of coincidence with the line-of-sight direction, the seam of the image is natural for any line-of-sight direction. A panoramic image can be obtained.
Further, since it is only necessary to set the parameter set so that only the seams around the reference direction overlap naturally, the parameter set can be easily set.

Embodiment 2. FIG.
In the first embodiment, the coincidence degree calculation unit 2 calculates the inner product of the line-of-sight direction vector and the plurality of reference direction vectors as the degree of coincidence between the line-of-sight direction and the plurality of reference directions. The degree calculation unit 2 may calculate a scalar value of an outer product of a vector in the line-of-sight direction and a plurality of reference directions.
The degree of coincidence between the line-of-sight direction and the reference direction indicates that the degree of coincidence is higher as the scalar value of the vector outer product is smaller.

The parameter set generation unit 3 calculates the scalar value of the vector outer product of the line-of-sight direction and the reference directions a to d as the degree of matching of the reference directions a to d, based on the scalar value of the vector outer product. The weight coefficients of the parameter sets a to d corresponding to the reference directions a to d are set.
As the weight coefficient of the parameter sets a to d corresponding to the reference directions a to d, a larger weight coefficient is set as the scalar value of the vector outer product in the reference direction is smaller (the matching degree in the reference direction is higher).
When the parameter set generation unit 3 sets the weight coefficients a to d of the parameter sets a to d corresponding to the reference directions a to d, the weight set a to d is used as the reference in the same manner as in the first embodiment. A weighted average of the parameter sets a to d corresponding to the directions a to d is calculated, and the parameter set after the weighted average is output to the panoramic image generation unit 4 as a parameter set used for generating a panoramic image.

Here, an example is shown in which the weighted average of all the parameter sets a to d stored in the parameter set storage unit 1 is shown. However, the reference direction in which the scalar value of the vector outer product is smaller than a predetermined threshold (the degree of coincidence is An average or a weighted average of only parameter sets corresponding to a reference direction higher than a predetermined threshold value may be calculated.
Further, a reference direction having the smallest vector cross product scalar value (reference direction having the highest degree of coincidence) is selected from the four reference directions a to d, and the parameter set corresponding to the reference direction is selected from the parameter set storage unit 1. May be read out.

However, it should be noted that the scalar value of the vector outer product takes the maximum value when the angle between the line-of-sight direction and the reference direction is 90 degrees, and decreases again when the angle is 90 degrees or more.
That is, when the scalar value of the vector cross product is used as the degree of coincidence, if the angle between the line-of-sight direction and the reference direction is 90 degrees or more, the parameter set corresponding to the reference direction is excluded from the target of the weighted average by some method. There is a need to. For example, the inner product value may be calculated separately, and the determination may be made based on whether the inner product value is positive or negative, or the angle may be calculated for determination.

When calculating the scalar value of the vector outer product as the degree of coincidence between the line-of-sight direction and a plurality of reference directions, the degree of coincidence in the case of the best match is always 0 (depending on the scalar value of the vector in the reference direction when using the vector inner product) , The best match value is different).
Therefore, it is possible to limit the target of the weighted average by simple threshold processing for the degree of coincidence.
Since it is possible to refer to the reference direction of the range set as a threshold value by the degree of coincidence, a suitable overlay result can be obtained when there are many parameter sets and the relative position of the camera changes drastically depending on the reference direction.

Embodiment 3 FIG.
In the first embodiment, the coincidence degree calculation unit 2 calculates the inner product of the line-of-sight direction vector and the plurality of reference direction vectors as the degree of coincidence between the line-of-sight direction and the plurality of reference directions. The degree calculation unit 2 may calculate the degree of approximation of the yaw value and pitch value between the line-of-sight direction and a plurality of reference directions.
As the value indicating the degree of approximation, any value may be calculated as long as it is a value indicating the degree of approximation between the scalar values. For example, the absolute value of the difference between the yaw value and the pitch value, or the square value of the difference Etc. may be calculated. Further, the cosine value of the angle formed by the vectors may be calculated as the degree of approximation.

When the coincidence calculation unit 2 calculates the degree of approximation of the yaw value and the pitch value between the line-of-sight direction and the plurality of reference directions as the degree of coincidence between the line-of-sight direction and the plurality of reference directions, In addition, if a reference direction is set, it is possible to easily predict parameter set candidates corresponding to the reference direction having a high degree of coincidence from the input yaw value / pitch value in the line-of-sight direction.
FIG. 7 is an explanatory diagram illustrating an example in which a reference direction is set at regular intervals.
In the example of FIG. 7, the reference direction is set every 45 degrees of the yaw value / pitch value.
For example, if the line-of-sight direction is in the range of the yaw value of 0 to 45 degrees and the pitch value of 90 to 135 degrees, the parameter set candidates corresponding to the reference direction having the highest degree of coincidence are in four ranges (yaw value-pitch value). 0 degree-90 degrees, 45 degrees-90 degrees, 0 degrees-135 degrees, or 45 degrees-135 degrees).

Therefore, since the parameter set having a high matching degree can be limited before calculating the matching degree, it is possible to reduce the number of targets for calculating the matching degree.
The degree of coincidence may be calculated only for the above four surrounding parameter sets, or a parameter set with a wider range or another number of parameter sets with a narrower range may be used as the degree of coincidence calculation. . Of course, the degree of coincidence can be calculated for all parameter sets.

In the third embodiment, a reference direction with a high degree of coincidence can be predicted in advance by setting a reference direction for the yaw value / pitch value at regular intervals. The parameter set can be easily excluded from the calculation target.
Compared to the case of using the inner product value or the scalar product of the outer product, the number of terms of the weighted average can be greatly reduced, so that a suitable overlapping result can be obtained with a smaller calculation amount.

  In the present invention, within the scope of the invention, any combination of the embodiments, or any modification of any component in each embodiment, or omission of any component in each embodiment is possible. .

  DESCRIPTION OF SYMBOLS 1 Parameter set memory | storage part (parameter set memory | storage means), 2 Concordance degree calculation part (coincidence degree calculation means), 3 Parameter set production | generation part (parameter set production | generation means), 4 Panorama image generation part (panoramic image production | generation means)

Claims (7)

A parameter set storage that stores a parameter set indicating a relative positional relationship between the projection planes of each camera in the reference direction for each reference direction that is a direction from the viewpoint with respect to each position of the projection plane projected by a plurality of cameras. Means,
A degree of coincidence calculating means for calculating a degree of coincidence of a plurality of reference directions with respect to the line-of-sight direction;
A parameter set used for generating a panoramic image is generated from parameter sets corresponding to the plurality of reference directions stored in the parameter set storage unit according to the matching degrees of the plurality of reference directions calculated by the matching level calculation unit. Parameter set generation means;
An image synthesizing apparatus comprising: a panoramic image generating unit configured to generate a panoramic image in the line-of-sight direction by combining the images of a plurality of cameras using the parameter set generated by the parameter set generating unit.   The parameter set generation means obtains a weighted average of the parameter sets corresponding to the plurality of reference directions stored by the parameter set storage means, using the degree of coincidence of the plurality of reference directions calculated by the degree of coincidence calculation means as a weight. The image synthesizing apparatus according to claim 1, wherein the averaged parameter set is output as a parameter set used for generating a panoramic image.   The parameter set generating means is a parameter corresponding to the reference direction having the highest degree of coincidence with respect to the line-of-sight direction calculated by the coincidence degree calculating means from among the parameter sets corresponding to the plurality of reference directions stored by the parameter set storage means. 2. The image composition apparatus according to claim 1, wherein a set is selected and the parameter set is output as a parameter set used for generating a panoramic image.   The coincidence degree calculating means calculates a vector dot product value between the line-of-sight direction and the plurality of reference directions as the degree of coincidence of the plurality of reference directions with respect to the line-of-sight direction. The image composition device according to claim 1.   The coincidence degree calculating means calculates a scalar value of a vector outer product of the line-of-sight direction and the plurality of reference directions as the degree of coincidence of the plurality of reference directions with respect to the line-of-sight direction. The image composition device according to any one of the preceding claims.   The degree-of-match calculation means calculates the degree of approximation of the yaw value and pitch value between the line-of-sight direction and the plurality of reference directions as the degree of match of the plurality of reference directions with respect to the line-of-sight direction. The image composition device according to any one of the above. For each reference direction that is a direction from the viewpoint for each position of the projection plane projected by a plurality of cameras, a parameter set indicating a relative positional relationship between the projection planes of each camera in the reference direction is stored in advance.
A degree-of-matching calculating means for calculating a degree of matching of a plurality of reference directions with respect to the line-of-sight direction;
The parameter set generating means uses the parameter set used for generating the panoramic image from the parameter sets corresponding to the plurality of reference directions stored in advance according to the matching degrees of the plurality of reference directions calculated in the matching degree calculation processing step. A parameter set generation processing step to be generated;
A panorama image generation unit that combines the images of a plurality of cameras using the parameter set generated in the parameter set generation processing step to generate a panoramic image in the line-of-sight direction. Synthesis method.

Images