JP2001052146A - Image synthesizing method and image display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method capable of exactly determining the correlation of images by excluding the influence of occlusion concerning the images for which not only the viewpoints/viewing directions of cameras but also zooms are different. SOLUTION: This method is provided with a process for storing plural input source images for which the viewing directions, viewpoints, zooms and resolutions are different, a process for correlating the images while suppressing influences caused by a lightness difference, the occlusion, the zooms and the resolutions from the source images, a process for measuring a lightness distribution in the area overlapped between the images, a process for generating a panoramic image by transforming the coordinates of images on the basis of the provided correlation of images and partially controlling lightness so that the information of an image having the wider lightness distribution can take priority in the overlapped area, and a process for segmenting and displaying a display image from the panoramic image corresponding to the display viewpoint, the viewing direction and the display resolution.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] The present invention relates to a plurality of viewpoint positions,
The present invention relates to an image synthesis method for generating a panoramic image by combining images having different directions and zooms, and an image display method for selecting and displaying a desired image.

[0002]

2. Description of the Related Art Many techniques have been proposed for generating and displaying panoramic images. Viewpoint position,
For a plurality of images in different directions, a technique of detecting and combining overlapping parts to generate a panoramic image, a technique of selecting a plurality of images when displaying, or a technique of generating an intermediate image,
At the time of photographing, there are techniques using a special imaging device so that a superimposed portion becomes smooth.

[0003]

However, none of them take into consideration various problems that occur between images having different zooms.
For example, if you zoom in on a dark part in the field of view, the aperture of the camera changes and that part appears brighter. Then, a different thing may be seen in the brightened portion. In this way, the brightness is different despite the same part,
No technology assumes that occlusion (visible on the one hand and invisible on the other) will occur. Also, when there are a plurality of images for the same location in display, there is no technique that takes into account the difference in resolution between the images.

[0004] It is an object of the present invention to provide a method capable of appropriately combining or displaying a plurality of images in response to a phenomenon that occurs when not only the viewpoint of a camera but also a zoom is changed.

[0005]

In order to solve the above-mentioned problems, an image synthesizing and displaying method according to the present invention comprises a step of storing a plurality of input original images having different viewing directions, viewpoints, zooms, and resolutions; Obtaining a correlation between the images by suppressing the effects of brightness difference, occlusion, zooming and resolution from the images; measuring a brightness distribution for a region to be overlapped between the images; and correlating the obtained images. Performing a coordinate transformation of the image based on the image data, partially adjusting the brightness of the superimposed area so as to give priority to the information of the image having the wider brightness distribution, and generating a panoramic image. Cutting out and displaying a display image in accordance with the viewing direction.

[0006] According to the above configuration, even if an object in an image is visible or invisible between a plurality of images having different zooms and resolutions as well as viewpoints and viewing directions, it is not affected by occlusion. The correlation between images can be determined accurately. At the time of display, the object in the image is not saturated brightly or darkly due to the brightness adjustment.

Preferably, when generating a panoramic image,
In the superimposition area, the brightness is adjusted so that the information of the image with the higher resolution is prioritized.

Preferably, when displaying the panoramic image, after extracting the panoramic image in accordance with the display viewpoint and viewing direction, the extracted image is enlarged or reduced in accordance with the display resolution, and the brightness of the display area is changed to the display brightness. Performs brightness conversion to fit within

Further, the image display method according to the present invention includes a step of storing a plurality of input original images having different viewing directions, viewpoints, zooms, and resolutions, and an effect of the brightness difference, occlusion, zoom, and resolution from the original images. Suppressing and obtaining a correlation between images, calculating and storing the viewpoint / viewing direction and resolution of the original image based on the obtained correlation between the images, and storing the viewpoint / viewing direction and resolution of the stored original image And selecting and displaying an original image in accordance with the display viewpoint / viewing direction and the display resolution based on the above data.

[0010] According to the above configuration, for a plurality of images having different zooms in addition to the viewpoint and the viewing direction, even if an object in the image is seen or not seen between the images, the image is not affected by occlusion and is not affected by occlusion. Can be accurately determined. Further, it is possible to easily browse a desired image without storing panorama image data in advance.

Preferably, at the time of display, the original image is selected in accordance with the display viewpoint, viewing direction, and resolution, and the viewpoint and the viewpoint of the original image are selected.
If the viewing direction / resolution is significantly different from the display viewpoint / viewing direction / resolution, the original image is coordinate-transformed and enlarged / reduced for display.

Preferably, at the time of display, the brightness in the display area is normalized to the displayable brightness before displaying.

Further, the image display method according to the present invention includes a step of storing a plurality of input original images having different viewing directions, viewpoints, zooms, and resolutions, and a method for detecting the influence of brightness difference, occlusion, zoom, and resolution from the original images. Suppressing and obtaining a correlation between the images, calculating the viewpoint / viewing direction and resolution of the image based on the obtained correlation between the images, and measuring and storing the brightness distribution of the superimposed region; Based on the viewpoint, viewing direction, and resolution data of the original image,
A step of selecting a plurality of original images according to a viewing direction and a display resolution; a step of determining a combination ratio based on a difference between superimposed areas and a difference in lightness distribution; and a step in which a lightness value of all image information in the display area is displayed in a predetermined manner. Adjusting the brightness of each image so as to be uniformly within the brightness and combining and displaying the images.

According to the above configuration, the same effect as that of the above-mentioned other invention can be obtained even when a plurality of original images are displayed in combination, and the balance between the images is optimally adjusted and displayed. be able to.

[0015] Preferably, at the time of image synthesis, the brightness adjustment is performed for each image in the superimposed region so that the information of the image with the highest resolution is prioritized.

[0016] Preferably, at the time of image synthesis, brightness adjustment is performed for each image so that information of an image having the widest brightness distribution is prioritized in the superimposed area.

Preferably, based on the brightness and the determination as to whether or not the image is within the superimposed area, the brightness distribution of the superimposed area is adjusted in accordance with the partial portion of the image so as to be uniformly within the display brightness, and then synthesized. To display. At this time, it is preferable to adjust the brightness of the superimposed area in accordance with a part of the image so that the information of the image with the highest resolution is prioritized. Alternatively, in the superimposition region, it is preferable to perform brightness adjustment according to a portion of the image such that information of an image having the widest brightness distribution is prioritized.
Further, the brightness distribution of the superimposed area can be measured at the time of composite display, instead of being measured and stored in advance.

In the present invention described above, the viewing direction, the viewpoint,
It is also possible to adopt a configuration in which a plurality of images having different viewing directions, viewpoints, and zooms are photographed by a photographing device capable of measuring the zoom and the resolution, and the process of calculating the correlation between the images is omitted.

An apparatus for implementing the above-described image synthesizing and displaying method according to the present invention comprises: an original image storage unit for storing a plurality of input original images having different viewing directions, viewpoints, zooms and resolutions; An image correlation acquisition unit that obtains a correlation between images while suppressing the effects of brightness difference, occlusion, zoom, and resolution; an original image coordinate conversion unit that performs image coordinate conversion based on the obtained correlation between images; Is an original image brightness value partial adjustment unit that partially adjusts the brightness so as to give priority to the information of the image with a wider brightness distribution, and a panorama based on the output of the original image coordinate conversion unit and the original image brightness value partial adjustment unit. From a panoramic image synthesis unit that generates an image and a panoramic image,
The image display unit cuts out a display image according to the image display viewpoint and the viewing direction, and enlarges and reduces the image cut out according to the display resolution to display the image.

An image synthesizing and displaying device for implementing the above-described image synthesizing and displaying method according to the present invention is an original image storage unit for storing a plurality of input original images having different viewing directions, viewpoints, zooms and resolutions. And an image correlation acquisition unit that obtains a correlation between images by suppressing the effects of brightness difference, occlusion, zoom, and resolution from the original image, and calculates a viewpoint and a viewing direction of the original image based on the correlation between the obtained images. A viewpoint / viewing direction calculation unit, a viewpoint / viewing direction storage unit for storing the calculated viewpoint / viewing direction of the original image, and a viewpoint / viewing direction storage unit for the stored original image.
Based on the viewing direction and resolution data, the display viewpoint / viewing direction,
It has a configuration including a display original image selection unit that selects an original image according to the display resolution, and an image display unit that displays an image based on the selected original image.

A recording medium according to the present invention is a computer-readable recording medium in which a program for causing a computer to execute a process of synthesizing and displaying an image from an original image is recorded, and has different viewing directions, viewpoints, zooms, and resolutions. Storing a plurality of input original images; and
Brightness difference, occlusion, a step of obtaining a correlation between images by suppressing the influence of zoom and resolution, a step of measuring a brightness distribution for a region to be overlapped between the images, and based on the obtained correlation between the images Generating a panoramic image by performing coordinate conversion of the image and partially adjusting the brightness of the superimposed area so that information of the image having a wider brightness distribution is prioritized; And a step of cutting out and displaying a display image in accordance with the program.

[0022]

(Embodiment 1) Hereinafter, an apparatus for implementing an image synthesizing and displaying method according to Embodiment 1 of the present invention will be described with reference to the drawings. FIG.
FIG. 2 is a block diagram illustrating a configuration of an image synthesis / display device.

Reference numeral 1 denotes an image input device which supplies a plurality of image signals obtained by photographing a target scene while changing the viewpoint position, the viewing direction, and the zoom. The image signal is input to the original image storage unit 2 and stored.

The image correlation acquisition section 3 reads the images stored in the original image storage section 2 and obtains a correlation between the images. In obtaining the correlation, first, the correlation for the entire image is obtained. On the basis of the result, the area for performing the correlation calculation is re-determined by the correlation calculation area determining unit 4, and again, the image correlation obtaining unit 3
Finds the correlation between the images. The above processing is repeated until the correlation calculation area determining unit 4 does not need to modify the correlation calculation area.

Based on the inter-image correlation information obtained by the image correlation acquisition unit 3, the camera viewpoint / view direction calculation unit 5 and resolution calculation unit 6 calculate the camera viewpoint / view direction and resolution for each image. The brightness distribution of the area to be superimposed between the images is measured by the superimposed area lightness distribution measuring unit 7.

Using the data of the camera viewpoint and the viewing direction obtained as described above, the original image coordinate conversion unit 8 performs coordinate conversion of the original image. Similarly, using the resolution data, the original image resolution adjustment unit 9 performs resolution adjustment for equalizing the resolution between images. The original image brightness value partial adjustment unit 10
Using the data of the lightness distribution of the superimposed area, a partial lightness adjustment for adjusting the lightness of the superimposed area for each image is performed.

The original images processed as described above are joined together by the panoramic image synthesizing unit 11.
The synthesized panoramic image is stored in the panoramic image storage unit 13 by the panoramic image output unit 12.

The display viewpoint / viewing direction acquisition unit 14 obtains a display viewpoint / viewing direction from the viewer. In accordance with the obtained display viewpoint and viewing direction, the panorama image cutout unit 15 cuts out a part to be displayed from the panorama image stored in the panorama image storage unit 13. Then, according to the display resolution obtained by the display resolution acquisition unit 16, the image is enlarged or reduced by the panorama image enlargement / reduction unit 17. The image finally obtained is displayed on the image output device 19 by the image display unit 18.

In the above configuration, when the image correlation is obtained by the image correlation acquisition section 3 and the correlation calculation area determination section 4, as described above, the lightness difference between images having different viewing directions, viewpoints, and zooms, The adjustment is performed in consideration of the occlusion so that the optimum processing is performed. The method will be described below.

First, an orthogonal three-dimensional coordinate system is set for a scene. This is called a scene coordinate system. The viewing direction and viewpoint of the camera that captured each image are represented using a scene coordinate system. The viewpoint can be represented by a three-dimensional vector, and the viewing direction can be represented by the three-dimensional vector and its rotation. Since changing the zoom is equivalent to moving the camera viewpoint in the depth direction,
It can be represented by a three-dimensional vector of viewpoint information.

The fact that there is a correlation between images means that an image can be converted into a plane model, a cylinder model, or a three-dimensional model that is actually based on an object based on the viewpoint and viewing direction of a camera. When projected by perspective projection, orthographic projection, pseudo perspective projection, or the like, the brightness value, hue value, saturation value, RGB value, or YI value of each pixel in the superimposition area
That is, the Q value and the like are approximated.

Unless the brightness difference and the occlusion are taken into account, if the index representing the correlation function is the inconsistency, the inconsistency is regarded as the difference in the image pattern of the superimposed area, and the difference in the color information of the image in the corresponding portion is referred to as the Euclidean distance. The degree of inconsistency may be represented by a section distance, and the degree of inconsistency may be searched by a brute force method, an iterative approximation method, a coarse / fine search method, or another search method that minimizes the value accumulated over the superimposed region.

In order to consider the lightness difference, the lightness I of one image is given by the lightness I ′ of another image as I ′ = a × I, I ′ = I + b, I ′ '= C × I
The unknowns in the conversion equation may be obtained at the same time, assuming that they are converted into + d and the like.

In order to consider occlusion, when accumulating the degree of mismatch after provisional search, instead of targeting the entire superimposed region for each image, the search is performed again excluding the portion with a high degree of mismatch. Just repeat the work.

The calculation of the degree of inconsistency may be limited to only a characteristic portion. For example, a characteristic portion may be filtered by an interest operator, a Sobel operator, a Gaussian filter or the like to derive an edge element, and may be defined as a portion having a high edge strength. The histogram may be defined using a statistical method, such as creating a histogram of color information and defining it as a portion corresponding to an isolated portion on the histogram. Then, for a characteristic portion, the feature amount used as a reference when the feature is derived, that is, in the above example, the degree of inconsistency with respect to edge strength, edge position, or color information amount, statistic, etc. You just have to calculate.
The portion where occlusion has occurred is excluded because the above feature values do not match.

It is also effective to position the images in consideration of the difference in resolution. For example, when a zoomed-in image exists in a certain portion, when calculating a brightness value correlation to determine the positional relationship between the images, the weight of the brightness value of an image with high resolution is suppressed. In other words, even in a video image of the same object, the pattern and boundary are clear in a high-resolution image, but ambiguous in a low-resolution image. Therefore, if the correlation between such images is directly obtained, an error occurs in the positional relationship detection. Therefore, for an image with a high resolution, if the correlation is obtained by blurring, a high accuracy in positional relationship detection can be obtained.

The unknowns appearing during the above processing may be obtained simultaneously or sequentially. Alternatively, the temporarily determined value may be modified by an iterative method.

As described above, the flow of the process of obtaining the image correlation in consideration of the brightness difference and the occlusion performed by the image correlation acquisition unit 3 and the correlation calculation region determination unit 4 will be described.
This will be described with reference to FIG. First, two images are prepared (step S101). Next, the region R is set as a region covering one entire image (step S102). Furthermore,
The area Q is set to be empty (step S103). Next, a search is made for a corresponding position and a lightness evaluation parameter at which the accumulation of the difference between the lightness evaluation values I ′ of the superimposed region with respect to the region R is minimized (step S104). Based on the result, it is determined whether or not there is an area S in which the brightness evaluation value I ′ is equal to or larger than a predetermined value in the area R (step S105). If not, the process ends. If there is, the region S is excluded from the region R (step S10).
6). Next, a region P in which the difference between the brightness evaluation values I ′ is equal to or smaller than a predetermined value in the region Q is added to the region R (Step S107). Further, the region P is excluded from the region Q (step S108),
After adding the area S to the area Q (step S9), the process returns to the step S104.

Next, a method of measuring the brightness distribution of an image by the superimposed area brightness distribution measuring section 7 will be described. In order to measure the lightness distribution, the lightness may be calculated from the color information of the image and a histogram thereof may be created. Color information is RG
When represented by the B value, the lightness I is I = (R + G
+ B) / 3 or I = 0.30 × R + 0.59 × G +
It can be calculated by 0.11 × B. The histogram of the lightness distribution may be stored. Further, only the maximum value, the minimum value, the average value, and the standard deviation value of the histogram may be stored. Also, instead of the maximum and minimum values of the histogram,
The upper limit value and the lower limit value of the portion excluding the exceptional value from the histogram may be used. The exceptional value of the histogram is defined as the part where the deviation value is more than twice the variance value or an arbitrary multiple, or the part that is not included in 90% or any percentage of the whole centered on the median value, etc. May be defined by the following statistical method.

Next, a description will be given of a resolution adjustment process for adjusting the resolution between images in the original image resolution adjustment section 9.

There is a correlation between two predetermined images in the input image, and after the original image coordinate conversion by the original image coordinate conversion unit, the two images are superimposed as shown in FIG. Suppose. At this time, as a panoramic image obtained by combining the two images, an output image including a circumscribed rectangle including both the first image p1 and the second image p2 is generated. The original image resolution adjusting unit 9 generates an image according to the following procedure.

1) When the first image p1 and the second image p2 are coordinate-transformed and projected onto the circumscribed rectangle, the pixels in the original image, such as pixel A shown in FIG. Find the smallest projected pixel.

2) The size of the output image is calculated so that the pixel obtained in 1) corresponds to one pixel in the output image.

3) The first image p1 and the second image p2 are projected on the output image having the size determined in 2). in this case,
Non-minimum pixels will be magnified. At the time of enlargement, interpolation or the like is performed.

By performing the processing described above, the smallest pixel will not be crushed. The brightness values of the pixels in the superimposed area are determined by the original image brightness value partial adjustment unit 10. When there are three or more input images, the size of the output image is calculated and projected so that the smallest projected pixel of all images corresponds to the size of one pixel in the output image.

Next, a description will be given of a partial brightness adjustment method for adjusting the brightness of the superimposed area in each image by the original image brightness value partial adjustment section 10.

First, the range of the brightness distribution of each image to be superimposed is examined based on the measurement result of the superimposed area brightness distribution measuring section 7. Then, the relative relationship of the brightness distribution is obtained by relying on the portion where the superimposed portion is reflected, and the entire brightness distribution is calculated based on one of the images. Then, primary conversion or secondary or higher conversion of the brightness is performed so that the upper and lower limits of the entire brightness distribution coincide with the upper and lower limits of the display brightness. When calculating the overall brightness distribution, a higher resolution image may be prioritized. In that case, the brightness is converted such that the upper and lower limits of the brightness distribution of the image with the highest resolution match the upper and lower limits of the display brightness. If another image is out of display brightness, the upper or lower limit of the display brightness of the image is adjusted to the upper or lower limit of the display brightness of the image with the highest resolution.

When calculating the overall brightness distribution, the image having the widest brightness distribution may be prioritized. In this case, the brightness is converted such that the upper and lower limits of the brightness distribution of the image having the widest brightness distribution match the upper and lower limits of the display brightness. If another image is out of display brightness, the upper limit or lower limit of the display brightness of the image is adjusted to the upper limit or lower limit of the display brightness of the image having the widest brightness distribution. The effect of the brightness adjustment may be suppressed near the boundary between the superimposition region and the non-overlapping region.

Next, a method of extracting and displaying a part of a panoramic image in the panoramic image extracting section 15 will be described. First, based on the display viewpoint / viewing direction obtained from the display viewpoint / viewing direction obtaining unit 14 and the resolution obtained from the display resolution obtaining unit 16, the scene coordinate system defined in the description of the method of calculating the correlation of the above images is used. And the area that the viewer is looking at. Then, a panoramic image included in the area may be cut out. If the size of the clipped image does not match the display area, it may be enlarged or reduced according to the display area. The enlargement / reduction may be performed uniformly on the cut-out area, or may be performed adaptively according to the display pixels.

As a method of acquiring the display viewpoint, the viewing direction, and the resolution from the viewer by the display viewpoint / viewing direction acquiring unit 14 and the display resolution acquiring unit 16, a method equivalent to a widely used three-dimensional browser is used. be able to.

(Embodiment 2) An apparatus for implementing an image synthesizing and displaying method according to Embodiment 2 of the present invention will be described with reference to the drawings. FIG. 4 is a block diagram showing a configuration of the image synthesizing / displaying device.

The image input device 1, the original image storage unit 2, the image correlation acquisition unit 3, the correlation calculation area determination unit 4, the camera viewpoint / viewing direction calculation unit 5, and the resolution calculation unit 6 are the same as those in the first embodiment.
Has the same function as the element given the same number in, and the description is omitted. The viewpoint and resolution obtained by the camera viewpoint / view direction calculation unit 5 and the resolution calculation unit 6 are stored in the viewpoint / view direction / resolution storage unit 21. In the second embodiment, unlike the first embodiment,
A configuration in which a panoramic image is generated and stored in advance is not employed, and only the positional relationship between input images is checked as a process after image input and before display. Then, at the time of display, the relative brightness between all the images included in the field of view is checked, normalized, and displayed.

When displaying an image, the display viewpoint / viewing direction acquisition unit 22 acquires the display viewpoint / viewing direction from the viewer, and the display resolution acquisition unit 23 acquires the display resolution. The display viewpoint / viewing direction acquisition unit 22 and the display resolution acquisition unit 23 have the same functions as the elements with the same names in the first embodiment, and the same elements can be used. The display original image selection unit 24 selects an original image closest to the obtained display viewpoint / viewing direction and display resolution. The selected original image is
The image is displayed on the image output device 26 by the image display unit 25.

The display original image selecting section 24 selects each image stored in the viewpoint / view direction / resolution storage section 21 when selecting an image from a plurality of original images in accordance with the display viewpoint / view direction / resolution. An image is selected using the data of the viewpoint, viewing direction, and resolution already measured for the image. For example, the viewpoint closest to the display viewpoint is selected. If there are a plurality of viewpoints within a certain range, the viewpoint closest to the image and the display direction is selected. Further, when there are a plurality of close ones within a predetermined range, the one having the closest image resolution and display resolution may be selected. Or
The difference between the viewpoint of the image and the viewpoint of the display, the difference between the viewing direction of the image and the viewing direction of the image, and the difference between the resolution of the image and the resolution of the display are mixed by the Euclidean distance, the block distance, the weighted Euclidean distance, the weighted block distance, etc. Then, the image having the smallest value may be selected.

When the selected image is displayed on the image display section 25 and the display viewpoint / viewing direction and resolution are significantly different, the following is performed. In other words, the selected image may be projected onto a model of the scene coordinate system, and a video when the projected image is observed based on the display viewpoint / viewing direction and the display resolution may be generated and displayed. If the video is to observe a part of the original image, the brightness conversion may be performed so that the upper and lower limits of the brightness of the observation area match the upper and lower limits of the display brightness.

With the above arrangement, when switching and displaying images, it is desirable to perform switching and display in consideration of occlusion. For example, when the pattern of the same portion changes, the switching timing or the ratio of image mixing is adjusted so as not to give a stranger to the viewer.

(Embodiment 3) An apparatus for implementing an image synthesizing and displaying method according to Embodiment 3 of the present invention will be described with reference to the drawings. FIG. 5 is a block diagram showing the configuration of the image synthesizing / displaying device.

The image input device 1, the original image storage unit 2, the image correlation acquisition unit 3, the correlation calculation region determination unit 4, the camera viewpoint / view direction calculation unit 5, the resolution calculation unit 6, and the superimposition region lightness distribution measurement unit 7, Are the same as those of the first embodiment with the same reference numerals, and the description is omitted. The viewpoint / viewing direction, resolution, and brightness distribution obtained by the camera viewpoint / viewing direction calculation unit 5, the resolution calculation unit 6, and the superimposed area brightness distribution measurement unit 7 are stored in the viewpoint / viewing direction / resolution / brightness distribution storage unit 31. Is stored. Similarly to the second embodiment, the third embodiment does not employ a configuration in which a panoramic image is synthesized and stored in advance, and only a positional relationship between input images is checked as a process after image input and before display. . Then, at the time of display, adjustment between images is performed using the data.

The display original image selection unit 35 selects a plurality of original images that are close to the display viewpoint / view direction obtained by the display viewpoint / view direction acquisition unit 32 and the display resolution obtained by the display resolution acquisition unit 33. Here, the display viewpoint / viewing direction acquisition unit 32 and the display resolution acquisition unit 33 have the same functions as the elements with the same names in the first embodiment, and the same elements can be used.

The original image coordinate conversion unit 36 performs image coordinate conversion including enlargement / reduction according to display based on the display viewpoint / viewing direction and display resolution.

The display area lightness distribution obtaining section 34 obtains the lightness distribution of an area related to display in a plurality of selected images. The original image lightness value part adjusting unit 37 determines the lightness adjustment amount by comparing the lightness distribution of the superimposed area, and adjusts the lightness in the superimposed area of the original image.

The inter-synthetic image difference detecting section 40 obtains a portion of the image selected by the display original image selecting section 35 where a difference appears even if the brightness of an overlapping area between the images is changed. Based on the obtained data of the difference between the images, the display viewpoint / viewing direction, and the display resolution, the combining ratio determination unit 39 determines a combining ratio when combining a plurality of images. Based on the obtained composition ratio, the display image composition unit 38
Synthesizes a plurality of images obtained by the original image coordinate conversion unit 36. The synthesized image is displayed on the image output device 42 by the image display unit 41.

In order to detect the location of the difference between the superimposed areas by the composite image difference detecting section 40, a portion where the color information is different when the image is projected onto the plate model in the scene space, or a portion where the color information differs to a certain extent or more. May be used.

In order for the display original image selection unit 35 to select an image from a plurality of original images in accordance with the display viewpoint, viewing direction, and resolution, it is necessary to select the measured viewpoint, viewing direction, and resolution for each image. What is necessary is just to select an image based on proximity. For example, the viewpoint of the image is selected to be closest to the display viewpoint, and when there are a plurality of close viewpoints within a certain range, the viewpoint in which the viewing direction of the image and the viewing direction of the display are the closest is selected. When there are a plurality of close ones within a predetermined range, the one having the closest image resolution and display resolution may be selected. Or, the difference between the viewpoint of the image and the viewing viewpoint, the difference between the viewing direction of the image and the viewing direction of the image, the difference between the resolution of the image and the resolution of the display, the Euclidean distance, the block distance, the weighted Euclidean distance, the weighted block distance, etc. And the image may be selected based on the small value. When a plurality of images are selected, a plurality of higher-order images may be selected according to the order of the former or the value of the latter.

If the selected image is significantly different from the display viewpoint / viewing direction and resolution, it is modified and displayed. For example, the selected image is projected onto a model of a scene coordinate system, and a video is generated when the projected image is observed based on a display viewpoint / viewing direction and a display resolution. If the video is to observe a part of the original image, the brightness conversion may be performed so that the upper and lower limits of the brightness of the observation area match the upper and lower limits of the display brightness. The above-described processing is performed on the plurality of selected images, and the images are combined by the display image combining unit 38.

Next, a description will be given of a process of determining the combining ratio when combining a plurality of images in the original image brightness value adjusting section 37. This processing is performed to combine a plurality of original images at the time of display based on the stored correspondence between the original images.

The composition ratio is determined for each pixel. Here, the resolution when the pixel of the first image corresponding to any point of the display device is projected on the display device is r ₁ , and the resolution when the pixel of the second image is projected on the display device is r ₂ And As shown in FIG. 6, and _{r 2 <r 1, r 2} <r 3 < such that r ₄ <r _1, sets the r _3, r _4.

The composition ratio of the pixels of the first image is represented by t (t ≦ t
If expressed by 1), the composition ratio of the pixels of the second image is (1−1).
t). Display resolution as shown in the graph of FIG.
Degree r is r _Four<R <r₁In this case, t = 1. This place
In this case, the composition ratio of the pixels of the first image and the pixels of the second image is
1: 0. r_Two<R <r_ThreeIn the case of, t = 0.
In this case, the synthesis of the pixels of the first image and the pixels of the second image
The ratio will be 0: 1.

If r ₃ <r <r ₄ and no difference is detected between the two pixels by the composite image difference detecting section 40, the composite ratio is determined according to a first-order linear curve as shown by a line a. Set to change smoothly. If a difference is detected, the transition of the composition ratio is determined by a higher-order function as shown by the line b so that the composition ratio of the two pixels changes rapidly.

If the density value of the pixel of the first image after the brightness conversion is I ₁ and the density value of the pixel of the second image is I ₂ ,
The density value of the display image when the two images are synthesized at the synthesis ratio t is expressed as I = tI ₁ + (1−t) I ₂ .

N sheets (n ≧ 3) at any point on the display device
If the images are overlapped, the following processing is performed. Here, the pixel on each image corresponding to that point is P _i , and the resolution of each pixel is r _i (1 ≦ i ≦ n, r _i <r _{i + 1} ). The composition ratio is represented by an n-dimensional vector W having a weight w _{i of} each pixel as an element. At this time, the display resolution r
Is r <r ₁ , w ₁ = 1, w _i = 0 (i> 1), r
> In the case of r _n, and _{w n = 1, w i =} 0 (i <n).
In addition, r _i <r <in the case of _{r i + 1, w j =} 0 (j <i∧i
+1 <j), and w _i and w _{i + 1} are determined according to the case of combining two images.

In the above embodiment, the data relating to the viewing direction, the viewpoint, and the zoom are obtained by the process of obtaining the correlation between the images. These data may be measured. For example, the visual direction and the viewpoint use a motion control camera, mount a GPS, fix the camera on a trolley and move the trolley on a graduated rail, or use a graduation marked on the joint of a tripod. It can be measured by attaching. The zoom can be measured by using a motion control camera, or by marking a handle for changing the zoom.

Each function of the image processing apparatus described in the first to third embodiments can be realized by a computer program. FIG. 7 shows a processing flow of a program for realizing the image processing apparatus according to the first embodiment.

In FIG. 7, first, a plurality of images of the target scene photographed by changing the viewpoint position, the viewing direction, and the zoom are input (step S201). Next, the correlation between the images is once obtained (step S202), and it is checked whether the correlation calculation area at that time is appropriate (step S203). If it is not appropriate, the correlation calculation area is corrected (step S204),
The correlation of the image is obtained again (step S202). After this repetition, if the correlation calculation area becomes appropriate, the camera viewpoint / viewing direction is calculated using the correlation data (step S205). Based on the result, coordinate conversion of the original image is performed (step S206), the resolution of the original image is further adjusted (step S207), and the brightness value of the original image is partially adjusted (step S208). Next, a panoramic image is synthesized using the data obtained as described above and stored (step S209). Next, display viewpoint / viewing direction data is obtained from the viewer (step S210), and a panoramic image is cut out and displayed based on the data (step S210).
211).

As shown in FIG. 8, the program for realizing the image processing apparatus of the present invention can be stored in a portable recording medium 61 such as a CD-ROM 61a or a flexible disk 61b and used. Further, it can be stored and used in any of the other storage device 63 provided at the end of the communication line 62 and the recording medium 65 such as a hard disk or a RAM of the computer 64. When using the image processing apparatus of the present invention, the stored program is loaded on a computer and executed on the main memory.

[0076]

As described above, according to the present invention, image synthesis and
According to the display method, for a plurality of images having different zooms as well as the viewpoint and the viewing direction, even if the aperture of the camera is significantly different due to zooming in and zooming out, the brightness adjustment causes the target in the image to become bright or dark and saturated. Therefore, even if an object in an image is seen or not seen between images due to a change in aperture, the correlation between images can be accurately determined without being affected by occlusion.

[Brief description of the drawings]

FIG. 1 is a functional block diagram illustrating a schematic configuration of an apparatus that performs an image synthesizing and displaying method according to a first embodiment of the present invention;

FIG. 2 is a flowchart showing a process for obtaining a correlation between images according to the first embodiment of the present invention;

FIG. 3 is a view for explaining resolution adjustment processing in an original image resolution adjustment unit of the apparatus in FIG. 1;

FIG. 4 is a functional block diagram illustrating a schematic configuration of an apparatus that performs an image synthesizing and displaying method according to a second embodiment of the present invention;

FIG. 5 is a functional block diagram illustrating a schematic configuration of an apparatus that performs an image combining / displaying method according to a third embodiment of the present invention;

FIG. 6 is a view for explaining processing in a combination ratio determination unit of the apparatus in FIG. 5;

FIG. 7 is a flowchart showing a processing flow of a program for executing the image synthesizing / displaying method according to the first embodiment of the present invention;

FIG. 8 is a diagram showing an example of a recording medium storing a program for executing the image processing / display method according to the embodiment of the present invention.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 image input device 2 original image storage unit 3 image correlation acquisition unit 4 correlation calculation area determination unit 5 camera viewpoint / viewing direction calculation unit 6 resolution calculation unit 7 and superimposition area lightness distribution measurement unit 8 original image coordinate conversion unit 9 original image resolution Adjustment unit 10 Original image brightness value adjustment unit 11 Panorama image synthesis unit 12 Panorama image output unit 13 Panorama image storage unit 14 Display viewpoint / viewing direction acquisition unit 15 Panorama image cutout unit 16 Display resolution acquisition unit 17 Panorama image enlargement / reduction Unit 18 image display unit 19 image output device 21 viewpoint / view direction / resolution storage unit 22 display viewpoint / view direction acquisition unit 23 display resolution acquisition unit 24 display original image selection unit 25 image display unit 26 image output device 31 viewpoint / view direction Resolution / brightness distribution storage unit 32 Display viewpoint / viewing direction acquisition unit 33 Display resolution acquisition unit 34 Display area brightness distribution acquisition unit 3 5 display original image selection unit 36 original image coordinate conversion unit 37 original image brightness value partial adjustment unit 38 display image synthesis unit 39 synthesis ratio determination unit 40 synthesized image difference detection unit 41 image display unit 42 image output device

Continued on the front page F-term (reference) 5B057 BA02 CD00 CD05 CE10 CE11 5C023 AA02 AA11 AA37 AA38 BA03 BA07 BA11 DA04 EA05 5C082 AA01 AA27 BA20 CA33 CA34 CA55 CA59 CB01 DA51 DA87 MM05 MM10

Claims (4)

[Claims] A step of storing a plurality of input original images having different viewing directions, viewpoints, zooms, and resolutions; and suppressing correlations between the images from the original images by suppressing effects due to brightness differences, occlusion, zooming, and resolution. Obtaining, a step of measuring a lightness distribution for an area to be superimposed between the images, and performing coordinate transformation of the image based on the obtained correlation between the images, and for the superimposed area, an image having a wider lightness distribution. Generating a panoramic image by partially adjusting the brightness so as to prioritize the information of, and cutting out and displaying a display image from the panoramic image in accordance with a display viewpoint / viewing direction and a display resolution. An image synthesizing and displaying method characterized by the following. 2. A process for storing a plurality of input original images having different viewing directions, viewpoints, zooms and resolutions, and from the original images, suppressing the effects of brightness difference, occlusion, zooming and resolution to reduce correlation between the images. Obtaining, calculating and storing the viewpoint / viewing direction and resolution of the original image based on the obtained correlation between the images; and displaying based on the stored data of the viewpoint / viewing direction and resolution of the original image. Selecting and displaying an original image according to the viewpoint / viewing direction and display resolution. Storing a plurality of input original images having different viewing directions, viewpoints, zooms, and resolutions; and suppressing correlations between the images from the original images by suppressing the effects of brightness difference, occlusion, zooming, and resolution. Obtaining, calculating the viewpoint / viewing direction and resolution of the image based on the obtained correlation between the images, measuring and storing the lightness distribution of the superimposed region, and storing the viewpoint / viewing of the stored original image. A step of selecting a plurality of original images according to the display viewpoint / viewing direction and display resolution based on the direction and resolution data; a step of determining a combination ratio based on a difference between superimposed regions and a difference in brightness distribution; A method of adjusting the brightness of each image so as to synthesize and display the images so that the brightness values of all the image information fall uniformly within the display brightness. 4. A computer-readable recording medium in which a program for causing a computer to execute a process of synthesizing and displaying an image from an original image is recorded. Storing, from the original image, obtaining a correlation between images by suppressing the effects of brightness difference, occlusion, zoom and resolution; and measuring a brightness distribution for a region to be superimposed between the images, Generating a panoramic image by performing coordinate conversion of the image based on the obtained correlation between the images and partially adjusting the brightness of the superimposed region so that information of the image having a wider brightness distribution is prioritized; And a step of cutting out and displaying a display image from the panoramic image in accordance with a display viewpoint and a viewing direction, A computer-readable recording medium a program to be executed.