JPH09128519A - Image compositing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To composite a high-quality image while shortening the processing time without extending the configuration even in the case in which information required for composition is not contained in overlapped area or rotation error is generated, by enabling local extension/reduction in various directions to partial images to be composited at need. SOLUTION: At the time when a partial image 6 is composited with partial images 1-5 which have been composited, for example. First of all, matching due to the partial images 5 and 6 is performed and a coordinate A (x1 , y1 ) at the upper left corner of the partial image 6 is provided. Similarly, matching by the partial images 1 and 6 is performed and a coordinates A" (x1 ', y1 ') at the upper left side of the partial image 6 is provided. As a result, when the partial image 6 is matched with the partial images 5 and 1, the difference of (δx , δy )=A"-A is generated. When δy is positive the picture elements in the overlapped image between the partial images 5 and 6 are reduced by y1 '-y1 in (y) direction and the other partial image 6 is translated just for these picture elements. Further, similar deformation is performed in (x) direction as well.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a still image pickup device, wherein an image of a subject on an image forming surface is picked up by an image pickup device.
The present invention relates to an image synthesizing apparatus that obtains a high-definition image by capturing as a divided image having an overlapping region, checking the matching state of the overlapping region, and synthesizing the images.

[0002]

2. Description of the Related Art Since a high-resolution image pickup device is expensive, conventionally, in order to obtain a high-definition image at a low cost, an image obtained from a subject is divided into a plurality of images, and the images are taken and then combined. Therefore, a method of improving the resolution in one image is considered. For example, in the "imaging device" of Japanese Patent Laid-Open No. 6-6686, an optical image transmitted through a lens is divided by a reflector having a plurality of reflecting surfaces, and an image captured by an imaging unit provided for each divided optical image. A method of obtaining a high-resolution image exceeding the resolution of the image sensor by synthesizing

Further, in the "still image pickup device" of Japanese Patent Application Laid-Open No. 7-107379, in order to input a still image having a large capacity, the image pickup element is moved and photographed while providing an overlapping area on the image forming plane. Also disclosed is a method for obtaining a high-resolution image that exceeds the resolution of the image sensor by performing movement, extracting movement information from the obtained overlapping region of the images, and combining the images.

[0004]

However, in the "imaging device" of Japanese Unexamined Patent Publication No. 6-6686, in order to perform the above-mentioned image composition processing, the accuracy error of the imaging mechanism or the assembly at the time of manufacture is required. Since the accuracy, particularly the assembling accuracy of optical system devices such as lenses and reflectors, has a great influence, it is necessary to improve the above accuracy in order to reduce the displacement and distortion of the combined image. However, there is a problem that the cost from the design to the manufacturing is high.

On the other hand, the matching method used in the "still image pickup device" of Japanese Unexamined Patent Publication No. 7-107379 to synthesize images has a uniform density difference in the overlapping area and an image having periodicity. When the pattern is formed, there is a problem that the combination position is not uniquely determined. In the above publication, the relative position between the images to be combined is obtained by detecting the movement amount of the image sensor. However, in such a synthesizing process, there is a large error with respect to the value obtained from the matching, so it is necessary to improve the detection accuracy of the motion amount in order to eliminate the image shift. As expected, it will be accompanied by cost increase.

The present invention has been made in order to solve the above-mentioned problems, and an object of the present invention is to provide a structure even when the information necessary for synthesis is not included in the overlapping area or a rotation error occurs. Without increasing the number and reducing the processing time,
An object of the present invention is to provide an image synthesizing device capable of synthesizing high quality images.

[0007]

In order to solve the above-mentioned problems, an image synthesizing device according to the invention of claim 1 shoots a subject and moves the shooting means on a plane.
A partial image storage unit that stores the captured partial images and a composite image storage unit that stores the combined images of the partial images while capturing the partial images by the dimension moving unit so as to include an overlapping region between the adjacent partial images Means and a combination information calculation means for obtaining a position where the partial images are combined based on the overlapping area are provided, and the partial image is drawn in the combined image storage means according to the combination position obtained by the combination information calculation means. An image synthesizing device for synthesizing images is characterized by being provided with image deforming means for locally enlarging and contracting partial images in different directions. As described above, the partial images to be combined can be locally enlarged / reduced in different directions as needed, so that partial images that are adjacent in different directions can be individually processed. The image shift can be eliminated.

In order to solve the above-mentioned problems, an image synthesizing apparatus according to a second aspect of the present invention adjoins a partial image by a photographing means for photographing a subject and a two-dimensional moving means for moving the photographing means on a plane. A partial image is captured such that an overlapping region is included between the partial images, a partial image storage unit that stores the captured partial image, a composite image storage unit that stores an image that combines the partial images, and a partial image based on the overlapping region. In an image synthesizing device for synthesizing an image, which is provided with a synthesizing information calculating unit for obtaining a position where images are synthesized, and in which a partial image is drawn in the synthesizing image storing unit according to the synthesizing position obtained by the synthesizing information calculating unit, A photographing position detecting means for detecting a position on the plane of the photographing means and a synthetic information calculation assisting means for obtaining an image area in which information to be synthesized by using the synthetic information calculating means exist are provided. It is characterized in that. With the above configuration, when the information to be combined is present in the overlapping area, the images are matched, and when the information to be combined is not present, the photographing position obtained by the photographing position detecting means is obtained. It is possible to combine the partial images based on the above, and it is possible to improve the speed of the combining process while suppressing the distortion of the images at the time of combining.

In order to solve the above-mentioned problems, an image synthesizing apparatus according to a third aspect of the present invention adjoins a partial image by a photographing means for photographing a subject and a two-dimensional moving means for moving the photographing means on a plane. A partial image is captured such that an overlapping region is included between the partial images, a partial image storage unit that stores the captured partial image, a composite image storage unit that stores an image that combines the partial images, and a partial image based on the overlapping region. In an image synthesizing device for synthesizing an image, which is provided with a synthesizing information calculating unit for obtaining a position where images are synthesized, and in which a partial image is drawn in the synthesizing image storing unit according to the synthesizing position obtained by the synthesizing information calculating unit, A plurality of the partial image storage means are provided, and a partial image memory control means for performing the image transfer processing to the composite image storage means and the processing of the composite information calculation means in parallel is provided. It is characterized by being. With the above configuration, since the image can be transferred to the composite image storage unit and the composite information can be calculated in parallel with the image capture, the processing speed can be improved.

[0010]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of an image synthesizing apparatus according to the present invention will be described below with reference to FIGS. The image synthesizing apparatus described here, for example, as shown in FIG.
When the capturing range of the image sensor is narrow with respect to the image to be captured, an overlapping region (hereinafter referred to as an overlapping image) is provided between adjacent partial images, and the partial images 1 to 6 are captured. After that, the images having the original size are combined by performing matching with the overlapping images.

As shown in FIG. 2, the image synthesizing device includes an image capturing section 11 using an image pickup device, and a partial image storage section 1 for capturing image data from the image capturing section 11.
2, a combination information calculation unit 13 that obtains information for combining the obtained partial images, a combination information buffer 14 that stores the combination information obtained by the combination information calculation unit 13, and image transformation based on the combination information. An image transforming unit 15 for performing the above, a combined image storage unit 16 for storing a combined image formed by the data of the partial image and the corrected overlapping region image, and an overlapping image storage unit for storing the overlapping image for matching. 17. An address table 18 for performing coordinate conversion using a conversion table for image transformation is provided.

Based on the above configuration, the flow of information and the processing procedure will be described. First, the partial image is the image capturing unit 11
And is transferred to the partial image storage unit 12. Then, among the partial images stored in the partial image storage unit 12, the overlapping image required for the calculation of the composite information is transferred to the overlapping image storage unit 17. The composite information calculation unit 13 performs a matching operation based on the data of the overlapping image, and the composite information obtained by this matching operation, that is, the relative coordinate values between the partial images and the read obtained from these coordinate values. The memory address on the write side and the memory address on the write side are stored in the composite information buffer 14. On the basis of the above-mentioned composition information stored in the composition information buffer 14, the corresponding part of the address table 18 is presented, and the image transformation unit 15 can refer to it. The image transforming unit 15 refers to the composition information and the conversion table of the address table 18 to obtain a read address for the partial image storage unit 12 and a write address for the composite image storage unit 16 and transfers the data. However, in the above, when the partial image to be captured is the first one, the compositing process cannot be performed. Therefore, after the image is captured and stored in the partial image storage unit 12, it is immediately stored in the composite image storage unit 16. Will be transferred.

For example, when an original image is divided and partial images are taken in to form a composite image as shown in FIG. 3, the partial images 1 to 6 are picked up in order from the partial image 1 to the partial image 6. However, at that time, the overlapping images generated between the adjacent partial images are detected, and the partial images are sequentially stored in the combined image based on the coordinate values obtained as a result of performing the matching between the partial images. It is transferred to the section 16.

However, in FIG. 3, when there is no information necessary for direct composition such as the overlapping image between the partial image 1 and the partial image 2, the matching process is used to perform the partial image 1 and the partial image 2. When trying to perform composition so that there is no shift in information, the positions of the partial image 6 with respect to the partial image 1 are determined, and then the position of the partial image 5 with respect to the partial image 6 is determined. It is necessary to perform synthesis by the procedure of determining the position. In the above case,
Since the partial image 6 is captured last, the image combining process cannot be started until all the partial images have been captured. During that time, the partial images 1 to 6 must be stored. If the storage area required for the partial image storage unit 12 is the maximum, the number of divided images will be required.

Therefore, as a result of performing the matching process, when the information necessary for direct composition between the partial images (for example, characters straddling between the partial images) does not exist, the image capturing unit 11 The combining process is performed based on the position detected from the movement amount. In the overlapping area between the partial image 1 and the partial image 2 described above, there is no area that causes a problem when a shift occurs,
As a result of combining based on the position detected from the movement amount, even if a few dots are displaced from the original image, the image quality at the connection portion between the partial images is hardly affected. In this way, the images are combined in the order in which the partial images are captured.

At this time, when there are partial images adjacent to each other in the xy directions, such as partial images 5 and 6, etc., each partial image (for partial image 6, for example, The composite information is obtained from the partial images 1 and 5), but as described above, between the partial images 1 and 2,
An error may occur as a result of performing the combining process based on the detection of the movement amount of the image sensor, and the combining information obtained from a plurality of adjacent partial images does not always match.

Therefore, by partially enlarging / reducing the partial images based on the respective combined information, the displacement of the images in the overlapping area between the partial images can be eliminated. If the connection between the partial images is natural even if it is partially displaced from the original position by a few dots, it is possible to improve the sense of quality of the overall composite image. For example, as shown in FIG. 4, image composition of partial images 1 to 5 is completed, and
Consider a case where the partial images 6 are combined. Here,
The partial images 2 to 4 are omitted.

In the above, if the obtained partial image 6 is simply combined with the partial image 5 to perform image synthesis, as shown in FIG. 5, the partial image 1 and the partial image 6 are straddled. Consider a case in which the drawn character string "ABC" is displaced in the y direction at the position of "B", and the positions in the x direction do not match. At this time, first, matching is performed between the partial images 5 and 6 as shown in FIG. 5, and the coordinates A (x ₁ , y ₁ ) of the upper left corner of the partial image 6 at this time are obtained. Other coordinates are B (x ₂ , y ₁ ), C (x ₁ , y ₂ ), D
Let it be (x ₂ , y ₂ ). Similarly, the partial image 1 and the partial image 6 are matched, and the partial image 6 at this time is matched.
Suppose you have the upper left coordinate A "(x ₁ ', y ₁ ') (Fig. 1).
These pieces of information are once written in the composite information buffer 14. As a result, when the partial image 6 is matched with both the partial image 5 and the partial image 1, the coordinates of the upper left corner are (δx, δy) = A "-A = (x ₁ '-x ₁ , y ₁ '-y ₁ ).

Therefore, when δy is positive, the image transformation unit 15 reduces the overlapping image of the partial image 5 and the partial image 6 by y ₁ '-y ₁ pixels in the y direction, except for the overlapping image in the partial image 6. If the portion of is translated by the above reduction amount, that is, by y ₁ '-y ₁ pixels in parallel, the coordinates A (x ₁ , y ₁ ) and B (x ₂ , y ₁ ) are A'
It moves to (x ₁ ', y ₁ ) and B' (x ₂ , y ₁ ') and is transformed into a partial image 6' (FIG. 6). On the other hand, if δy is negative, the duplicated image is y ₁ '-
You can enlarge y ₁ pixels and similarly translate by y ₁ '-y ₁ pixels. This eliminates the vertical shift in the image. Further, the deformation in the x direction is performed, but in this case, the same consideration can be applied. When δx is positive as shown in FIG. 6, the overlapping images of the partial images 1 and 6 are reduced by x ₁ '-x ₁ pixels in the x direction, and the portion other than the overlapping images in the partial image 6 is reduced, that is, x. ₁ 'when -x ₁ pixel by translating the coordinates A'
(x ₁ ', y ₁ ), C (x ₁ , y ₂ ) are A "(x ₁ ', y ₁ '), C' (x ₁ ', y
₂ ), and it is transformed into a partial image 6 ″ (Fig. 1). Conversely, if δx is negative, the overlapping image is enlarged by x ₁ '-x ₁ pixel, and the part other than the overlapping image is x ₁ '. -x ₁ pixel only needs to be translated, which eliminates left-right misalignment in the image. As described above, when the partial image 5 and the partial image 6 are matched first, the coordinates are adjusted. The value of D (x ₂ , y ₂ ) does not change.

The above-mentioned partial enlargement or reduction processing is performed by referring to the address table 18 when the image transformation unit 15 generates the read address and the write address. For example, as shown in FIG. 7, when δx and δy are given as offset values and each address of the overlapping area is given as an input address, the address table 18 transforms the output address according to the offset value into an image. It is output to the unit 15. In the example of FIG.
When -8 is given as the offset value and 0,1,2, ..., 49,50,51 is given as the input address, 0,1,2, ..., 45,45, You get 46. That is, the generation condition of the output address with respect to the input address can be changed according to the change of the offset value.

At this time, if the data is read a plurality of times from each address in a certain address range, local expansion is performed in the above address range, and conversely, data in a certain address range is thinned out. If it is output, that part will be locally reduced. The address table 18 stores the above correspondence in association with the offset value. This can be easily realized even in the case of partial rotation if the same method is considered.

Further, as shown in FIG. 8, a composite information calculation assisting section 19 for selecting an area in which the matching processing is required in the composite information calculating section 13 may be added to the image forming apparatus. Since the calculation of the combined information needs to be performed on a pixel-by-pixel basis, the processing takes a long time. Therefore, if the matching process can be omitted so that it can be omitted if the matching process is unnecessary, and if the matching process is determined not to be necessary, the time is saved. It can be shortened.

However, when the matching process is required in the overlapping area of all the partial images, the processing time is shortened by simply performing the matching process. Therefore, in parallel with the image synthesizing process, if the matching process is necessary or not, the image synthesizing process is stopped at the stage when the matching process is determined to be unnecessary, and the synthesizing position obtained from the moving means is determined. Partial image composition processing is performed on
It is possible to continue the next image composition process,
The calculation load on the composition information calculation unit 13 can be reduced, and the processing speed of image composition can be improved. The composite information calculation assisting unit 19 determines whether or not the matching process is necessary, for example, by obtaining the degree of change in the brightness value in each pixel. In this case, if the degree of change in the brightness value is smaller than a certain threshold value, it is determined that the image is flat and no matching process is performed.

When it is determined that the matching processing is not suitable for all the overlapping areas, the combined position is detected from the moving means, and therefore the result learned in advance as the characteristic of the moving means is written in the combined information buffer 14. Need to leave. The characteristics of the above-mentioned moving means include, for example, a value obtained by inferring from a numerical value detected from the moving amount of the image deforming unit 15 or a numerical value such as a relative position between already-combined partial images.

Further, as shown in FIG. 9, a memory control unit 20 including a comparator, an adder, a counter, a decoder and an encoder and a plurality of partial image storage units 12 and 12 are provided in the image synthesizing apparatus of FIG. May be. Further, a data selector 21 is further arranged in the partial image storage section 12 for memory control.

The memory control unit 20 is based on the address data of the partial image provided from the image capturing unit 11 in order to transfer the image from the image capturing unit 11 to the partial image storage unit 12 and the overlapping image storage unit 17. The duplicate image storage unit 17
To generate the address and data of the overlapping image to be output to, the address of the partial image to be output to the partial image storage unit 12, and the enable signal to the data selector 21. Further, when the data to be sent to the composite image storage unit 16 is complete, a control start signal is sent to the composite information calculation unit 13.

With the above configuration, if the partial image storage unit 12 has a storage capacity of at least two partial images, the image acquisition unit 11 captures the image and the composite image storage unit 16 stores the image. Since writing and writing can be performed at the same time, by performing parallel processing as shown in FIG. 10, image combining processing can be performed at higher speed.

In the composition of FIG. 1 and FIGS. 4 to 6, the image is exaggeratedly expressed, but since the actual deviation is several dots, the distortion of the image does not occur even when enlarging / reducing the level of several dots. It can be said that the quality of an image is improved rather than leaving a discrepancy in the image that is not conspicuous and is easily recognized as a step or the like.

As a result, a high-resolution and high-quality image exceeding the resolution of the image pickup device can be obtained, so that a high-quality image can be obtained with an inexpensive structure.

[0030]

The image synthesizing apparatus according to the first aspect of the invention is
As described above, the image capturing unit that captures the subject and the two-dimensional moving unit that moves the image capturing unit on the plane capture the partial images so that the overlapping regions are included between the adjacent partial images, and the captured regions are captured. A partial image storage unit for storing an image, a combined image storage unit for storing an image obtained by combining the partial images, and a combination information calculation unit for obtaining a position where the partial images are combined based on the overlapping area are provided. An image synthesizing device that draws a partial image in the synthetic image storing means and performs image synthesizing according to the synthesizing position obtained by the calculating means is provided with an image transforming means for locally enlarging / shrinking the partial image in different directions It is a configured structure. Therefore, when trying to combine partial images that are adjacent to each other in different directions, by performing local enlargement / reduction even if the partial images that are adjacent to one side are misaligned. Since it is possible to eliminate the shift of the image in each direction, it is possible to obtain a high quality image.

In the image synthesizing apparatus according to the second aspect of the present invention, as described above, the overlapping area between the adjacent partial images by the photographing means for photographing the subject and the two-dimensional moving means for moving the photographing means on the plane. While capturing a partial image so as to include the partial image, a partial image storage unit that stores the captured partial image, a composite image storage unit that stores an image obtained by combining the partial images, and the partial images are combined based on the overlapping area. And a combination information calculating unit for obtaining a position, and a plane of a photographing unit in an image combining apparatus that draws a partial image in the combined image storage unit according to the combining position obtained by the combining information calculation unit and performs image combination. With a configuration provided with a photographing position detecting means for detecting the upper position and a combined information calculation assisting means for obtaining an image area in which information to be combined using the combined information calculating means exists. That. Therefore, in the overlapping region between the partial images, the image matching is performed in the overlapping region in which the information to be combined exists, and the overlapping position in which the information to be combined does not exist is obtained by the photographing position detecting means. It is possible to combine the partial images based on the obtained shooting position, and it is possible to improve the speed of the combining process while suppressing the distortion of the images at the time of combining.

In the image synthesizing apparatus according to the third aspect of the present invention, as described above, the overlapping area between the adjacent partial images by the photographing means for photographing the subject and the two-dimensional moving means for moving the photographing means on the plane. While capturing a partial image so as to include the partial image, a partial image storage unit that stores the captured partial image, a composite image storage unit that stores an image obtained by combining the partial images, and the partial images are combined based on the overlapping area. And a composite information calculating unit for obtaining a position to be obtained, and the partial image is drawn in the composite image storing unit according to the composite position obtained by the composite information calculating unit to perform image composition. A plurality of storage means are provided, and a partial image memory control means for causing the image transfer processing to the composite image storage means and the processing of the composite information calculation means to be performed in parallel is provided. It is a configuration. Therefore, since the synthetic information can be calculated in parallel with the image capturing, the processing speed can be improved.

[Brief description of the drawings]

FIG. 1 is a schematic diagram showing a result of image composition performed by an image composition apparatus according to the present invention.

FIG. 2 is a block diagram showing a configuration example of an image synthesizing apparatus according to the present invention.

FIG. 3 is a schematic diagram of a combined image obtained by combining images captured by dividing.

FIG. 4 is a diagram for explaining a process of synthesizing partial images when images are present in two directions of a region to be synthesized,
It is a schematic diagram of a synthetic image showing a state of each partial image before synthesis.

FIG. 5 is a diagram following FIG. 4, and is a schematic diagram of a combined image showing a result of performing matching processing on one partial image.

FIG. 6 is a diagram subsequent to FIG. 5, and is a schematic diagram of a composite image showing a result of partial reduction in the y direction.

FIG. 7 is a schematic diagram showing a relationship between an input address and an output address in an address table.

FIG. 8 is a block diagram showing a configuration example in the case where a synthesis information calculation assisting unit is provided in the image synthesis apparatus shown in FIG.

9 is a block diagram showing a configuration example when a memory control unit is provided in the image synthesizing apparatus shown in FIG.

10 is a time chart showing a memory control process of the image synthesizing apparatus shown in FIG.

[Explanation of symbols]

 Reference Signs List 11 image capturing unit 12 partial image storage unit 13 composite information calculation unit 14 composite information buffer 15 image transformation unit 16 composite image storage unit 17 overlapping image storage unit 18 address table 19 composite information calculation auxiliary unit 20 memory control unit 21 data selector ( Switching means)

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Shoji Hirosawa 22-22 Nagaike-cho, Abeno-ku, Osaka-shi, Osaka Inside Sharp Corporation (72) Inventor Kazuyuki Nago 22-22 Nagaike-cho, Abeno-ku, Osaka-shi, Osaka Within Sharp Corporation

Claims (3)

[Claims] 1. A partial image is photographed by a photographing means for photographing a subject and a two-dimensional moving means for moving the photographing means on a plane so as to include an overlapping area between adjacent partial images, and the photographed portion is taken. A partial image storage unit for storing an image, a combined image storage unit for storing an image obtained by combining the partial images, and a combination information calculation unit for obtaining a position where the partial images are combined based on the overlapping area are provided. An image synthesizing device that draws a partial image in the synthetic image storing means according to the synthetic position obtained by the calculating means to perform image synthesizing is provided with an image deforming means for locally enlarging / contracting the partial image in different directions. An image synthesizing device characterized by being provided. 2. A partial image is photographed by a photographing means for photographing a subject and a two-dimensional moving means for moving the photographing means on a plane so as to include an overlapping area between adjacent partial images, and the photographed portion. A partial image storage unit for storing an image, a combined image storage unit for storing an image obtained by combining the partial images, and a combination information calculation unit for obtaining a position where the partial images are combined based on the overlapping area are provided. In an image synthesizing device that draws a partial image in the synthetic image storage means according to the synthetic position obtained by the calculating means and synthesizes images, a photographing position detecting means for detecting the position on the plane of the photographing means, and a synthetic information calculation An image synthesizing apparatus, comprising: a synthetic information calculation assisting means for obtaining an image area in which information to be synthesized using the means exists. 3. A partial image is photographed by a photographing means for photographing a subject and a two-dimensional moving means for moving the photographing means on a plane so as to include an overlapping region between adjacent partial images, and the photographed portion. A partial image storage unit for storing an image, a combined image storage unit for storing an image obtained by combining the partial images, and a combination information calculation unit for obtaining a position where the partial images are combined based on the overlapping area are provided. In an image synthesizing device that draws partial images in the synthetic image storage unit and synthesizes images in accordance with the synthetic position obtained by the calculation unit, switching between input and output to the plurality of partial image storage units and each partial image storage unit The switching means is provided, and the partial image memory control means for performing the image transfer processing to the composite image storage means and the processing of the composite information calculation means in parallel is provided. An image synthesizing device characterized by the above.