JP3333012B2 - Image synthesis device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing apparatus for suitably synthesizing a plurality of digital images (binary images) having mutually overlapping areas.

[0002]

2. Description of the Related Art Conventionally, when two or more digital images (binary images) and the like are combined, an overlap region, that is, an overlap region, is formed at the end of each image by combining. In this overlap region, a difference occurs between the position of the pixel on the image side to be combined and the position of the pixel on the image side to be combined. Has been synthesized.

For example, Japanese Patent Application Laid-Open No. 63-64180 discloses that when a large image is input as an image using a hand scanner, the range that can be photographed by the hand scanner is limited to a certain extent. A description is given of a synthesizing method in which, when the image is divided into blocks, captured and input, and reproduced, an image image is superimposed and synthesized for each block (matching) to reproduce a synthesized image or a large image.

Japanese Patent Application Laid-Open No. Hei 3-182976 discloses a method of converting images to be synthesized so that the densities of the images to be synthesized become equal.
A synthesizing method is described in which a binarization process is performed, a specific particle-shaped center of gravity is obtained for each image, and matching is performed.

[0005]

In each of the above publications, a block (reference block) is used as a reference for a certain range of area (area) in the overlap area when combining.
, And based on this, various corrections are performed to synthesize a plurality of images.

[0006] That is, the setting of the reference block includes a method in which the reference block is fixed at a certain position in the image, and a method in which the operator manually specifies an arbitrary position in the image. The mask pattern described in JP-A-63-64180 is set at a fixed position in advance,
In the method described in JP-A-3-182976,
By limiting the number of pixels in a portion represented in a particle shape on the screen to a specific range, two reference points are determined at arbitrary positions.

When the image synthesizing apparatus is mounted on a camera or the like, and is used for photographing a blackboard, a posted material, an OHP (overhead projector), a book, or the like during a meeting, depending on the contents, the photographing is immediately performed. Sometimes you need to play the image. For that purpose, high-speed processing is required, and detection of a reference block and extraction of a corresponding point at the time of synthesis need to be automatically performed on the spot after shooting.

Furthermore, since the data of the actually photographed image is various, when the reference block is set at a fixed position, the average density in the reference block is extremely high (for a binary image, There is a possibility that the number of white pixels is extremely larger than the number of black pixels or true white) and the average density may be extremely low. If corresponding points are extracted based on such a reference block, correct matching can be obtained. Will not be.

As described in JP-A-63-64180, extraction of corresponding points in an overlap region between images to be combined is arranged in a reference block of a predetermined size. Of course, the larger the reference block is, the larger the deviation of the image can be dealt with, but the processing always takes time.

Therefore, the fixed-size reference block always has a range in which a proper corresponding point can be extracted, and is not always an efficient size. Also, JP-A-3-1829
The system described in Japanese Patent Publication No. 76 can be realized for a crystal photograph for inspection of a specific metal structure with a characteristic particle, and is configured to be used for general-purpose use which is the gist of the present invention. Since it is not performed, and it is not necessary to reproduce the image immediately after shooting, no consideration is given to speeding up and shortening of the processing time, and high speed processing cannot be realized by this method.

Accordingly, the present invention provides an image processing method in which an interpolated image is obtained from a reference block having an optimum average density based on a search area of an arbitrary size capable of varying the amount of displacement between images to be synthesized and a search line, and an image synthesized at a high speed. It is an object to provide a synthesis device.

[0012]

According to the present invention, in order to achieve the above object, a first search area is set in an overlapping area when image data is superimposed with a predetermined width. A second reference block having a predetermined size having a correlation based on the density of the image data in the search area, and extending along the longitudinal direction of the overlap area from the first reference block Reference block setting means for setting a second reference block at a position having the correlation in an area, and a position of a pixel to be interpolated from a predetermined pixel in each reference block set by the reference block setting means upon synthesis Displacement amount detecting means for detecting a displacement amount to the image data; interpolating means for interpolating an image to be synthesized based on the displacement amount detected by the displacement amount detecting means; Providing an image synthesizing apparatus constituted by an image synthesizing means for synthesizing the image data interpolated by said interpolating means.

[0013]

According to the image synthesizing apparatus having the above structure, a search area (search line) having a size selected on the basis of a predetermined size is set in an area where a plurality of images to be synthesized overlap each other. A reference block is set at a lower position, and a reference block is set at a position more suitable in the search area (search line) according to the result of the correlation determination. Corresponding point detection processing is performed based on the reference block. Are interpolated by the displacement amount calculated from the images to synthesize the images.

[0014]

Embodiments of the present invention will be described below in detail with reference to the drawings. First, with reference to FIG. 1, the outline of image synthesis of the image synthesis apparatus according to the present invention will be described.

This image synthesizing apparatus determines a search area at a position appropriate for setting a reference block, extracts a reference block having a correlation, and performs a corresponding point detection process based on the extracted reference block. The processing of interpolating and synthesizing an image from the calculated displacement amount is quickly performed without user operation. In other words, this image synthesizing device is used in an overlapping area of the images to be synthesized.
In order to calculate the amount of displacement between images, a movable search area is set, and the position of the reference block is automatically set based on the determination result of the correlation. Here, the reference size of the reference block and the overlap area is arbitrarily set and stored in advance by the user according to the size of the image and the degree of overlap. In this embodiment, the reference block is about 17 × 17 pixels, and the overlap area is 15 to 20% of the area of the image (638 × 500 pixels).
(100 to 130 pixels in the horizontal direction).

FIG. 1 shows an example of combining two images.
The left image 1 is used as an image to be synthesized (reference image), and the right image 2
Is the image to be synthesized. The overlapping area of each image is referred to as an overlap area 3.

A search area 4 of an arbitrary size is provided in the overlap area 3 of the reference image 1, and a correlation block described later is determined in this area, and a reference block 5 is set at an appropriate position. Then, the set reference block 5
The extension line (search line) 7 is extended from the center point toward the lower part 3b, and it is determined whether a reference block 6 having a correlation can be set on the search line 7. The reference block 6 having a correlation is detected by searching only on the search line 7, the calculation amount for the search is reduced, and the coordinate conversion in the subsequent synthesis processing is simplified.

Since the area where the reference block is provided on the search area 4 and the search line 7 is represented by binary image data, the average density of each block is checked by the following process for each pixel, and A reference block is set at a location that satisfies the expression (1).

V ₀ = Min | C ₀ −C / 2 | or V ₁ = Min | C ₁ −C / 2 | (1) where C ₀ ; the number of pixels of 0 in the block; C ₁ ; in the block 1, the number of pixels, C; the total number of pixels in the block (C ₀ + C ₁ ).

On the other hand, the average density of the search area 4 of the overlap area 3 and the area where the reference block is provided on the search line 7 differs depending on the photographed image.
May be extremely high or extremely low. In this case, the probability of incomplete synthesis (mismatching) increases. In order to solve this problem, the average density of the reference block set is evaluated by the following equation (2) (hereinafter, referred to as correlation judgment).

| V ₀ -C / 2 | <p · C / 2 or | V ₁ -C / 2 | <p · C / 2 (2) where 0 <p <20%. If the condition of equation (2) is satisfied, it can be considered that there is no correlation. This range of p is a numerical value that is considered to be suitable based on the results of a test.

FIG. 2 shows a configuration example in which the above-described image synthesizing apparatus is mounted on an electronic camera or the like which outputs a binary image as the first embodiment. This configuration includes an imaging unit 11, an image synthesizing device 12, a printing unit 13 serving as an output device, and a display unit 14. This imaging unit 11 is disclosed in Japanese Patent Application No. 5-63.
No. 978, a configuration in which a mirror provided on the front face is rotated to obtain images in different directions,
A configuration in which a V camera is rotated to obtain a continuous image signal is used.

The image signal picked up by the image pick-up unit 11 is digitized by the A / D unit 15 of the image synthesizing unit 12 and further binarized by the binarizing circuit 16. This binarization processing is performed, for example, in Japanese Patent Application No. 5-9455 proposed by the present applicant.
Using a fast binarization method as described in No. 4,
Perform value processing. Here, the binarization process is performed because the image to be output last is a binary image, so that the binarization process needs to be performed sometime, and the reference block and the corresponding point for the binary image to be performed later will be described. The binarization processing is applied to the image immediately after the imaging, because the detection processing can be speeded up.

The image data output from the binarization circuit 16 is temporarily stored in a frame memory A17.
Then, image data of a predetermined frame is read out from the frame memory A17, and a synthesizing process described later is performed by the image synthesizing circuit 18.
19 is stored.

The composite image stored in the frame memory B19 is read and printed by the printing unit 13 or converted into an analog signal by the D / A unit 20 and displayed on the display unit 14 such as a cathode ray tube.

Next, FIG. 3 shows a specific configuration example of the image synthesizing circuit shown in FIG. 2 and will be described. This image synthesizing circuit includes a reference block setting circuit 21 for setting a reference block from image data read from the frame memory A17.
A displacement detection circuit 22 for detecting a displacement of a corresponding point from the set reference block and the image data; and an interpolation circuit 2 for interpolating an image to be synthesized based on the detected displacement.
And a combining circuit 2 for combining the combined image read from the frame memory A17 with the interpolated combined image.
And 4.

The displacement detecting circuit 22, the interpolating circuit 23, and the synthesizing circuit 24 are the same as those disclosed in Japanese Patent Application No. Hei.
It is the same as that described in No. 42402. FIG. 4 shows a specific configuration example of the reference block setting circuit shown in FIG. 3 and will be described.

The reference block setting circuit includes an upper search area setting circuit 31 which receives image data from a frame memory A17 (not shown) and sets a search area 4 in the upper part 3a shown in FIG.
1 is a memory 32 for storing numerical values of a block size and a size of an overlap area, which are used as references when setting the search area 4, and an optimum average density calculation for calculating an optimum average value of image data densities in the search area. Similarly, a circuit 33, a lower search line setting circuit 34 for setting a search line (reference block 6) in the lower portion 3b, and an optimum average density calculation circuit 35 for calculating an optimum average value of image data densities in the search area. Be composed. The upper and lower reference blocks are output from the upper and lower optimum average density calculation circuits 33 and 35 to the displacement amount detection circuit 22 shown in FIG.

Next, FIG. 5 shows a specific configuration example of the optimum average density calculation circuit shown in FIG. 4 and will be described. This optimum average density calculation circuit counts the number of pixels of either "0" or "1" from a block reading circuit 36 for reading a block in a range corresponding to a reference block from image data of a set search area. A counter circuit 37;
1/2 block size (1/2 of the total number of pixels in a block)
A memory 38 for storing 2), a subtractor 39 for subtracting a half block size from the numerical value counted up to a predetermined number by the counter circuit 37, an arithmetic circuit 40 for obtaining an absolute value of the result, and a numerical value obtained. And a minimum value detection circuit 41 for detecting the minimum value from the above.

Here, the reason why the memory stores the 1/2 block size and subtracts it from the counted value (the number of pixels on the "0" or "1" side in the block) is shown in the above-mentioned equation (1). As described above, since the value of コ ン ト ラ ス ト is a value having a high contrast, the value is obtained by subtracting the value from the count number to obtain the minimum value.

In the image processing apparatus constructed as described above, the overlap area 3 as shown in FIG.
In some cases, the average density of the reference blocks on the search area 4 and the search line 7 may not be appropriate, that is, there may be no correlation, and it is necessary to move the search area or the search line and perform the search again.

FIG. 6 shows and describes a reference block setting circuit (2) as an image processing apparatus according to the second embodiment. Since each component of the reference block setting circuit performs the same operation as the component of the reference block setting circuit shown in FIG. 4, detailed description is omitted, and the flowchart shown in FIG. 7 and the overlap area shown in FIG. The configuration and operation will be described with reference to the arrangement relationship between a search area (search line) and a reference block.

In this image processing apparatus, it is determined whether or not there is a correlation between the reference blocks set in the search area. If there is no correlation, the search area or the search line is moved, a new reference block is set, and a new reference block is set. By performing a search, various image signal data can be handled.

First, the upper search area setting circuit 50
A search area 4 as shown in FIG. 8A is set based on the block size read from the memory 51 and the size of the overlap area. Then, the average density of the reference block 5 set in the search area 4 is calculated by the optimum average density calculation circuit 52, and the presence or absence of the correlation is determined by the correlation determination circuit 53 (steps S1 and S2).

If it is determined in step S2 that there is a correlation (YES), the lower search line setting circuit 54 extends the search line 7 to the lower part 3b of the overlap area 3, and the search line 7 The correlation of the reference block 6 set above is searched (step S
3), the presence / absence of the correlation is determined by the correlation determination circuit 56 (step S4).

If it is determined by the correlation determination circuit 56 that there is a correlation (YES), a reference block as shown in FIG. 1 is set, and the displacement amount detection circuit 22 shown in FIG. Displacement amounts R and S (R: rotation vector, S: displacement vector) are calculated (step S5), and an interpolation process is performed based on the displacement amounts R and S to synthesize an image (step S6). However, if it is determined that there is no correlation (NO), the lower search line resetting circuit 57
The extended search line 7a is extended from the lower part 3b toward the search area 4 as shown in FIG. 8A, and the correlation of the reference block 6 is searched again by the optimum average density calculation circuit 58 and the correlation judgment circuit 59. It is determined whether there is a correlation (steps S7, S8). If it is determined in step S8 that there is a correlation (YES), the process proceeds to step S5, where the displacement amounts R and S are calculated, and the images are combined. However, if it is determined that there is no correlation (NO),
The state becomes as shown in FIG.
2, the displacement amount S is calculated (step S9), and based on the displacement amount S, an image combining process of only translation is performed (step S10).

In the determination in step S2,
When it is determined that there is no correlation (NO), the correlation of the reference block 6 provided in the lower search area 8 set by the lower search area setting circuit 60 is searched. The presence or absence is determined (steps S11 and S12).

If it is determined in step S12 that there is a correlation (YES), as shown in FIG. 8C, the upper search line extends from the lower portion 3b of the overlap region 3 toward the upper portion 3a. The extension search line is extended by the resetting circuit 63, the correlation of the reference block 5 is searched again, and the presence or absence of the correlation is determined (steps S13 and S13).
14).

If it is determined in step S14 that there is no correlation (NO), the state becomes as shown in FIG. 8D, and the flow shifts to step S9 where an image of only parallel movement is obtained based on the displacement S. A combining process is performed. However, if it is determined that there is a correlation (YES), there moves to step S5, Displacement amount R, S is calculated, the image is synthesized. If it is determined in step S12 that there is no correlation (NO), the state is as shown in FIG. 8 (e), and the images are simply superimposed and combined (solid) (step S15).

As described above, in the present embodiment, a movable search area and a search line are provided for setting a reference block, and a place having an optimum average density is searched for. By evaluating the average density, high-speed automatic processing can be performed and various image signal data can be handled.

Next, in the first and second embodiments, the size of the overlap area is fixed, and the reference point (corresponding point) is extracted. Also, in Japanese Patent Application Laid-Open No. Hei 3-182976, the extraction of the reference point is performed by binarizing an already known image in the overlap area and then performing the extraction processing.

However, in the case where the size of the overlap area is fixed, it is possible to cope with the difference between the images to be synthesized if the difference is not large. If the difference is too large, as shown in FIG. Since there is no area for detecting a corresponding point, the occurrence of mismatching is further increased. In particular, in a hand-held imaging device, there is a possibility that the image to be synthesized is largely shifted due to camera shake.

Therefore, as a third embodiment according to the present invention, an image synthesizing apparatus which changes the size of the overlap area will be described. FIG. 10 shows the positional relationship between the overlap area, the reference block to be set, and the reference point.
2 shows a configuration example of an image synthesizing circuit in an image synthesizing apparatus that performs the setting process.

FIG. 11 shows a specific configuration of the image synthesizing circuit. It is assumed that the overall configuration of the image synthesizing device is the same as the configuration shown in FIG. The configuration of the image synthesizing circuit includes an overlap area detecting circuit 71 for detecting the size of the actual overlap area from the image data from the frame memory A, and a search line 7 from the detected overlap area and the upper reference block. A reference block setting circuit 72 for setting an upper lower reference block; a corresponding point detection circuit 74 for detecting a lower corresponding point 26b for setting a reference point 10 from the set reference block and the image data; A displacement coefficient calculation circuit 73 for calculating a displacement coefficient (displacement amount) from the region, the reference block, and the corresponding point;
It comprises a binary interpolation circuit 75 for performing binary interpolation of image data based on the calculated displacement amounts R and S, and a combining circuit 76 for combining the interpolated image data and the image data from the frame memory A.

In the third embodiment, as shown in FIG. 10, first, the actual overlap area is detected and the search area 4 is detected.
a is set to the right end line of the left image (corresponding to a search line). Further, the size of the upper reference block 5a is set to, for example, about 51 × 51 pixels, which is larger than that of the first embodiment. Then, along the right edge line of the left image,
Find a highly correlated location. The lower reference block 6 has the size of the reference block (17 ×
17 pixels).

The upper corresponding point 26a is detected in a search area 25a whose lateral direction is set relatively large. When the corresponding point is detected by the matching process, the position of the right end line of the block indicates the actual size of the overlap area. The corresponding point search area 25b at the bottom is the same as in the above-described embodiment.

In the third embodiment, since the upper reference block which is set to be large is searched from the search area provided on the right end line of the left image, even if there is a large shift between the images, it is possible to cope with the right image. Can be detected, and by knowing the actual size of the overlap area,
The search area for detecting the corresponding point at the lower part of the right image is properly set, and accurate and efficient processing can be performed.

In the above-described first to third embodiments, an image synthesizing apparatus according to a fourth embodiment will be described in which detection of a corresponding point of a right image with respect to a reference block (reference point) of a left image is further accelerated. This corresponding point is a corresponding point of the right image corresponding to the reference point of the overlap area of the left image.

The image synthesizing apparatus according to the fourth embodiment has a feature in a displacement detection circuit, and the other components are the same as those in the first embodiment, and a description thereof will be omitted. First, well-known template matching, structure matching, and the like are known as methods for realizing high-speed corresponding point detection.
In the present embodiment, corresponding points are detected by applying template matching using overlapping images as search areas.

In this template matching, as described in, for example, JP-A-63-64180, an area for matching is generally fixed in advance. When finding the appropriateness in detecting the corresponding point, the larger the area is set, the more accurate it is. On the other hand, the area to be matched is widened, so that the calculation amount increases and high-speed processing becomes difficult. Conversely, if the area is set small, there is a possibility of mismatching if the displacement between the images to be synthesized is large.

FIG. 12 shows and describes the configuration of the displacement amount detection circuit. This displacement amount detection circuit is roughly divided into a corresponding point detection circuit 81 and a displacement coefficient calculator 82. The corresponding point detection circuit 81 receives, for example, the reference data output from the reference block setting circuit 21 shown in FIG. 3 and the image data read from the frame memory A (not shown), and provides a small area search area. An in-small-area matching unit 83 that performs matching processing by using an extended-area matching unit 84 that performs matching processing again in an extended-area search area obtained by expanding the search area when the value is equal to or greater than a preset threshold value; In the case of (2), the reference block is expanded or a reference block expansion matching unit 85 that performs a matching process using two pairs of reference blocks arranged at positions that are not adjacent to each other.

Each of the matching units has the same configuration, and the configuration will be described with reference to FIG. The matching section includes a search area data reading circuit 86 for reading input image data from a search area (search area block) of a predetermined size based on the size of the reference block, and reading out reference data (reference block). NO to take exclusive OR described later with the searched search area data
An R circuit 87, an adder 88 for adding a point (point) that does not match with the output from the NOR circuit 87, and comparing the added values to detect a minimum value, and calculating a displacement coefficient calculator shown in FIG. And a minimum value detection circuit 89 that outputs the signal to the output circuit 82.

A matching process in the image synthesizing apparatus according to the fourth embodiment will be described with reference to FIGS. In this matching process, since the image to be synthesized is a binary image, the calculation is speeded up using an exclusive OR shown in the following equation.

[0054]

(Equation 1) Here, T (x) is a reference block, and S (x) is a search area block.

First, as shown in FIG. 14, a relatively small area is set as the search area 27, and the matching process within the small area 83 is performed. The center point of this search area is obtained by mapping from the position of the center point (reference point) of the reference block detected in the left image based on the size of the overlap area.

Next, if the result from the in-small area matching section 83 is larger than a preset threshold value (for example, 10% of the reference block pixel number), it is determined that no matching has been achieved, and FIG. As shown, after the search area 28 is further expanded by the in-extended region matching section 84, the matching process is performed again.

In this extended search area, if the value exceeds the preset threshold value and matching is not possible,
As shown in FIG. 16, a matching process is performed by enlarging a template block serving as a reference. In this case, by setting the evaluation threshold to be slightly wider than the above-described threshold for matching, for example, to 20%, it is possible to cope with a relatively large deviation between the images to be combined. Can be.

In the case where the value is equal to or larger than the threshold value even after performing such a matching process, as shown in FIG. 17, a reference block is set at a position not adjacent to each other in the search area.
The matching process may be performed using two pairs of reference blocks.

In this case, the corresponding point is determined by the difference between the two obtained rotation angles (one of the displacement coefficients). If the difference is large due to a shift or the like, the smaller absolute value is adopted. If the difference becomes smaller, the respective displacement coefficients are obtained and averaged.

Although two pairs of reference blocks are used, more precise matching processing may be performed using three or more pairs of reference blocks. Further, a large number of reference blocks may be set, and the result of each matching process may be analyzed and obtained.

Therefore, the image synthesizing apparatus according to the fourth embodiment enlarges the image corresponding to the image to be synthesized without fixing the size of the search area and the reference block in which the matching processing is performed in the corresponding point detection processing between the images. In addition, by performing the matching process in multiple stages, it is possible to obtain the same accuracy as that obtained by fixing a large area in advance and to perform high-speed corresponding point detection.

The means for sequentially expanding the search area for performing the matching process can be applied to the third embodiment. Next, the interpolation of a binary image by the image synthesizing apparatus according to the fifth embodiment of the present invention will be described.

As for the interpolation of an image performed at the time of image synthesis, a method such as a spline or a linear method is generally known. For example, Japanese Patent Application No. Hei 5-42402 proposed by the present applicant.
In the reference (1), linear interpolation is mainly used for an image pickup apparatus capable of obtaining a high-resolution image.

However, depending on the subject to be photographed and the imaging device, an image may be required immediately after photographing, and a high-quality binary image is required by fast processing. conventionally,
As shown in FIG. 18, in the linear interpolation, Japanese Patent Application No.
Even if the interpolation coefficients are discretized as in the third embodiment described in Japanese Patent No. 89090, at least four multiplications and three additions are required for each pixel. Furthermore, the data calculated by this interpolation operation is not binary data distinguished between “1” and “0” in many cases, and it is necessary to perform re-binarization processing, which further increases the processing time. .

In the fifth embodiment, an image synthesizing circuit of an image synthesizing apparatus for performing high-speed interpolation processing on a binary image will be described. In this embodiment, components other than the image composition circuit are the same as those in the first embodiment, and a description thereof will be omitted.

As shown in FIG. 19, this image synthesizing circuit includes a reference block setting circuit 91 for setting a reference block based on image data to be synthesized read from the frame memory A17, and a frame similar to the set reference block. A displacement amount detection circuit 92 for detecting a displacement amount from the image data to be combined read from the memory A17, a binary interpolation circuit 93 for performing a binary interpolation on the combined image data based on the detected displacement amounts R and S; And a synthesizing circuit 94 for synthesizing the binary-interpolated image data and the synthesized image data read from the frame memory A17.

FIG. 20 shows a specific configuration of the binary interpolation circuit. The binary interpolation circuit converts a coordinate of a specified pixel (original pixel) on the image to be synthesized based on the displacements R and S from the displacement detection circuit 92 shown in FIG. Neighbor distance calculating circuit 96 for calculating the distance between the coordinate-converted new pixel and the original pixel adjacent thereto
And a distance determining circuit 97 for determining which area the position of the new pixel depends on based on the calculated distance, and image data to be read out as interpolation of image data (original pixel value) to be synthesized based on the determined result. And a readout circuit 98.

The operation of such an image synthesizing circuit will be described with reference to FIGS. As shown in FIGS. 21A to 21F, image data to be combined (original image)
Is a value of “1” or “0”.
As shown in the figure, when there is a new pixel V ′ whose coordinate has been transformed in the area 100 surrounded by four pixels of binary “1”, the value of this pixel is determined to be “1”, and the opposite is true. The value of the pixel in the area 101 shown in FIG.

In other cases, as shown in FIGS. 7C to 7F, the value of the new pixel is determined by the value of the closest original pixel, evenly divided. Therefore, as shown in FIG. 22, the four original pixels Va and V near the new pixel
The area surrounded by b, Vc, and Vd is divided into four equal parts, and a new pixel V ′ depends on one of the equally divided areas,
An area surrounded by the value of the closest original pixel in the dependent area is divided into a maximum of four areas of “1” or “0” value. The value of the pixel to be interpolated is determined by the position in any of these areas.

The operation of pixel interpolation will be described with reference to the flowchart of FIG. First, the distances m and p between the values m and p [coordinates; (V'x, V'y)] of the pixel V 'to be interpolated and the neighboring original image are calculated (step S21).
These values m and p indicate the distance from the pixel Va as shown in FIG. However, 0 ≦ m ≦ 1, 0 ≦ p ≦
Let it be 1.

Next, it is determined whether or not the obtained value m is smaller than 1/2 (step S22). In this determination, the value m is 1/2.
If less than (YES), it is determined whether or not the value p is p <１ ／ (step S23). If it is determined in step S23 that the value p is less than （(YES), the pixel V ′ has the value of Va, but if the value p is 以上 or more (N
O), the pixel V 'has the value of Vc.

Then, the value m is determined in step S22.
Is greater than or equal to １ ／ (NO), it is determined whether the value p is p <同 様, as in step S23 (step S2).
6). If it is determined in step S26 that the value p is less than （(YES), the pixel V ′ becomes the value of Vb, but if the value p is 以上 or more (NO), the pixel V ′ becomes Vd
Value.

In the division of the area surrounded by the four pixels, if the pixel value is 3: 1, the area is also 3: 1 in FIGS. 21E and 21F described above. 24A, and considering the distance between pixels, FIG.
As shown in (b), there is a concept of dividing the area into 8: 1.

Therefore, in the fifth embodiment, with respect to the interpolation processing of a binary image, the pixel to be interpolated divides the pixel area surrounded by the four neighboring pixels into equal parts, and interpolates the value of the pixel to be interpolated to the nearest pixel value. This is suitable for high-speed processing. In addition, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications and applications are possible without departing from the spirit of the invention.

[0075]

As described above in detail, according to the present invention, a reference block having an optimum average density based on a search area of an arbitrary size in which the displacement between images to be combined is movable and variable, and a search line. It is possible to provide an image synthesizing apparatus that performs high-speed processing of an interpolated image obtained by setting and detecting corresponding points. Furthermore, according to the present invention, a plurality of
Suitable for areas where the images to be combined overlap each other
The reference block is set at the position where
The corresponding point is detected based on the
The image is interpolated based on the displacement calculated from the
Image synthesizing device that synthesizes images appropriately
Can be provided.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining an outline of image synthesis of an image synthesis device according to the present invention.

FIG. 2 is a diagram illustrating a configuration example in which an image combining device is mounted on an electronic camera or the like that outputs a binary image according to a first embodiment of the present invention.

FIG. 3 is a diagram illustrating a specific configuration example of the image synthesis circuit illustrated in FIG. 2;

FIG. 4 is a diagram showing a specific configuration example of a reference block setting circuit shown in FIG. 3;

FIG. 5 is a diagram showing a specific configuration example of an optimum average density calculation circuit shown in FIG. 4;

FIG. 6 is a diagram showing a configuration of a reference block setting circuit of an image synthesizing apparatus according to a second embodiment of the present invention.

FIG. 7 is a flowchart for explaining the operation of a reference block setting circuit of the image synthesizing apparatus according to the second embodiment.

FIG. 8 is a diagram illustrating a positional relationship between a search area and a reference block in an overlap area of an image to be combined by the image combining apparatus according to the second embodiment.

FIG. 9 is a diagram showing an arrangement relationship between a search area and a reference block when there is no area for detecting a corresponding point.

FIG. 10 is a diagram showing a positional relationship between an overlap area, a reference block to be set, and a corresponding point by an image synthesizing apparatus according to a third embodiment of the present invention.

FIG. 11 is a diagram illustrating a configuration example of an image combining circuit in an image combining device that performs a setting process according to a third embodiment.

FIG. 12 is a diagram illustrating a configuration example of a displacement amount detection circuit of an image synthesizing apparatus as a fourth embodiment according to the present invention.

13 is a diagram illustrating a specific configuration example of a matching unit in the displacement amount detection circuit illustrated in FIG. 12;

FIG. 14 is a diagram showing an arrangement relationship between a search area for setting a relatively small area as a search area and a reference block in the fourth embodiment.

FIG. 15 is a diagram showing an arrangement relationship between a search area for setting an extended area as a search area and a reference block in the fourth embodiment.

FIG. 16 is a diagram showing an arrangement relationship between a search area for enlarging and setting a reference template block and a reference block in the fourth embodiment.

FIG. 17 is a diagram showing an arrangement relationship between a search area and a reference block when matching processing is performed using a plurality of reference blocks in the fourth embodiment.

FIG. 18 is a diagram for explaining conventional linear interpolation.

FIG. 19 is a diagram showing an image synthesizing circuit of an image synthesizing device as a fifth embodiment according to the present invention.

20 is a diagram illustrating a specific configuration example of the binary interpolation circuit illustrated in FIG. 19;

FIG. 21 is a diagram illustrating pixel values of pixels to be interpolated located in an area divided into four pixels.

FIG. 22 is a diagram showing a divided state of a region surrounded by four pixels.

FIG. 23 is a flowchart for determining a pixel value of a pixel to be interpolated located in an area divided into four pixels.

FIG. 24 is an example of a division example of a region divided into four pixels.

[Description of Signs] 1 left image (reference image: image to be synthesized), 2 right image (image to be synthesized), 3 overlapping area, 3a upper part, 3b lower part, 4, 8, 25a, 25b ... search area, 5, 6 ... reference block, 7 ... extension line (search line), 9, 10 ... reference point, 11 ... imaging unit, 12 ... image synthesis unit, 13 ... printing unit, 14 ... display unit, 15 ... A / D part,
16 binarization circuit, 17 frame memory A, 18 image synthesis circuit, 19 frame memory B, 20 D / A
Part, 21: reference block setting circuit, 22: displacement amount detection circuit, 23: interpolation circuit, 24: synthesis circuit, 26a, 26b
... corresponding points, 27 ... small research area, 28 ... extended research area, 31 ... upper search area setting circuit, 32, 3
8: memory, 33: optimum average density calculation circuit, 34: lower search line setting circuit, 35: optimum average density calculation circuit
36: block readout circuit, 37: counter circuit, 39
... Subtracter, 40 arithmetic circuit, 41 minimum value detection circuit.

────────────────────────────────────────────────── ─── Continued on the front page (58) Fields investigated (Int. Cl. ⁷ , DB name) G06T 3/00 400 G06T 3/00 300 G06T 1/00 280-340 H04N 1/387 Patent file (PATOLIS) JICST File (JOIS)

Claims (7)

(57) [Claims] 1. A first search area is set in an overlapping area when image data is overlapped with a predetermined width, and has a correlation based on the density of the image data in the first search area. A first reference block having a predetermined size is set, and a second reference block is located at a position having the correlation in a second search area extending along the length direction of the overlap area from the first reference block. Reference block setting means for setting, a displacement amount detecting means for detecting a displacement amount from a predetermined pixel in each reference block set by the reference block setting means to a position of a pixel to be interpolated at the time of synthesis, and the displacement amount Interpolating means for interpolating an image to be synthesized based on the amount of displacement detected by the detecting means; and synthesizing the image data and the image data interpolated by the interpolating means. Image synthesizing apparatus characterized by comprising an image synthesizing unit. Wherein setting the first search area for setting the first reference block image data in a region overlapping when not polymerize a predetermined width on the edge line of a predetermined picture image, this It sets a first reference block of a predetermined size having a correlation based on the optimal average value of the density of the image data in the first search area along from the first reference block in the longitudinal direction of the O-bus Rappu region A search is performed by setting a second search area on the extended search line or including the search line and having a size smaller than the first search area, and performing a correlation based on the optimum average value of the density in the second search area. Reference block setting means for setting a second reference block at a position having a characteristic, a corresponding point on another image corresponding to the first reference block based on the set first reference block, and Displacement amount calculating means for detecting a lap area to detect a corresponding point corresponding to the second reference block, and calculating a displacement amount between the images from the reference block and the corresponding point; image data based on the calculated displacement amount And an image synthesizing unit that synthesizes the image data and the image data interpolated by the interpolation unit. 3. The displacement amount detecting means of the image synthesizing device according to claim 1, wherein the first and second reference blocks set in the first image to be synthesized are respectively set on the basis of a predetermined overlap area.
Search area extension for performing matching processing by sequentially expanding the search area when the processing result of corresponding block detection that maps to an image and performs matching in the corresponding first and second search areas exceeds a preset range When the range exceeds the range defined by the matching area and the search area expansion matching section, the reference block and the corresponding block are enlarged or two pairs of non-adjacent reference and corresponding blocks are used.
A corresponding point detection means comprising a reference block expansion matching unit for performing a matching process, and a displacement coefficient calculation means for calculating a displacement amount from a result from each search area expansion matching unit and the reference block expansion matching unit of the corresponding point detection unit. An image synthesizing apparatus, further comprising: 4. A reference block setting means for setting a reference block of a predetermined size in an area overlapping each other when binarized image data is superimposed at a predetermined width; Displacement amount detecting means for detecting the displacement amount from the data, a coordinate transformation unit for transforming the coordinates of a pixel (original pixel) on the image side to be synthesized based on the detected displacement amount, and a new coordinate transformed pixel and its vicinity A neighborhood distance calculation unit that calculates a distance to an original pixel to be processed, a distance determination unit that determines a position of a new pixel based on the calculated distance, and an image data reading unit that interpolates image data from the determined position. comprising a binary interpolation means, and images synthesizing means you combine the image data to be combined with binary interpolated image data, and the reference block setting means is larger than the reference block
Set a search area and search within that search area.
An image synthesizing device, wherein an area of a predetermined size having relevancy is used as a reference block . 5. An image synthesizing apparatus for synthesizing first image data and second image data having an overlapping area, wherein two or more reference blocks are set in an overlapping area of the first image data. Reference block setting means for detecting a corresponding block in the second image data corresponding to the set reference block position, and obtaining the first and second blocks from the reference block position and the corresponding block position.
A displacement amount detecting means for detecting a displacement amount between the images, and interpolating at least one of the first image data and the second image data based on the displacement amount, and then combining the respective images. comprising an image combining unit, wherein the reference block setting means is larger than the reference block
Set a search area and search within that search area.
An image synthesizing device, wherein an area of a predetermined size having relevancy is used as a reference block . 6. The size of the search area or the size of the reference block is variable.
6. The image synthesizing device according to 4 or 5 . 7. The image synthesizing apparatus according to claim 5, wherein said image data is binary image data.