JPH08181903A - Image pickup device - Google Patents

Abstract

PURPOSE: To pickup an image with high quality by picking up an image by dividing and reducing the time of pasting and improving the precision of the processing when partial images are composited into an entire image. CONSTITUTION: Plural images picked up by dividing having overlapped areas are obtained by an image pickup element 4 by using a turning mirror 4. A reference block (reference point) is set and a corresponding block (corresponding point) matching the reference block in the overlapped area toward a pasted image is retrieved and the reference block and the corresponding block or the reference point and the corresponding point match the paste by using a pasting section 25 and predict information based on each position of the reference block and the corresponding block for preceding paste is provided. In the case of succeeding paste, the pasted position (retrieval range) is predicted by the predict information for the paste.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup apparatus which combines a plurality of divided images taken by dividing them and restores them into one image.

[0002]

2. Description of the Related Art Generally, flying spot scanners, facsimiles, copiers and the like are widely known as image pickup devices for picking up an image of a subject which is drawn on a paper surface and spreads in two dimensions. These imaging devices do not scan the subject area purely electronically at the time of shooting, but scan it mechanically or
Alternatively, a line sensor is used to electronically scan one dimension and mechanically scan the other one dimension to obtain a two-dimensional image.

In recent years, it has been called an "OA camera" or a "blackboard camera" and the like, which is attached near a whiteboard or a blackboard to photograph information such as written characters and figures at once and collect information very easily. There is a camera to do. This camera is
It is a portable camera having a normal image pickup format in which a single image pickup device such as a CCD is incorporated to obtain one image pickup screen for one subject.

Generally, image information such as characters is fine, and it is necessary to obtain a resolution such that the character information can be read when a conventional two-dimensional image pickup device is used to capture an image by pure electronic scanning. Since the number of pixels of the element becomes extremely large, the structure becomes difficult and the cost becomes high. Further, in the above-described pure mechanical scanning or the combination method of the line sensor and the mechanical scanning, there is a problem that the configuration of the apparatus becomes complicated and the size becomes large, and the image capturing time becomes long.

Therefore, as compared with an "OA camera" which obtains one image-capturing screen for one subject, a higher-definition image can be obtained, which is easy to manufacture and has a relatively low number of practical pixels. An image pickup system is configured by a single image pickup device such as a CCD, and further, a screen division photographing type “OA camera” in which a movable mirror for scanning is arranged in the image pickup optical system can be considered. This split-screen image capturing type "OA camera" captures an image of a divided subject portion corresponding to the rotating position by an image sensor when the movable mirror is at each scanning rotating position, and the plurality of image capturing operations are performed. By sticking the screens together,
It is for obtaining information of one captured image.

This camera divides a subject region into a plurality of regions, for example, 2 to 10 or more, rotates the movable mirror in each subject region, scans, and sequentially photographs. After shooting, the obtained image information is combined with the plurality of partial areas to obtain one subject image. As a result,
A portable size device can capture a high-definition image in a large area in a relatively short time. As such a camera, for example, there is a camera described in Japanese Patent Application No. 5-246390 proposed by the present applicant.

In the screen division type "OA camera", as described above, the process of combining the image information of the photographed partial areas is performed. The stitching process includes not only the process of simply joining the partial images obtained separately into one large overall image, but also the process of correcting the deviation or inclination between the images at that time. It is a thing.

Such processing will be described by taking a stitched panoramic photograph using a silver salt film as an example. Conventionally, as a method of joining a plurality of partial images to obtain one large overall image, in silver halide photography, it has been mainly used for landscape photography that captures a broad horizontal and horizontal area. is called.

What is important when taking this joint panoramic photograph is that the photographing range of the camera is accurately shifted in the horizontal direction. Normally, the camera is fixed to a tripod and horizontally rotated by a tripod head. Is used.

However, even if the horizontal rotation of the platform is used, it is extremely difficult to completely eliminate the deviation and the inclination.
There is also a method of simply realizing this by hand-held by the photographer when there is no tripod at the time of photographing, and in this case, the deviation or inclination of the photographing range obviously occurs.

Therefore, when the final image is joined to one whole image (performed in the cutting and pasting of the print or the printing process), the "connecting state" of the connecting portions of the adjacent partial images, that is, the continuity at the boundary. The position and inclination are corrected while visually determining the characteristics and the correlation of the images. Therefore, at the time of shooting, the shooting range is set so that the area near the connection portion is commonly included in the adjacent partial images. At this time, the region commonly included is referred to as an overlap region.

Even on the screen of the "OA camera", each partial image is not obtained at the same time, but is obtained sequentially by scanning. Therefore, if camera shake (camera shake) or movement of the subject occurs during shooting, each portion Lamination errors such as misalignment and tilt occur in the image. A similar error also occurs due to the scanning mechanism.

That is, in the case of a mechanical scanning mechanism in which a mirror is rotated by a motor, the component parts often have a manufacturing error within a predetermined allowable range, and when assembled with these parts, The mirror vibrates due to the rotation and stop, and the fixed (stop) position of the mirror varies. These vibrations and variations also cause an error in the captured image. At this time, the former is a constant error that always occurs with the same value in every photographing, but the latter is an accidental error in which the value varies each time photographing is performed due to the variation of the fixed position. The error caused by the above-mentioned camera shake or movement of the subject is an accidental error.

The process of joining a plurality of partial images including these errors into one large whole image while correcting the position and the inclination is the stitching process. Note that this process is not always performed all at once, and is sequentially performed for each pair or set of adjacent partial images depending on the procedure (algorithm) for connecting partial images to the entire image.
At that time, also in each subsequent process, it is called a laminating process.

Such a laminating process is performed by calculating the laminating error by calculating the correlation of the images in the overlapping portion. That is, "joining" is realized by assigning (replacement) the coordinates of the data of each partial image to the coordinates of the entire image. At that time, the coordinates may be corrected based on the calculated bonding error. .

A specific technique for performing the laminating process based on such a correlation operation is described in, for example, Japanese Patent Application No.
The detailed description is omitted here.

[0017]

However, in the pasting processing described above, in the actual "OA camera", each image information data is taken into the digital memory and the data (including address information) is digitally processed. It is realized by performing data processing such as calculation.

Generally, several tens of ms to process the data.
It takes several 100 ms or more. From the viewpoint of production cost and shape (miniaturization), it is advantageous to minimize the capacity of the digital memory required for processing.

When the apparatus is created based on such an idea (first case), it is better to execute the stitching process sequentially every time each partial image is captured, so that all partial images are temporarily stored in the buffer memory. Better than the one that is taken up and then processed. That is, the latter requires a buffer memory for all images, but the former can be realized if there is a buffer memory for several partial images.

However, if the stitching process takes a long time, the shooting of the next partial image cannot be started until the process is completed, so the shooting time interval becomes long and the condition of the subject changes. was there.

On the other hand, since the bonding process utilizes the correlation of the image data as described above, there is no subject having a certain constant contrast in the area used for the correlation calculation in the overlap portion, Alternatively, if there is a subject such as a repeated pattern that causes an erroneous judgment in the calculation, there is a problem that the bonding becomes impossible.

Further, the same error may be always corrected for the stationary error caused by the above-mentioned scanning, but the correction amount is unknown, and as a result, the efficiency of the calculation is lowered and the calculation time is reduced. It became a problem that it became longer and the risk of misjudgment increased.

Further, the time required for the above calculation generally differs depending on the situation (the size of the bonding error, the pattern of the overlapping portion, etc.) each time, and due to other factors, the photographing time interval (exposure timing) of each partial image. Will vary by accident. For this reason, the effect of camera shake and the effect of movement of the subject fluctuate more and more with each calculation, which also causes a problem of reduction in calculation efficiency.

The "first case" has been described above. On the other hand, however, the cost and the shape are not always prioritized, but the performance is prioritized, or the ease of design is considered. In some cases.

If such a case is referred to as a "second case", in the second case, in the OA camera constituted by the buffer memory having a limited capacity as described above, each of the OA cameras is dependent on the time required for the bonding process. There is a problem that the efficiency of calculation is reduced because the timing of shooting (exposure) of the partial image is restricted and the amount of information that can be used for the combining process is limited.

In consideration of the situation that the cost, shape, performance, etc. are affected by the capacity of the bonding process buffer memory, in the conventional OA camera, the buffer memory and a digital memory as a recording medium are configured. Since it is completely separated and cannot be used for both purposes, there is a problem that cost, shape and performance cannot be secured at the same time.

Further, in addition to the capacity of the buffer memory, if the exposure time and the exposure level at the time of shooting each partial image are not kept constant, the image quality variation in a broad sense of the partial image becomes large. There is a risk that the image quality of the entire image will be poor, or that a problem will occur in the laminating process.

Therefore, according to the present invention, an image is divided and photographed,
It is an object of the present invention to provide an image pickup device which realizes a reduction in the time of the bonding process and an improvement in accuracy when restoring a partial image to an entire image, and which picks up an image with high image quality.

[0029]

In order to achieve the above object, the present invention divides an image-capturing target area so that at least one end side of the screen has an area overlapping with another screen and captures a plurality of images. An image pickup unit for obtaining a partial image and a reference block including an arbitrary reference point in an overlapping area on the pasted image side in the partial image are set, and the reference block is set in the overlapping area on the pasting image side. Image matching means for searching for a corresponding block matching with or a corresponding point matching the reference point, matching the reference block with the corresponding block, or matching the reference point with the corresponding point, and performing a bonding process on the partial images; By the laminating means,
At least the reference block or reference point and the information based on each position of the corresponding block or corresponding point that matched when pasted before is held as prediction information, and when the next partial image is pasted, the prediction information An image pickup apparatus is provided, which is configured with a pasting position, and a pasting control unit that instructs the pasting position to the image pasting unit.

[0030]

In the image pickup apparatus having the above-described structure, a part of the screen is provided with a region overlapping with an image of another screen, and one shooting target region is divided and shot to generate a plurality of partial images. Then, when connecting the partial images, set at least two reference points or reference blocks in the overlap area of the images to be pasted, and paste the corresponding points or corresponding blocks that match the image data at this location. A search is performed within the overlapping area of the images to be matched, and the matching point or the corresponding block that has been searched is overlapped so as to match the reference point or the reference block and the bonding processing is performed. Then, the reference point, the reference block, the corresponding point, and the position information of the corresponding block obtained during the bonding process are stored, and the position information is predicted to determine the bonding position when performing the bonding from the next time. Using as information, a rough search area for searching for corresponding points or corresponding blocks is set within the overlap area of the images to be pasted.

[0031]

Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 1 shows a schematic configuration of an image pickup apparatus as a first embodiment according to the present invention and will be described.

In this image pickup apparatus, a mirror 2 for reflecting a subject image (subject) 1, a mirror drive section 3 for rotating the mirror 2 in a predetermined direction, and a subject image 1 reflected by the mirror 2 are picked up. AF for focusing on the element 4
A photographing lens 5 and a diaphragm 6 having a function, an image pickup system driving section 7 for driving the photographing lens 5 and the diaphragm 6, an image pickup element 4 and an image pickup circuit for photoelectrically converting the formed subject image 1 to generate a video signal. 8, an image sensor driving unit 9 that drives the image sensor 4, an A / D converter 10 that digitally converts a video signal from the image sensing circuit 8, and a memory 11 that stores digitally converted image data. A memory drive unit 12 that drives the memory 11, a pattern matching unit 13 that performs pattern matching of a pasted portion of a captured partial image, which will be described later, and a recording medium on which the obtained entire image is detachable, such as a memory card 14. A card driving unit 15 for recording the image on the printer, a printer driving unit 17 for printing out an image from the printer 16 via the card driving unit 15,
A system control unit 18 for driving and controlling the entire drive unit, a sync signal generator for sending a predetermined sync signal to the image pickup circuit 8, the image pickup device drive unit 9, the A / D conversion unit 10, and the system control unit 18. EEPR that is detachably attached to the unit 19 and the system control unit 18 and stores component error information and learning information
The OM 20, the camera shake sensor 21 for detecting the rotation and shift amount of the camera housing, the display unit 22 for displaying the state of the entire system, the trigger switch for starting the shooting, and the setting of the number of partial shootings. Input switch 2 for
3 and 3.

The system control unit 18 is composed of a sequence processing unit 24 for controlling the entire system and a bonding processing unit 25 for performing a bonding process of images. The system control unit 18 roughly performs two controls, one of which starts when the sequence processing unit 24 presses a trigger switch of the input switches 23, and drives the mirror 2 as described above. Then, the image pickup circuit 8 and the memory 11 are controlled to perform a series of sequence control of recording on the memory card 14 and outputting to the printer 16. Memory 11 for matching
And the pattern matching by the pattern matching unit 13 are controlled.

Image pickup by the image pickup apparatus configured as described above will be described. First, the subject image 1 is reflected by the rotating mirror 2 and guided to the taking lens 5. This mirror 2
Is rotated by the mirror drive unit 3 controlled by the system control unit 18 so as to divide and capture the subject image, for example, to perform several scans (four times in the embodiment).

The subject image 1 is formed on the image pickup element 4 by the taking lens 5 and photoelectrically converted. The subject image 1
Image signal is processed by the image pickup circuit 8, and the A / D conversion unit 1
When it is 0, it is converted into a digital signal and stored in the memory 11 as image data. At this time, the image pickup device driving unit 9
The image pickup circuit 8 and the A / D converter 10 are synchronized signal generator 19
It operates in synchronization with the sync signal from the control unit, and the sync signal is also input to the system control unit 18 to synchronously control the entire sequence.

The image data read from the memory 11 is input to the pattern matching section 13 and pattern matching is performed on the bonded portions of each of the four scans.
The amount of deviation is detected, and the memory drive unit 12 is driven again via the system control unit 18 to read from the memory 11 and obtain a large whole image for one sheet. The entire image thus obtained is controlled by the system control unit 18 and the card drive unit 1
Is recorded in the memory card 14 through 5, or
The printer driver 17 is driven to output the data from the printer 16.

Next, the memory 11 described above will be described with reference to FIG. In the present embodiment, the memory 11 will be described as a memory for sequentially recording. This sequential recording is 1
When a large whole image is divided into four, and the images are photographed by four scans and the partial images are pasted together and restored, pattern matching processing is performed for each photographing and then recorded in the memory card. Is.

Therefore, since the partial image is only temporarily stored, the memory capacity is small. FIG. 2A shows an image captured the third time of the partial images stored in the memory 11 and including an area to be attached to the partial image captured the second time. That is, if the whole image is shown by the one-dot chain line, the amount of data stored in the memory 11 at one time is about 1/4 of the partial image capacity of the entire image, and the amount of data stored in the memory 11 is inward from the end of the second captured image. The third partial image including the image area of the pasted part necessary for pasting is stored.

An image that has been pasted by performing pattern matching and a pasting process on the portion necessary for pasting to the end portion of the partial image taken at the second time and the partial image taken at the third time. Are sequentially recorded in the memory card for each photographing (scan).

Next, as another example of the configuration of another memory as shown in FIG. 2B, a memory capable of storing partial images for four times of photographing will be described. In this memory, four times of photographing (all scans) are performed, all the partial images are stored in the memory 11, the pasting process is performed to generate the entire image, and the entire image is recorded in the memory card 14 at once. Therefore, the memory 11 requires at least a memory capacity for storing four partial images (one whole image). Although this memory has a large memory capacity, it is suitable for learning and joint position prediction described later.

Next, FIG. 3 shows an example of the construction of an image pickup apparatus as a second embodiment according to the present invention. Here, in the constituent members shown in FIG. 3, the same members as the constituent members shown in FIG. 1 are designated by the same reference numerals, and the description thereof will be omitted.

This image pickup device is characterized in that it does not have a memory mechanism corresponding to the memory 11 described above. Similar to the first embodiment, the subject image 1 formed by the taking lens 5 is photoelectrically converted by the image pickup device 4, and the image pickup circuit 8
The image signal (partial image) is generated by digitizing the video signal processed by the A / D converter 10.

Next, the digitized partial image is directly recorded on the memory card 14 via the card driving unit 15. Next, the card drive unit 15 takes out the image data from the memory card 14, performs the pattern matching and the bonding process with the pattern matching unit 13 and the bonding processing unit 25, and again through the card driving unit 15 to the memory. It is recorded on the card 14.

Therefore, in this embodiment, since the memory mechanism is not provided in the apparatus, downsizing and cost reduction can be realized. The memory card used in this embodiment will be described with reference to FIG.

First, the partial images indicated by the dotted lines are sequentially photographed, and the entire partial image not subjected to the bonding process is recorded on the memory card 14. After that, read the image of the boundary part required to perform the bonding, that is, the bonded part,
The pattern matching unit 13 performs pattern matching,
Lamination is performed by the laminating processing unit 25.

Then, the images resulting from the laminating process (images shown by solid lines) are sequentially recorded in the memory card 14. This means that the partial image recorded on the memory card is subjected to coordinate conversion due to image shift, and the image after lamination shown by the solid line is recorded.

Further, in order to execute such processing,
When performing coordinate conversion, a buffer memory having a capacity of at most one photographing (scanning) for performing interpolation processing or coordinate conversion processing is required. However, using a part of the storage capacity of the memory card, for example, It can be used as a buffer memory by using the work area.

Next, the laminating process in this embodiment will be described with reference to FIG. The pattern matching used in the above-mentioned bonding process will be described. First, as shown in FIG. 2, shooting is performed so that the edge of the previously captured partial image overlaps the edge of the partial image. That is, FIG.
As shown in (a), the trailing edge (boundary) of image A and image B
Shoot so that there is overlap in the width at the front edge (boundary) of. This overlapping portion is referred to as an overlap area (overlap portion). The image A and the image B are pasted together by pasting the overlapping areas.

First, a block of a predetermined size (a block of 8 pixels × 8 pixels in this embodiment) is located at an arbitrary position in the overlap area of the image A, and is a reference block A1, A.
2, and the image information of those blocks is taken into the pattern matching unit 13. Here, in order to simplify the explanation, it is assumed that the reference blocks A1 and A2 have a certain degree of contrast and are fixed within the overlap region of the image A.

Then, after the information of the reference blocks A1 and A2 is fetched, a block of the image corresponding to the reference block A1 is searched for in the overlap area of the image B and set as the corresponding block B1. Block A
Find a block of the image corresponding to 2 and find the corresponding block B2
And set. This is to sequentially compare with the reference block while moving the block of 8 pixel × 8 pixel unit that can be set in the overlap area of the image B. At the time of the comparison, the correlation between the blocks is obtained. Here, the correlation means the sum of absolute values of the subtraction values of the pixels forming the block.

Corresponding blocks B1 and B have the maximum correlation, that is, the sum of the absolute values of the subtraction values is the minimum.
Set to 2. Next, the corresponding block B1 corresponding to the reference block A1 and the corresponding block B2 corresponding to the reference block A2
After setting, the movement amount and the rotation amount are calculated based on the respective coordinates. Here, the movement amount means that when the bonding is performed,
This means the distance by which the image B is moved in parallel, and the degree of displacement between the upper reference block A1 and the corresponding block B1 is the displacement amount S1, and the displacement amount S1 is equal to the displacement amount. Also, the degree of deviation between the lower reference block A2 and the corresponding block B2 is referred to as the deviation amount S2.

Then, the angle formed by the images A and B is obtained as the rotation amount from the shift amount S1 and the shift amount S2. Based on these movement amounts and rotation amounts, the image B can be moved, and the images A and B can be pasted together to restore a single image as shown in FIG.

As described above, the movement amount is originally shown in FIG. 5B when the origin “0” of the image A and the origin “0” of the image B are overlapped with each other when there is no camera shake or mechanical error. Thus, the reference block A1 and the corresponding block B1 and the reference block A2 and the corresponding block B2 overlap one point. However, in reality, camera shake and mechanical error will occur,
Similarly, when the origins “0” of the two images A and B are overlapped, as shown in FIG. 5C, the reference block A1 and the corresponding block B1 and the reference block A2 and the corresponding block B are obtained.
Two discrepancies occur. That is, the images are displaced and joined together.

It should be noted that the image B is moved and bonded by the amount of movement and the amount of rotation obtained as described above, but the shift amounts S1 and S2 are used to determine the amount of movement or rotation required for bonding. The procedure for recalculating will be described later.

Next, in the above-mentioned laminating process, the image A
The same image as the reference block of is searched out from the image B and set as the corresponding block B1. Regarding the search in this setting, a normal case and a case of using the predicted position will be described.

Here, in this embodiment, the range in which the corresponding block B1 in the image B as shown in FIG. 7A exists is set as the search area C, and is set to, for example, 64 pixels × 64 pixels. The search area C is determined on the basis of the maximum camera shake amount, the maximum mechanical error, etc., in consideration of the range in which the images can be pasted without loss of images.

In the search that is usually performed, an 8 × 8 block is shifted from the edge of the search area C by one pixel and searched.
With this method, the correlation of 57 × 57, that is, 3249 blocks must be obtained, and the search takes a very long time. Therefore, it is conceivable to reduce the number of calculations and shorten the search time by some prediction.

As a search method using the first predicted position,
As shown in FIG. 8A, there is a method of searching only a range in which the predicted position is shifted by one pixel in the vertical and horizontal directions. This technique
The search is sequentially performed within nine times around the predicted position in order from one pixel on the upper right side shifted by one pixel. After a total of nine correlations are calculated, if the correlation value of the predicted position exceeds the reference value set in advance and the correlation is the maximum, the position is set as the corresponding block.

As a search method using the second predicted position, as shown in FIG. 8B, a total of 25 correlations are obtained by shifting by 2 pixels (1 pixel to 2 pixels) vertically and horizontally. Then, as in the case described above, there is a method of setting the position as the position to be set in the corresponding block if the correlation of the predicted position is the maximum after exceeding the preset reference value.

When the search using such a predicted position is used, the search can be performed only by the calculation of the correlation value at most 9 times or 25 times, and the corresponding block can be found at an overwhelming speed compared with the conventional method. You can

Further, as a search method using the third predicted position, as shown in FIG. 11A, a search area of 64 pixels × 64 pixels is divided, and a divided area containing the predicted position is divided from the end. There is a method of obtaining the correlation in order. Here, in the present embodiment, an example will be described in which an area of 64 pixels × 64 pixels is divided into four, and a correlation in a range of 32 pixels × 32 pixels is obtained.

According to this method, the correlation is calculated 1024 times in total, and the search time is shortened as compared with the case where the correlation of the entire search area of 64 pixels × 64 pixels is calculated.

Next, specifically, a method of shortening the search time by using the predicted position from the overlapping area of the image B on the side where the corresponding block B1 is pasted will be described. How to obtain this predicted position will be described.

As a first method for obtaining the predicted position, a method for obtaining the predicted position from the displacement amount used in the previous pasting will be described for the second and subsequent pasting when a plurality of images are pasted together. . Here, it is assumed that when the images taken at one time are pasted a plurality of times, an approximate shift is made.

Shift amount S1 obtained by one-time laminating
The predicted position is calculated by the following equation using (x, y) and S2 (x, y) as relative values. α1x = A1x + (previous S1x) ... H1 α1y = A1y + (previous S1y) ... H2 α2x = A2x + (previous S2x) ... H3 α2y = A2y + (previous S2y) ... H4 Here, the coordinate of the predicted position: α, Predicted position of corresponding block B1 is α1, predicted position of corresponding block B2 is α2, coordinates in x direction and y direction are x and y, and reference block A1.
The x coordinate of x plus the x coordinate of the previous shift amount:
α1x, the Y coordinate of the previous displacement added to the Y coordinate: α1y, the x coordinate of the reference block A2, the X coordinate of the previous displacement added: α2x, the previous Y coordinate of A2 The sum of the y-coordinates of the deviation amount of
2y.

In the OA camera as in this embodiment, the influence of camera shake is the largest. Therefore, as a method of obtaining the second predicted position, the shake amount is detected by the shake sensor,
A method of obtaining the predicted position from the calculated camera shake amount will be described. As shown in FIG. 1, a camera shake sensor 21 that can detect the amount of camera shake in the xy directions is provided. In this example, camera shake detection is always performed at the time of partial image capturing.

Α1x = A1x + (camera shake 1x component during each measurement) ... H5 α1y = A1y + (camera shake 1y component during each measurement) ... H6 α2x = A2x + (camera shake 2x component during each measurement) ... H7 α2y = A2y + (camera shake 2y component at each measurement) ... H8 As described above, α1x of the predicted position adds the camera shake amount (x component) obtained during partial imaging to the x coordinate of the reference block A1. Hereinafter, the predicted position of α2 is similarly obtained.

When the camera shake sensor 21 detects the camera shake amount, the measurement is performed every time, and therefore the sum of camera shake amounts is not the same value. For example, if four times of photographing (scanning) is performed, four times of camera shake (scanning) is performed. However, the amount of camera shake having a different value is added to the coordinates of A.

Since two points, the upper side B1 side and the lower side B2, have to be obtained when the shift amount is obtained, camera shake is detected at two points, the upper side and the lower side of the camera. Then, the upper hand shake amount is divided into xy components and set as 1x and 1y, which are used when obtaining the predicted position α1. Also, the amount of camera shake on the lower side is divided into xy components to be 2x and 2y, which are used when obtaining α2 of the lower predicted position.

Next, as a method of obtaining the third predicted position, the error when the mirror 2 is rotated (scanned) at the time of photographing peculiar to each camera is recorded in the EEPROM 20, and the EEPROM 20 is used at the time of prediction. This is a method of reading the information from to obtain the predicted position. Here, the error at the time of mirror scanning is a mechanical error such as a positional deviation of the mirror that occurs when the mirror rotates, and an error of an electrical control system caused by a variation in characteristics of electric components, or It consists of optical errors caused by optical parts such as lenses.

The error information recorded in the EEPROM 20 is recorded with the error generated for each scan so that the variation in the error for each photographing can be eliminated.

Α1x = A1x + (Error 1x component stored in EEPROM) ... H9 α1y = A1y + (Error 1y component stored in EEPROM) ... H10 α2x = A2x + (Error 2x component stored in EEPROM) ... H11 α2y = A2y + (EEPROM The error 2y component stored in) ... H12 The predicted position α1x is added with the x component obtained by adding the mechanical error, the control system error, and the optical error stored in the coordinate of A1x. Hereinafter, the predicted position of α2 is similarly obtained.

Next, as a fourth method of obtaining the predicted position, there is a method of determining the predicted position by providing a learning function such as a neural network and learning the habit of hand shaking of the user and mechanical error. As an example of such a method, here, an average value of the shift amounts generated in the previous shooting is stored, and the average value is added to the coordinate of A1,
Let it be the predicted position.

Α1x = A1x + (average value 1x component up to the previous time) ... H13 α1y = A1y + (average value 1y component up to the previous time) ... H14 α2x = A2x + (average value 2x component up to the previous time) ... H15 α2y = A2y + (previous time) The average value up to 2y component) ... H16 Above, the above-described predicted positions can be used independently,
Also, they can be used in combination.

Next, with reference to the flow chart shown in FIG. 9, the bonding process using the predicted position will be described. Here, the reference numerals of the constituent members indicate the reference numerals of the constituent members shown in FIG.

First, the number of divisions of a desired image to be photographed by the user is set, that is, the number N of times of photographing partial images is set (step S1). Next, the number of shots I
Initial value "1" is set (step S2), and a start switch (release switch) not shown is turned on to
The partial shooting of the first sheet is started (step S3). Mirror 2
Rotates to form the next subject partial image on the image sensor (step S4). After the rotation of the mirror 2 is completed, the number of times of photographing I is incremented (step S5). The partial photographing is performed again (step S6).

After the shooting, the movement amount and the rotation amount are calculated using the predicted position (step S7). Based on the calculated movement amount and rotation amount, the image is moved and rotated, and the bonding process is performed (step S8).

Next, until the number of times I of photographing has reached the number of times N of photographing set by the user, the above-described photographing and bonding processing is performed and the processing is terminated (step S9). Next, FIG.
Calculation of the movement amount and the rotation amount in the bonding process will be described with reference to the flowchart of FIG.

Here, in this embodiment, the movement amount is
S1x, S1y, that is, the displacement amount of points A1, B1 themselves on the boundary portion is the movement amount, and the rotation amount is A1 and B1.
The amount of rotation required until the points A2 and B2 below the boundary coincide with each other when 1s are superposed is defined as the rotation amount.

First, the image of the reference block A1 is captured (step S11), and subsequently, the image of the reference block A2 (lower side) is captured (step S12). Next, using one of the methods for obtaining the predicted position described above,
Input the predicted position of the corresponding block B1 (step S1)
3). Here, the predicted positions that are input are α1x and α1y. However, if it is impossible to predict and there is no information, it is necessary to input as “no information”.

Next, the corresponding block B1 is searched based on the predicted position (step S14). Using the corresponding block B1 obtained by this search, the shift amount S1 between the reference block A1 and the corresponding block B1 is calculated using the following formula: S1x = B1x−A1x ... H17 S1y = B1y−A1y. Step S15).

Similar to the above, the predicted positions α2x and α2y of the corresponding block B2 are input (step S16). Then, the corresponding block B2 is searched based on the predicted positions α2x and α2y (step S17). Using the corresponding block B2 obtained in this search, the above equations H17, H18
Similarly, the shift amount S2 between the reference block A2 and the corresponding block B2 is calculated (step S18). Then, the calculated shift amount S1x is the movement amount in the X direction, and the shift amount S1y is Y.
The amount of movement in the direction is set (step S19). And the amount of rotation is

[0083]

[Equation 1] Is calculated (step S20), and the process returns.

Next, the search process for the corresponding block B will be described. In the present embodiment, an example of performing a search process for the corresponding block B1 with a block size of 8 pixels × 8 pixels will be described.

Now, with reference to FIG. 11B, the calculation of the correlation used for the search will be described. FIG. 11B
, The first upper left pixel of the reference block of image A is DA (0,0), and the upper left pixel of the block for which the actual correlation with this DA (0,0) is Dp (0,0). To do.

This pixel DA (0,0) and pixel Dp (0,
0) from the lower right pixel DA (7,7) and pixel Dp
The difference up to (7, 7) is sequentially obtained, and the absolute value is added to obtain the correlation value. If this is expressed by an equation,

[0087]

[Equation 2] Becomes Here, q is a correlation error and indicates the degree of correlation. The smaller the value of q, the larger the correlation. The smaller the value of q, the better the correlation.

Further, FIG. 12 shows how to represent the position (coordinates) of the blocks for deriving the correlation between the respective reference blocks. The position of the reference block of the image A is based on the position of the pixel at the upper left of the block and is the coordinates of the block. Therefore, the coordinates of the reference block are A1x and A1y, and the coordinates of the corresponding block at the predicted position of the image B are α1x and α1y.
And

Further, the block for actually obtaining the correlation is
Assuming that the search is performed in a range shifted by 2 pixels from the position of the first block to the periphery, for example, the position of the block shifted to the upper left of the first block is represented by α1x-2, α1y-2, and the peripheral block is the first block. The position of the lower right block is α1x +
2, represented by α1y + 2.

Next, the search process will be described with reference to the flowchart shown in FIG. First, it is determined whether there is predicted position information (step S31). If the predicted position is not set (NO), a search [normal search] is performed for all normal pixels (step S3).
2). When performing this normal search, for example, if the search area is 64 pixels × 64 pixels, the block correlation is calculated a total of 3249 times. However, if it is determined in this determination that there is the predicted position information (YES), the correlation of the predicted position is obtained based on the following equation as described above (step S33).

[0091]

(Equation 3) Here, q is the above-mentioned correlation error. Dα1 indicates the value of the data itself of one pixel in the upper prediction position block, and DA1 indicates the value of the data itself of one pixel forming the upper reference block. n represents an allowable amount, and when the correlation value q of the predicted position is lower than the preset allowable amount, it means that the reliability of the search based on the predicted position is low as will be described later, and the normal search is performed.

Next, the calculated correlation value q of the predicted position,
The allowable value n is compared (step S34). If the correlation value q of the predicted position is equal to or smaller than the allowable value n in this comparison (NO), that is, it is determined that the reliability of the search based on the predicted position is low, the process proceeds to step S32, and the normal search is performed.

However, if the correlation value q of the predicted position is larger than the allowable value n in the comparison in step S34 (YE
S), the search is performed within the range of two pixels around the predicted corresponding block. At this time, the coordinates of the position for which the correlation is first compared with the predicted position are α1x−2 for I and α1y for J.
-2 is substituted (step S35), and the correlation value γ is obtained by the following equation (step S36).

[0094]

[Equation 4] Here, q is the above-mentioned correlation error, and Dp1 represents the value of the data of the pixels forming the block around the predicted position, that is, the block for which the correlation is currently calculated.

Next, the calculated correlation value γ is calculated in step S
It is determined whether it is larger than the correlation value q obtained in 33 (step S37). If the correlation value γ is less than the correlation value q in this determination (NO), that is, if the correlation value γ is determined to have a higher correlation, it is assumed that the corresponding point is more accurate than the predicted position. , I as α1x and J as α1y as predicted positions (step S38).

Further, the correlation value γ is replaced with the correlation value q as a new predicted position (step S39), and step S35.
Then, the correlation error is obtained again. However, if it is determined in step S37 that the correlation value γ is greater than the correlation value q (YES), it is determined that q has a higher correlation, that is, the predicted position has a better correlation, then I and α1x + 2.
Are compared (step S40), I is incremented until α1x + 2 is reached (step S41), the process returns to step S36, and the same processing is repeated.

Similarly, until J reaches α1y + 2 (step S42), I is temporarily returned to α1x-2 (step S43), J is further incremented (step S44), and the same processing is performed. This I is once α1x-
To return to 2, the search is performed 5 times in the horizontal direction and 5 times in the vertical direction.
Since it has to be performed 25 times in total, the operation is performed once in the lateral direction and then when the J is incremented once, the operation is performed again in the lateral direction.

After such processing is performed a set number of times, the final predicted position α1x is set as the corresponding point, and B
The process is terminated by setting 1x and α1y to B1y (step S45).

Although this flowchart describes the search for the corresponding block (corresponding point) B1, the process for obtaining the corresponding block (corresponding point) B2 is the same. Next, an example in which characters or pictures are not described in the overlap area and a reference block having contrast data cannot be set will be described.

In such a case, one method is as follows.
It is conceivable to attach them at the default position. The default position is, for example, a position where the origins of two overlapping areas are overlapped.

Description will be made with reference to the flowchart in FIG. First, when determining the movement amount and the rotation amount, it is determined whether or not there is data of the reference block (step S51).
In this judgment, if there is data in the reference block (YE
S), as described above, the movement amount and the rotation amount are calculated (step S52). However, if there is no data in the reference block (NO), if a predetermined default position, for example, the origin is superposed, "0" is set to the movement amount and the rotation amount, respectively (step S53).

Then, the pieces are pasted at their default positions, the user is notified of the fact, and the process is terminated (step S54). In this way, the bonding at the default position is not limited to when the bonding process cannot be performed because there is no data in the reference block. For example, when bonding at the time of manufacturing the camera without using prediction Alternatively, when a scene that is difficult to shoot is to be taken and pasting must be performed beyond the prediction of the camera, it is effective to perform pasting at the default position without using the prediction due to user switching.

As another method in the case where there is no data in the reference block, there is a method of pasting at the predicted position according to the procedure of the flowchart shown in FIG. First, when calculating the movement amount and the rotation amount, it is determined whether or not there is data in the reference block (step S61). With this judgment,
If the reference block has data (YES), the movement amount and the rotation amount are calculated as described above (step S6).
2). However, if there is no data in the reference block, (N
O), and input predicted positions α1 and α2 (step S6)
3).

Since the predicted position is used for the position of the corresponding block (corresponding point) B1, α1 is set as the corresponding point B1 and α2 is set as the corresponding point B2 (step S64).
With these settings, the movement amount and the rotation amount using the corresponding points B1 and B2 are calculated (step S65). Then, the pasting is performed at the predicted position, the user is notified of the fact, and the process is ended (step S66).

However, in the present embodiment, the reference point is described as being fixed within the boundary, but the present invention is not limited to this.
The position of the reference point may be changed each time the bonding is performed. For example, if a block having a characteristic that the number of white pixels and the number of black pixels are the same is used as the reference block, the accuracy of the search result is further improved.

In the first embodiment described above with reference to FIG. 16, an example in which the memory 11 in the apparatus has a capacity capable of storing all partial images to be photographed will be described. First, all the partial images (all areas) to be photographed are exposed (step S71). Next, the amount of movement and the amount of rotation are calculated (step S72). Regarding the calculation of the movement amount and the rotation amount, the same processing as the processing described in step S7 of FIG. 9 described above is performed.

Then, using the calculated movement amount and rotation amount, the respective partial images are combined (step S73). This laminating process is the same process as step S8 of FIG. 9 described above.

Next, it is judged whether or not the screens are pasted over all the partial images (step S74). In this determination, if the combining has not been completed over all the partial images (NO), the process returns to step S72 and the same process is performed. However, when the pasting of the screens is completed over all the partial images (YES), the process ends.

As described above, in order to continuously expose all the areas to be photographed, the time from the start of the exposure of the first image to the end of the exposure of the final image is shorter than that in the case where the bonding process is performed each time, and a camera shake or hand shake occurs. It is possible to reduce image blurring on the subject side.

Next, the case where all of the movement amount and the rotation amount can be calculated in step S72 of FIG. 16 has been described, but the case where there is a partial image that cannot be calculated will be described. This will be described with reference to the flowchart shown in FIG.

First, all the partial images (all areas) to be photographed are exposed (step S81). Next, the correlation between the partial images in all areas is examined (step S8).
2). Then, it is determined whether or not the correlation has been established for all areas (step S83). In this determination, if all the correlations between all the partial images are obtained (YES), the movement amount,
The rotation amount is calculated (step S84). This calculation is based on the procedure shown in step S7 of FIG. 9 described above.

Then, the laminating process is performed based on the calculated movement amount and rotation amount (step S85), and the process returns. However, if it is determined in step S83 that no correlation can be obtained between any of the partial images (NO), it is determined whether all correlations between all the partial images have not been obtained (step S86). If no correlation is found among all the partial images in this determination (YES), error processing is performed (step S87), and the process returns. In this case, as a part of the error processing, the error display to that effect or the pasting with the default value as described above may be performed, or appropriate processing may be performed.

However, in step S86, if not all correlations can be obtained, but at least some correlations are obtained (NO), the movement amount and rotation amount of the non-correlation portions are calculated (step S88), The process proceeds to step S85, the bonding process is performed, and the process returns.

Next, referring to the flow chart shown in FIG. 18, the movement amount in step S88 of FIG.
The calculation of the rotation amount will be described in detail. Here, it is assumed that the partial image is divided into four, and each image is composed of a set of four areas A, B, C, and D. Further, the movement amount and the rotation amount between the areas A and B are the movement amount 1, the rotation amount 1, and the movement amount 2 between the areas B and C are 2, respectively.
The rotation amount 2, the movement amount between the areas C and D, and the rotation amount are set as the movement amount 3 and the rotation amount 3.

First, it is judged whether or not the correlation can be taken only between the areas A and B and between the areas B and C (step S9).
1). If there is no correlation in this judgment (YES),
The amount of movement and the amount of rotation between AB and BC are calculated (step S92), and the process returns. Here, the amount of movement and the amount of rotation between AB and BC are calculated by the procedure of the flowchart shown in FIG.

As the method of calculating the movement amount and rotation amount between AB and BC, since the movement amount and rotation amount between CDs are finally obtained, the movement amount between each AB is 1. For the rotation amount 1, the calculated values of the movement amount 3 between CDs and the rotation amount 3 are input and calculated. Similarly, the values of the movement amount 3 and the rotation amount 3 are respectively substituted and calculated for the movement amount 2 and the rotation amount 2 between BCs.

However, if the correlation is obtained in the step S91 (NO), it is next determined whether or not the correlation can be obtained only between the areas B and C and between the areas C and D (step S93). If there is no correlation in this judgment (YE
S), the amount of movement and the amount of rotation between the BCs and between the CDs are calculated (step S94), and the process returns. Here, between BC,
The movement amount and the rotation amount between CDs are calculated by the procedure of the flowchart shown in FIG. Between this BC,
In the calculation between CDs, it is a prerequisite that AB is calculated, and between BC and C that are not calculated, respectively.
The movement amount 2 and movement amount 3 between D, and the movement amount 1 and rotation amount 1 between AB are input and calculated for the rotation amount 2 and rotation amount 3, respectively.

However, if the correlation is obtained in step S93 (NO), then it is determined whether or not the correlation can be obtained only between the areas A and B and between the areas C and D (step S95). If there is no correlation in this judgment (YE
S), the movement amount and rotation amount between AB and CD are calculated (step S96), and the process returns. Here, between AB,
The amount of movement and the amount of rotation between BCs are calculated by the procedure of the flowchart shown in FIG. This AB, C
Calculation between D is assumed to be between BC and BC
The amount of movement 2 and the amount of rotation 2 are input to the amount of movement between AB and the amount of rotation between CDs 1, the amount of rotation 1, the amount of movement 3, and the amount of rotation 3, respectively, and calculated.

However, if the correlation is obtained in step S95 (NO), then it is determined whether or not the correlation can be obtained only between the areas A and B (step S97). If no correlation is obtained in this determination (YES), the amount of movement and the amount of rotation between the ABs are calculated (step S98), and the process returns. Here, calculation of the amount of movement and the amount of rotation between AB is performed with reference to FIG.
It is calculated by the procedure of the flowchart shown in (d).
Calculation of the movement amount 1 and the rotation amount 1 between AB is performed between AB and B
It is assumed that the movement amount 2, the rotation amount 2, the movement amount 3, and the rotation amount 3 between C are calculated, and using these two, the movement amount 1 is a value obtained by subtracting the movement amount 3 from the movement amount 2. A value obtained by adding the movement amount 2 is input to and calculated. Similarly, the rotation amount is obtained by adding the rotation amount 2 to the value obtained by subtracting the rotation amount 3 from the rotation amount 2
The amount of rotation 1 is input and calculated.

However, if the correlation is obtained in step S97 (NO), then it is determined whether or not the correlation can be obtained only between the areas B and C (step S99). If no correlation is obtained in this determination (YES), the amount of movement and the amount of rotation between these BCs are calculated (step S100), and the process returns. Here, the calculation of the amount of movement and the amount of rotation between BCs is performed as shown in FIG.
It is calculated by the procedure of the flowchart shown in 9 (e).

The calculation between the BCs is performed by the movement amount 2 and the rotation amount 2
When determining, the movement amount between AB and CD, the rotation amount 1,
It is calculated using the movement amount 3 and the rotation amount 3. As a method of obtaining the value, a value obtained by adding the movement amount 1 and the movement amount 3 to the movement amount 2 and dividing by 2 is used. Similarly, the rotation amount is calculated by adding the rotation amount 1 and the rotation amount 3 and dividing by 2.

However, if the correlation is obtained in step S99 (NO), then it is determined whether or not the correlation can be obtained only between the areas C and D (step S101). If no correlation is found in this determination (YES), the amount of movement and the amount of rotation between these CDs are calculated (step S102), and the process returns. Here, the amount of movement and the amount of rotation between CDs are calculated by the procedure of the flowchart shown in FIG.

The calculation between the CDs is performed by calculating the moving amount 3 between the CDs.
Since the rotation amount 3 has been calculated, it is calculated using the movement amount 2 between AB and BC, the rotation amount 2, the movement amount 1, and the rotation amount 1. The movement amount 3 is a value obtained by subtracting the movement amount 1 from the movement amount 2,
A value obtained by adding the movement amount 2 is input and calculated. As the rotation amount 3, the value obtained by subtracting the rotation amount 1 from the rotation amount 2 and adding the rotation amount 2 is input.

When the correlation is obtained in step S101 (NO), the process returns and ends. In the present embodiment, the case where the correlation could not be obtained by using the interpolation or the like has been described, but in addition, the movement amount and the rotation amount may be relative amounts from the default position for absorbing mechanical errors and the like. .

Next, an image pickup apparatus as the third embodiment will be described. The configuration of the image pickup apparatus of this embodiment is the same as that of the first embodiment described above, but the operation is different. This imaging device is characterized in that the exposure interval is constant.
That is, in the first embodiment, in the second and subsequent bonding processes, the previous shift amounts (S1 _x , S1 _y , S2
_{Although the} predicted position is determined by _x , S2 _y ), the exposure interval at the time of capturing each divided image is not particularly defined. In this embodiment, by making the exposure interval constant,
The calculation of the predicted position is facilitated.

FIG. 20 shows a captured image in this embodiment. By rotating the mirror, A, B,
The partial images divided into C and D are sequentially captured. Figure 21
[Fig. 6] is a time chart for explaining exposure timing.

As a reference time, each processing is performed in synchronization with the vertical synchronizing signal VD. First, the divided image A is exposed, and then the mirror driving (driving to the mirror position for exposing B) and the signal processing are performed. In signal processing,
Reading from the image sensor and bonding processing are performed (however, after the exposure of A, only reading is performed).

Thereafter, similarly, B, C, and D exposure, mirror driving, and signal processing are performed. Here, th ₁ to TH ₄ is shows the processing time, and if this time is smaller correlation at that varying (predicted position largely by the state of the object, if there is no predicted position information, the search processing time of bonding become longer).

Generally, as soon as the signal processing is completed, the exposure of the next divided image is started.
1 t _R1 , t _R2 , t _R3 ) changes every time, and as a result, the movement amount due to camera shake also changes from the previous time.

Therefore, the exposure interval is made constant (t _R1 = t
_R2 = t _R3 ), the amount of movement due to camera shake is
It can be considered as the same value as the last time. Previous movement amount S1
_x , S1 _y , S2 _x , S2 _y can be used as they are. That is, the above formulas H1 to H4 can be used.

Further, by setting the exposure interval set at this time to the assumed maximum time, the exposure interval will not be insufficient for any subject. Next, an image pickup apparatus as a fourth embodiment will be described. The configuration of the image pickup apparatus of this embodiment is the same as that of the first embodiment described above,
The behavior is different.

Normally, when a plurality of images are combined to generate one image, the brightness of each partial image differs depending on the part of the subject, so that each partial image is exposed appropriately. The level may be adjusted. However, it is better to keep the exposure level constant in terms of performing accurate correlation detection in the bonding process or making the seams of images inconspicuous.

Therefore, in the present embodiment, the first image (see FIG.
The exposure level is determined when A) shown in 0 is taken in,
The exposure time, the aperture, the gain of the image pickup circuit, etc. are made constant, and are fixed while the subsequent (B, C, D) images are captured.

Here, in order to keep the exposure level constant, it is possible to keep the total exposure level unchanged even if the exposure time, aperture, gain, etc. are changed. It is constant.

Also, making the exposure time and the aperture constant has their own effects. When the exposure time is changed and when the control is performed to keep the exposure interval constant in the third embodiment, the exposure interval must be defined in the middle of each exposure period, which requires very complicated control. By making the exposure time constant, the exposure interval can be defined by the exposure start time as shown in FIG. 21, and the control of making the exposure interval constant becomes easy (it can also be defined by the exposure end time).

With respect to the aperture, when the aperture changes, the degree of blurring depending on the depth of each divided image also changes, so the joints of the images become unnatural. Therefore, the aperture is also set to a constant value.

The exposure level can also be determined based on the information of the entire screen by performing a prescan of the mirror before photographing. This takes more time, but more accurate exposure level determination is possible.

In the third embodiment in which the exposure interval is constant as described above, since the previous movement amount can be used as it is in each bonding process, the scale of hardware and software is reduced, but the actual signal processing time is reduced. Even if the length is short, there is a drawback that the shooting time becomes long.

Therefore, as a fifth embodiment, an image pickup apparatus will be described in which the exposure interval is not fixed and the next exposure is started as soon as the signal processing is completed, and the predicted position is changed according to the length of the exposure interval. FIG. 22 shows a time chart of the fifth embodiment. Here, the description of each symbol is the same as that of the third embodiment (FIG. 21), and thus will be omitted.

When each signal processing is completed, the exposure of the next divided image is started in synchronization with the next VD. Therefore, the exposure interval is t _R1 ≠ t _R2 ≠ t _R3 . Next, how to obtain the predicted position at the time of bonding will be described.

When the exposure interval changes from the previous time, it can be considered that the movement amount due to camera shake increases or decreases substantially in proportion to the exposure interval. Therefore, the predicted position at the time of bonding can be expressed by the following formula.

When the divided images B and C are combined, α1x = A1x + (previous S1x) × t _R2 / t _R1 ... S1 α1y = A1y + (previous S1y) × t _R2 / t _R1 ... S2 α2x = A2x + (previous time) S2x) × t _R2 / t _R1 ... S3 α2y = A2y + (previous S2y) × t _R2 / t _R1 ... S4. When the divided images C and D are combined, α1x = A1x + (previous S1x) × t _R3 / t _R2 ... S5 α1y = A1y + (previous S1y) × t _R3 / t _R2 ... S6 α2x = A2x + (previous S2x) × t _R3 / t _R2 ... S7 α2y = A2y + (previous S2y) × t _R3 / t _R2 ... S8.

With this image pickup apparatus, the photographing time can be shortened while performing the predicting operation. Although the description has been given based on the above embodiment, the present invention also includes the following inventions. (1) Image pickup means for obtaining a plurality of partial images by dividing an image pickup target area so as to have an area that overlaps with another screen at least on one side of the screen, and a pasted image side with the partial images Set a reference block including an arbitrary reference point in the overlapping area of, and search for a corresponding block that matches the reference block or a corresponding point that matches the reference point in the overlapping area on the stitched image side. , A reference block and a corresponding block, or a reference point and a corresponding point are matched, the image pasting means for pasting a partial image and the above-mentioned reference that is met at least when pasted by the image pasting means. Information based on each position of the block or reference point and corresponding block or corresponding point is retained as prediction information, and the next partial image is pasted In this case, a pasting position is predicted based on the prediction information, and a pasting control unit for instructing the pasting position to the image pasting unit is included.

Therefore, the range and position of the search area can be easily set, and the bonding position can be determined by the search in the area. Therefore, the bonding process time is shortened and the bonding process accuracy is improved. (2) The image pickup apparatus according to (1), wherein the image combining means includes means for combining all the partial images for forming one image and then generating as one image.

Therefore, it is considered that the amount of movement of the image due to the bonding processing shows a value close to the result of the processing previously performed, and by performing prediction based on this, the bonding processing with high speed and high accuracy is realized. . (3) The image pickup device according to (1), further comprising a shake detection unit that detects a shake amount when a partial image is picked up, and the information includes shake information based on the shake amount. apparatus.

Therefore, the displacement between the partial images is almost caused by camera shake. By detecting the camera shake amount and using it for prediction information, a high-speed and highly accurate stitching process is realized. (4) The image pickup device includes error information storage means for storing error information based on inherent mechanical and / or electrical error of a member forming the image pickup device, and the information includes the error information. The imaging device according to (1) above, which includes information.

Therefore, by storing the mechanical error that occurs in the same amount each time by using the storage means, high speed and high accuracy of the bonding process can be realized. (5) The image pickup device according to (4), wherein the image pickup device has a learning unit that changes the error information in the error information storage unit according to a usage pattern.

Therefore, the mechanical error and the habit of the user are learned, and the high speed and the high accuracy of the bonding process are realized by utilizing them. (6) The image stitching unit searches the corresponding block matching the reference block provided in the overlapping region of the images to be stitched, based on the prediction information, the stitched image overlapping region. The image pickup device according to any one of (1) to (5), further comprising means for setting a search area having a predetermined size therein.

Therefore, high speed is realized by using the prediction information and surely searching for a small search area in the corresponding block. (7) The laminating means further comprises means for laminating images based on the prediction information when the reference block provided in the overlapping area of the images to be pasted does not exist. The imaging device according to any one of (1) to (6).

Therefore, even if there is no image in the overlapping area, the images can be pasted together. (8) The image pickup apparatus further includes partial image storage means having a capacity capable of storing image data corresponding to at least one of the partial images and image data corresponding to the overlapping area. To (7) the imaging device.

Therefore, the memory capacity is small, and the size and cost of the device can be reduced. (9) The image pickup device according to any one of (1) to (7), further including an all-image storage unit having a capacity capable of storing all image data corresponding to the image pickup target area.

Therefore, the order of pasting partial images can be freely set. Also, the result of pasting the partial images captured later in time can be used as the prediction information. (10) The imaging device includes an all-image storage unit having a capacity capable of storing all image data corresponding to the imaging target region, and the bonding unit performs a bonding process a plurality of times, A unit for generating a high-definition image is further included, and when the reference block does not exist in the overlapping area of the image to be pasted, the pasting position of the pasting process different from the pasting process is used as the prediction information. The imaging device according to (1) above, including means for use.

Therefore, even if there is no image in the overlapping area, the pasting can be performed. (11) When the bonding processing means performs the bonding process without the reference block that should be provided in the overlapping area of the image to be bonded and without using the prediction information. The imaging device according to any one of (1) to (6), including means for performing a bonding process on the bonded image and the bonded image at a predetermined position.

Therefore, by bonding at a predetermined position,
The entire image is always output. (12) Image pickup means for obtaining a plurality of partial images by dividing an image pickup target area so that at least one end side of the screen has an area overlapping with another screen, and a pasted image side with the partial images Set a reference block including an arbitrary reference point in the overlapping area of, and search for a corresponding block that matches the reference block or a corresponding point that matches the reference point in the overlapping area on the stitched image side. It is impossible or unsuitable to perform the combining by the image combining means for matching the reference block and the corresponding block, or the reference point and the corresponding point and combining the partial images, and the image combining means. In some cases, when performing the pasting process without using the prediction information, a unit for performing the pasting process of the pasting image and the pasted image at a predetermined position. An image pickup apparatus comprising:

Therefore, when the image pasting means cannot be pasted together or when the use is inappropriate, an appropriate whole image output can be obtained. (13) The image pickup device according to any one of (9) to (10), wherein the all-image storage unit is a storage medium that is detachably attached to the image pickup device.

Therefore, by utilizing a part of the main storage medium for storing images, it is not necessary to mount a memory in the image pickup apparatus. (14) The image pickup apparatus further includes an operating unit, and the image pasting unit includes a unit that generates a high-definition image by performing a pasting process a plurality of times, and is based on the operation of the operating unit. The number of times of the above-mentioned plurality of bonding processes,
The imaging device according to (1) above, which can be set within the range.

Therefore, the subject can be photographed with the aspect ratio and the resolution desired by the user. (15) The image pickup device includes a unit that obtains a high-definition image by performing a plurality of stitching processes as the image stitching unit, and makes each exposure interval in image pickup of each of the partial images substantially constant. The image pickup apparatus according to (1), further including exposure interval control means for

Therefore, camera shake and the like can be considered as a temporally linear component. In that case, if the exposure interval is constant, the prediction becomes easy. (16) The image pickup apparatus according to (1), wherein the exposure interval is set based on the longest period of each signal processing period of the image pickup signal by the image pickup for each partial image.

Therefore, if the exposure interval is set based on the longest time, the signal processing period can be surely set. (17) The image pickup device according to (1), wherein the image pickup device includes an exposure level control unit that keeps an exposure level related to image pickup for each of the partial images constant.

Therefore, a good laminated image can be obtained with the same exposure level. (18) The image pickup device according to (1), wherein the image pickup device has an exposure control unit for keeping a constant exposure time for image pickup for each of the partial images.

Therefore, the exposure interval and the exposure level can be easily controlled. (19) The image pickup device according to (1), wherein the image pickup device includes an exposure control unit that keeps an aperture value for image pickup for each of the partial images constant.

Therefore, a good laminated image can be obtained with the same exposure level. (20) The exposure start timing for the image pickup apparatus to set the exposure start timing for the image pickup of the partial image based on the processing end timing of the image pickup signal by the image pickup of another partial image performed immediately before the image pickup. The image pickup apparatus according to (1) above, which has setting means. Therefore, the exposure start timing can be set as early as possible and the total shooting time can be shortened.

[0163]

As described above in detail, according to the present invention, an image is divided and photographed, and when the partial image is restored to the whole image, the time for the bonding process is shortened and the accuracy is improved, so that a high image quality is achieved. An imaging device that captures an image can be provided.

[Brief description of drawings]

FIG. 1 is a diagram showing a schematic configuration of an image pickup apparatus as a first embodiment according to the present invention.

FIG. 2 is a diagram showing a subject image divided into partial images stored in a memory according to the first embodiment.

FIG. 3 is a diagram showing a configuration example of an image pickup apparatus as a second embodiment according to the present invention.

FIG. 4 is a diagram showing a subject image divided into partial images stored in a memory according to a second embodiment.

FIG. 5 is a diagram for explaining a laminating process.

FIG. 6 is a diagram showing one subject image restored by a process of combining partial images.

FIG. 7 is a diagram showing a search area for performing a normal search.

FIG. 8 is a diagram showing an example of performing a search by a position based on prediction information according to the present embodiment.

FIG. 9 is a flowchart for explaining a bonding process using predicted positions according to this embodiment.

FIG. 10 is a movement amount in the bonding process of the present embodiment,
It is a flow chart for explaining calculation of the amount of rotation.

FIG. 11 is a diagram showing a search area for performing a search according to this embodiment.

FIG. 12 is a diagram showing positions (coordinates) of reference blocks and blocks for deriving a correlation.

FIG. 13 is a flowchart for explaining a search process according to this embodiment.

FIG. 14 is a flowchart for explaining a movement amount and a rotation amount according to the present embodiment.

FIG. 15 is a flowchart for explaining a method of pasting at a predicted position according to the present embodiment.

FIG. 16 is a flowchart for explaining an example in which there is a capacity capable of storing all partial images to be captured.

FIG. 17 is a flowchart for explaining a case where a movement amount and a rotation amount cannot be calculated in FIG.

FIG. 18 is a flowchart for explaining calculation of a movement amount and a rotation amount in FIG.

FIG. 19 is a flowchart for explaining calculation of a movement amount and a rotation amount in FIG.

FIG. 20 is a diagram showing images that are sequentially captured as the mirror rotates.

FIG. 21 is a time chart for explaining exposure timing.

FIG. 22 is a time chart for explaining the fifth embodiment.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Subject image (subject), 2 ... Mirror, 3 ... Mirror drive part, 4 ... Imaging element, 5 ... Photographing lens, 6 ... Aperture, 7 ... Imaging system control part, 8 ... Imaging circuit, 9 ... Imaging device drive part 10,
... A / D converter, 11 ... Memory, 12 ... Memory driver,
13 ... Pattern matching part, 14 ... Memory card, 1
5 ... Card drive unit, 16 ... Printer, 17 ... Printer drive unit, 18 ... System control unit, 19 ... Sync signal generation unit,
20 ... EEPROM, 21 ... hand shake sensor, 22 ... display section, 23 ... input switch, 24 ... sequence processing section, 2
5 ... Lamination processing section.

 ─────────────────────────────────────────────────── ─── Continued Front Page (72) Akio Terane 2-43-2 Hatagaya, Shibuya-ku, Tokyo Inside Olympus Optical Co., Ltd. (72) Kazuya Kobayashi 2-43 Hatagaya, Shibuya-ku, Tokyo No. 2 Olympus Optical Co., Ltd.

Claims (20)

[Claims] 1. An image pickup means for obtaining a plurality of partial images by dividing an image pickup target area so that at least one end side of the screen has an area overlapping with another screen, and the partial images are pasted together. Set a reference block including an arbitrary reference point in the overlapping area on the image side, and set a corresponding block that matches the reference block or a corresponding point that matches the reference point in the overlapping area on the laminated image side. By searching, matching the reference block and the corresponding block, or matching the reference point and the corresponding point, and combining the partial images, the image combining means and the image combining means match at least the time before the previous combining. The reference block or reference point and information based on each position of the corresponding block or corresponding point is held as prediction information, and the next partial image During mating, the predict the adhesion position by the prediction information, an imaging apparatus characterized by comprising a cemented control means for instructing the adhesion position in the image laminating means. 2. The image pickup apparatus according to claim 1, wherein the image combining unit includes a unit that combines all partial images for forming one image and then generates the image as one image. 3. The image pickup apparatus further comprises a shake detection unit that detects a shake amount when a partial image is picked up, and the information includes shake information based on the shake amount. Imaging device. 4. The image pickup device comprises error information storage means for storing error information based on inherent mechanical and / or electrical error of a member forming the image pickup device, and the information includes: The image pickup apparatus according to claim 1, wherein the image pickup apparatus includes the error information. 5. The image pickup apparatus according to claim 4, wherein the image pickup apparatus has a learning unit that changes the error information in the error information storage unit according to a usage pattern. 6. The image pasting means, based on the prediction information, seeks a corresponding block matching the reference block provided in an overlapping area of the pasted images, and the pasted image is pasted based on the prediction information. The image pickup apparatus according to claim 1, further comprising a unit that sets a search area having a predetermined size in the image overlapping region. 7. The pasting means further comprises means for pasting images based on the prediction information when the reference block provided in the overlapping area of the pasted images does not exist. The image pickup apparatus according to claim 1, wherein 8. The image pickup apparatus further comprises partial image storage means having a capacity capable of storing image data corresponding to at least one of the partial images and image data corresponding to the overlapping area. The imaging device according to any one of claims 1 to 7. 9. The image pickup apparatus according to claim 1, wherein the image pickup apparatus further comprises an entire image storage unit having a capacity capable of storing all image data corresponding to the image pickup target area. 10. The image pickup device includes an all-image storage unit having a capacity capable of storing all image data corresponding to the image pickup target region, and the joining unit performs a plurality of joining processes. According to the above, a unit for generating a high-definition image is included, and when the reference block does not exist in the overlapping area of the images to be pasted, the pasting position of the pasting process different from the pasting process is predicted. The image pickup apparatus according to claim 1, further comprising means used as information. 11. The pasting processing unit performs the pasting process without the reference block to be provided in an overlapping region of the pasted images and without using the prediction information. The imaging device according to claim 1, further comprising a unit that performs a bonding process on the bonded image and the image to be bonded at a predetermined position. 12. An image pickup means for obtaining a plurality of partial images by dividing an image pickup target area so that at least one end of the screen has an area overlapping with another screen, and the partial images are pasted together. Set a reference block including an arbitrary reference point in the overlapping area on the image side, and set a corresponding block that matches the reference block or a corresponding point that matches the reference point in the overlapping area on the laminated image side. It has an image pasting unit that searches and matches the reference block and the corresponding block or the reference point and the corresponding point and performs a stitching process on the partial images. It is impossible to perform the stitching by the image stitching unit. Or, when it is inappropriate and when performing the bonding process without using the prediction information, the bonding image and the bonding image are bonded at a predetermined position. Imaging apparatus characterized by comprising means for management. 13. The image pickup apparatus according to claim 9, wherein the all-image storage means is a storage medium that is detachably attached to the image pickup apparatus. 14. The image pickup apparatus further includes an operating unit, and the image combining unit includes a unit that generates a high-definition image by performing a combining process a plurality of times. The image pickup apparatus according to claim 1, wherein the number of times of the plurality of pasting processes can be set based on the range. 15. The image pickup device performs a plurality of bonding processes as the image bonding means,
The image pickup apparatus according to claim 1, further comprising an exposure interval control unit that includes a unit that obtains a high-definition image, and that makes each exposure interval in the image pickup of each of the partial images substantially constant. 16. The exposure interval in the image pickup for each partial image is set based on the longest period of each signal processing period of the image pickup signal by the image pickup for each partial image. The imaging device described. 17. The image pickup apparatus according to claim 1, wherein the image pickup apparatus has an exposure level control unit for keeping a constant exposure level for image pickup for each of the partial images. 18. The image pickup apparatus according to claim 1, wherein the image pickup apparatus has an exposure control unit for keeping a constant exposure time for image pickup for each of the partial images. 19. The image pickup apparatus according to claim 1, wherein the image pickup apparatus has an exposure control unit for keeping a constant aperture value for image pickup for each of the partial images. 20. The exposure apparatus sets an exposure start timing for capturing an image of the partial image based on an end timing of processing an image capturing signal by capturing another partial image immediately before the image capturing. The image pickup apparatus according to claim 1, further comprising start timing setting means.