JP6717354B2 - Image processing apparatus, image processing method and program - Google Patents

Description

The present invention relates to an image processing device.

Conventionally, there is a panoramic image pickup function as one function of an image pickup apparatus.
The same function moves the digital camera in the horizontal direction while maintaining the state in which the user has pressed the shutter switch, the imaging device executes a plurality of imaging processes during that time, and each of the plurality of imaging processes is performed. This is a function of generating image data of a panoramic image by synthesizing a plurality of resulting image data in the horizontal direction (horizontal direction).

In Patent Document 1, a digital camera causes a horizontally long frame to be superimposed and displayed on an image which is being currently captured, and a bar extending horizontally in the horizontally long frame is used to capture a panoramic image. A technique of presenting the progress state (hereinafter, referred to as "technology of Patent Document 1") is disclosed.

JP, 06-303562, A

However, in the case of the technique of Patent Document 1, the user can confirm the history of image capturing so far during image capturing, for example, what kind of image has been captured since the start of panoramic image capturing. Can not.
Therefore, the panoramic image recorded as image data after the end of the panoramic imaging may lack a necessary image, or conversely may include an extra image. In order to obtain a panoramic image, it was necessary to repeat the imaging operation many times.

The present invention has been made in view of such a situation, and an object of the present invention is to allow a user to easily visually recognize the history of imaging so far during panoramic imaging.

In order to achieve the above object, an image processing device according to one embodiment of the present invention is
An image pickup means for continuously picking up an image of a subject and sequentially outputting image data;
An image including a first image synthesizing unit that synthesizes a plurality of image data sequentially output by the image pickup unit while the subject is continuously imaged while the image pickup range of the image pickup unit is moved in a substantially predetermined direction. Synthesizing means ,
Determination means for determining whether or not the image range of the combination area to be combined by the first image combination means has reached a limit combination range;
If the subject is continuously being captured by the imaging means, an image corresponding to the sequentially output image data by the imaging unit, the synthesis corresponding to the image data which are sequentially combined by the first image synthesizing means Display control means for displaying the image of the area on the display means at the same time,
While the image corresponding to the image data sequentially output by the display control unit and the image of the combination area are displayed on the display unit, an image pickup for combining the image data by the first image combination unit Is ended at the timing when it is determined by the discriminating means that the image range of the composite area has reached the limit compositing range or by the discriminating means that the image range of the composite area has reached the limit compositing range. Instructing means to instruct at a timing earlier than,
Equipped with
The image synthesizing means includes a second image synthesizing means for synthesizing a plurality of image data captured by the instruction means until the instruction to end the image capturing,
It is characterized by further comprising recording control means for causing the recording means to record the combined image data combined by the second image combining means when the instruction means instructs the end of the imaging.

According to the present invention, the user can easily visually recognize the history of imaging so far during panoramic imaging.

It is a block diagram which shows the hardware constitutions of the digital camera as one Embodiment of the imaging device which concerns on this invention. FIG. 2 is a functional block diagram showing a functional configuration for the digital camera of FIG. 1 to perform an imaging process. FIG. 3 is a diagram illustrating an image capturing operation when a normal image capturing mode and a panoramic image capturing mode are selected as operation modes of the digital camera in FIG. 1. It is a figure which shows an example of the panoramic image produced|generated by the panoramic imaging mode shown in FIG. FIG. 2 is a back view showing an external configuration of the digital camera of FIG. 1 in which a panoramic image is displayed. FIG. 3 is a diagram illustrating a combining process of the image processing unit in FIG. 2. It is a flowchart which shows an example of the flow of an imaging process. 8 is a flowchart showing a detailed flow of a panoramic image pickup process in the image pickup process of FIG. 7.

Hereinafter, embodiments according to the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a hardware configuration of a digital camera 1 as an embodiment of an image pickup apparatus according to the present invention.

The digital camera 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an optical system 15, an imaging unit 16, and an image processing unit 17. The storage unit 18, the display unit 19, the operation unit 20, the communication unit 21, the angular velocity sensor 22, and the drive 23 are provided.

The CPU 11 executes various processes according to the program stored in the ROM 12 or the program loaded from the storage unit 18 into the RAM 13.
The ROM 12 also appropriately stores data necessary for the CPU 11 to execute various processes.

For example, in the present embodiment, a program that realizes each function of the imaging control unit 51 to the display control unit 54 of FIG. 2 described later is stored in the ROM 12 or the storage unit 18. Therefore, the CPU 11 executes the processes according to these programs, so that each function of the image pickup control unit 51 to the display control unit 54 of FIG.
Note that at least some of the functions of the imaging control unit 51 to the display control unit 54 in FIG. 2 described later can be transferred to the image processing unit 17.

The CPU 11, ROM 12, and RAM 13 are connected to each other via a bus 14. Also connected to the bus 14 are an optical system 15, an image pickup unit 16, an image processing unit 17, a storage unit 18, a display unit 19, an operation unit 20, a communication unit 21, an angular velocity sensor 22, and a drive 23.

The optical system 15 is composed of a lens that condenses light, such as a focus lens and a zoom lens, for photographing a subject. The focus lens is a lens that forms a subject image on the light receiving surface of the image pickup device of the image pickup unit 16. The zoom lens is a lens that freely changes the focal length within a certain range. The optical system 15 is also provided with peripheral devices for adjusting the focus, exposure, etc., if necessary.

The imaging unit 16 is composed of a photoelectric conversion element, an AFE (Analog Front End), and the like. The photoelectric conversion element is composed of, for example, a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element. The photoelectric conversion element photoelectrically converts (captures) an optical signal of a subject image that is incident and accumulated during the fixed time, and sequentially supplies an analog electric signal obtained as a result to the AFE.
The AFE performs various signal processing such as A/D (Analog/Digital) conversion processing on the analog electric signal, and outputs a digital signal obtained as a result as an output signal of the imaging unit 16.
Note that, hereinafter, the output signal of the imaging unit 16 is referred to as “image data of captured image”. Therefore, the image data of the captured image is output from the image capturing unit 16 and appropriately supplied to the image processing unit 17 and the like.

The image processing unit 17 includes a DSP (Digital Signal Processor), a VRAM (Video Random Access Memory), and the like.
The image processing unit 17 cooperates with the CPU 11 to perform image processing such as noise reduction, white balance, and camera shake correction on the image data of the captured image input from the image capturing unit 16.
Here, the image data of the captured image input from the image capturing unit 16 at regular intervals will be referred to as “frame image data” below. In this embodiment, the frame is adopted as a processing unit. That is, the image processing unit 17 outputs various types of image processing to the image data of the frame supplied from the imaging unit 16.

The storage unit 18 is configured by a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores the image data of the frame output from the image processing unit 17, the image data of a panoramic intermediate image described later, and the like. The storage unit 18 also stores various types of data necessary for various types of image processing.

The display unit 19 is configured as a flat display panel including an LCD (Liquid Crystal Device) and an LCD driving unit, for example. The display unit 19 displays an image represented by the image data supplied from the storage unit 18 or the like, for example, a through image described below in frame units.

In addition to the shutter switch 41, the operation unit 20 has a plurality of switches such as a power switch, an imaging mode switch, and a reproduction switch, which are not shown. When a predetermined switch among the plurality of switches is pressed, the operation unit 20 supplies the CPU 11 with a command assigned to the predetermined switch.

The communication unit 21 controls communication with another device (not shown) via a network including the Internet.

The angular velocity sensor 22 is composed of a gyro or the like, detects the angular displacement amount of the digital camera 1, and supplies a digital signal indicating the detection result (hereinafter, simply referred to as “angular displacement amount”) to the CPU 11. The angular velocity sensor 22 also has a function of detecting the direction of the earth's magnetism, and accordingly, the angular velocity sensor 22 also exerts the function of the azimuth sensor as needed.

A removable medium 31 including a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 23. Then, the program read from the removable medium 31 is installed in the storage unit 18 as needed. Further, the removable medium 31 can also store various data such as image data stored in the storage unit 18 similarly to the storage unit 18.

2 is a series of processes (hereinafter, referred to as “imaging process”) of imaging a subject and recording image data of a resulting captured image in the removable medium 31 among the processes executed by the digital camera 1 of FIG. Is a functional block diagram showing a functional configuration for executing (referred to as).

The CPU 11 is provided with an imaging control unit 51 that controls execution of imaging processing.
For example, the imaging control unit 51 can selectively switch the normal imaging mode and the panoramic imaging mode as the operation mode of the digital camera 1 and execute the processing according to the switched operation mode.

FIG. 3 is a diagram illustrating an image capturing operation when the normal image capturing mode and the panoramic image capturing mode are selected as the operation modes of the digital camera 1 of FIG.
Specifically, FIG. 3A is a diagram illustrating an image capturing operation in the normal image capturing mode. FIG. 3B is a diagram illustrating an image capturing operation in the panoramic image capturing mode.
In each of FIG. 3(A) and FIG. 3(B), the picture in the back of the digital camera 1 shows the state of the real world including the subject of the digital camera 1. Further, the vertical dotted lines shown in FIG. 3B indicate the respective positions a, b, c in the moving direction of the digital camera 1. The moving direction of the digital camera 1 refers to the direction in which the optical axis of the digital camera 1 moves when the user changes the image pickup direction (angle) of the digital camera 1 about the body of the user.

The normal imaging mode refers to an operation mode for imaging an image of a size (resolution) corresponding to the angle of view of the digital camera 1.
In the normal imaging mode, as shown in FIG. 3A, the user presses the shutter switch 41 of the operation unit 20 to the lower limit with the digital camera 1 fixed. The operation of pressing the shutter switch 41 to the lower limit in this way is hereinafter referred to as "full-press operation" or simply "full-press".
The imaging control unit 51 controls execution of a series of processes until the image data of the frame output from the image processing unit 17 immediately after the full-press operation is performed is recorded on the removable medium 31 as a recording target.
Hereinafter, a series of processes executed under the control of the imaging control unit 51 in the normal imaging mode will be referred to as “normal imaging process”.

On the other hand, the panoramic image capturing mode refers to an operation mode when capturing a panoramic image.
In the panoramic imaging mode, as shown in FIG. 3B, the user moves the digital camera 1 in the direction of the black arrow in the figure while maintaining the full-press operation of the shutter switch 41.
The imaging control unit 51 controls the image synthesizing unit 52 while the full-press operation is maintained, and every time the angular displacement amount from the angular velocity sensor 22 reaches a constant value, outputs from the image processing unit 17 immediately thereafter. The storage of the image data of the selected frame in the storage unit 18 is repeated.
After that, the user gives an instruction to end the panoramic imaging by releasing the full-press operation, that is, releasing the finger or the like from the shutter switch 41 (hereinafter, such an operation is referred to as a “release operation”). ..
When the end of panoramic imaging is instructed, the imaging control unit 51 horizontally combines the image data of the plurality of frames stored in the storage unit 18 so far in the stored order to generate a panoramic image. Generate image data.
Then, the imaging control unit 51 records the image data of the panoramic image on the removable medium 31 as a recording target.
As described above, the imaging control unit 51 controls a series of processes in the panoramic imaging mode, in which image data of a panoramic image is generated and is recorded on the removable medium 31 as a recording target.
Hereinafter, a series of processes executed under the control of the imaging control unit 51 in the panoramic imaging mode will be referred to as “panoramic imaging process”.

FIG. 4 shows image data of a panoramic image obtained as a result of the panoramic image capturing process.
That is, in the panoramic imaging mode, when the imaging operation as shown in FIG. 3B is performed, the imaging control unit 51 controls the execution of the panoramic imaging process. As a result, image data of the panoramic image P3 as shown in FIG. 4 is generated and recorded in the removable medium 31.

In the present embodiment, the imaging control unit 51 of FIG. 2 further executes a through image capturing process and a through image displaying process regardless of the normal image capturing mode and the panoramic image capturing mode.
That is, the user presses the shutter switch 41 of the operation unit 20 halfway (a predetermined position that does not reach the lower limit) to cause the digital camera 1 to execute AF (Auto Focus) processing or the like before performing a full-press operation. The operation can be performed. The operation of pressing the shutter switch 41 halfway (a predetermined position that does not reach the lower limit) in this manner is hereinafter referred to as “half-press operation” or simply “half-press”.
When the half-press operation is performed, the imaging control unit 51 controls the imaging unit 16 and the image processing unit 17 to continue the imaging operation by the imaging unit 16. Then, the image capturing control unit 51 stores the image data of the frames sequentially output from the image processing unit 17 via the image capturing unit 16 in the memory (in this embodiment, while the image capturing operation by the image capturing unit 16 is continued). Part 18) for temporary storage. A series of control processes performed by the imaging control unit 51 as described above is the “through imaging process”.
Further, the imaging control unit 51 controls the display control unit 54 to sequentially read out each image data temporarily recorded in the memory (the storage unit 18 in the present embodiment) at the time of through-imaging, and to correspond to each frame. The images are sequentially displayed on the display unit 19. A series of control processes performed by the imaging control unit 51 as described above is the “through display process”. The frame image displayed on the display unit 19 by the through display process is hereinafter referred to as “through image”.

In the present embodiment, when the full-press operation is further performed while the panoramic image capturing mode is selected, an image showing a region captured up to the present in the panoramic image to be generated (hereinafter, “panorama intermediate image”). Is called) is displayed on the display unit 19 while being superimposed on the through image.

FIG. 5 is a rear view showing the external configuration of the digital camera 1 of FIG. 1 in which a panoramic image is displayed.
FIG. 5A shows the external configuration of the digital camera 1 when it has moved to the position a in FIG. 3B. FIG. 5B shows an external configuration of the digital camera 1 when it has moved to the position b in FIG. 3B. FIG. 5C shows the external configuration of the digital camera 1 when it has moved to the position c in FIG. 3B.
That is, the transition of the display state of the digital camera 1 when the user sequentially moves the digital camera 1 to the position a to the position c in the panoramic image capturing mode is shown in FIGS. 5A to 5C. Has been done.
Specifically, when the digital camera 1 sequentially moves to the position a, the position b, and the position c, each of the through images PA1, PB1, and PC1 on the display unit 19 corresponds to each of the positions a, b, and c. Displayed sequentially. In the present embodiment, each of the intermediate images PA2, PB2, and PC2 is displayed so as to be superimposed on each of such through images PA1, PB1, and PC1.
Since the user can operate the panoramic image pickup while visually checking the panoramic image PA2, PB2, and PC2, the user can easily and accurately visually recognize the image pickup history from the start of the panoramic image pickup to the present. You can As a result, the user can easily and easily obtain a desired panoramic image without repeating the imaging operation many times.

As shown in FIG. 2, the CPU 11 of the digital camera 1 further includes an image combining unit 52, an image reducing unit 53, and A display controller 54 is provided.
Note that, as described above, the respective functions of the image synthesizing unit 52, the image reducing unit 53, and the display control unit 54 do not need to be mounted on the CPU 11 as in the present embodiment, and among these respective functions. It is also possible to transfer at least a part to the image processing unit 17.

The image synthesizing unit 52 executes a process of sequentially synthesizing the image data of the frames sequentially supplied from the image processing unit 17 in the horizontal direction in the supplied order (hereinafter, referred to as “image synthesizing process”). .. As a result of the image synthesizing process, image data of the intermediate panoramic image is obtained and supplied to the image reducing unit 53.
Specifically, for example, the imaging control unit 51 monitors the detection result of the angular velocity sensor 22 and issues an acquisition command to the image synthesis unit 52 every time the cumulative value of the angular displacement amount reaches a constant value. The image composition unit 52 acquires the image data of the frame supplied from the image processing unit 17 immediately after receiving the acquisition command as a compositing target. That is, the image synthesizing unit 52 sequentially acquires the image data of the frames to be synthesized, each time an acquisition command is issued from the imaging control unit 51 after the shutter switch 41 is fully pressed.
Then, the image combining unit 52 sequentially combines the image data of a plurality of frames acquired as a combining target from the time the full-press operation of the shutter switch 41 is started to the current in the acquired order in the horizontal direction. ..
As a result of such an image synthesizing process, the image data of the intermediate panoramic image is obtained by the image synthesizing unit 52 and supplied to the image reducing unit 53.

Here, the image data to be subjected to the image combination processing is not particularly limited, and may be, for example, image data of each of a plurality of frames acquired from the start of the full-press operation of the shutter switch 41 to the present. It may be. However, in the present embodiment, the image data of the intermediate panoramic image generated in the previous combining process and the image data of the frame acquired as the combining target immediately before are adopted as the targets of the combining process.
That is, in the present embodiment, the size (resolution) of the intermediate panoramic image generated by the image combining unit 52 is the same fixed size as the maximum size of the panoramic image that can be finally generated.
Therefore, during panorama imaging, that is, while the digital camera 1 is moving, even if the image data of some frames acquired so far are combined in the horizontal direction, the size in the horizontal direction is less than the maximum size. Instead, some areas cannot be filled with the effective image of the frame. In the present embodiment, such an area is an area to which a predetermined color (for example, black in the example of FIG. 5) is uniformly applied. Such a region, that is, a region in which all pixel values are constant (for example, 0 which indicates black is constant) is hereinafter referred to as a “blank region”.
Therefore, in the present embodiment, the image combining unit 52 uses the image data of the previous panoramic image and the image data of the frame acquired immediately before as the compositing target in the blank area of the previous panoramic image to be combined. The image compositing process is executed to overwrite the frame.

FIG. 6 is a diagram for explaining the combining process of the image combining unit 52 of FIG.
In this embodiment, the storage area 101 as shown in FIG. 6 is provided in the storage unit 18, for example.
The storage area 101 is divided into N frame storage areas 101-1 to 101-N. Here, N is an arbitrary integer value of 2 or more. However, the size of the frame storage areas 101-1 to 101-N in the horizontal direction (the direction of the x axis in the figure) is the horizontal size (resolution) of the intermediate panoramic image, that is, the horizontal size of the panoramic image that can be finally generated. It is the same as the maximum size in the direction. In addition, N matches the maximum number of image data of frames that can be acquired by the image combining unit 52 as a combining target in one panorama image capturing process.
That is, when the K-th acquisition command is issued after the shutter switch 41 is fully pressed to start the panoramic imaging process, at least one of the image data of the frames acquired as the composition target by the image composition unit 52. The set is stored in the frame storage area 101-K.
Here, the description of at least a part of the image data of the frame is that the size of the image data of the frame may be larger than the size of the frame storage area 101-K. In such a case, only image data corresponding to a predetermined part of the frame is stored in the frame storage area 101-K.

As described above, in this embodiment, the entire image data stored in the storage area 101 is the image data of the intermediate panoramic image. In the storage area 101, the same pixel value (for example, 0 indicating black) is uniformly stored in the initial state. That is, in the initial state, all the N frame storage areas 101-1 to 101-N are blank areas.
Therefore, at the timing acquired by the image combining unit 52 of the Kth frame to be combined, the frame storage areas 101-1 to 101-(K-1) among the N frame storage areas 101-1 to 101-N are acquired. ) Includes image data of an effective image. Here, the effective image is an image in which (at least a part) of each of the first to (K-1)th frames to be combined is combined in the horizontal direction (x-axis direction in FIG. 6) in that order. .. On the other hand, the frame storage areas 101-K to 101-N are blank areas.
Of the frame storage areas 101-K to 101-N, which are blank areas, at least a part of the image data of the frame acquired as the combining target for the Kth time is stored in the frame storage area 101-K. As a result, the entire data stored in the storage area 101 is generated as image data of the intermediate panoramic image.
After that, the image synthesizing unit 52 in FIG. 2 reads the entire data stored in the storage area 101 as image data of the intermediate panoramic image and supplies it to the image reducing unit 53.

The image reduction unit 53 performs reduction processing for reducing the size (resolution) on the image data each time the image data of the panoramic image is supplied from the image synthesis unit 52. Here, the reduction ratio can be arbitrarily set within a range in which the panoramic intermediate image after reduction is smaller than the through image (frame).
The image data of the intermediate panoramic image whose size (resolution) has been reduced in this way is supplied from the image reduction unit 53 to the display control unit 54.

The display control unit 54 controls the display unit 19 to display a frame represented by the image data temporarily stored in the storage unit 18 as a through image.
Further, during the panoramic image capturing process, the display control unit 54 displays the image represented by the reduced image data supplied from the image reduction unit 53, that is, the reduced panoramic intermediate image, on the through image. The control to be displayed on the unit 19 is executed.

After that, when the user's finger or the like is released from the shutter switch 41, that is, when the release operation is performed, the imaging control unit 51 executes control to record the image data to be recorded on the removable medium 31.
Specifically, in the normal image pickup mode, the image pickup control unit 51 executes control to record the image data of the frame output from the image processing unit 17 in the removable medium 31 as the image data to be recorded.
On the other hand, in the panoramic image pickup mode, the image pickup control unit 51 executes control to record the image data of the panoramic image on the removable medium 31 as the image data of the recording target via the image combining unit 52.
For example, the image composition unit 52 reads out the image data of the effective image portion other than the blank area from the data stored in the storage area 101 (FIG. 6), and appropriately performs image processing such as reduction processing and enlargement processing. Give. As a result, image data of the panoramic image to be recorded is generated. The image data of the panoramic image generated in this way is supplied to the drive 23 and recorded in the removable medium 31 as image data to be recorded.

Next, with reference to FIG. 7, an image capturing process executed by the digital camera 1 having such a functional configuration will be described.

FIG. 7 is a flowchart showing an example of the flow of the imaging process.
In the present embodiment, the imaging process starts when a power source (not shown) of the digital camera 1 is turned on.

In step S1, the imaging control unit 51 of FIG. 2 executes an operation detection process and an initial setting process.
The operation detection process is a process of detecting the state of each switch of the operation unit 20. The imaging control unit 51 can detect whether the normal imaging mode or the panoramic imaging mode is set as the operation mode by executing the operation detection process.
Further, as one of the initial setting processes of the present embodiment, a process of setting a constant value of the angular displacement amount and a threshold value (for example, 360 degrees) that is the maximum limit of the angular displacement amount is adopted.
Specifically, the constant value of the angular displacement amount and the threshold value (for example, 360 degrees) that is the maximum limit of the angular displacement amount are stored in advance in the ROM 12 of FIG. 1, are read from the ROM 12, and are written in the RAM 13. Be set. The constant value of the angular displacement amount is used in the determination process of step S35 of FIG. 8 described later. On the other hand, a threshold value (for example, 360 degrees) that is the maximum limit of the amount of angular displacement is used in the determination process of step S45 in FIG.
In addition, in the present embodiment, as shown in steps S34 and S40 of FIG. 8 to be described later, the angular displacement amounts detected by the angular velocity sensor 22 are cumulatively added, and the cumulative angular displacement amount and the total angular displacement amount as the cumulative addition value and The amount of angular displacement (the difference between the two will be described later) is stored in the RAM 13. Therefore, the process of resetting these accumulated angular displacement amount and total angular displacement amount to 0 is adopted as one of the initial setting processes of the present embodiment. The cumulative angular displacement amount is compared with the above-described constant value in the determination process of step S35 of FIG. 8 described later. On the other hand, the total angular displacement amount is compared with the above-described constant value in the determination process of step S45 of FIG. 8 described later.
Furthermore, a process of resetting the error flag to 0 is adopted as one of the initial setting processes of this embodiment. The error flag is a flag that is set to 1 when an error occurs during the panorama image capturing process (see step S44 in FIG. 8 described later).

In step S2, the imaging control unit 51 determines whether or not the shutter switch 41 has been half pressed.
If the shutter switch 41 is not half pressed, it is determined as NO in step S2, and the process proceeds to step S12.

In step S12, the imaging control unit 51 determines whether or not an instruction to end the process has been issued.
The processing end instruction is not particularly limited, but in the present embodiment, it is assumed that a notification that the power source (not shown) of the digital camera 1 has been turned off is adopted.
Therefore, in the present embodiment, when the power is turned off and the fact is notified to the imaging control unit 51, YES is determined in step S12, and the entire imaging process ends.
On the other hand, when the power supply is in the on state, no notification that the power supply is in the off state is made. Therefore, it is determined as NO in step S12, the process is returned to step S2, and the subsequent steps are performed. The process is repeated. That is, in the present embodiment, as long as the power is maintained in the ON state, the loop processing of step S2: NO and step S12: NO is repeatedly executed until the shutter switch 41 is half pressed, and the imaging processing is It will be in a standby state.

If the shutter switch 41 is half-pressed in the standby state, YES is determined in step S2, and the process proceeds to step S3.

In step S3, the imaging control unit 51 starts the through image capturing process and the through display process described above.
As a result, the through image is displayed on the display unit 19. In addition, in the present embodiment, it is assumed that the through image is continuously displayed on the display unit 19 until the normal image capturing process of step S8 or the panoramic image capturing process of step S9 described later is completed.

In step S4, the image pickup unit 16 is controlled to execute so-called AF processing.

In step S5, the imaging control unit 51 determines whether or not the shutter switch 41 has been fully pressed.
If the shutter switch 41 is not fully pressed, it is determined as NO in step S5. In this case, the process is returned to step S4, and the subsequent processes are repeated. That is, in the present embodiment, the loop processing of step S4 and step S5: NO is repeatedly executed until the shutter switch 41 is fully pressed, and the AF processing is executed each time.

After that, when the shutter switch 41 is fully pressed, YES is determined in step S5, and the process proceeds to step S6.
In step S6, the imaging control unit 51 determines whether the currently set imaging mode is the panoramic imaging mode.
If it is not the panoramic imaging mode, that is, if the normal imaging mode is currently set, it is determined as NO in step S6, and the process proceeds to step S7.
In step S7, the imaging control unit 51 executes the above-described normal imaging process.
That is, one frame of image data output from the image processing unit 17 immediately after the full-press operation is performed is recorded on the removable medium 31 as a recording target. As a result, the normal imaging process of step S8 ends, and the process proceeds to step S12. Since the processing after step S12 has been described above, the description thereof is omitted here.

On the other hand, when the panoramic imaging mode is currently set, it is determined to be YES in step S6, and the process proceeds to step S8.
In step S8, the imaging control unit 51 executes the panoramic imaging process described above.
Although details of the panoramic image capturing process will be described later with reference to FIG. 8, in principle, image data of a panoramic image is generated and recorded in the removable medium 31 as a recording target. As a result, the panoramic image capturing process in step S8 ends, and the process proceeds to step S9.

In step S9, the imaging control unit 51 determines whether the error flag is 1.
Although details will be described later with reference to FIG. 8, when the image data of the panoramic image is recorded on the removable medium 31 as a recording target and the panoramic image capturing process of step S8 is properly finished, the error flag becomes 0. There is. In such a case, NO is determined in step S10, and the process proceeds to step S12. Since the processing after step S12 has been described above, the description thereof is omitted here.
On the other hand, if any error occurs during the panoramic image capturing process in step S8, the panoramic image capturing process ends improperly. In such a case, the error flag is 1, so YES is determined in step S9, and the process proceeds to step S10.

In step S10, the imaging control unit 51 displays the content of the error on the display unit 19. A specific example of the content of the displayed error will be described later.
In step S11, the imaging control unit 51 cancels the panoramic imaging mode and resets the error flag to 0.
After that, the process is returned to step S1, and the subsequent processes are repeated. That is, the imaging control unit 51 prepares for the next new imaging operation by the user.

The flow of the imaging process has been described above with reference to FIG. 7.
Next, with reference to FIG. 8, a detailed flow of the panoramic image capturing process of step S9 in the image capturing process of FIG. 7 will be described.

FIG. 8 is a flowchart illustrating the detailed flow of the panoramic image capturing process.
As described above, when the shutter switch 41 is fully pressed in the panorama imaging mode, if YES is determined in step S6 of FIG. 7, the process proceeds to step S8, and the following panoramic imaging process is performed. Is executed.

That is, in step S31 of FIG. 8, the imaging control unit 51 acquires the angular displacement amount from the angular velocity sensor 22.

In step S32, the imaging control unit 51 determines whether or not the angular displacement amount acquired in the process of step S31 is larger than 0.
In a state where the user has not moved the digital camera 1, the angular displacement amount is 0, so it is determined to be NO in step S32, and the process proceeds to step S33.

In step S33, the imaging control unit 51 determines whether or not the angular displacement amount 0 has continued for a predetermined time. As the predetermined time, for example, an appropriate time longer than the time required from when the user fully presses the shutter switch 41 to when the movement of the digital camera 1 is started can be adopted.
If the predetermined time has not elapsed, it is determined to be NO in step S33, the process returns to step S31, and the subsequent processes are repeated. That is, when the duration of the state in which the user does not move the digital camera 1 is shorter than the predetermined time, the imaging control unit 51 repeatedly executes the loop processing of steps S31 to S33: NO, thereby performing the panoramic imaging. Put the process in the standby state.

When the user moves the digital camera 1 in this standby state, the angular displacement amount from the angular velocity sensor 22 becomes a value larger than zero. In such a case, YES is determined in step S32 and the process proceeds to step S34.

In step S34, the imaging control unit 51 updates the cumulative angular displacement amount by adding the angular displacement amount acquired in the process of step S32 to the cumulative angular displacement amount so far (cumulative angular displacement amount= Cumulative angular displacement + angular displacement). That is, the value stored as the accumulated angular displacement amount in the RAM 13 is updated.

The cumulative angular displacement amount is a value obtained by cumulatively adding the angular displacement amounts in this way, and indicates the movement amount of the digital camera 1.
Here, in the present embodiment, every time the user moves the digital camera 1 by a certain amount, one frame of image data (combining target) for generating image data of a panoramic image is transferred from the image processing unit 17 to the image combining unit 52. Shall be supplied to.
In order to realize this, the cumulative angular displacement amount corresponding to the "constant amount" as the moving amount of the digital camera 1 is previously given as the "constant value" by the initial setting process of step S1 in FIG.
That is, in the present embodiment, each time the cumulative angular displacement amount reaches a fixed value, one frame of image data (combining target) is supplied from the image processing unit 17 to the image compositing unit 52, and the cumulative angular displacement amount becomes 0. Is reset to.
Such a series of processing is executed as the processing of the next step S35 and thereafter.

That is, in step S35, the imaging control unit 51 determines whether or not the cumulative amount of angular displacement has reached a constant value.
If the cumulative amount of angular displacement has not reached the fixed value, NO is determined in step S35, the process returns to step S31, and the subsequent processes are repeated. That is, the imaging control unit 51 repeatedly executes the loop processing of steps S31 to S35 unless the cumulative angular displacement amount reaches a certain value by the user moving the digital camera 1 by a certain amount.
After that, when the user moves the digital camera 1 by a certain amount and the accumulated angular displacement amount reaches a certain value, YES is determined in step S35, and the process proceeds to step S36.

In step S36, the image composition unit 52 acquires the 1-frame image data from the image processing unit 17 under the control of the imaging control unit 51.
That is, the imaging control unit 51 issues an acquisition command to the image synthesizing unit 52 when the process proceeds to step S36 on the assumption that the cumulative amount of angular displacement has reached a certain value.
The image combining unit 52 that has received this acquisition command acquires image data of one frame from the image processing unit 17 as the process of step S36.

In step S37, the image synthesizing unit 52 overwrites the acquired one-frame image data (at least a part thereof) in the blank area of the storage area 101 (FIG. 6) to generate image data of the intermediate panoramic image. To do.
When the image data of the intermediate panoramic image is supplied from the image synthesizing unit 52 to the image reducing unit 53, the process proceeds to step S38.

In step S38, the image reduction unit 53 reduces the size (resolution) of the image data of the supplied intermediate panoramic image.
When the image data of the intermediate panoramic image after reduction is supplied from the image reduction unit 53 to the display control unit 54, the process proceeds to step S39.

In step S39, the display control unit 54 causes the display unit 19 to display the image represented by the image data supplied from the image reduction unit 53, that is, the reduced panoramic intermediate image, on the through image.

In step S40, the imaging control unit 51 updates the total angular displacement amount by adding the current angular displacement amount (=substantially constant value) to the total angular displacement amount so far (total angular displacement amount). =Total angular displacement + Cumulative angular displacement). That is, the value stored as the total angular displacement amount in the RAM 13 is updated.

In step S41, the imaging control unit 51 resets the cumulative angular displacement amount to 0. That is, the value stored as the accumulated angular displacement amount in the RAM 13 is updated to 0.

In this way, the cumulative amount of angular displacement is used to control the timing at which one frame of image data (synthesis target) is supplied from the image processing unit 17 to the image synthesis unit 52, that is, the timing of issuing the acquisition command. Therefore, the cumulative amount of angular displacement is reset to 0 every time the constant is reached and the acquisition command is issued.
Therefore, the image capturing control unit 51 cannot recognize how far the digital camera 1 has moved from the start of the panoramic image capturing process to the present even by using the cumulative angular displacement amount.
Therefore, in order to enable such recognition, in the present embodiment, the total angular displacement amount is adopted in addition to the cumulative angular displacement amount.
That is, the total amount of angular displacement is a value obtained by cumulatively adding the amount of angular displacement, but is not reset to 0 even when a certain amount is reached, until the panoramic image capturing process ends (details will be described later). It is a value that continues cumulative addition until the processing of step S47 is executed.

In this way, when the total angular displacement amount is updated in the process of step S40 and the cumulative angular displacement amount is reset to 0 in the process of step S41, the process proceeds to step S42.

In step S42, the imaging control unit 51 determines whether a release operation has been performed.
When the release operation is not performed, that is, when the user fully presses the shutter switch 41, it is determined as NO in Step S42, and the process proceeds to Step S43.

In step S43, the imaging control unit 51 determines whether or not an image acquisition error has occurred.
The error in image acquisition is not particularly limited, but in the present embodiment, for example, the fact that the digital camera 1 has moved in a diagonal direction, a vertical direction, or an opposite direction by a certain amount or more is adopted as the error.

If no error has occurred in the image acquisition, NO is determined in step S43, and the process proceeds to step S44.
In step S44, the imaging control unit 51 determines whether the total angular displacement amount exceeds the threshold value.
As described above, the total angular displacement amount is a cumulative addition value of the angular displacement amounts from the start of the panoramic imaging process (after the full-press operation is performed) to the time when the process of step S42 is executed. ..
Here, in the present embodiment, the maximum amount of movement by which the user can move the digital camera 1 during panoramic imaging is determined in advance. The total angular displacement amount corresponding to the “maximum movement amount” as the movement amount of the digital camera 1 is given in advance as the “threshold value” by the initial setting process of step S1 of FIG.
As described above, in the present embodiment, that the total angular displacement amount reaches the threshold means that the digital camera 1 has moved by the maximum movement amount.
Therefore, if the total amount of angular displacement does not reach the threshold value, that is, if the amount of movement of the digital camera 1 has not reached the maximum amount of movement, the user can still continue moving the digital camera 1, so NO in step S44. Is determined, the process is returned to step S31, and the subsequent processes are repeated.
That is, if one of the errors is that the angular displacement amount 0 continues for a predetermined time (the digital camera 1 does not move for a predetermined time), as long as the full-push operation is continued without an error. The loop processing of steps S31 to S44 is repeatedly executed.
During this time, as described above with reference to FIG. 5, the display of the through image and the display of the intermediate panoramic image are updated as the digital camera 1 moves.
Since the user can operate the panoramic image capturing while visually observing the panoramic image which is continuously updated in this way, it is possible to easily and accurately visually recognize the image capturing history from the start of the panoramic image capturing to the present. .. As a result, the user can easily and easily obtain a desired panoramic image without repeating the imaging operation many times.

After that, in a state where no error occurs, whether the release operation is performed (whether it is determined to be YES in step S42) or the digital camera 1 has moved to the maximum movement amount (YES in step S45). Processing is performed), the process proceeds to step S46.
In step S46, the imaging control unit 51 generates the image data of the panoramic image via the image combining unit 52, and causes the removable medium 31 to record the image data as the image data to be recorded.

Then, in step S47, the imaging control unit 51 resets the total angular displacement amount to 0.
As a result, the panoramic image capturing process ends properly. That is, the process of step S8 in FIG. 7 ends properly, and it is determined to be NO in the process of step S9. The process after the determination of NO in step S9 has been described above, and thus the description thereof is omitted here.

If any error occurs during the series of processes described above, that is, if it is determined to be YES in the process of step S33 or YES in the process of step S43, the process is It proceeds to step S44.
In step S44, the imaging control unit 51 sets the error flag to 1.
In this case, the process of step S46 is not executed, that is, the image data of the panoramic image is not recorded, and the panoramic image capturing process ends improperly.
That is, the process of step S8 in FIG. 7 ends improperly, it is determined to be YES in the process of step S9, and the content of the error is displayed in the process of step S10.
The display of the error content in this case is not particularly limited as described above, but for example, a message display such as “Failed to acquire image” or “Time is over” can be adopted.

As described above, the digital camera 1 according to the present embodiment includes the image capturing unit 16 that sequentially outputs, as frame image data, each piece of image data that is sequentially obtained by capturing an image at predetermined time intervals.
The digital camera 1 of the present embodiment also includes an image synthesizing unit 52 and a display control unit 54.
The image synthesizing unit 52 outputs the image data of the frame output from the image capturing unit 16 to the image processing unit 17 each time the digital camera 1 moves by a predetermined amount (each time the above-described cumulative angular displacement amount reaches a constant value). It is acquired through the storage unit 18 and stored in the storage unit 18. Then, the image combining unit 52 sequentially generates the image data of the intermediate panoramic image (composite image) by combining at least a part of the image data of the plurality of frames cumulatively held in the storage unit 18 so far. To do.
The display control unit 54 controls to simultaneously display a frame (that is, a through image) sequentially output as image data from the imaging unit 16 and a panorama midway image (composite image) sequentially generated as image data by the image combining unit 52. To execute.
As a result, as described above with reference to FIG. 5, the display of the through image and the display of the intermediate panoramic image (composite image) are updated as the digital camera 1 moves.
Since the user can operate the panoramic image pickup while visually checking the panoramic image (composite image) that is continuously updated in this way, the image pickup history from the start of the panoramic image pickup to the present can be easily and accurately viewed. can do. As a result, the user can easily and easily obtain a desired panoramic image without repeating the imaging operation many times.

In particular, in the digital camera 1 of the present embodiment, the display control unit 54 executes control for superimposing the intermediate panoramic image (composite image) on the through image and displaying the superimposed image on the display unit 19.
In this case, the user can visually recognize the through-the-lens image and the intermediate panoramic image at the same time, so that the user can visually recognize from the through-the-lens image what the subject is currently capturing, and the subject is included in which part of the panoramic image. It becomes possible to visually recognize whether or not.

The digital camera 1 of the present embodiment further includes an image reduction unit 53 that reduces the image data of the intermediate panoramic image.
Accordingly, the display control unit 54 can superimpose the reduced panoramic intermediate image on the through image and display it on the display unit 19.
In this case, the through image is not largely hidden by the mid-panoramic image, so that the user can reliably visually recognize the next subject to be imaged from the through image.

In the digital camera 1 of the present embodiment, when a predetermined condition such as a release operation or a total angular displacement amount reaching a threshold value is satisfied, the image synthesizing unit 52 generates a panorama as image data of a synthetic image for recording. Generate image data for an image.
In this case, the imaging control unit 51 executes control to record the image data of the panoramic image generated for recording on the removable medium 31.
In this way, the user can easily and easily record the image data of the desired panoramic image on the removable medium 31 without repeating the imaging operation many times.

It should be noted that the invention is not limited to the above-described embodiments, and modifications, improvements, etc. within a range in which the object of the invention can be achieved are included in the invention.

For example, in the above-described embodiment, the reduced panoramic image is superimposed on the through image and displayed on the display unit 19, but the through image and the panoramic image are displayed simultaneously. The form is not limited to this.

For example, the screen of the display unit 19 may be vertically divided into two, and the through image may be displayed on one screen of the upper side and the lower side and the panoramic image may be displayed on the other screen.
In this way, by clearly distinguishing and displaying the through image and the mid-panoramic image,
The user can visually recognize the panoramic image showing the process of forming the panoramic image more clearly. As a result, the user can more easily and accurately view the history of imaging from the start of panoramic imaging to the present, and can easily and easily obtain a desired panoramic image with less imaging operation. It will be possible.

Further, for example, the intermediate panorama image may be displayed without being reduced, if necessary.
Further, in the present embodiment, the image combining unit 52 uses the image data of the previous panoramic image and the image data of the frame acquired immediately before as the compositing target in the blank area of the previous panoramic image to be combined. Although the image synthesizing process for overwriting the frame No. is executed, it is also possible to display only the intermediate image of the panorama without displaying the blank area.

Further, in the above-described embodiment, the image reducing unit 53 is provided in the latter stage of the image combining unit 52, but may be provided in the former stage of the image combining unit 52. In this case, the image reduction unit 53 sequentially reduces the image data of the frames sequentially supplied from the image processing unit 17. Therefore, the image synthesizing unit 52 synthesizes the image data of the reduced frames to generate the image data of the intermediate panoramic image.

Further, in the above-described embodiment, the angular velocity sensor 22 detects the angular displacement of the digital camera 1, but the method of detecting the angular displacement amount is not particularly limited to this.
For example, a method of detecting the angular displacement of the digital camera 1 by image processing of analyzing a through image may be adopted.

Further, in the above-described embodiment, the intermediate panoramic image and the panoramic image have a horizontally long configuration, but the present invention is not particularly limited to this, and the configuration becomes longer in a direction conforming to the moving direction of the digital camera 1, for example, a vertically long configuration. It may be configured.

Further, for example, in the above-described embodiments, the image pickup apparatus to which the present invention is applied has been described as an example configured as the digital camera 1.
However, the present invention is not particularly limited to this, and can be applied to electronic devices generally having an imaging function capable of capturing a panoramic image. For example, the present invention relates to a portable personal computer, a portable navigation device, It is widely applicable to portable game machines and the like.

The series of processes described above can be executed by hardware or software.

When the series of processes is executed by software, a program forming the software is installed in the image pickup apparatus or a computer controlling the image pickup apparatus from a network or a recording medium. Here, the computer may be a computer incorporated in dedicated hardware. Alternatively, the computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

The recording medium including such a program is provided not only with the removable medium 31 that is distributed separately from the apparatus main body to provide the program to the user, but also provided to the user in a state of being pre-installed in the apparatus main body. It is composed of a recording medium etc. The removable medium 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. In addition, the recording medium provided to the user in a state of being pre-installed in the apparatus main body is constituted by, for example, the ROM 12 in which the program is recorded, the hard disk included in the storage unit 18, or the like.

In the present specification, the steps of writing the program recorded on the recording medium are not limited to the processing performed in time series according to the order, but are not necessarily performed in time series, and may be performed in parallel or individually. It also includes the processing to be executed.

DESCRIPTION OF SYMBOLS 1... Digital camera, 11... CPU, 12... ROM, 13... RAM, 15... Optical system, 16... Imaging part, 17... Image processing part, 18... Storage part, 19... Display part, 20... Operation part, 21... Communication unit, 22... Angular velocity sensor, 23... Drive, 31... Removable media, 41... Shutter switch, 51... Imaging control unit, 52... Image combining unit, 53... Image reduction unit, 54... Display control unit

Claims (14)

An image pickup means for continuously picking up an image of a subject and sequentially outputting image data;
An image including a first image synthesizing unit that synthesizes a plurality of image data sequentially output by the image pickup unit while the subject is continuously imaged while the image pickup range of the image pickup unit is moved in a substantially predetermined direction. Synthesizing means ,
Determination means for determining whether or not the image range of the combination area to be combined by the first image combination means has reached a limit combination range;
If the subject is continuously being captured by the imaging means, an image corresponding to the sequentially output image data by the imaging unit, the synthesis corresponding to the image data which are sequentially combined by the first image synthesizing means Display control means for displaying the image of the area on the display means at the same time,
While the image corresponding to the image data sequentially output by the display control unit and the image of the combination area are displayed on the display unit, an image pickup for combining the image data by the first image combination unit Is ended at the timing when it is determined by the discriminating means that the image range of the composite area has reached the limit compositing range or by the discriminating means that the image range of the composite area has reached the limit compositing range. Instructing means to instruct at a timing earlier than,
Equipped with
The image synthesizing means includes a second image synthesizing means for synthesizing a plurality of image data captured by the instruction means until the instruction to end the image capturing,
The image processing apparatus , further comprising a recording control unit that causes the recording unit to record the combined image data combined by the second image combining unit when the instruction unit instructs the end of the imaging. An image pickup means for continuously picking up an image of a subject and sequentially outputting image data;
An image synthesizing unit for synthesizing a plurality of sequentially output image data corresponding to a subject imaged while the image capturing range of the image capturing unit is moved in a substantially predetermined direction,
Discriminating means for discriminating whether or not the image range of the combined area to be combined by the image combining means has reached the limit combining range;
While the subject is continuously imaged by the image pickup means, an image corresponding to the image data sequentially output by the image pickup means and an image of the combined area corresponding to the image data sequentially combined by the image combiner Display control means for displaying simultaneously and on the display means,
While the image corresponding to the image data sequentially output by the display control unit and the image of the combination area are displayed on the display unit, the image combining unit ends the imaging for combining the image data. The timing at which the determination unit determines that the image range of the combination area has reached the limit combination range, or before the determination unit determines that the image range of the combination area has reached the limit combination range. Instructing means to instruct at the timing of
Recording control means for recording composite image data of a plurality of image data picked up by the image pickup means in a recording means before the image pickup end is instructed by the instruction means; Equipped with
The display control means overwrites the image corresponding to the sequentially output image data with the image data finally synthesized in the synthesis area as the synthesis of the image in the synthesis area and the sequentially synthesized image data progresses. An image of a predetermined display area including an image of an uncombined area, which is displayed on the display means,
The instructing unit sets the end of the image capturing at an arbitrary timing when the image of the predetermined display region includes the image of the uncombined region while the image capturing unit continuously captures the image of the subject. in the image processing apparatus you wherein the instructing. An image pickup means for continuously picking up an image of a subject and sequentially outputting image data;
An image synthesizing unit for synthesizing a plurality of sequentially output image data corresponding to a subject imaged while the image capturing range of the image capturing unit is moved in a substantially predetermined direction,
Discriminating means for discriminating whether or not the image range of the combined area to be combined by the image combining means has reached the limit combining range;
While the subject is continuously imaged by the image pickup means, an image corresponding to the image data sequentially output by the image pickup means and an image of the combined area corresponding to the image data sequentially combined by the image combiner Display control means for displaying simultaneously and on the display means,
While the image corresponding to the image data sequentially output by the display control unit and the image of the combination area are displayed on the display unit, the image combining unit ends the imaging for combining the image data. The timing at which the determination unit determines that the image range of the combination area has reached the limit combination range, or before the determination unit determines that the image range of the combination area has reached the limit combination range. Instructing means to instruct at the timing of
Recording control means for recording composite image data of a plurality of image data picked up by the image pickup means in a recording means before the image pickup end is instructed by the instruction means; Equipped with
Said instructing means, the image processing apparatus you characterized by instructing the end of the imaging by a user operation. An image pickup means for continuously picking up an image of a subject and sequentially outputting image data;
An image synthesizing unit for synthesizing a plurality of sequentially output image data corresponding to a subject imaged while the image capturing range of the image capturing unit is moved in a substantially predetermined direction,
Discriminating means for discriminating whether or not the image range of the combined area to be combined by the image combining means has reached the limit combining range;
While the subject is continuously imaged by the image pickup means, an image corresponding to the image data sequentially output by the image pickup means and an image of the combined area corresponding to the image data sequentially combined by the image combiner Display control means for displaying simultaneously and on the display means,
While the image corresponding to the image data sequentially output by the display control unit and the image of the combination area are displayed on the display unit, the image combining unit ends the imaging for combining the image data. The timing at which the determination unit determines that the image range of the combination area has reached the limit combination range, or before the determination unit determines that the image range of the combination area has reached the limit combination range. Instructing means to instruct at the timing of
Recording control means for recording composite image data of a plurality of image data picked up by the image pickup means in a recording means before the image pickup end is instructed by the instruction means; Equipped with
Wherein the display control unit, the image processing apparatus you characterized in that the synthetic area to be displayed on said display means so as to overlap the image corresponding to the image data to which the sequentially output. The image synthesizing unit synthesizes a plurality of image data sequentially output by the image capturing unit while the subject is continuously imaged while the image capturing range of the image capturing unit is moved in a substantially predetermined direction. The image processing device according to any one of claims 2 to 4 . The display control means overwrites the image corresponding to the sequentially output image data with the image data finally synthesized in the synthesis area as the synthesis of the image in the synthesis area and the sequentially synthesized image data progresses. The image processing apparatus according to any one of claims 1 to 5 , wherein an image of a predetermined display area including an image of an uncombined area is displayed on the display unit. Wherein the display control unit, according to the image which the image data to which the sequentially synthesized is reduction processing in any one of 6 claim 1, characterized in that to be displayed on said display means as an image of the synthesis area Image processing device. The image pickup unit continues to output an image to be displayed on the display unit even when the instruction of the image pickup is completed by the instruction unit.
Wherein the display control unit, an image processing apparatus according to any one of claims 1 7, characterized in that to display an image that is sequentially outputted continuously by the image pickup means on the display means. 9. The recording control unit causes the recording unit to record the composite image data generated by the instruction unit instructing the end of the image capturing, according to any one of claims 1 to 8. The image processing device described. The image processing apparatus according to claim 3 , wherein the instructing unit instructs the image processing apparatus to perform the user operation on a predetermined location as the end of the imaging. Further comprising detection means for detecting the amount of change in the imaging range,
The determination means determines whether or not the total change amount of the imaging range detected by the detection means has reached a threshold value, and when the total change amount of the imaging range reaches the threshold value, the image combination means combines them. The image processing apparatus according to any one of claims 1 to 10 , characterized in that it is determined that the image range of the combined area has reached the limit combined range. The detection means detects the amount of change in the imaging range by detecting the amount of directional displacement in the imaging direction by the imaging means,
The discriminating means discriminates whether or not the total directional displacement amount obtained by accumulating the directional displacement amounts reaches a predetermined threshold value, and when the total directional displacement amount reaches the predetermined threshold value, the image synthesizing means performs synthesis. The image processing apparatus according to claim 11 , wherein the image processing apparatus determines that the image range of the area has reached the limit composition range. An image processing method for displaying an image on a display means,
An image synthesizing step of synthesizing sequentially output image data corresponding to the subject continuously imaged while the imaging range of the image pickup means is moved in a substantially predetermined direction,
A determining step of determining whether or not the image range of the combined area to be combined by the image combining step has reached a limit combining range;
While the subject is continuously imaged by the image pickup means, an image corresponding to the image data sequentially output by the image pickup means and an image of the combined area corresponding to the image data sequentially combined at the image combining step. And a display control step of simultaneously displaying on the display means,
While the image corresponding to the image data sequentially output by the display control step and the image of the combination area are displayed on the display unit, the end of imaging for combining the image data by the image combination step is terminated. , At a timing when it is determined that the image range of the composition area has reached the limit composition range by the determination step or before it is determined that the image range of the composition area has reached the limit composition range by the determination step. Instruction step to instruct at the timing of
When the instruction step gives an instruction to end the imaging according to a user operation, the recording means records composite image data of a plurality of image data captured by the imaging means until the imaging end is instructed. A recording control step,
An image processing method comprising: Computer,
An image synthesizing unit for synthesizing a plurality of sequentially output image data corresponding to the subject imaged while the image capturing range of the image capturing unit is moved in a substantially predetermined direction,
Discriminating means for discriminating whether or not the image range of the combined area to be combined by the image combining means has reached the limit combining range,
While the subject is continuously imaged by the image pickup means, an image corresponding to the image data sequentially output by the image pickup means and an image of the combined area corresponding to the image data sequentially combined by the image combiner Display control means for displaying and on the display means at the same time,
While the image corresponding to the image data sequentially output by the display control unit and the image of the combination area are displayed on the display unit, the image combining unit ends the imaging for combining the image data. The timing at which the determination unit determines that the image range of the combination area has reached the limit combination range, or before the determination unit determines that the image range of the combination area has reached the limit combination range. Instruction means to instruct at the timing of
When the end of the image pickup is instructed by the instruction unit according to a user operation, composite image data of a plurality of image data picked up by the image pickup unit is recorded in the recording unit until the end of the image pickup is instructed. Recording control means,
A program characterized by making it function as.

Images