JP4851392B2 - Image composition apparatus, image composition method, and image composition program - Google Patents

Description

  The present invention relates to image processing for synthesizing a plurality of still images taken by an imaging apparatus, and more particularly to a technique for obtaining an effect that is not found in an original image by superimposing images such as camera shake correction.

  2. Description of the Related Art Today, a camera shake correction technique for generating an image free from camera shake by combining short-exposure continuous shot images has been developed. For example, a method of generating a high-quality image by switching processing methods and parameters according to the number of synthesized images has been developed.

  There are a hardware method and a software method for performing the above-described image stabilization. For example, camera shake in a digital camera occurs when light reaching a sensor (for example, a CMOS sensor) shifts with time as the hand moves during exposure. There is an optical camera shake correction (sensor shift type or lens shift type) for suppressing such camera shake. This optical image stabilization uses a dedicated hardware that bends the optical path by moving the sensor and lens in conjunction with the movement of the hand, and suppresses the deviation of the light reaching the sensor regardless of the movement of the hand.

On the other hand, there is a method of reducing camera shake by software. Since camera shake is caused by movement of the hand during the exposure time, it can be suppressed by shortening the exposure time.
However, if the exposure time is shortened, the amount of light decreases, and the entire image becomes dark. Therefore, it is necessary to brighten the entire image. Therefore, the sensitivity of the sensor is increased, and the pixel value is changed to a large value in terms of software (gain increase).

However, since increasing the gain also increases the noise, the method of performing the short exposure + gain increase results in a noisy image.
Therefore, in software image stabilization, multiple short-time exposures and gain-up images are taken continuously, the subject of each image is aligned, the noise component is detected by superimposing the images, and noise is detected based on the detection result. We are reducing. In other words, if there is no noise, the pixel values of the corresponding pixel positions of the superimposed images are the same, but since noise is generated randomly, the pixel values are not the same even at the same pixel position. Using such a principle, noise components are detected to reduce noise.

There is also a method of reducing only the noise component by superimposing the images and averaging the pixel values.
However, when noise is reduced by superimposing the plurality of images, if the subject moves, the subject motion area is multiplexed like a ghost on a television. In view of this, a method has been proposed in which the presence / absence of subject movement is detected after the positional deviation is matched, and the processing is changed depending on whether the subject moves or not.

  In this method, for example, an area where there is no movement of the subject is synthesized as usual, and only one image is used for an area where the movement of the subject is present. In this way, multiplexing due to subject movement can be suppressed.

  However, after the image is taken as described above, the image is aligned, and all the series of processes shown above are performed. When the processed image is finally saved with high image quality, the image is as shown in (1) in FIG. Processing takes time and user response is poor.

Therefore, conventional methods (2) to (4) shown in FIG. 18 and methods disclosed in Patent Documents 1 to 3 have been proposed.
(2) The viewer image (stored image) that has been subjected to simple processing is displayed to the user, and the image that is finally stored is processed separately from the processing for generating the viewer image. In this case, the viewer image is stored with low image quality. The final image to be saved is stored for a long time with high image quality. However, if the processing for generating the viewer image and the high-quality image is performed by, for example, one CPU, the processing time for performing the processing for generating the viewer image is longer than that in (1). In addition, when the processing for viewer and the processing for high image quality are performed in parallel using two CPUs, the processing time is shortened, but the size of the CPU and its peripheral circuits increases, resulting in an increase in the imaging device. .

(3) The process is simplified and completed so that the user does not feel stress. The image to be finally saved is saved with low image quality in a short time.
(4) A low-quality viewer image is generated and stored in a time when the user does not feel stress, and thereafter, processing is performed in the background based on the viewer image. The high image quality generation process performed in the background of (4) is not the process of generating a high image quality shown in (1) and (2) but a process for improving the image quality for the viewer. Since an image to be finally stored is generated from the viewer image, medium image quality can be finally stored although the image quality is not high as in (1) and (2).

Further, according to Japanese Patent Laid-Open No. 2004-260, a proposal has been made to brighten a single image before synthesizing when performing image synthesizing processing with continuous shot images, and display the brightened image as a viewer.
According to Patent Document 2, in the process of extracting feature points from a plurality of captured images and determining the amount of variation between the images, the reference image is divided into at least two and the amount of variation is calculated from the feature points in one region. Proposals to ask for have been made.

  According to Patent Document 3, a motion area is detected when combining continuously shot images, and the motion areas are substantially matched to be combined. In other words, a method has been proposed in which a series of movements of a moving object is held as a multiple image by substantially matching the movement areas.

In addition, Patent Documents 4 and 5 are also proposed.
However, when an image is output immediately after shooting by an imaging device such as a digital camera or a camera-equipped mobile phone, in order to allow the user to check the shooting contents, an attempt is made to provide a high-quality image after the synthesis process. Output takes time and user response is poor. Therefore, there is a problem that the image quality of the provided image deteriorates if the processing is postponed to improve the user response.

Further, in order to correct a camera shake by synthesizing a plurality of still images acquired by an imaging device to obtain one image, it takes time for the synthesis process and also requires large resources (memory capacity, disk capacity, etc.). Become.
JP 2006-045498 A JP 2004-219765 A JP 2006-025312 A JP 2006-139758 A JP 2003-108416 A

  The present invention has been made in view of the above-described circumstances, and an image composition apparatus and an image composition method that can maintain a high image quality with good user response and can ultimately reduce the memory capacity. And an image composition program.

  An image composition device that synthesizes a plurality of images according to one aspect of the present invention, and an intermediate data holding unit that holds detection results obtained by detecting the movement of a subject based on at least the plurality of images as intermediate data; An image composition unit that generates a viewer image that is a composite image for the plurality of images, an image storage unit that retains the viewer image, and a high-quality image that is generated using the viewer image and the intermediate data And a reprocessing unit that performs high image quality processing.

In image composition, the number of images used for composition is determined for each region, and image composition is performed. After combining, the combined number information is stored together with the image when the image is stored.
In the present invention, the detection result of the subject moving area and the non-moving area generated at the time of image composition is held as intermediate data, so that the subject moving area and the non-moving area can be easily separated from the saved image and different processing is performed. Can be made.

In addition, the intermediate data used during the image quality enhancement process uses the determination result information of the motion area and the non-motion area (hereinafter referred to as a motion detection map), so that the motion area and the non-motion area can be reproduced during the image quality enhancement process. Different processing can be performed for a motion region and a non-motion region without making a determination. If there is a motion detection result, there is no need to hold the original continuous shot image, so the resources used can be greatly reduced.

  The background processing is automatically performed when the embedded device that stores the image is in a predetermined status state, thereby pressing down other processing performed on the embedded device without any user operation. The processing can be carried out without doing so. For example, in the case of a mobile phone, the quality of the final image can be improved without applying a load to other processing by not performing background processing during a call or during execution of another program.

  Before the background processing, only a saved image generated by simple processing can be presented to the user, so the user may evaluate that the image is bad by looking at the image. Therefore, an icon indicating that the image is before final processing is displayed on the image before starting the background processing when the image is displayed. It is detected whether or not there is intermediate data, and if there is intermediate data, the user is notified that it is before the final image.

  At that time, it is considered that the user has a request to confirm the final quality of the image quickly. Immediate execution of background processing is assigned to a key or menu, and image quality improvement processing for high quality similar to background processing is immediately performed by user operation.

  By storing the intermediate data in the header area of the saved image, it is possible to manage the intermediate data without managing files that maintain the correspondence between the intermediate data and the saved image. For example, by storing intermediate data in an area (comment field) where an image creator can store arbitrary information such as JPEG or Exif, the intermediate data can be held while maintaining the compatibility of the file format.

  Since the intermediate data is held, the data amount of the image file temporarily increases. Since the intermediate data is binary data, the data may be compressed by a simple run length method or the like. This intermediate data is unnecessary data when the background processing is completed and a final image is obtained. By executing the background processing and deleting the intermediate data from the final image when the processing is completed, an increase in the final file size can be suppressed.

  According to the present invention, the user response is good and the quality of the finally recorded image can be kept high, and the memory capacity can be further reduced.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
(Principle explanation)
FIG. 1 shows a principle diagram of the present invention.

  The continuous shot image shown in FIG. 1 is a plurality of images taken by the user with the imaging device. The stored image (low image quality) is a viewer image. The intermediate data is data used to generate a high-quality image that is finally stored from the viewer image.

  First, a subject is photographed with a relatively short exposure time te, and a plurality of images are acquired in time series. For example, as shown in FIG. 2, images 1 and 2 are acquired by continuous shooting. At this time, shooting parameters such as exposure time te and gain value g are also acquired.

  Next, the entire image 1 and image 2 are aligned, a motion detection process is further performed, a motion region of the subject is detected, and a motion determination result is stored. Here, the white area shown in the image 3 is a non-motion area, and the black area is a movement area.

  Thereafter, from the image 2 taken later in time, the region determined as the motion region by the motion determination result is excluded and combined with the image 1 to obtain a composite image. By the above processing, a viewer image (first stored image) is generated in a short time based on the continuous shot image, and the viewer image is displayed to the user.

  In the detection process of the motion area and the non-motion area, the threshold value d_exposure or the threshold value d_gain is determined according to the shooting parameter. The threshold d_exposure is determined according to the exposure time te, and the threshold d_gain is determined according to the gain value g.

  A difference value Δa between corresponding individual pixels (x, y) of two frames of image data (image 1, image 2) is calculated. Then, the difference value Δa is determined using the threshold value d_gain or the threshold value d_exposure. If difference value Δa> threshold value d_gain, or difference value Δa> threshold value d_exposure, the pixel (x, y) is regarded as a motion region. If the difference value Δa ≦ the threshold value d_gain or the difference value Δa ≦ the threshold value d_exposure, the pixel (x, y) is regarded as a non-motion area. This processing is executed for all the pixels (regions) of image 1 and image 2 to generate a viewer image.

  At that time, the created intermediate data is temporarily held and used when an image to be finally saved is generated. Here, the intermediate data is data generated when a high-quality image is generated. For example, a detection result (image 3) of a motion area and a non-motion area generated at the time of image synthesis.

  In the intermediate data, the number of combined images for each moving area and non-moving area of the image 3 is held as combined number information. The moving area is an area where the number of synthesized images is small and the image quality is low, and the non-motion area is an area where the number of synthesized images is large and the image quality is high.

  Since the viewer image (saved image (low image quality)) of the conventional example (4) shown in FIG. 18 is after image synthesis, when creating the saved image (high image quality), the motion region and the non-moving area are determined only from the viewer image. It is not possible to separate the motion areas and perform different processing. That is, when different processing is performed on a single image for a motion region and a non-motion region, it is necessary to detect the motion region and the non-motion region from one image, but this is difficult.

Next, after generating the viewer image and intermediate data, a process for obtaining a high-quality image in the background is performed.
In order to improve the quality of the viewer image, which is a composite image free from obstacles such as multiple copying, a process for generating a high quality image based on the intermediate data is performed. At this time, processing for generating different high-quality images is performed in the motion region and the non-motion region.

By doing so, it is possible to synthesize continuous shot images to generate a viewer image and intermediate data, and to generate a high-quality image from the viewer image and intermediate data.
Example 1
FIG. 3 shows the configuration of the image processing unit 1 of the imaging apparatus incorporated in the digital camera or camera-equipped mobile phone according to the present invention.

  The image processing unit 1 includes an image photographing unit 2, a photographed image holding unit 3, a positional deviation detecting unit 4, a subject motion detecting unit 5, an intermediate data holding unit 6 (motion detection result holding unit), an image composition unit 7, and an image holding unit. 8, a reprocessing unit 9 and a status monitoring unit 10. A viewer image process to be displayed to the user after shooting and a high-quality image to be finally saved are generated based on the viewer image.

  The image capturing unit 2 captures an optical image from an imaging lens to an image sensor that converts the optical image into an electric signal via a shutter having a diaphragm function to capture an object, and outputs an analog signal output from the image sensor by an A / D converter. Convert the output to a digital signal. Further, when shooting once, images are continuously shot at a shutter speed that does not cause camera shake, and a plurality of images are acquired.

  The captured image storage unit 3 stores a plurality of images captured by the image capturing unit 2. For example, two images are held in the photographic image holding unit 3 in the case of two continuous shooting, but in the case of three continuous shooting and four continuous shooting, three or four images are also stored in the photographic image holding unit 3. Hold on.

For example, in the case of two-sheet continuous shooting, the position shift detection unit 4 detects the position shift amount of two images. There is a template matching method or the like as the detection of the displacement.
A case will be described in which two images are continuously shot and the images 1 and 2 held in the photographed image holding unit 3 are combined to generate one image. First, a minute region composed of specific pixels in the image 1 is held as a feature point, and where the feature point is in the image 2 is scanned and detected. A difference between the position of the feature point of the image 1 and the position of the detected feature point of the image 2 is calculated and used as a displacement amount.

Also, there is a Moravec operator as a typical method for detecting the amount of deviation (reference: computer image processing edited by Hideyuki Tamura, Ohmsha Publishing Bureau p243).
After calculating the amount of deviation between images in this way, the subject motion detection unit 5 detects the subject motion by comparing the difference between the pixel values of the images after alignment with a preset threshold value. be able to.

  The difference between the pixels at the corresponding position (for example, Y difference of YUV) is calculated in the image with the deviation amount combined, and the difference is compared with a preset threshold (for example, ΔY = 20 to 25) to reach the threshold. It is determined whether or not. When the threshold is reached, the pixel is regarded as a motion region.

  Here, as shown in the image 3 of FIG. 2, the values of movement (for example, “1”, black in the figure) and no movement (for example, “0”, white in the figure) are held for each pixel. If there is intermediate data (motion detection map) as a result of this motion detection, it is not necessary to retain the original continuous shot image, so that the resources to be used can be greatly reduced (three original images are 9 bytes per pixel, motion detection). Since the result is 1 bit per pixel, the resource amount may be 1/72).

The intermediate data holding unit 6 holds a motion detection result for each pixel as, for example, intermediate data as shown in FIG.
The image composition unit 7 performs an image composition process. At this time, intermediate data may or may not be used. When using the intermediate data, according to the motion detection map stored in the intermediate data storage unit 6, for example, in FIG. 2, the pixel value of the image 1 is used for the motion region, and the composite result of the image 1 and the image 2 is used for the non-motion region. To do.

The image holding unit 8 once holds the image combined by the image combining unit 7.
In response to the execution instruction of the image quality enhancement processing from the status monitoring unit 10, the reprocessing unit 9 displays the image data (viewer image) in the image holding unit 8 and the motion detection map (intermediate data) in the intermediate data holding unit 6. To perform high image quality processing. Specifically, contour enhancement processing is performed on the non-motion region, and noise removal processing is performed on the motion region.

  The status monitoring unit 10 monitors the status of the digital camera or camera-equipped mobile phone in which the imaging device is incorporated, and is in a preset state (for example, only a minimum necessary program is operating or the CPU load is equal to or less than the set value). Or the like, and when a preset state is reached, the re-processing unit 9 is instructed to execute an image quality enhancement process.

After the image quality enhancement processing is completed, the reprocessing unit 9 overwrites the viewer image stored in the image holding unit 8 with the high quality image to be finally stored.
Next, the operation will be described.

FIG. 4 shows an operation flowchart of the present invention.
In step S41, images are continuously shot. The image photographing unit 2 continuously photographs the subject to obtain a plurality of photographed images and stores them in the photographed image holding unit 3.

  In step S42, the images are synthesized. The positional deviation detection unit 4 detects the positional deviation of the plurality of captured images stored in the captured image holding unit 3 and the subject movement detection unit 5 aligns the positions of the images so that the difference between the pixel values is set in advance. The movement of the subject is detected by comparing with a threshold value. Then, the image composition unit performs image composition processing according to intermediate data (motion detection map) held in the intermediate data holding unit 6 described later.

In step S43 (intermediate data holding step), the intermediate data at the time of synthesis is held in the intermediate data holding unit 6.
In step S44 (image composition step), the composite image is held. The image holding unit 8 once holds the viewer image generated by the image combining unit 7. The user can see the viewer image.

  In step S45 (reprocessing step), the image stored using the intermediate image in the background is subjected to high image quality processing. The viewer image is converted into a high-quality image by using the determination result information of the motion area and the non-motion area which are intermediate data stored in the detection result holding unit 6. Since the intermediate data is held in this way, different processing can be performed for the moving region and the non-moving region without re-determination of the moving region and the non-moving region at the time of high image quality processing. As processing, for example, 1) noise removal (blurring) is performed in a motion region, and contour enhancement (sharpening) is performed in a non-motion region. 2) It is possible to remove the minute motion region (change to the non-motion region) in the motion region, or do nothing in the non-motion region.

In step S46, the viewer image and intermediate data are deleted and updated to a high-quality image that is finally stored.
Further, as a modification, the process executed in the background is automatically performed when the imaging apparatus storing the composite image is in a predetermined status state, so that there is no user operation. The processing can be performed without pressing other processing performed on the embedded device.

  For example, in the case of a mobile phone, it is possible to realize a high image quality of an image to be finally stored without applying a load to other processing by not performing background processing during a call or during execution of another program.

The operation will be described with reference to the operation flowchart of FIG.
In step S51, as in step S41, continuous shooting is performed to obtain a plurality of images.
In step S52, the number of combined images is determined. After detecting the amount of displacement as shown in step S42, the motion area and the non-motion area are detected, and the image composition information is assigned to the motion area and the non-motion area.

In step S53, image composition is performed based on the number of composites in each area determined in step S52. In step S54, the composite number of each pixel is stored.
In step S55, it is determined whether scanning of all pixels has been completed. If image composition has been completed for all pixels, the process proceeds to step S56. If not completed, the process proceeds to step S52.

In step S56, the viewer image, which is the composite image generated in step S53, and the intermediate data used for generation are stored.
The diagram shown in FIG. 17 is a diagram showing a specific data example of the intermediate data output in S56 of FIG. 5 or S76 of FIG.

  The upper block diagram of FIG. 17 uses the original image 1 and the original image 2 to be combined by an image combining unit to generate a viewer image, and the original image 1 and the original image 2 are used to generate intermediate data (motion detection map). It is the figure which showed the structure which produces | generates.

  The lower data in FIG. 17 represents pixel values in the thick line square range A of the original image 1, the thick line square range B of the original image 2, and the thick line square range D of the viewer image. The data in the range C represents flag data with the motion area as 0 and the non-motion area as 1. For example, if the viewer image is a non-motion region, the pixel value is calculated based on the value of the original image 1, and if it is a motion region, based on the intermediate value between the original image 1 and the original image 2.

  In step S57, the status of the imaging device is detected. For example, in the case of a mobile phone, another process may be performed without a user operation such as an incoming call. Stop background processing when the state of the imaging device shifts to a predetermined state during background processing, or when it is no longer in a predetermined state (when a thread other than background processing is newly started), Or interrupt. If the CPU of the imaging device is empty, the process proceeds to step S58 in order to create an image to be finally saved. If it is not empty, it loops and waits at step S57 until other processing is completed.

In step S58, an image to be finally saved is created.
In step S59, the intermediate data is deleted, and the high-quality image to be saved is updated in the viewer image.

Due to the above, when the state of the imaging device shifts to a predetermined state during background processing (for example, when an incoming call is received) or when the state is no longer predetermined (when a thread other than background processing is present) New processing (such as a telephone call) can be executed without load by canceling or interrupting the background processing (for example, when it is newly started).

(Example 2)
The second embodiment of the present invention shown in FIG. 6 has a configuration in which a work area holding unit 63 for storing the intermediate result in the case of the interruption described above is added to FIG. 3 described in the first embodiment.

  The reprocessing unit 62 receives the determination result from the status monitoring unit 10 that detects the operation state of the imaging apparatus including the image composition device and determines whether the operation state is a preset operation state. The image quality enhancement process is automatically executed in the background when the operation state is set in advance.

  In other words, the status monitoring unit 10 outputs an interruption signal to the reprocessing unit 62 when the user operates the imaging device or receives an incoming call during the high image quality processing. The reprocessing unit 62 is the same as the reprocessing unit 9 described in the first embodiment, but includes (or is connected to) a work area holding unit 63. When the reprocessing unit 62 receives an interruption signal, the image quality enhancement processing is interrupted or stopped. In the case of interruption, the work progress is held in the work area holding unit 63, and the process is continued when the execution instruction is received from the status monitoring unit 10 again. In the case of cancellation, all intermediate results are discarded, and when the execution instruction is received again from the status monitoring unit 10, the image quality improvement processing is restarted from the beginning.

Next, the operation of the second embodiment will be described with reference to the flowchart of FIG.
Steps S71 to S77 shown in FIG. 7 are the same processes as the processes of steps S51 to S57 shown in FIG.

In step S78, the re-processing unit 62 performs a process for improving the image quality for the pixels in each of the motion area and the non-motion area in order to create a final image.
In step S79, it is determined whether the creation of a high-quality image to be finally saved has been completed. It is determined whether or not processing has been completed for all the pixels of the viewer image. If completed, the process proceeds to step S711. If not completed, the process proceeds to step S710.

  In step S710, it is determined whether an interruption signal is output from the status monitoring unit 10, and if an interruption signal is output, the process proceeds to step S712. If no interruption signal is output, the process proceeds to step S78.

Step S711 is the same as step S59.
In step S712, when the reprocessing unit 62 (work area holding unit 63) receives the interruption signal, the process is interrupted or stopped. In the case of interruption, the progress is held in the work area holding unit 63, and again, When the execution instruction is received from the status monitoring unit 10, the process is continued. In the case of cancellation, all intermediate results are discarded, and when an execution instruction is received again from the status monitoring unit 10, the process is restarted from the beginning. The data is output to the reprocessing unit 62.

  As described above, when the state of the imaging device shifts to a predetermined state during background processing (for example, when an incoming call is received) or when the state is no longer predetermined (a thread other than background processing is new) By canceling or suspending background processing (such as when a phone call is started), new processing (such as a telephone call) can be executed without load.

(Example 3)
The image processing unit 81 according to the third embodiment of the present invention shown in FIG. 8 includes an image capturing unit 2, a captured image holding unit 3, a positional deviation detection unit 4, a subject motion detection unit 5, an image composition unit 82, and an intermediate data addition unit. 83, an image holding unit 84, a reprocessing unit 9, and a status monitoring unit 10. A viewer image process to be displayed to the user after shooting and a high-quality image to be finally saved are generated based on the viewer image.

  In the third embodiment, the intermediate data (motion detection map or the like) shown in the first or second embodiment is transmitted to the intermediate data adding unit 83 included in the image synthesizing unit 82 without being held in another region, and the intermediate data adding unit 83 is transmitted. Thus, intermediate data (motion detection map) is stored in the header area of the synthesized image synthesized by the image synthesis unit 82 and held in the image holding unit 8. That is, the intermediate data holding unit 6 is provided in the intermediate data adding unit 83 and the storage location of the intermediate data is set as the header area.

  As described above, by storing the intermediate data in the header area of the saved image 1, the intermediate data can be managed without a management file that holds the correspondence between the intermediate data and the saved image 1.

  For example, an image format such as JPEG or Exif has an area (comment field) where an image creator can store arbitrary information. By using this area as a header area and storing intermediate data, a file format The intermediate data can be held while maintaining compatibility without changing.

The image holding unit 84 can temporarily store the viewer image in the same manner as the image holding unit 8, but can also hold intermediate data.
By doing as described above, the intermediate data holding unit 6 need not be provided, so that a management table and a management mechanism for associating the intermediate data with the stored data become unnecessary. If it is configured by hardware, the circuit scale can be reduced, and if realized by software, the memory size can be reduced.

Example 4
A fourth embodiment of the present invention will be described with reference to FIGS. The image processing unit 91 shown in the fourth embodiment further includes a final image determination unit 92 that determines whether the image is a final image (an image subjected to high image quality processing) in the configuration shown in the third embodiment. An image based on the result is displayed on the image display unit 93.

  The final image determination unit 92 can determine whether there is a final image by detecting whether there is intermediate data (motion detection map) in the image holding unit 94. If it is not the final image, the user is notified that it is not the final image.

  The present invention has an effect that a high-quality stored image can be provided without the user being aware of it. However, before the background processing, only the viewer image generated by simple processing can be presented to the user, and thus the user may see the image and evaluate that the quality is poor. Therefore, an icon indicating that the image is before final processing is displayed on the image before starting the background processing when the image is displayed. It is detected whether or not there is intermediate data, and if there is intermediate data, the user is notified that it is before the final image.

  FIG. 10 shows an example of notifying that a high-quality image has not been generated. For example, in the case of a mobile phone, an icon or the like is displayed on the display scene (FIG. 10 shows a bill icon). Here, an icon is shown in this example, but the present invention is not limited to this, and it is sufficient that the user can be notified.

(Example 5)
Next, it is considered that there is a request for the user to confirm the final quality of the image as soon as possible, and the background processing is assigned to the operation key or the display menu, and the background processing is performed by the user's operation. Immediately implement a high image quality process for high quality.

  Embodiment 5 of the present invention will be described with reference to FIG. In the fifth embodiment, a reprocessing instruction unit 113 is further provided in the configuration of the fourth embodiment, and the reprocessing instruction unit 113 instructs the reprocessing unit 114 to perform high image quality processing according to a user instruction from the operation unit 112. Here, the reprocessing unit 114 is the same as the reprocessing unit 9 but receives a signal from the reprocessing instruction unit 113 and immediately generates a high-quality image from the viewer image.

Here, the operation unit 112 may be an operation key (an operation unit (button switch) such as a mobile phone or a digital camera), or may be assigned to a screen and controlled by software.
Although not shown in the figure, execution of image quality improvement processing by operation and image quality improvement processing execution by status monitoring may be mixed. When the user gives an instruction, the image quality improving process is immediately executed, and when not instructed, the process is automatically executed in the background when the state of the device becomes a specific state.

  Since the intermediate data is held in the first to fifth embodiments, the data amount of the image file temporarily increases. For example, when a motion area detection result is stored as intermediate data in an image of VGA (640 × 480 pixels), binary data of 38,400 bytes (307, 200 bits) is stored to store 1 bit of information per pixel. Need to hold. The file size increases accordingly.

  For example, if the intermediate data is binary data, the data may be compressed by a simple run length method or the like. In addition, since the intermediate data is unnecessary data when the background processing is completed and the final image is obtained, the background data is executed, and the intermediate data is deleted from the final image when the final processing is completed. Increase in file size can be suppressed.

(Modification)
Next, an application capable of obtaining a viewer image and a high-quality image by aligning and superimposing images will be described. For example, according to processing such as (1) camera shake correction, (2) panorama processing, and (3) super-resolution, feature points are extracted from a common area (overlapping area) of multiple images, and are aligned and combined. The purpose of each is different.

  (1) Camera shake correction reduces noise caused by short exposure time by overlapping pixels at the same position. (2) In panorama formation, a wide-angle image is created across a common area. (3) Super-resolution uses high-resolution interpolation pixels by using pixels that are slightly deviated, thereby increasing the resolution.

  As described above, in any image processing, it is assumed that a plurality of images used for composition are the same image. When the subject moves, there are problems such as inconveniences such as multiplexing and reduced accuracy.

  As with camera shake correction, panoramaization and super-resolution can be performed by detecting motion, storing motion detection results, etc. as intermediate data, and performing high-quality processing using intermediate data in the background. Conceivable.

An example in the case of panorama is shown below.
In panorama processing, images are combined across a common area. In the example of panorama composition shown in FIG. 12, the left side of the original image 1 (FIG. 12A) and the right side of the original image 2 (FIG. 1B) are common areas (FIG. 1C). When the amount of shift between the two images is calculated and combined using the common area, an image in which the car has run out can be obtained, for example, as shown in FIG.

Therefore, the motion of the common area is detected, and the common area is the left side of the original image 1 and the right side of the original image 2.
In the motion detection map of FIG. 13A, since the shadow reaches the left side surface, it can be seen that the subject with this motion is the object in the second image. Therefore, if the motion area range of the second image is cut out along the line of the motion area (car) and overwritten on the common area of the panorama composite image of FIG. 12D, as shown in FIG. It is possible to create an unnatural image that does not cut the car.

  In this case, the image held as the intermediate data is a motion detection map (FIG. 13A) and a second common area (FIG. 13B). Also, if the image is simply cut out and pasted, it may result in an unnatural image such as an outline rising. If this process is performed, a more natural image can be created.

Hereinafter, an example in the case of super-resolution will be shown.
Super-resolution image composition is the same as image stabilization. As shown in FIG. 14, an image with high resolution is generated.

  As shown in FIG. 14, there is no problem if there is no motion, but in the case of an image with subject motion, regions with subject motion are multiplexed. In addition, problems such as a wrong pixel interpolation result occur. For this reason, as in the case of camera shake correction, a motion area is detected, non-motion areas are interpolated using two images (interframe interpolation), and motion areas are interpolated using only one image (intraframe interpolation). )I do. Intra-frame interpolation is less accurate than inter-frame interpolation, and high-quality areas and high-quality areas are mixed in the image. In order to improve the quality of intra-frame interpolation, it is necessary to perform complicated processing (for example, bicubic interpolation).

  Therefore, in the first step, the motion region in FIG. 14C is enlarged by simple copying, and the motion detection map shown in FIG. 15 is held as intermediate data. Since a reduced image is used for display in the viewer, even if the common area is copy enlargement, it becomes difficult to understand the difference in quality. Moreover, since it is a copy, processing can be completed at high speed. Later, based on the motion detection map, high-precision interpolation calculation is performed by bicubic etc. from the pixels of the motion region (the region where the copy is enlarged) to improve the quality of the motion region.

  Further, using a PC or the like, a predetermined program (multi-channel data transfer program) whose source is the operation or step shown in the above embodiment is stored in a memory (ROM or the like) and written into a computer at the time of startup. Steps can be performed.

  In addition, using a single computer (including an information processing device such as a CPU or FPGA), a predetermined program that uses the steps shown in the above embodiments as a source is stored in a memory (such as a ROM), and the computer is activated at the time of startup. And the above steps can be performed.

  In addition, a control program for causing a CPU of a standard computer (for example, an image processing apparatus) to perform the processing of the flowchart as described above is created and recorded on a computer-readable recording medium, and the program is recorded. The present invention can be implemented even if it is read from a medium into a computer and executed by a CPU.

An example of a recording medium from which the recorded control program can be read by the computer system is shown in FIG. As such a recording medium, for example, recording is performed by being inserted into a storage device 161 such as a ROM or a hard disk device provided as an internal or external accessory device in the computer system 160, or a medium driving device 162 provided in the computer system 160. A portable recording medium 163 such as a flexible disk, an MO (magneto-optical disk), a CD-ROM, a DVD-ROM, or the like that can read the control program that has been read can be used.

  The recording medium may be a storage device 166 included in a computer system that functions as the program server 165 and is connected to the computer system 160 via the communication line 164. In this case, a transmission signal obtained by modulating a carrier wave with a data signal representing a control program is transmitted from the program server 165 to the computer system 160 through the communication line 164 as a transmission medium, and the computer system 160 receives the transmission signal. By demodulating the transmitted signal and reproducing the control program, the control program can be executed by the CPU of the computer system 160.

The present invention is not limited to the above-described embodiment, and various improvements and modifications can be made without departing from the gist of the present invention.
(Appendix 1)
An image composition device for compositing a plurality of images,
An intermediate data holding unit for holding detection results obtained by detecting the movement of the subject based on at least the plurality of images as intermediate data;
An image composition unit for generating a viewer image that is a composite image for the plurality of images;
An image holding unit for holding the viewer image;
A reprocessing unit for performing high image quality processing for generating a high quality image using the viewer image and the intermediate data;
An image synthesizing apparatus comprising:

(Appendix 2)
The intermediate data is
The image synthesizing apparatus according to claim 1, further comprising region information obtained by dividing the image according to the number of synthesized images.

(Appendix 3)
The reprocessing unit
An operation preset by receiving the determination result from a status monitoring unit that detects an operation state of the imaging apparatus including the image composition device and determines whether the operation state is a preset operation state. The image synthesizing apparatus according to appendix 1 or 2, wherein the image quality enhancement process is executed in the background when in a state.

(Appendix 4)
The high image quality processing is
When the determination result becomes a preset operation state during execution of the image quality enhancement process, the image quality enhancement process is stopped or interrupted,
In the case of cancellation, all the intermediate results are discarded, and when the execution instruction is received again from the status monitoring unit, the image quality improvement processing is restarted from the beginning,
In case of interruption, keep track
The image synthesizing device according to Supplementary Note 3, wherein

(Appendix 5)
An image display unit for displaying the viewer image and the high-quality image;
A final image determination unit for determining whether or not the intermediate data exists;
The image synthesizing apparatus according to appendix 1 or 2, wherein when the final image determination unit determines that no intermediate data exists, the image display unit is notified that the image quality enhancement processing has not been completed.

(Appendix 6)
The image synthesizing apparatus according to claim 5, further comprising a reprocessing instruction unit that is a user interface that immediately executes the image quality improvement processing when the image quality improvement processing is incomplete.

(Appendix 7)
The intermediate data holding unit is provided in the image holding unit as an intermediate data adding unit, and when the image data held by the image holding unit has a header area, the intermediate data is held in the header area. Or the image synthesizing apparatus according to 2;

(Appendix 8)
8. The image composition apparatus according to appendix 7, wherein the intermediate data stored in the header area is deleted after the image quality enhancement processing is completed.

(Appendix 9)
An image composition method for combining a plurality of images,
An intermediate data holding step for holding, as intermediate data, a detection result obtained by detecting the movement of the subject based on at least the plurality of images;
An image synthesis step of generating a viewer image that is a synthesized image for the plurality of images;
An image holding step for holding the viewer image;
A reprocessing step of performing a high image quality process for generating a high quality image using the viewer image and the intermediate data;
An image synthesizing method characterized by comprising:

(Appendix 10)
The intermediate data is
The image composition method according to appendix 9, characterized in that it has region information obtained by dividing the image into regions according to the number of composite images.

(Appendix 11)
The reprocessing step includes
An operation state of an imaging device including the image composition device is detected, the determination result is received from a status monitoring unit that determines whether the operation state is a preset operation state, and the determination result is The image synthesizing method according to appendix 9 or 10, wherein the image quality enhancement process is executed in the background in the set operation state.

(Appendix 12)
The high image quality processing is
When the determination result becomes a preset operation state during execution of the image quality enhancement process, the image quality enhancement process is stopped or interrupted,
In the case of cancellation, all the intermediate results are discarded, and when the execution instruction is received again from the status monitoring unit, the image quality improvement processing is restarted from the beginning,
In case of interruption, keep track
The image synthesizing method according to appendix 11, which is characterized in that.

(Appendix 13)
The image synthesizing method according to appendix 9 or 10, wherein when there is no intermediate data, notification is made that the image quality enhancement processing is incomplete.

(Appendix 14)
14. The image composition method according to appendix 13, wherein when the image quality improvement process is incomplete, an instruction to immediately execute the image quality improvement process is given.

(Appendix 15)
11. The image composition method according to appendix 9 or 10, wherein when the image data has a header area, the intermediate data is held in the header area.

(Appendix 16)
The image composition method according to appendix 15, wherein the intermediate data stored in the header area is deleted after the image quality enhancement processing is completed.

(Appendix 17)
In the computer of the image processing apparatus that acquires a plurality of images and creates a composite image,
An intermediate data holding step for holding, as intermediate data, a detection result obtained by detecting the movement of the subject based on at least the plurality of images;
An image synthesis step of generating a viewer image that is a synthesized image for the plurality of images;
An image holding step for holding the viewer image;
A reprocessing step of performing a high image quality process for generating a high quality image using the viewer image and the intermediate data;
A program for an image processing apparatus that causes a computer to execute.

(Appendix 18)
The program of the image processing apparatus according to appendix 17, wherein the intermediate data includes area information obtained by dividing the image according to the number of synthesized images.

(Appendix 19)
The reprocessing step includes
An operation state of an imaging device including the image composition device is detected, the determination result is received from a status monitoring unit that determines whether the operation state is a preset operation state, and the determination result is The program for an image processing apparatus according to appendix 18, wherein the image quality improvement processing is automatically executed in the background in the set operation state.

(Appendix 20)
The high image quality processing is
When the determination result becomes a preset operation state during execution of the image quality enhancement process, the image quality enhancement process is stopped or interrupted,
In the case of cancellation, all the intermediate results are discarded, and when the execution instruction is received again from the status monitoring unit, the image quality improvement processing is restarted from the beginning,
The program of the image processing apparatus according to appendix 20, wherein an intermediate process is retained in the case of interruption.

(Appendix 21)
The program of the image processing apparatus according to appendix 17 or 18, wherein when there is no intermediate data, notification is made that the image quality enhancement processing is incomplete.

(Appendix 22)
The program of the image processing apparatus according to appendix 21, wherein the image quality improvement process is immediately executed when the image quality improvement process is incomplete.

(Appendix 23)
19. The image processing apparatus program according to appendix 17 or 18, wherein the intermediate data is held in the header area when the image data has a header area.

(Appendix 24)
24. The image processing apparatus program according to appendix 23, wherein the intermediate data stored in the header area is deleted after the image quality enhancement processing is completed.

(Appendix 25)
The high image quality processing is
The image processing apparatus according to claim 1, wherein contour enhancement processing is performed on the non-motion region, and noise removal processing is performed on the motion region.

(Appendix 26)
The intermediate data is
The image processing apparatus according to claim 1, wherein the image processing apparatus is data generated by a camera shake correction process, a panorama process, and a super-resolution process.

It is a figure which shows the principle of this invention. It is a figure which shows the principle of a motion detection map. It is a figure which shows the structure of the image process part of the imaging device integrated in a digital camera, a mobile telephone with a camera, etc. FIG. 6 is a diagram illustrating an operation flow of the first embodiment. FIG. 6 is a diagram illustrating an operation flow of the first embodiment. FIG. 6 is a diagram illustrating a configuration of an image processing unit according to a second embodiment. FIG. 10 is a diagram illustrating an operation flow of the second embodiment. FIG. 10 is a diagram illustrating a configuration of an image processing unit according to a third embodiment. FIG. 10 is a diagram illustrating a configuration of an image processing unit according to a fourth embodiment. It is a figure which shows the example which notifies that the high quality image is not produced | generated. FIG. 10 is a diagram illustrating a configuration of an image processing unit according to a fifth embodiment. It is a figure which shows the example of a synthesis | combination of panorama. It is a figure which shows the example of a synthesis | combination of panorama. It is a figure which shows the synthesis example of super-resolution. It is a figure which shows a motion detection map. It is a block diagram which shows a control system. The upper block diagram is a configuration in which the original image 1 and the original image 2 are combined by the image combining unit to generate a viewer image, and the original image 1 and the original image 2 are used to generate intermediate data (motion detection map). The lower data represents the values of the pixels in the thick line square range A of the original image 1, the thick line square range B of the original image 2, and the thick line square range D of the viewer image. The data in the bold square range C represents the flag data with the moving area set to 1 and the non-moving area set to 0. It is a figure which shows a prior art example.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing part 2 Image imaging part 3 Captured image holding part 4 Position shift detection part 5 Subject motion detection part 6 Intermediate data holding part 7 Image composition part 8 Image holding part 9 Reprocessing part 10 Status monitoring part 61 Image processing part 62 Re Processing unit 63 Work area holding unit 81 Image processing unit 82 Image composition unit 83 Intermediate data adding unit 84 Image holding unit 91 Image processing unit 92 Final image determination unit 93 Image display unit 94 Image holding unit 112 Operation unit 113 Reprocessing instruction unit 114 Reprocessing unit 160 Computer system 161 Storage device 162 Medium drive device 163 Portable recording medium 164 Communication line 165 Program server 166 Storage device

Claims (4)

An image composition device for compositing a plurality of images,
An intermediate data holding unit for holding detection results obtained by detecting the movement of the subject based on at least the plurality of images as intermediate data;
An image composition unit for generating a viewer image that is a composite image for the plurality of images;
An image holding unit for holding the viewer image;
A reprocessing unit that performs high image quality processing to generate a high quality image using the viewer image and the intermediate data ,
The reprocessing unit
From the status monitoring unit that detects the operating state of the imaging apparatus including the image synthesizing device and determines whether only the minimum necessary program is operating or the CPU load is equal to or less than a set value, the determination When the result is received and the operation state is set in advance, the image quality enhancement process is executed in the background,
The high image quality processing is
When the determination result becomes a preset operation state during execution of the image quality enhancement process, the image quality enhancement process is stopped or interrupted,
In the case of cancellation, all the intermediate results are discarded, and when the execution instruction is received again from the status monitoring unit, the image quality improvement processing is restarted from the beginning,
In case of interruption, keep track
Image synthesizing apparatus according to claim and this. An image display unit for displaying the viewer image and the high-quality image;
A final image determination unit for determining whether or not the intermediate data exists;
The image synthesizing apparatus according to claim 1, wherein when the final image determination unit determines that there is no intermediate data, the image display unit is notified that the image quality enhancement processing has not been completed.   The intermediate data holding unit is provided in the image holding unit as an intermediate data adding unit, and when the image data held by the image holding unit has a header area, the intermediate data is held in the header area. 2. The image composition device according to 1. An image composition method for combining a plurality of images,
An intermediate data holding step for holding, as intermediate data, a detection result obtained by detecting the movement of the subject based on at least the plurality of images;
An image synthesis step of generating a viewer image that is a synthesized image for the plurality of images;
An image holding step for holding the viewer image;
A reprocessing step of performing a high image quality process for generating a high image quality image using the viewer image and the intermediate data ,
The reprocessing step includes
The determination result from the status monitoring unit that detects the operating state of the image pickup apparatus including the image composition device and determines whether only the minimum necessary program is operating or the CPU load is equal to or less than a set value. And when the determination result is a preset operation state, the image quality enhancement process is executed in the background,
The high image quality processing is
When the determination result is in a preset operation state during execution of the image quality enhancement process, the image quality enhancement process is canceled or interrupted. In the case of cancellation, all the intermediate results are discarded and the status monitoring unit When the execution instruction is received again, the image quality improvement process is restarted from the beginning.
Image composition wherein the this.