JP2017228904A - Image processing method and image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To eliminate the problem in that if manual zooming speed varies, a moving image or the like captured by manual zooming operation performed by a user may impart discomfort to a viewer.SOLUTION: An image processing method comprises an image processing step of performing at least a magnification varying process or a replacement process, the magnification varying process being such that each of at least some images of a plurality of images captured while a zooming operation is performed is enlarged or reduced so as to smooth a change in zooming magnification when the plurality of images are reproduced after processed, and the replacement process being such that replacement is carried out using some of the plurality of images thus captured. In the image processing step, the magnification varying process is performed for each of the plurality of images when a first condition is satisfied, and the replacement process is performed when a second condition different from the first condition is satisfied.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing method, and more particularly to a method for correcting a zoom magnification of an image.

  In a digital camera or the like having a lens unit, a moving image can be captured while zooming in and out by manually rotating the zoom ring. Furthermore, in a mobile phone with a camera function such as a smartphone, by performing a pinch-out operation that moves two fingers on the touch panel so as to spread, an electronic zoom is performed to enlarge or reduce the image according to the user's finger. You can shoot movies. However, when performing these operations, the zoom speed may become non-uniform. For example, when the zoom ring is manually rotated, the rotation speed may not be stable, and the zoom speed may be uneven. Alternatively, it is not easy to perform pinch out slowly at a uniform speed, and there may be a moving image with a nonuniform zoom speed.

  With respect to the above problem, Patent Document 1 discloses a technique for generating a smooth moving image with a stable zoom speed by enlarging or reducing an image of each frame with respect to a moving image with a non-uniform zoom speed. ing.

JP 2011-40945 A

  However, when generating a smooth video with a stable zoom speed by enlarging or reducing the image of each frame for a video with a non-uniform zoom speed, the enlargement ratio for enlarging the image of each frame is too high. There is a problem that the image is blurred. Further, if the reduction ratio for reducing each frame image is too high, there is a problem that an output image size to be generated becomes smaller and an area where image information is missing occurs.

  In the above-described Patent Document 1, an example in which each frame image is enlarged or reduced to generate a smooth moving image is described. However, smoothness in consideration of a blurred area of an image at the time of enlargement or a missing area of image information at the time of reduction. There is no mention of the creation of a simple video.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to generate a moving image with improved zoom speed stability while suppressing deterioration in image quality.

  In the present invention, for each of at least a part of a plurality of images captured while performing a zoom operation, the zoom magnification or the change when reproducing the plurality of images after processing is smoothed. An image processing step for performing at least one of a scaling process for performing reduction and a replacement process for replacing part of the plurality of captured images, and each of the plurality of images in the image processing step On the other hand, an image processing method is provided in which the scaling process is performed when a first condition is satisfied, and the replacement process is performed when a second condition different from the first condition is satisfied. To do.

  According to the present invention, it is possible to reduce a deterioration in image quality of a moving image captured while performing a zoom operation, and to smoothly change the zoom magnification of the image.

It is a figure for showing the structure of the digital camera in the present invention. It is a flowchart which shows the process for improving the stability of the zoom of the moving image in 1st Embodiment. It is a figure for demonstrating the relationship between the zoom magnification and time in 1st Embodiment. It is a figure for demonstrating the reduction process in this invention. It is a figure for demonstrating the expansion process and trimming process in this invention. It is a figure for demonstrating the replacement process in 1st Embodiment. It is a flowchart which shows the process for improving the stability of the zoom of the moving image in 2nd Embodiment. It is a figure for demonstrating the position of the digital camera 100 calculated in 2nd Embodiment. It is a flowchart which shows the process for improving the stability of the zoom of the moving image in 3rd Embodiment. It is a figure for demonstrating the comparison of the signal value of the 1st image and 2nd image in 3rd Embodiment. It is a figure which shows the non-moving body area | region and moving body area | region in 3rd Embodiment.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, a digital camera will be described as an example. However, the present invention is not limited to a digital camera, and a moving image captured by performing a zoom operation can be acquired. The present invention can be applied to any image processing apparatus that can perform processing. In the following, the case of zooming in (zooming to the telephoto side) will be described, but the present invention can also be applied to the case of zooming out (zooming to the wide angle side). Here, an example in which the zoom lens is driven by the user operating the zoom ring during moving image capturing will be described, but the present invention is not limited to this. The present invention can also be applied to the case where the zoom lens is driven by the operation of the zoom lever or the case where the electronic zoom is performed to enlarge or reduce the image by the operation of pinch out.

(First embodiment)
The digital camera 100 can capture still images and moving images, and can perform a zoom operation during imaging. Furthermore, the digital camera 100 can perform a scaling process for performing an enlargement process or a reduction process on an image captured and stored or an image input from the outside. FIG. 1 is a block diagram showing the structure of a digital camera 100 according to this embodiment.

  The control unit 101 is a signal processor such as a CPU or an MPU, for example, and controls each part of the digital camera 100 while reading a program built in the ROM 105 described later. A command from the user is also input to the digital camera 100 by an operation unit 110 described later, and reaches each part of the digital camera 100 through the control unit 101. For example, as will be described later, the control unit 101 instructs the imaging unit 104 described later to start and end imaging. Alternatively, an image processing command is issued to an image processing unit 107 to be described later based on a preset setting.

  The drive unit 102 is configured by a motor or the like, and mechanically operates an optical system 103 to be described later under a command from the control unit 101. For example, based on a command from the control unit 101, the driving unit 102 moves the position of the focus lens included in the optical system 103 and adjusts the focus position of the optical system 103.

  The optical system 103 includes a zoom lens, a focus lens, a diaphragm, and the like. The diaphragm is a mechanism that adjusts the amount of transmitted light. In the present embodiment, the zoom operation by the optical system 103 is performed by the user operating the zoom ring included in the operation unit 110 as described above. The optical system 103 is provided with a sensor that detects the position of the zoom lens, and the output of this sensor is transmitted to the control unit 101. Therefore, even when the user performs a zoom operation by operating the zoom ring, the control unit 101 can grasp the zoom magnification of the optical system 103 based on the output of the sensor.

  The imaging unit 104 is a photoelectric conversion element, and performs photoelectric conversion that converts an incident optical signal into an electrical signal. For example, a CCD sensor, a CMOS sensor, or the like can be applied to the imaging unit 104. The imaging unit 104 is provided with a moving image capturing mode, and can capture a moving image using a plurality of temporally continuous images as respective frames of the moving image.

  The ROM 105 is a read-only nonvolatile memory, and stores parameters necessary for the operation of each block, in addition to the operation program for each block provided in the digital camera 100. The RAM 106 is a rewritable volatile memory, and is used as a temporary storage area for data output in the operation of each block included in the digital camera 100.

  The image processing unit 107 performs various image processing, such as white balance adjustment, color interpolation, and filtering, on the image output from the imaging unit 104 or image signal data recorded in the internal memory 109 described later. . Further, compression processing is performed on the data of the image signal captured by the imaging unit 104 in accordance with a standard such as JPEG. In particular, the image processing unit 107 can perform a scaling process for performing an enlargement process or a reduction process with an electronic zoom on an image stored in the imaging unit 104 or a built-in memory 109 described later.

  The image processing unit 107 includes an integrated circuit (ASIC) that collects circuits for performing specific processing. Alternatively, the control unit 101 may perform some or all of the functions of the image processing unit 107 by performing processing according to a program read from the ROM 105. When the control unit 101 has all the functions of the image processing unit 107, it is not necessary to have the image processing unit 107 as hardware.

  The display unit 108 is a liquid crystal display, an organic EL display, or the like for displaying an image temporarily stored in the RAM 106, an image stored in an internal memory 109 described later, a setting screen of the digital camera 100, or the like.

  The built-in memory 109 is a place for recording an image captured by the imaging unit 104, an image obtained by the processing of the image processing unit 107, and the like. A memory card or the like may be used instead of the built-in memory.

  The operation unit 110 is, for example, a button, switch, key, or mode dial attached to the digital camera 100, or a touch panel that is also used as the display unit 108. A command from the user reaches the control unit 101 via the operation unit 110. The operation unit 110 also includes a zoom ring attached to the optical system 103. When the user operates the zoom ring, the position of the zoom lens included in the optical system 103 is changed by the mechanism connected to the zoom ring without using the control unit 101.

  The apparatus motion detection unit 111 is a device configured by a gyro sensor to detect the motion of the digital camera 100. The device motion detection unit 111 detects the motion of the digital camera 100 in the yaw direction and the pitch direction based on the angular change per unit time, that is, the angular velocity. To detect.

  FIG. 2 is a flowchart showing a process for improving the zoom stability of a moving image in the present embodiment.

  When a user moves a zoom ring and takes a moving image that has been zoomed, or reads a moving image shot in this way from the built-in memory 109, the control unit 101 starts the process shown in the flowchart of FIG. At the time of shooting a moving image, information on the zoom magnification detected by the control unit 101 for each frame of the moving image is stored in the built-in memory 109 together with the moving image. Note that the process shown in FIG. 2 may be performed immediately after imaging, or may be performed when a captured moving image is played back.

  In step S201, the control unit 101 acquires the imaging zoom magnification in each captured image. When not only the optical zoom by the optical system 103 during imaging but also the electronic zoom is performed in the image processing unit 107, the zoom magnification obtained by multiplying the zoom magnification by the optical zoom and the zoom magnification by the electronic zoom is set as the imaging zoom magnification. Get every frame.

  In step S202, the control unit 101 calculates a corrected zoom magnification for smoothing the change in the zoom magnification. The calculation of the correction zoom magnification, a known method may be used, and an example thereof will be briefly described below.

  FIG. 3 is a diagram illustrating a relationship between the zoom magnification and time. A broken line 301 represents a change in imaging zoom magnification. Between time t0 and time t1, the user performs a zoom operation so as to enlarge the subject image, and the imaging zoom magnification increases. Between time t1 and time t3, the user does not perform a zoom operation, and the imaging zoom magnification is kept constant. Between time t3 and time t4, the user again performs a zoom operation so that the user enlarges the image of the subject, and the imaging zoom magnification increases.

  When the user performs a zoom operation, the lens ring may not be rotated by a single operation, and thus the user may have to perform the operation in multiple steps. As described above, when the user performs a zoom operation, it often happens that the user pauses. Since such a pause causes unnaturalness in the captured moving image, it is necessary to smooth the zoom operation. As an example, processing is performed as follows. From time t0 to time t4, the correction zoom magnification is calculated and the zoom magnification is adjusted so that the zoom magnification changes at a constant change rate. A straight line 302 represents the corrected zoom magnification. The control unit 101 performs enlargement or reduction processing so that the zoom magnification of the image captured at each time becomes the correction zoom magnification. At time t2, when the imaging zoom magnification and the correction zoom magnification are equal, the control unit 101 does not perform enlargement or reduction processing.

  The above correction zoom magnification is proportional to time, but this is only an example, and the correction zoom magnification is not necessarily calculated so as to be proportional to time. The correction zoom magnification may be a smoothing curve that follows a change in the imaging zoom magnification to some extent.

  In step S203, the imaging zoom magnification is compared with the correction zoom magnification. If the difference between the imaging zoom magnification and the correction zoom magnification is too large, the image quality may deteriorate if the image is enlarged or reduced. As an example, the ratio between the imaging zoom magnification and the correction zoom magnification is calculated and compared with a predetermined threshold value. If the ratio is less than the threshold value, the process proceeds to step S204. If the ratio is greater than the threshold value, the process proceeds to step S205. Here, for convenience of calculation, in calculating the ratio between the imaging zoom magnification and the correction zoom magnification, the larger one may be always used as the dividend and the smaller one may be used as the divisor.

  Next, how to determine the threshold value will be described with an example.

When the corrected zoom magnification is larger than the imaging zoom magnification, a process for enlarging the captured image is necessary to set the zoom magnification of the image to the corrected zoom magnification. In order to determine the threshold value, it is necessary to calculate a limit that allows image blurring when the image is enlarged. For example, in the captured image, the edge strength is calculated by edge detection filter processing represented by a differential filter and a Sobel filter. Using such edge strength, a threshold value for enlarging processing is determined according to (Equation 1) below. Note that a and b are constants.
Threshold value during enlargement processing = (a ÷ edge strength + b) (Expression 1)

  When the edge strength is low, the image is composed of a flat portion, so that blurring is not noticeable even when enlarged. For this reason, the threshold value is increased. On the other hand, when the edge strength is high, the image is composed of a texture portion, and blurring tends to be noticeable when enlarged. In order to solve this, the threshold value is reduced.

When the corrected zoom magnification is smaller than the imaging zoom magnification, a process of reducing the captured image is necessary to set the image zoom magnification to the corrected zoom magnification. FIG. 4 is a diagram for explaining how to determine a threshold value for reduction. A region 400 in FIG. 4 is an image captured by the imaging unit 104 and input to the control unit 101 or the like when each image is captured, and is referred to as an original image. An area 401 corresponds to a display range when a captured image is displayed on the display unit 108. Here, the area 401 is referred to as a display area. In this embodiment, since it is assumed that the image is reduced in order to smooth the change in zoom magnification, image data having the number of pixels corresponding to the area 401 smaller than the entire image is set in the display area. Yes. When the image processing unit 107 reduces the image so that the image becomes smaller than the size of the display area, a blank area appears around the image. In order to prevent this, it is necessary to set a threshold value for the reduction process as in (Equation 2).
Threshold during reduction processing = (size of original image) ÷ (size of display area) (Expression 2)

  When different calculation results are obtained in both the horizontal size and the vertical size of the image, the smaller one of the calculation results is set as the threshold value.

  As described above, the control unit 101 sets a threshold value using a different threshold value determination method depending on whether each image is subjected to enlargement processing or reduction processing.

  If the difference between the imaging zoom magnification and the correction zoom magnification is less than the threshold value in step S203, the process proceeds to step S204. In step S204, the control unit 101 performs enlargement processing or reduction processing so that the zoom magnification of the captured image becomes the correction zoom magnification, and generates a new image. Hereinafter, the image generated in step S204 is referred to as a first image.

  FIG. 5 is a diagram showing an enlargement process in the present embodiment. FIG. 5A shows image data captured at the imaging zoom magnification, and enlargement processing is performed on this image. First, enlargement processing is performed according to the correction zoom magnification. The output at this time becomes the image data after enlargement in FIG. However, since the size of the image data has increased, the control unit 101 performs trimming processing so that the size does not change from that in FIG. The output after the control unit 101 performs the trimming process is shown in FIG. 5C, and the region to be trimmed is a region 501 in FIG. There are various methods for determining the position of the area 501 such as a method of detecting the main subject and the tracking subject and setting the center of the trimming position, and the center position of the front and rear frames. When performing the reduction process, similarly, the area to be trimmed can be adjusted according to the positions of the main subject and the tracking subject. The first image can be generated by the above processing.

  Note that the magnitude of noise may change because the enlargement / reduction processing is performed in the generation of the first image. If the noise levels of the images of a plurality of frames of a moving image are different, the viewer may feel uncomfortable. In order to keep the noise level the same over a plurality of images, correction processing may be performed on noise as necessary. For example, low-pass filter processing is performed on the first image, and a noise component is detected from difference information between the images before and after the processing. Enlargement processing is performed on the detected noise component, and when the first image is generated, the reduction processing is performed on the noise component at a reduction ratio for returning to the image before the enlargement processing. Then, the detected noise component of the first image is replaced with a noise component that has been subjected to reduction processing. Alternatively, an operation of adding random noise is performed on the first image having a low noise level. Or you may perform the noise reduction process which replaces with the image data after the filter process of a 1st image with respect to the noise component of a 1st image.

  If the difference between the imaging zoom magnification and the correction zoom magnification is greater than the threshold value in step S203, the process proceeds to step S205. In step S <b> 205, the control unit 101 performs replacement processing for replacing the captured image using an image captured at another time. The image used for replacement in step S205 is hereinafter referred to as a second image. The process in step S205 will be described with reference to FIGS.

  For example, it is assumed that the control unit 101 compares the imaging zoom magnification and the correction zoom magnification with respect to the image captured by the imaging unit 104 at time t3 and determines that the image is equal to or greater than the threshold value. If the enlargement process is performed with the corrected zoom magnification, a defective image may be generated due to blur or the like. Instead, an image having an imaging zoom magnification equal to the correction zoom magnification of the image at time t3 (represented by a point 305 in FIG. 3) is used for replacement.

  By replacing the second image captured at a different time with the above-described processing, it is possible to prevent defects such as blurring and blanking of the image.

  FIG. 6 is a diagram showing a state of replacement over a plurality of images. FIG. 6A shows an image group made up of images taken continuously in time by the imaging unit 104 during the zoom operation, and FIG. 6B is generated by the processing in step S204 or step S205 described above. The image group which consists of a 1st image or a 2nd image is shown. The numbers written on the images 601 to 608 indicate the imaging zoom magnification. The numbers written on the images 611 to 618 mean the correction zoom magnification. Since the zoom magnifications of the images 601 to 603 have not changed, it is necessary to adjust the zoom magnification as described above in order to smooth the zoom operation. Here, as an example, the zoom magnifications of the images 601 to 608 are adjusted so as to be an equidistant sequence having a tolerance of 1 and an initial term of 9. Then, the magnification of each image becomes the magnification of the numbers written on the images 611 to 618 in FIG.

  Here, first, for each of the images 601 to 608, an enlargement ratio (ratio between the imaging zoom magnification and the correction zoom magnification in step S203) when the first image is generated is calculated. The control unit 101 compares the enlargement ratio with the above-described predetermined threshold value, and determines whether to generate the first image or the second image. Here, the threshold is assumed to be 1.3.

  Since the ratio between the imaging zoom magnification and the correction zoom magnification of the image 601 is 9 ÷ 8 = 1.125 <1.3, the control unit 101 determines to generate the first image in the processing of the image 601. That is, the control unit 101 enlarges the image 601 to the correction zoom magnification, and generates the image 611. Similar processing is also applied to images 602, 606, 607, and 608 in which the ratio between the imaging zoom magnification and the correction zoom magnification is less than the threshold.

  On the other hand, since the ratio between the imaging zoom magnification and the correction zoom magnification of the image 603 is 11 ÷ 8 = 1.373 ≧ 1.3, the control unit 101 determines to generate the first image in the processing of the image 603. . In other words, the control unit 101 replaces the image 606 having the imaging zoom magnification equal to the correction zoom magnification of the image 603 as the second image 613 in the processing of the image 603. Similar processing is also applied to images 604, 605, and 606 in which the ratio between the imaging zoom magnification and the correction zoom magnification is greater than or equal to a threshold value.

  According to the above-described processing, the imaging zoom magnification and the correction zoom magnification are compared, and the first image and the second image are selectively generated according to the comparison result, thereby realizing a smooth zoom operation. it can. However, since the first image and the second image are captured in a non-continuous time, there may be a difference in image quality between the first image and the second image. In order to solve this, it may be necessary to perform various types of image processing on some images. For example, there are known distortion aberration correction, peripheral light amount correction, brightness correction, white balance correction, geometric conversion processing, and the like. The control unit 101 compares the images processed in step S204 or step S205 that are temporally continuous, and performs necessary image processing as appropriate.

  In the above example, an image having an imaging zoom magnification equal to the correction zoom magnification is used as it is as the second image, but the present invention is not limited to this. For example, in the example shown in FIG. 6, the image 605 to be processed is replaced using the image 607 instead of the image 608 having the imaging zoom magnification equal to the correction zoom magnification. In this case, since the imaging zoom magnification of the image 607 is different from the corrected zoom magnification of the image 605, the image 607 is not replaced directly, but is replaced after the image 607 is enlarged to the corrected zoom magnification of the image 605. Such a method is useful when there is a defect in the image 608 due to blurring at the time of imaging, and it is preferable not to use it directly. Alternatively, a replacement method using the image 608, a replacement after enlarging the image 607, or both replacement methods may be prepared and the user may select them.

  According to the first embodiment, the zoom magnification is adjusted for a plurality of images captured while performing a zoom operation, and a replacement process is performed by using images captured at different times for some images. Zoom operation can be realized, and deterioration of image quality due to adjustment of zoom magnification can be prevented.

(Second Embodiment)
In the second embodiment, unlike the first embodiment, when the difference between the imaging zoom magnification and the correction zoom magnification is less than the threshold, the control unit 101 does not directly generate the first image. Select whether to generate an image. FIG. 7 is a flowchart showing a process for improving the zoom stability of a moving image in the second embodiment. Hereinafter, the second embodiment will be described. Note that the same parts as those in the first embodiment are omitted.

  In step S701, the control unit 101 acquires the imaging zoom magnification in the same manner as in step S201 of the first embodiment. In step S702, the control unit 101 calculates a corrected zoom magnification in the same manner as in step S202 of the first embodiment. In step S703, as in step S203 of the first embodiment, the control unit 101 calculates the difference between the imaging zoom magnification and the correction zoom magnification and compares it with a predetermined threshold value.

  In step S703, when the control unit 101 determines that the difference between the imaging zoom magnification and the correction zoom magnification is equal to or greater than the threshold value, the control unit 101 proceeds to step S705 and generates a second image.

  In step S703, when the control unit 101 determines that the difference between the imaging zoom magnification and the correction zoom magnification is less than the threshold value, the control unit 101 proceeds to step S704. Here, when the control unit 101 determines to generate the first image, the control unit 101 proceeds to step S706 and generates the first image. When determining that the first image is not generated, the control unit 101 proceeds to step S705 and generates the second image.

  As a method for selecting an image in step S704, there is a method in which the user is presented and selected one by one. However, such a method is troublesome for the user.

  Hereinafter, an example in which the control unit 101 automatically determines which image to generate will be described. It is assumed that the digital camera 100 moves greatly due to blurring while capturing a plurality of images, and then returns to the original location by a user operation. In such a case, the device motion detection unit 111 of the digital camera 100 detects speed information. The control unit 101 calculates based on the detected speed information and calculates position information. FIG. 8 shows an example of the relationship between the calculated position information and time. In the period 801, the user performs zoom imaging in a state where the movement of the digital camera 100 is small and is almost within the position. The period 802 is a period in which the digital camera 100 moves greatly due to camera shake or some cause and an operation for returning the position of the digital camera 100 is performed. In the period 803, the image is taken so as not to move back to the original position. In this case, the subject intended by the user may not be captured in the period 802. Therefore, the control unit 101 selects to generate the second image in the period 802 or a section in which the calculated distance information is larger than the predetermined threshold (exceeds the horizontal line 811).

  In the above-described automatic determination method, when it is determined that the difference between the imaging zoom magnification and the correction zoom magnification is equal to or greater than the threshold value, the control unit 101 further performs blurring through the speed information detected by the apparatus motion detection unit 111 during imaging. Determine if there is. If there is blurring, the process proceeds to step S705 to generate a second image.

  According to the second embodiment, when it is determined that the difference between the imaging zoom magnification and the correction zoom magnification is equal to or greater than the threshold value, the determination condition can be further set once until the first image is generated. With such a setting, it is possible to prevent the influence of blur and the like, and it is possible to further improve the image quality of the image after adjusting the zoom magnification.

(Third embodiment)
Unlike the first embodiment, the third embodiment determines whether there is a moving object, and does not generate the second image when there is a moving object. FIG. 9 is a flowchart illustrating a process for improving the zoom stability of a moving image according to the third embodiment. Hereinafter, a third embodiment will be described. Note that the same portions as those in the first embodiment or the second embodiment are omitted.

  In step S901, the control unit 101 acquires the imaging zoom magnification in the same manner as in step S201 of the first embodiment. In step S902, the control unit 101 calculates a corrected zoom magnification in the same manner as in step S202 of the first embodiment. In step S903, as in step S203 of the first embodiment, the control unit 101 calculates the difference between the imaging zoom magnification and the correction zoom magnification and compares it with a predetermined threshold value.

  If the difference between the imaging zoom magnification and the correction zoom magnification is less than a predetermined threshold value in step S903, the process proceeds to step S906, and the control unit 101 generates a first image.

  If the difference between the imaging zoom magnification and the correction zoom magnification is greater than or equal to a predetermined threshold value in step S903, the process proceeds to step S904. In step S904, both the first image and the second image are generated by the same method as in step S204 and step S205 in the first embodiment.

  In step S905, it is determined whether there is a moving object in the image. If there is a moving object and it is replaced with an image at a different time, there is a possibility that the time series of the frames of the moving image may be switched, resulting in an unnatural moving image. For this reason, when there is a moving object, it is necessary to select the first image without selecting the second image.

  As described above, when determining in step S905 that there is no moving object, the control unit 101 proceeds to step S907 and selects the second image. In step S905, when the control unit 101 determines that there is a moving object, the control unit 101 proceeds to step S908 and selects the first image. Hereinafter, the determination of the moving object in the control unit 101 in step S905 will be described with an example.

  FIG. 10 is a diagram for explaining determination of a moving object.

  In step S905, the first image acquired in step S904 is compared with the second image, and the control unit 101 determines whether there is a moving object. As an example of the determination method, a signal value difference between frames of the first image and the second image is calculated.

  For example, FIG. 10A shows an example of signal values of image data when a moving object is included. The signal value 1001 of the first image is indicated by a solid line, and the signal value 1002 of the second image is indicated by a broken line. The horizontal axis represents the coordinates of the horizontal or vertical image data, and the vertical axis represents the signal value of the image data.

  In the graph shown in FIG. 10A, since the difference between the signal value 1001 of the first image and the signal value 1002 of the second image is significant in the range 1011, it can be predicted that a moving object exists in the range 1011. A threshold value for the difference in signal value is provided in the control unit 101 in advance, and when a region that is equal to or greater than the threshold value is detected, it is determined that a moving object exists.

  However, for example, in the graph shown in FIG. 10B, when the difference between the signal value 1003 of the first image and the signal value 1004 of the second image is kept constant over the entire range, changes in AE and weather There is a high possibility that it is caused by something other than a moving object. In such a case, the control unit 101 determines that there is no moving object.

  For comparing one or both of the first image and the second image when comparing the signal values of the first image and the second image. The difference may be taken after low-pass filter processing or high-pass filter processing is performed. Thereby, it is possible to acquire a difference between signal values excluding a difference in frequency bands held between the first image and the second image.

  Thus, a moving body can be detected by comparing the signal values of the first image and the second image. In accordance with the detection result, the control unit 101 selects the first image or the second image.

  In addition, this invention is not limited by the above description, A various deformation | transformation and change are possible within the range of the summary. In the following, a method in which a moving object exists only in a part of an image, and an image is synthesized using a second image part corresponding to the part and a first image part corresponding to the part other than the part. Will be described.

  For example, when it is determined that there is a difference in signal value between the first image and the second image, the second image is used for the region where the difference in signal value occurs, and the region other than that region is used. On the other hand, an operation of combining using the first image may be performed. FIG. 11 shows the composition using the first image and the second image. An area 1102 (non-moving object area) where a signal value difference is detected is synthesized using the first image, and an area 1101 (moving object area) where no signal value difference is detected is used using the second image. A composite image can also be generated. Thereby, it is possible to output a sharp image in which the sense of resolution of the image is not lost in a region where no moving object is detected.

  According to the third embodiment, a moving object can be detected by generating both the first image and the second image and comparing the signal values. Further, when a moving object is detected, it is possible to prevent the selection of the first image and generation of an unnatural moving image.

(Other embodiments)
The above embodiment has been described based on the implementation with a digital camera, but is not limited to a digital camera. For example, it may be implemented by a portable device with a built-in image sensor or a network camera capable of capturing an image.

  In the above description, the zoom operation is based on the optical zoom operation. However, the present invention is not limited to this. For example, the electronic zoom can be performed by the user manually operating the touch panel during moving image capturing. The present invention can also be used when trying to adjust the unevenness of the operation in the electronic zoom.

  Note that the present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus execute the program. It can also be realized by a process of reading and operating. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 100 Digital camera 101 Control part 102 Drive part 103 Optical system 104 Imaging part 105 ROM
106 RAM
107 Image processing unit 108 Display unit 109 Built-in memory 110 Operation unit 111 Device motion detection unit

Claims (12)

A variable for enlarging or reducing each of at least some of the plurality of images captured while performing the zoom operation so that the zoom magnification changes smoothly when the plurality of images are reproduced after processing. An image processing step of performing at least one of a doubling process and a replacement process using a part of the plurality of captured images;
In the image processing step, for each of the plurality of images, the scaling process is performed when a first condition is satisfied, and the replacement process is performed when a second condition different from the first condition is satisfied. And an image processing method. A variable for enlarging or reducing each of at least some of the plurality of images captured while performing the zoom operation so that the zoom magnification changes smoothly when the plurality of images are reproduced after processing. An image processing step of performing at least one of a doubling process and a replacement process using a part of the plurality of captured images;
In the image processing step,
A first image is generated and output by performing the scaling process on an image satisfying a first condition among the plurality of images,
A second image and a third image are generated by performing the scaling process and the replacement process on an image satisfying a second condition among the plurality of images, and the second image is generated. And the third image are output, or an image obtained by combining the second image and the third image is output. A moving object detection step and a first selection step;
3. In the first selection step, when a moving object is detected in the image in the moving object detection step, a second image is selected and output as a processed image of the selected image. An image processing method described in 1. A moving object detection step and an image synthesis step;
In the moving object detection step, when a moving object is detected from a partial area of the image,
In the image synthesizing step, an image is generated by using a second image portion corresponding to a partial region of the image and a first image portion corresponding to a region other than the partial region of the image. The image processing method according to claim 2, wherein the images are combined and output as a processed image.   5. The image processing method according to claim 3, wherein the moving object is detected by comparing the first image and the second image in the moving object detection step. Further comprising generating a corrected zoom magnification,
In the replacement process, at least a part of the plurality of images is replaced with an image having an imaging zoom magnification equal to the correction zoom magnification of the image to be replaced.
6. The zoom magnification of another image is performed so as to be equal to the correction zoom magnification of the image, and the image is replaced with the image after the enlargement / reduction processing. The image processing method according to any one of the above.   The image processing method according to claim 6, wherein the correction zoom magnification changes more smoothly than the imaging zoom magnification. Having a step of predetermining a threshold value of a difference in zoom magnification;
The first condition is:
The difference between the corrected zoom magnification and the imaging zoom magnification when taking the image is
8. The image processing method according to claim 6, wherein the image processing method is less than a zoom magnification difference threshold value determined in the step of determining the zoom magnification difference threshold value. Having a second selection step;
The first condition is that an image to be processed is an image in which a difference between the correction zoom magnification and the imaging zoom magnification is less than a threshold value of the difference in magnification, and is selected in the second selection step. The image processing method according to claim 6, wherein the image processing method is an image processing method. An image capturing step of capturing an image using the imaging unit;
A detection step of detecting information related to the movement of the imaging unit,
The said 2nd selection process WHEREIN: When imaging in the imaging process of the said image, an image is selected based on the information regarding the motion of the said imaging part detected in the said detection process, The image of Claim 9 characterized by the above-mentioned. Image processing method.   The image according to any one of claims 6 to 10, wherein the second condition is that a difference between the correction zoom magnification and the imaging zoom magnification is equal to or greater than a threshold value of the difference in zoom magnification. Processing method. A variable for enlarging or reducing each of at least some of the plurality of images captured while performing the zoom operation so that the zoom magnification changes smoothly when the plurality of images are reproduced after processing. Image processing means for performing at least one of a doubling process and a replacement process using a part of the plurality of captured images;
The image processing means performs the scaling process when a first condition is satisfied for each of the plurality of images, and performs the replacement process when a second condition different from the first condition is satisfied. An image processing apparatus characterized by

Images