JP6929133B2 - Imaging device, control method and program of imaging device - Google Patents

Description

The present invention relates to an image pickup apparatus, and more particularly to an image pickup apparatus for generating a panoramic image.

A technique is known in which a plurality of images are captured while moving a digital camera or the like, and a panoramic image is generated by joining the plurality of captured images. For example, Patent Document 1 discloses a technique for generating a panoramic image using a moving image. However, in the technique disclosed in Patent Document 1, if a moving body is present, the same moving body may appear a plurality of times, resulting in a composite image that is uncomfortable for the user.

In Patent Document 2, the image sensor is divided into a plurality of regions and shooting is repeated while panning, and among the images generated by each shooting, the images corresponding to the same region of the image sensor are joined to form a plurality of panoramic images. To generate. Then, a method of generating a panoramic image in which a moving object is deleted by detecting a moving object from each of the plurality of panoramic images and synthesizing the plurality of panoramic images using a region in which the moving object does not exist is disclosed. In the technique disclosed in Patent Document 2, it is described that a plurality of regions for panoramic composition are set in an image captured during panoramic imaging.

Japanese Unexamined Patent Publication No. 2000-244814 Japanese Unexamined Patent Publication No. 2014-11782

However, in the technique disclosed in Patent Document 2, since it is necessary to generate a plurality of panoramic images before generating a panoramic image in which moving objects are deleted, there arises a problem that a large amount of memory is required.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an imaging device that realizes removal of unnecessary moving objects with a smaller amount of memory when moving objects are present in the composition of panoramic images. ..

The present invention provides an imaging means for capturing a plurality of images that are continuous in time, a first region for each of the plurality of images, and a second region for each part of the plurality of images. A control means for controlling storage in the storage means and deletion from the storage means with respect to the first region and the second region in the plurality of images in parallel with the imaging of the plurality of images. In each of the plurality of images, if the first region satisfies a predetermined condition, the image of the first region is used, and if the first region does not satisfy the predetermined condition, the image of the first region is used. A panoramic image using an image of the second region cut out from an image captured before the cut-out image, which has an angle of view overlapping with the first region and satisfies the predetermined conditions. The control means has an angle of view that overlaps with the first region if the first region satisfies the predetermined condition, and the control means is an image obtained by cutting out the first region. Provided is an image pickup apparatus characterized in that a second region cut out from an image captured before is deleted from the storage means.

According to the present invention, in panoramic composition, it is possible to create a panoramic image in which one moving object does not exist in duplicate by utilizing motion detection in different cutout regions in the captured image, and it is possible to reduce the amount of memory used.

It is a rear perspective view which shows the schematic structure of the digital camera which concerns on embodiment of this invention. It is a block diagram which shows the hardware structure of the digital camera which concerns on embodiment of this invention. It is a figure for demonstrating the relationship between the moving direction of a digital camera during panoramic imaging which concerns on embodiment of this invention, and the cut-out area of image data. It is a figure explaining the flow of the composition processing of the panoramic image which concerns on embodiment of this invention. It is a figure for demonstrating the problem of panoramic composition in the presence of a moving body. It is a flowchart for demonstrating embodiment of this invention. It is a figure for demonstrating the determination of the 2nd region in embodiment of this invention. It is a figure for demonstrating panoramic composition in embodiment of this invention. It is a figure for demonstrating speed-up of comparison of regions in embodiment of this invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(Embodiment of the present invention)
FIG. 1 is a rear perspective view showing a schematic configuration of a digital camera according to an embodiment of the present invention.

On the back surface of the digital camera 100, a display unit 101 for displaying images and various information, and an operation unit 102 including operation members such as various switches and buttons for receiving various operations by the user are provided. Further, on the back surface of the digital camera 100, a mode changeover switch 104 for switching an imaging mode or the like and a controller wheel 103 capable of rotation operation are provided. On the upper surface of the digital camera 100, a shutter button 121 for giving an imaging instruction, a power switch 122 for switching the power of the digital camera 100 on / off, and a strobe 141 for irradiating a subject with a flash of light are provided.

The digital camera 100 can be connected to an external device via the connection cable 111 and the connector 112, and can output image data (still image data, moving image data) or the like to the external device. A storage medium slot (not shown) that can be opened and closed by a lid 131 is provided on the lower surface of the digital camera 100 so that a storage medium 130 such as a memory card can be inserted and removed from the storage medium slot. There is.

The storage medium 130 stored in the storage medium slot can communicate with the system control unit 210 (see FIG. 2) of the digital camera 100. The storage medium 130 is not limited to a memory card or the like that can be inserted into and removed from the storage medium slot, and may be an optical disk such as a DVD-RW disk or a magnetic disk such as a hard disk, and is further digital. It may be built in the main body of the camera 100.

FIG. 2 is a block diagram showing a hardware configuration of the digital camera 100. The digital camera 100 includes a barrier 201, an image pickup lens 202, a shutter 203, and an image pickup unit 204. The barrier 201 covers the imaging optical system to prevent the imaging optical system from becoming dirty or damaged. The image pickup lens 202 is composed of a lens group including a zoom lens and a focus lens, and constitutes an image pickup optical system. The shutter 203 has an aperture function and adjusts the amount of exposure to the imaging unit 204. The image pickup unit 204 is an image pickup element that converts an optical image into an electric signal (analog signal), and is, for example, an image sensor such as a CCD sensor or a CMOS sensor having a Bayer array structure in which RGB pixels are regularly arranged. .. The shutter 203 may be a mechanical shutter or an electronic shutter whose storage time is controlled by controlling the reset timing of the image sensor.

The digital camera 100 includes an A / D converter 205, an image processing unit 206, a memory control unit 207, a D / A converter 208, a memory 209, and a system control unit 210. An analog signal is output from the image pickup unit 204 to the A / D converter 205, and the A / D converter 205 converts the acquired analog signal into image data composed of digital signals, and the image processing unit 206 or the memory control unit 207. Output to.

The image processing unit 206 performs correction processing such as pixel interpolation and shading correction, white balance processing, gamma correction processing, and color conversion for the image data acquired from the A / D converter 205 or the data acquired from the memory control unit 207. Perform processing, etc. Further, the image processing unit 206 realizes an electronic zoom function by cutting out an image and performing scaling processing. Further, the image processing unit 206 performs a predetermined calculation process using the image data of the captured image, and the system control unit 210 performs exposure control and distance measurement control based on the calculation result obtained in this way. For example, the system control unit 210 performs TTL (through-the-lens) AF (autofocus) processing, AE (autoexposure) processing, and EF (strobe pre-emission) processing. The image processing unit 206 performs a predetermined calculation process using the image data of the captured image, and the system control unit 210 performs a TTL method AWB (auto white balance) process using the obtained calculation result.

The image processing unit 206 has an image compositing processing circuit that synthesizes a panoramic image from a plurality of images continuously captured in time and determines the compositing result. The image composition processing circuit is not limited to simple addition and average composition, but also comparative bright composition and comparison in which pixels having the brightest value or the darkest value in each region of the image data to be synthesized are selected to generate one image data. Processing such as dark composition can be performed. In addition, the synthesis result is evaluated and judged based on a specific criterion. Instead of the configuration in which the image processing unit 206 includes the image composition processing circuit, the function of the image composition processing circuit may be realized by software processing by the system control unit 210.

The image data output from the A / D converter 205 is written to the memory 209 via the image processing unit 206 and the memory control unit 207, or via the memory control unit 207. The memory 209 also serves as an image display memory (video memory) for storing image data to be displayed on the display unit 101. The memory 209 has a storage capacity capable of storing a predetermined number of still images, panoramic images (wide-angle images), and panoramic image composition results. The memory 209 can also be used as a work area for developing a program or the like read from the non-volatile memory 211 by the system control unit 210.

The image display data (digital data) stored in the memory 209 is transmitted to the D / A converter 208. The D / A converter 208 converts the received digital data into an analog signal and supplies it to the display unit 101, whereby the image is displayed on the display unit 101. The display unit 101 is a display device such as a liquid crystal display or an organic EL display, and displays an image based on an analog signal from the D / A converter 208. The on / off of the image display on the display unit 101 is switched by the system control unit 210, and the power consumption can be reduced by turning off the image display. The digital signal stored in the memory 209 from the imaging unit 204 through the A / D converter 205 is converted into an analog signal by the D / A converter 208 and sequentially displayed on the display unit 101 to display a through image. The electronic viewfinder function can be realized.

The digital camera 100 includes a non-volatile memory 211, a system timer 212, a system memory 213, a detection unit 215, and a strobe control unit 217. The non-volatile memory 211 is a memory (for example, EEPROM or the like) that can be electrically erased or stored, and stores a program executed by the system control unit 210, constants for operation, and the like. Further, the non-volatile memory 211 has an area for storing system information and an area for storing user setting information, and the system control unit 210 has various types of areas stored in the non-volatile memory 211 when the digital camera 100 is started. Read and restore information and settings.

The system control unit 210 has a CPU and controls the overall operation of the digital camera 100 by executing various program codes stored in the non-volatile memory 211. The programs, operation constants, variables, and the like read from the non-volatile memory 211 by the system control unit 210 are expanded on the system memory 213. RAM is used for the system memory 213. Further, the system control unit 210 controls the display by controlling the memory 209, the D / A converter 208, the display unit 101, and the like. The system timer 212 measures the time used for various controls and the time of the built-in clock. The strobe control unit 217 controls the light emission of the strobe 141 according to the brightness of the subject. The detection unit 215 includes a gyro and a sensor, and acquires angular velocity information, attitude information, and the like of the digital camera 100. The angular velocity information includes information on the angular velocity and the angular acceleration at the time of panoramic imaging by the digital camera 100. Further, the posture information includes information such as the inclination of the digital camera 100 with respect to the horizontal direction.

The display unit 101, the operation unit 102, the controller wheel 103, the shutter button 121, the mode changeover switch 104, the power switch 122, and the strobe 141 shown in FIG. 2 are the same as those described with reference to FIG.

Various operation members constituting the operation unit 102 are used for selecting various function icons displayed on the display unit 101, for example, and when a predetermined function icon is selected, a function is assigned to each scene. That is, each operating member of the operating unit 102 acts as various function buttons. Examples of the function button include an end button, a back button, an image feed button, a jump button, a narrowing down button, an attribute change button, a DISP button, and the like. For example, when the menu button is pressed, a menu screen for making various settings is displayed on the display unit 101. The user can intuitively perform the setting operation by using the menu screen displayed on the display unit 101 and the up / down / left / right four-direction buttons and the SET button.

The controller wheel 103, which is an operating member capable of rotating, is used when designating a selection item together with a four-direction button. When the controller wheel 103 is rotated, an electric pulse signal corresponding to the amount of operation (rotation angle, number of rotations, etc.) is generated. The system control unit 210 analyzes this pulse signal and controls each unit of the digital camera 100.

The shutter button 121 has a first switch SW1 and a second switch SW2. The first switch SW1 is turned on in a half-pressed state during the operation of the shutter button 121, whereby a signal instructing the preparation for imaging is transmitted to the system control unit 210. When the system control unit 210 receives the signal that the first switch SW1 is turned on, the system control unit 210 starts operations such as AF processing, AE processing, AWB processing, and EF processing. The second switch SW2 is turned on when the operation of the shutter button 121 is completed, and a signal instructing the start of imaging is transmitted to the system control unit 210. When the system control unit 210 receives the signal that the second switch SW2 is turned on, the system control unit 210 performs a series of imaging operations from reading the signal from the imaging unit 204 to writing the image data to the storage medium 130.

The mode changeover switch 104 is a switch for switching the operation mode of the digital camera 100 between various modes such as a still image imaging mode, a moving image imaging mode, and a playback mode. The still image imaging mode includes a panoramic image imaging mode in which panoramic images are combined by panoramic imaging, in addition to an auto imaging mode and the like.

The digital camera 100 includes a power supply unit 214 and a power supply control unit 218. The power supply unit 214 is a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like, and supplies power to the power supply control unit 218. The power supply control unit 218 detects whether or not a battery is installed in the power supply unit 214, the type of battery, the remaining battery level, and the like, and based on the detection result and the instruction of the system control unit 210, a necessary voltage is applied for a necessary period. It is supplied to each part including the storage medium 130.

The digital camera 100 includes a storage medium I / F 216 for enabling communication between the storage medium 130 and the system control unit 210 when the storage medium 130 is installed in a storage medium slot (not shown). Since the details of the storage medium 130 have already been described with reference to FIG. 1, the description thereof will be omitted here.

Next, a method of panoramic imaging and a method of synthesizing a panoramic image from a plurality of captured images will be described. First, a process of cutting out a predetermined region from the image data of the captured image in order to synthesize the panoramic image will be described.

FIG. 3 is a diagram for explaining the relationship between the moving direction of the digital camera 100 during panoramic imaging and the cutout region of image data.

FIG. 3A shows an effective image area of the image sensor included in the image pickup unit 204, where “Wv” is the number of effective pixels in the horizontal direction and “Hv” is the number of effective pixels in the vertical direction. FIG. 3B shows a cutout region cut out from the image data of the captured image, where “Wcrop” is the number of cutout pixels in the horizontal direction and “Hcrop” is the number of cutout pixels in the vertical direction.

FIG. 3C is a diagram showing a cutout region for image data when panoramic imaging is performed while moving the digital camera 100 in the horizontal direction indicated by the arrow. The region S1 shown by hatching in FIG. 3 (c) indicates a region cut out from the image data, and satisfies the following (Equation 1) and (Equation 2).
Wv> Wcrop ... (Equation 1)
Hv = Hcrop ... (Equation 2)

Similarly, FIG. 3D is a diagram showing a cutout region for image data when panoramic imaging is performed while moving the digital camera 100 in the vertical direction indicated by the arrow. The area S2 shown by hatching in FIG. 3 (d) indicates a cut-out area of image data, and satisfies the following (Equation 3) and (Equation 4).
Wv = Wcrop ... (Equation 3)
Hv> Hcrop ... (Equation 4)

The cutout area of the image data of the captured image may be different for each image data. Further, with respect to the image data at the start of panoramic imaging and at the end of panoramic imaging, the cutout area may be widened in order to widen the angle of view. The method for determining the cutout region of the image data can be determined, for example, by the difference between the angle of the digital camera 100 immediately after imaging and the angle of the digital camera 100 one frame before. By cutting out and storing only the image data necessary for the panoramic image compositing process, the storage capacity of the memory 209 can be saved.

The cutout region described above should be set in the center of the captured image as much as possible. This is because the center of the captured image is generally less distorted, and a more natural composite image can be generated by setting the cutout region to the center of the image.

Next, a method of synthesizing panoramic images will be described. The system control unit 210 reads the cutout area saved at the time of panoramic image capture from the memory 209, and detects the positional deviation between the images corresponding to the read image data. As an example, the system control unit 210 divides the cutout area into small blocks of an arbitrary size, and calculates a corresponding point at which the brightness SAD (Sum of Absolute Difference) is minimized for each small block. The system control unit 210 can calculate the motion vector from the corresponding point where the calculated SAD is the minimum. The system control unit 210 may use the angular velocity information, the attitude information, and the like detected by the detection unit 215 in calculating the motion vector. Subsequently, the system control unit 210 corrects the positional deviation between the images in the cutout region based on the motion vector obtained between the images in the adjacent cutout region, and weights and adds the overlapping portion between the images in the cutout region. Synthesize.

FIG. 4 is a diagram illustrating a flow of a panoramic image compositing process. In FIG. 4, the dot-hatched area is an area schematically representing a row of trees in the field of view, and the shaded area is a cut-out area of image data. FIG. 4A shows a state in which the user presses the shutter button 121 and the first switch SW1 is turned on, and shows that the user focuses on the main subject. FIG. 4B shows the position where the second switch SW2 of the shutter button 121 is turned on, and the field of view is set according to one end of the panoramic image that the user is trying to synthesize. In FIG. 4B, the imaging unit 204 captures the image 410. 4 (c) to 4 (e) schematically show a state in which a panoramic image is taken while moving the digital camera 100 toward the other end of the panoramic image that the user is trying to synthesize. .. FIG. 4E shows a state in which the user stops pressing the shutter button 121 and panoramic imaging is completed. In FIGS. 4 (b) to 4 (e), the image pickup unit 204 has captured a total of seven images 410 to 470, but the images 430, 450 and 460 are not shown. The image processing unit 206 performs cutout processing on the images 410 to 470 captured by the image pickup unit 204 to generate cutout regions 411 to 471. In the system control unit 210, the width of the cutout region may be predetermined, but may be changed according to the moving speed of the digital camera 100 during panoramic imaging.

FIG. 4 (f) shows a panoramic image formed by the image processing unit 206 synthesizing a plurality of images captured by the image pickup unit 204. Here, since the upper side and the lower side of the cutout areas 411 to 471 do not match due to camera shake or the like, the image processing unit 206 also performs the cutout process in the vertical direction. As a result, the image processing unit 206 generates a panoramic image as shown by the area 400.

FIG. 4 shows an example in which the cutout regions 411 to 471 do not have overlapping regions and are adjacent to each other for ease of explanation. If there is an overlapping area, the image processing unit 206 outputs the pixel information of one cutout area on the left side and the other cutout area on the right side to the composite image with the center of the overlapping area as a boundary. do. Alternatively, the image processing unit 206 outputs a value obtained by synthesizing the pixel information of both cutout regions by 50% on the boundary, and as the distance from the boundary increases, one cutout area is displayed on the left side of the boundary and one cutout area is displayed on the right side of the boundary. Synthesis is performed while increasing the proportion of the other cutout area.

In the present embodiment, the panoramic image compositing process is started without waiting for the end of pressing the shutter button 121. Specifically, in parallel with the imaging process shown in FIG. 4 (c), the image processing unit 206 performs a composite process of the images of the rectangular regions 511 and 512 shown in FIGS. 4 (a) and 4 (b). May be good. By performing parallel processing by the image processing unit 206, the composition processing time of the entire panoramic image can be shortened. The image processing unit 206 temporarily stores the combined panoramic image in the memory 209 via the memory control unit 207. Finally, the system control unit records the panoramic image stored in the memory 209 on the storage medium 130 via the I / F 216.

However, the composition of the panoramic image as shown in FIG. 4 may generate an unnatural panoramic image in the presence of a moving object. In the following, an example of panoramic composition in the presence of a moving object will be described.

FIG. 5 is a diagram for explaining panoramic composition in the presence of a moving object. In the scene shown in FIG. 5, unlike FIG. 4, it is assumed that the person as the subject is a moving object and is moving during panoramic imaging.

5 (a) to 5 (e) are diagrams schematically showing each imaging scene of panoramic imaging in order. The rectangular regions 510, 520, 530, 540, and 550 represent the imaging angle of view of the image captured by the imaging unit 204 in each imaging. Similar to FIG. 4, the shaded hatched areas 511, 521, 513, 541, and 551 represent the cutout areas of the image data. The system control unit 210 can calculate the movement direction and the movement amount between a plurality of images that are continuous in time based on the angular velocity information and the posture information detected by the detection unit 215, and other motion vector information. FIG. 5 (f) schematically shows a panoramic image 500 which is a result of the image processing unit 206 synthesizing a plurality of images captured by the imaging unit 204 in the panoramic imaging shown in FIGS. 5 (a) to 5 (e).

However, in the scene shown in FIG. 5, since the person is a moving object, this person is reflected in all of the cutout areas 511 to 541. When the system control unit 210 uses such a cutout region for panoramic composition, as shown in the panoramic image 500 of FIG. 5 (f), an image in which a moving person is projected a plurality of times is created.

Therefore, in the present embodiment, the system control unit 210 does not generate the panoramic image having a sense of incongruity as described above by using an area satisfying a predetermined condition for synthesis while considering the memory capacity for image recording. To.

FIG. 6 is a flowchart for explaining the present embodiment. This flowchart is started when the user sets a mode for generating a panoramic image by operating the operation unit 102 and the user operates the shutter button 121. FIG. 7 is a flowchart for explaining the determination of the second region in step S605, which will be described later. FIG. 8 is a diagram for explaining the composition of panoramic images in the present embodiment. FIG. 8 shows that images 810 to 840 were captured while moving the digital camera 100. Hereinafter, the flow of the present embodiment will be described in detail with reference to FIGS. 6 to 8.

In step S601, the imaging unit 204 performs imaging. At this time, it is assumed that the image 810 shown in FIG. 8A is captured. The system control unit 210 temporarily stores the image 810 captured by the image pickup unit 204 in the memory 209. However, since the moving direction of the digital camera 100 is to the right, as will be described later, in the image 810, the portion to the left of the region 811 is not used for compositing, so this portion is not stored.

In step S602, the image processing unit 206 cuts out the first region 811 in the image 810. In FIG. 8A, the first region 811 is located closer to the left side than the center of the image 810, but for convenience of illustration only. As will be described later, in the composition in the present embodiment, the first region is preferentially used. Therefore, in consideration of distortion and the like, the image processing unit 206 generally preferentially sets the first region from the center of the image. It is preferable to cut it out.

In step S603, the imaging unit 204 performs the next imaging. At this time, it is assumed that the image 820 shown in FIG. 8B is captured. Since the user performs imaging while moving the digital camera 100, the angles of view of the image 810 and the image 820 are deviated. Similar to step S601, the system control unit 210 temporarily stores the image 820 in the memory 209.

In step S604, the image processing unit 206 cuts out the first region 821 in the image 820. Since the first region 811 and the first region 821 are adjacent to each other in the composition of the panoramic image, the image processing unit 206 cuts out so that there is no gap between the angles of view of both. Therefore, the image processing unit 206 may determine the position of the angle of view of the first region to be cut out from the image by referring to information such as the speed detected by the detection unit 215. The image processing unit 206 may determine the position of the angle of view of the first region from the captured image by using the SAD value, feature point extraction, or the like, but when the region occupied by the moving object is large, the stationary region There are cases where it is difficult to extract SAD values and feature points only by using only. Therefore, in step S604, it is desirable that the image processing unit 206 determines the position of the angle of view of the first region with reference to information such as the speed detected by the detection unit 215. Further, since the system control unit 210 does not use the portion to the left of the first region for synthesis, the portion to the left of the first region may be deleted from the memory 209 at this point.

In step S605, the system control unit 210 determines the second region. The details of step S605 will be described with reference to the flow shown in FIG.

In step S701, the image processing unit 206 cuts out a candidate for the second region from the image captured one frame before the image of the frame from which the first region was cut out in step S604. For example, the image processing unit 206 cuts out a candidate for the second region from the image 810 one frame before with respect to the first region 821 cut out from the image 820. As will be described later, in step S606, the system control unit 210 compares the candidates in the first region and the second region for motion detection, so that the candidates in the first region and the candidates in the second region overlap. It is necessary to have an angle of view. Therefore, the image processing unit 206 cuts out the candidate 812 of the second region having an angle of view overlapping with the first region 821 of the image 820 shown in FIG. 8B from the image 810. Similarly, the image processing unit 206 cuts out the first region 831 from the image 830, and cuts out the candidate 822 of the second region having an angle of view overlapping with the first region 831 from the image 820 one frame before the image 830.

In step S702, the system control unit 210 determines whether or not there is a moving object in the first region of the image in which the image processing unit 206 cuts out the candidate of the second region in step S701. When the system control unit 210 determines that there is a moving object, the system control unit 210 proceeds to step S703, and the image processing unit 206 cuts out the third region. When the system control unit 210 determines that there is no moving object, the system proceeds to step S707, and the candidate for the second region cut out by the image processing unit 206 in step S701 is set as the second region.

For example, with respect to the candidate 812 of the second region cut out from the image 810, since there is no moving body in the first region 811 cut out from this image (the moving body is not detected), the system control unit 210 proceeds to step S707 and proceeds to the second step. The candidate 812 of the two regions is set as the second region. Further, with respect to the candidate 822 of the second region cut out from the image 820, since there is a moving body in the first region 821 cut out from this image, the process proceeds to step S703. When the process proceeds to step S702 for the first time, the image obtained by cutting out the second candidate region corresponds to the image captured on the first image, and it is determined whether or not the first region of the image is a moving object. Since the above is not performed, it is assumed that the process proceeds to step S707.

In step S703, the image processing unit 206 sets the most captured timing among the frames before the candidate image of the second region cut out in step S701 and which is determined to have no moving object in the first region. A third region is cut out from an image of a close frame. As will be described later, in step S704, the system control unit 210 makes a comparison between the candidate in the second region and the third region for motion detection, so that the candidate in the second region and the third region have the same image. Must have horns. For example, the image processing unit 206 cuts out the first region 831 in the image 830, and cuts out the candidate 822 of the second region corresponding to the first region 831 in the image 820. At this time, a moving body is included in the first region 821 of the image 820 from which the candidate 822 of the second region is cut out. In such a case, from the image 810 which is a frame before the image 820 and in which no moving object is detected from the first region 811, the third region 813 having an angle of view overlapping with the candidate 822 of the second region is displayed. break the ice. Further, the image processing unit 206 cuts out the first region 841 in the image 840, and cuts out the candidate 832 in the second region having an angle of view overlapping with the first region 841 in the image 830. Since there is a moving object in the first region 821 of the image 820, the system control unit 210 cuts out a third region 814 having an angle of view overlapping with the candidate 832 of the second region in the image 810 which is a frame before that. ..

In step S704, the system control unit 210 compares the candidate for the second region cut out by the image processing unit 206 in step S701 with the third region cut out in step S703. The comparison in step S704 is for detecting a moving object, and based on this result, the system control unit 210 determines whether or not a moving object exists as a candidate for the second region in the next step S704. The method of comparison here may be a known method such as pixel value difference detection or image correlation determination. In the pixel value difference detection, the system control unit 210 calculates the difference in the pixel values of the pixels at the same positions in the candidate in the second region and the third region. Alternatively, the difference in the luminance signals (Y) of the pixels may be calculated. If this difference is equal to or greater than a predetermined value, the system control unit 210 counts the pixels to be calculated, and when the number of pixels counted in the entire area exceeds a predetermined threshold value, the system control unit 210 causes the system control unit 210 to count. , Judge that there is a moving object in that area. In the image correlation determination method, the system control unit 210 divides the two regions into a designated block size and calculates the average value of the luminance values (Y) in the blocks. Further, the system control unit 210 calculates the variance value of the average value of the brightness values of the blocks for each of the two regions. If the difference between the variance values of the two regions is equal to or greater than a predetermined threshold value, the system control unit 210 determines that a moving object exists.

In step S705, when the system control unit 210 determines that a moving object exists in the candidate of the second region, the system proceeds to step S706 and determines that the third region having an angle of view overlapping with the candidate of the second region is the second region. do. On the other hand, in step S704, when the system control unit 210 determines that the moving object does not exist in the candidate of the second region, the system proceeds to step S707 and determines the candidate of this second region as the second region.

For example, the system control unit 210 compares the candidate 822 of the second region in the image 820 with the third region 813 in the image 810 to detect a moving object. In the example of FIG. 8, since it is recognized that a moving object exists in the candidate 822 in the second region of the image 820 (the person in front of the tree is jumping), the system control unit 210 selects the candidate 822 in the second region. The third region 813 is determined to be the second region instead of the second region. On the other hand, the system control unit 210 compares the candidate 832 in the second region in the image 830 with the third region 814 in the image 810 and determines that there is no moving object, so that the candidate 832 in the second region is the second region. do.

The above has been described for determining the second region. In the above process, the system control unit 210 determines the region without the moving object as the second region, prevents an error in detecting the moving object in the first region, and uses the second region with the moving object for the composite image. Can be avoided.

Next, in step S606, the system control unit 210 makes a comparison for motion detection with respect to the first region cut out by the image processing unit 206 in step S604 and the second region determined in step S605. Since the method is the same as in step S703, detailed description thereof will be omitted.

In step S607, the system control unit 210 determines whether or not there is a moving object in the first region according to the result of the comparison in step S606. If there is no moving object in the first region, the process proceeds to step S608, and the system control unit 210 sets a flag in the first region. If there is a moving object in the first region, the process proceeds to step S609, and the system control unit 210 sets a flag in the second region determined in step S605.

Next, in step S610, the system control unit 210 deletes an unnecessary area from the memory 209. The unnecessary region referred to here refers to a region determined by the system control unit 210 to be reliably not used for synthesis at the time of step S610. Specifically, the system control unit 210 deletes the unflagged area in the direction opposite to the moving direction of the digital camera 100 from the angle of view of the area flagged in step S608 or S609.

For example, in the image 820 shown in FIG. 8B, the system control unit 210 determines that there is a moving object in the first area 821, and when the second area 812 is flagged, the memory 209 to the second area Delete the unflagged area to the left of the right edge. Specifically, the system control unit 210 deletes the second region 812 in the image 810 and the portion to the left of the left end of the first region 821 in the image 820.

Next, the system control unit 210 cuts out the first region 831 in the image 830 shown in FIG. 8 (c). Since the moving object exists in the candidate 822 of the second region, the image processing unit 206 cuts out the third region 813 as the second region in the image 810. As a result of comparing the first region 831 and the second region 813, the system control unit 210 sets a flag in the second region 813 because there is a moving object in the first region 831. In response to this, the system control unit 210 deletes the area on the left side of the right end of the angle of view of the second area 813 from the memory 209. Specifically, the system control unit 210 deletes the first region 831 and the candidate 822 of the second region. Further, since the moving body exists in the first region 821 of the image 820, the image processing unit 206 does not cut out the second region and the third region from the image 820 even after this point. Therefore, the system control unit 210 may also delete the portion to the right of the right end of the candidate 822 in the second region in the image 820. That is, the system control unit 210 sets a flag in the second region 813 and deletes the entire image 820.

Finally, the system control unit 210 cuts out the first region 841 in the image 840 shown in FIG. 8 (d). Since there is no moving object in the candidate 832 in the second region, the candidate 832 in the second region is set as the second region, and the system control unit 210 compares the first region 841 with the second region 832 and compares the first region 841 with the second region 832. Judge that there is no moving body. Therefore, the system control unit 210 sets a flag in the first region 841. At this point, the system control unit 210 deletes the second region 832 and the third region 814.

As described in steps S702 and S707 described above, when the image processing unit 206 has no moving object in the first region of the image obtained by cutting out the candidate for the second region, the system control unit 210 is the candidate for the second region. Is determined as the second region. Since there is no moving object in the first region 841 in the image 840, the image processing unit 206 also has the second region as the third region from the image captured before the image 841 after the time when the moving object is not detected in the image 841. Will not be cut out. Therefore, if the system control unit 210 does not detect a moving object in the image 841, the system control unit 210 deletes all the regions in the images 810 to 830 that are not flagged.

As described above, the system control unit 210 can minimize the capacity of the memory 209 used by determining and deleting an unnecessary area when the image area is flagged. It can be expected that the reduced capacity will contribute to the improvement of image quality.

In step S611, the system control unit 210 determines whether or not the panoramic imaging is completed. As a method of determination, the user may decide on the basis of stopping the pressing of the button, or if the number of images to be imaged or the range of the angle of view is determined in advance, it may be determined whether or not the range has been reached. If the system control unit 210 determines in step S611 that the panoramic imaging has been completed, the process proceeds to step S612, and if it determines that the panoramic imaging has not yet been completed, the system proceeds to step S603.

Finally, in step S612, the system control unit 210 synthesizes a panoramic image using the first region cut out by the image processing unit 206 in step S602 and the regions flagged in steps S608 and S609. The system control unit 210 detects the positional deviation between the images corresponding to the above-mentioned flagged region. As an example, the system control unit 210 divides the flagged area into small blocks of an arbitrary size, and calculates a corresponding point at which the brightness SAD (Sum of Absolute Difference) is minimized for each small block. The system control unit 210 can calculate the motion vector from the corresponding point where the calculated SAD is the minimum. The system control unit 210 may use the angular velocity information, the attitude information, and the like detected by the detection unit 215 in calculating the motion vector. Subsequently, the system control unit 210 corrects the positional deviation between the regions based on the motion vectors obtained between the images of the adjacent regions, and synthesizes the overlapping portions between the regions by weighted addition or the like.

In the above implementation method, if the moving object remains near the first region of the image of each frame, the flag continues to be set for the first image, so even if the angle of view of the image is used up. If the moving object does not disappear, the wide-angle panoramic image cannot be combined. To prevent this situation, the user may be warned if the moving object remains near the first region of each frame.

Further, in the comparison of the two regions in step S606 and step S704, the region to be compared can be limited and the speed can be further increased. FIG. 9 is a diagram for explaining the speeding up of comparison of regions in the present embodiment. The shaded area in FIG. 9A is used as the area to be compared. When the moving direction of the digital camera 100 is on the right, a part on the left side as shown by the shaded area in FIG. 9 (b), and when the moving direction of the digital camera 100 is on the left, the right side as shown by the shaded area in FIG. 9 (c). The similarity may be calculated only for a part. The reason is that when panoramic composition is performed with the image of the composite image unit 323, when the movement direction of the digital camera 100 is the right direction, the left side portion is connected, and when the movement direction of the digital camera 100 is the left direction. This is because the composition is performed so that the right parts are connected. That is, it is sufficient to determine whether or not there is a moving subject at the joint portion. Therefore, in a scene where it can be assumed that there is only a part of the moving subject, the area for calculating the similarity can be further reduced. For example, in a scene where you know that the subject should move only in the lower part (such as a landscape scene consisting of the sky and mountains where the upper half of the scene is stationary), reduce the shaded area to only the lower part. Is also possible. When the similarity calculation area becomes small in this way, there are merits such as shortening the calculation time.

According to this embodiment, it is possible to realize the removal of moving objects with a smaller memory capacity by using a digital camera that performs panoramic composition.

(Other embodiments)
In the above embodiment, the description is based on a digital camera for individuals, but if it is equipped with a panoramic imaging and compositing function, a mobile device, a smartphone, a network camera connected to a server, etc. It is also possible to apply to.

In the present invention, a program that realizes one or more functions of the above-described embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device program. It can also be realized by the process of reading and operating. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Digital camera 102 Operation unit 104 Mode changeover switch 130 Recording medium 204 Imaging unit 206 Image processing unit 207 Memory control unit 209 Memory 210 System control unit 215 Detection unit 216 I / F

Claims (14)

An imaging means that captures multiple images that are continuous in time,
A first region is set for each of the plurality of images, and a second region is set for each part of the plurality of images, and the plurality of images are captured in parallel with the imaging of the plurality of images. A control means for controlling storage in the storage means and deletion from the storage means with respect to the first region and the second region, and
In each of the plurality of images, if the first region satisfies a predetermined condition, the image of the first region is used, and if the first region does not satisfy the predetermined condition, the image of the first region is used. A panoramic image using an image of the second region cut out from an image captured before the cut-out image, which has an angle of view overlapping with the first region and satisfies the predetermined conditions. Has a generation means to generate
If the first region satisfies the predetermined condition, the control means has an angle of view that overlaps with the first region, and is from an image captured before the image obtained by cutting out the first region. An imaging device characterized in that the cut out second region is deleted from the storage means. It has a moving object detecting means for detecting a moving object,
The imaging device according to claim 1, wherein the predetermined condition is that the moving object does not exist. The imaging device according to claim 2, wherein the moving object detecting means detects whether or not a moving object is present in the plurality of regions having overlapping angles of view. The imaging device according to claim 3, wherein the moving object detecting means detects a moving object in the first region by comparing the first region with the second region having an angle of view overlapping with the first region. .. The imaging device according to any one of claims 1 to 4, wherein the control means sets the first region of each of the plurality of images so as to include a position at the center of the image. The control means sets a third region in at least a part of the plurality of images.
The generation means determines whether or not the second region satisfies the predetermined condition by comparing the second region with the third region having an angle of view overlapping with the second region. The imaging apparatus according to any one of claims 1 to 5, which is characterized. The sixth aspect of claim 6, wherein the control means sets the third region when the first region cut out from the image obtained by cutting out the second region does not satisfy the predetermined condition. Imaging device. The control means is the same as the image obtained by cutting out the second region from the images in which the first region cut out is captured before the image obtained by cutting out the second region and the cut out first region satisfies the predetermined condition. The imaging apparatus according to claim 7, wherein the third region is set on the image having the closest imaging timing. The control means according to any one of claims 6 to 8, wherein when the second region does not satisfy the predetermined condition, the third region is set as a new second region. The imaging apparatus described. The imaging device according to any one of claims 6 to 9, wherein the control means sets the widths of the second region and the third region equally. The imaging device according to any one of claims 1 to 10, wherein the control means sets the widths of the first region and the second region equally. The imaging device according to any one of claims 1 to 11, wherein the control means sets the first region at the same position of each of the plurality of images. Steps to capture multiple images that are continuous in time,
A first region is set for each of the plurality of images, and a second region is set for each part of the plurality of images, and the plurality of images are captured in parallel with the step of capturing the plurality of images. A step of controlling storage in the storage means and deletion from the storage means with respect to the first region and the second region in the image, and
In each of the plurality of images, if the first region satisfies a predetermined condition, the image of the first region is used, and if the first region does not satisfy the predetermined condition, the image of the first region is used. A panoramic image using an image of the second region cut out from an image captured before the cut-out image, which has an angle of view overlapping with the first region and satisfies the predetermined conditions. Has steps to generate
In the control step, if the first region satisfies the predetermined condition, the first region has an angle of view overlapping with the first region, and the image is taken before the image obtained by cutting out the first region. A control method for an imaging device, which comprises deleting a second region cut out from an image from the storage means. A computer program that runs on the computer of the image pickup device.
Steps to capture multiple images that are continuous in time,
A first region is set for each of the plurality of images, and a second region is set for each part of the plurality of images, and the plurality of images are captured in parallel with the step of capturing the plurality of images. A step of controlling storage in the storage means and deletion from the storage means with respect to the first region and the second region in the image, and
In each of the plurality of images, if the first region satisfies a predetermined condition, the image of the first region is used, and if the first region does not satisfy the predetermined condition, the image of the first region is used. A panoramic image using an image of the second region cut out from an image captured before the cut-out image, which has an angle of view overlapping with the first region and satisfies the predetermined conditions. Let me take the steps to generate
In the control step, if the first region satisfies the predetermined condition, the first region has an angle of view overlapping with the first region, and the image is taken before the image obtained by cutting out the first region. A program characterized by deleting a second region cut out from an image from the storage means.

Images