JP2019068243A - Image processing apparatus, imaging apparatus, control method of image processing apparatus, and program - Google Patents

Abstract

To provide an image processing apparatus capable of automatically creating a suitable panoramic image in which the same person image does not appear multiple times.SOLUTION: An image processing apparatus includes: recognition means for recognizing a foreground and a background from each of a plurality of images captured while performing a panning operation; and combining means for combining a foreground, which is an image corresponding to the foreground recognized in any one of the plurality of images and a plurality of backgrounds corresponding to the background recognized in each of the plurality of images to generate a panoramic image. The combining means preferentially performs combination of the foreground in a region of a background where the angle of view overlaps with the foreground from among the plurality of backgrounds.SELECTED DRAWING: Figure 7

Description

  The present invention relates to an image processing apparatus that combines a plurality of images to create a panoramic image.

  There is known a method of capturing a plurality of images while panning an imaging device such as a digital camera and connecting the captured images to generate a panoramic image. In a conventional imaging apparatus that generates a panoramic image, it is not assumed that a photographer takes a picture so as to be included in the panoramic image, and it is difficult to generate such a panoramic image. Therefore, Patent Document 1 discloses a method of generating a panoramic image in which an image capturing person is reflected by fixing and rotating an imaging device such as a tripod using a rotation mechanism.

JP, 2013-34158, A

  However, in the method disclosed in Patent Document 1, a fixing device such as a tripod is required, and the convenience is not good. On the other hand, a method is widely used in which a photographer captures an image of the imaging device toward himself (so-called "self-portrait"). A method is conceivable in which a panoramic image including an image of the image capturing person is generated by the image capturing person rotating while shooting the image capturing apparatus toward the user.

  FIG. 14 is a diagram for explaining the problem of self-portrait panoramic imaging. As shown in FIG. 14 (a), if an image pickup person takes an image while panning (rotating) with an image pickup device facing himself / herself with an image pickup device such as a digital camera, a panoramic image including the image pickup person is It is generated. However, when the conventional panoramic imaging method is used, as shown in FIG. 14B, a panoramic image in which a person image of the same image pickup person appears a plurality of times is generated.

  The present invention has been made in view of the above problems, and an object of the present invention is to create a suitable panoramic image in which the same human figure does not appear multiple times.

  In order to solve the above problems, the present invention relates to recognition means for recognizing a foreground and a background from each of a plurality of images captured while panning, and a foreground recognized in any one of the plurality of images. Combining a foreground image which is an image corresponding to a plurality of background images corresponding to the background recognized in each of the plurality of images to generate a panoramic image, the combining means The present invention provides an image processing apparatus characterized in that the foreground image is preferentially synthesized in a region where the angle of view overlaps with the foreground image among the plurality of background images.

  According to the configuration of the present invention, it is possible to provide an image processing apparatus capable of creating a suitable panoramic image in which the same person image does not appear multiple times.

FIG. 1 is a rear perspective view showing a schematic configuration of a digital camera according to an embodiment of the present invention. It is a block diagram showing hardware constitutions of a digital camera concerning an embodiment of the present invention. It is a figure for demonstrating the relationship between the moving direction of the digital camera in panorama imaging using the conventional method, and the cutting-out area | region of image data. It is a figure explaining the flow of synthetic processing of a panoramic image using the conventional method. It is a figure for demonstrating the self-portrait panoramic imaging in this invention. It is a flowchart for describing the embodiment of the present invention. It is a flow chart for explaining image composition in an embodiment of the present invention. It is a figure for demonstrating the area | region of the image of the process target in embodiment of this invention. It is a figure for demonstrating the case where only the background area | region in embodiment of this invention is synthesize | combined. It is a figure for demonstrating the case where the foreground area | region of the 1st image is used for a synthetic | combination image in synthetic | combination in embodiment of this invention. It is a figure for demonstrating an example in the case where the foreground area | region of the 3rd image is used for a synthetic | combination image in synthetic | combination in embodiment of this invention. It is a figure for demonstrating another example in the case where the foreground area | region of the 3rd image is used for a synthetic | combination image in synthetic | combination in embodiment of this invention. It is a figure for demonstrating the case where persons' persons other than the imaging person in embodiment of this invention exist in the view | field angle of panorama imaging. It is a figure for demonstrating the subject of self-portrait panoramic imaging.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a rear perspective view showing a schematic configuration of a digital camera according to an embodiment of the present invention.

  On the back of the digital camera 100, a display unit 101 for displaying an image and various information, and an operation unit 102 including operation members such as various switches and buttons for receiving various operations by the user are provided. Further, on the back of the digital camera 100, a mode change switch 104 for switching an imaging mode and the like, and a controller wheel 103 which can be operated to rotate are provided. On the upper surface of the digital camera 100, a shutter button 121 for instructing imaging, a power switch 122 for switching on / off the power of the digital camera 100, and a strobe 141 for emitting a flash to a subject are provided.

  The digital camera 100 can be connected to an external device via wired or wireless communication, and can output image data (still image data, moving image data) and the like to the external device. A recording medium slot (not shown) which can be opened and closed by a lid 131 is provided on the lower surface of the digital camera 100 so that the recording medium 130 such as a memory card can be inserted into and removed from the recording medium slot. There is.

  The recording medium 130 stored in the recording medium slot can communicate with the system control unit 210 (see FIG. 2) of the digital camera 100. The recording medium 130 is not limited to a memory card or the like which can be inserted into and removed from the recording medium slot, and may be a magnetic disk such as an optical disk or a hard disk. It may be done.

  FIG. 2 is a block diagram showing the hardware configuration of the digital camera 100. As shown in FIG. The digital camera 100 includes a barrier 201, an imaging lens 202, a shutter 203, and an imaging unit 204. The barrier 201 covers the imaging optical system to prevent the contamination and damage of the imaging optical system. The imaging lens 202 is constituted by a lens group including a zoom lens and a focus lens, and constitutes an imaging optical system. The shutter 203 has an aperture function, and adjusts the exposure amount to the imaging unit 204. The imaging unit 204 is an imaging device that converts an optical image into an electrical signal (analog signal), and is an image sensor such as a CCD sensor or a CMOS sensor having a Bayer array structure in which RGB pixels are regularly arranged. . The shutter 203 may be a mechanical shutter (hereinafter referred to as a mechanical shutter), or may be an electronic shutter that controls the accumulation time by controlling the reset timing of the imaging device.

  Alternatively, if the imaging unit 204 has a structure in which a plurality of photoelectric conversion units are provided in one pixel, which can acquire a stereo image, automatic focus detection (AF) processing to be described later can be performed more quickly.

  The digital camera 100 includes an A / D converter 205, an image processing unit 206, a memory control unit 207, a D / A converter 208, a memory 209, and a system control unit 210. An analog signal is output from the imaging unit 204 to the A / D converter 205, and the A / D converter 205 converts the acquired analog signal into image data composed of a digital signal, and the image processing unit 206 or the memory control unit 207. Output to

  The image processing unit 206 performs correction processing such as pixel interpolation and shading correction, white balance processing, γ correction processing, and color conversion on image data acquired from the A / D converter 205 or data acquired from the memory control unit 207. Perform processing etc. Further, the image processing unit 206 realizes an electronic zoom function by performing image clipping and scaling processing. Furthermore, the image processing unit 206 performs predetermined arithmetic processing using the image data of the captured image, and the system control unit 210 performs exposure control and distance measurement control based on the calculation result thus obtained. For example, the system control unit 210 performs TTL (through-the-lens) AF (auto focus) processing, AE (automatic exposure) processing, and EF (flash pre-emission) processing. The image processing unit 206 performs predetermined arithmetic processing using image data of a captured image, and the system control unit 210 performs TTL AWB (Auto White Balance) processing using the obtained arithmetic result.

  The image processing unit 206 has an image combining processing circuit that combines a panoramic image from a plurality of images and determines the combining result. The image synthesis processing circuit selects not only simple addition average synthesis but also pixels having the brightest value or dark value in each area of the image data to be synthesized to generate one image data. Processing such as dark composition can be performed. Also, the combined result is evaluated and determined based on specific criteria. For example, when the number of combined images does not satisfy a predetermined number, or when the length of the combined image does not meet a reference value, it is determined that combining has failed. Note that, instead of the configuration including the image processing unit 206, the configuration of the image combining process may be realized by software processing by the system control unit 210.

  Image data output from the A / D converter 205 is written to the memory 209 via the image processing unit 206 and the memory control unit 207 or via the memory control unit 207. The memory 209 also serves as an image display memory (video memory) for storing image data to be displayed on the display unit 101. The memory 209 has a storage capacity capable of storing a predetermined number of still images, a panoramic image (wide-angle image), and a panoramic image synthesis result. The memory 209 can also be used as a work area in which a program or the like read out from the non-volatile memory 211 by the system control unit 210 is developed.

  Image display data (digital data) stored in the memory 209 is transmitted to the D / A converter 208. The D / A converter 208 converts the received digital data into an analog signal and supplies the analog signal to the display unit 101, whereby an image is displayed on the display unit 101. The display unit 101 is a display device such as a liquid crystal display or an organic EL display, and displays an image based on an analog signal from the D / A converter 208. The image display on the display unit 101 is switched on / off by the system control unit 210, and power consumption can be reduced by turning off the image display. A through image is displayed by converting a digital signal accumulated in the memory 209 from the imaging unit 204 through the A / D converter 205 into an analog signal through the D / A converter 208 and sequentially displaying the analog signal on the display unit 101. An electronic view finder function can be realized.

  The digital camera 100 includes a non-volatile memory 211, a system timer 212, a system memory 213, a detection unit 215, and a flash control unit 217. The non-volatile memory 211 is a memory (for example, an EEPROM or the like) which can be electrically erased and stored, and stores a program executed by the system control unit 210, a constant for operation, and the like. The non-volatile memory 211 also has an area for storing system information and an area for storing user setting information. The system control unit 210 stores various information stored in the non-volatile memory 211 when the digital camera 100 is activated. Read and restore information and settings.

  The system control unit 210 includes a CPU, and controls various operations of the digital camera 100 by executing various program codes stored in the non-volatile memory 211. The program read from the non-volatile memory 211 by the system control unit 210, constants for operation, variables, and the like are expanded on the system memory 213. A RAM is used for the system memory 213. Furthermore, the system control unit 210 performs display control by controlling the memory 209, the D / A converter 208, the display unit 101, and the like. A system timer 212 measures time used for various controls and time of a built-in clock. The flash control unit 217 controls the light emission of the flash 141 according to the brightness of the subject. The detection unit 215 includes a gyro and a sensor, and acquires angular velocity information, posture information, and the like of the digital camera 100. The angular velocity information includes information on angular velocity and angular acceleration at the time of panoramic imaging by the digital camera 100. The posture information also includes information such as the tilt of the digital camera 100 with respect to the horizontal direction.

  The display unit 101, the operation unit 102, the controller wheel 103, the shutter button 121, the mode switching switch 104, the power switch 122, and the electronic flash 141 shown in FIG. 2 are the same as those described with reference to FIG.

  Various operation members constituting the operation unit 102 are used, for example, to select various function icons displayed on the display unit 101, and functions are assigned to each scene by selecting a predetermined function icon. That is, each operation member of the operation unit 102 functions as various function buttons. Examples of the function buttons include an end button, a back button, an image feed button, a jump button, a narrow-down button, an attribute change button, and a DISP button. For example, when the menu button is pressed, a menu screen for performing various settings is displayed on the display unit 101. The user can intuitively perform the setting operation using the menu screen displayed on the display unit 101 and the four-direction button or the SET button on the top, bottom, left, and right.

  The controller wheel 103, which is an operation member capable of rotating, is used together with a four-way button when designating a selection item. When the controller wheel 103 is rotated, an electrical pulse signal corresponding to the amount of operation (rotation angle, number of rotations, etc.) is generated. The system control unit 210 analyzes this pulse signal to control each unit of the digital camera 100.

  The shutter button 121 has a first switch SW1 and a second switch SW2. The first switch SW1 is turned on when the shutter button 121 is halfway pressed, and a signal instructing preparation for imaging is transmitted to the system control unit 210. When receiving the signal that the first switch SW1 is turned on, the system control unit 210 starts operations such as AF processing, AE processing, AWB processing, and EF processing. The second switch SW2 is turned on in the fully-pressed state in which the operation of the shutter button 121 is completed, whereby a signal instructing start of imaging is transmitted to the system control unit 210. When the system control unit 210 receives a signal that the second switch SW2 is turned on, the system control unit 210 performs a series of imaging operations from signal reading from the imaging unit 204 to writing of image data to the recording medium 130.

  The mode switching switch 104 is a switch for switching the operation mode of the digital camera 100 among various modes such as a still image capturing mode, a moving image capturing mode, and a reproduction mode. The still image pickup mode includes, in addition to the auto pickup mode and the like, a panoramic image pickup mode for synthesizing panoramic images by panoramic pickup.

  The digital camera 100 includes a power supply unit 214 and a power control unit 218. The power supply unit 214 is a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery or a Li battery, or an AC adapter, and supplies power to the power control unit 218. The power supply control unit 218 detects the presence or absence of a battery attached to the power supply unit 214, the type of the battery, the remaining battery capacity, and the like, and based on the detection result and the instruction of the system control unit 210, a necessary period It supplies to each part including the recording medium 130.

  The digital camera 100 includes a recording medium I / F 216 for enabling communication between the recording medium 130 and the system control unit 210 when the recording medium 130 is loaded into a recording medium slot (not shown). The details of the recording medium 130 have already been described with reference to FIG.

  Next, a method of panoramic imaging and a method of combining panoramic images from a plurality of captured images will be described. First, a process of cutting out a predetermined area from image data of a captured image to synthesize a panoramic image will be described.

  FIG. 3 is a diagram for explaining the relationship between the moving direction of the digital camera 100 during panoramic imaging using the conventional method and the cutout area of image data.

  FIG. 3A shows the effective image area of the imaging element of the imaging unit 204, where "Wv" is the number of effective pixels in the horizontal direction, and "Hv" is the number of effective pixels in the vertical direction. FIG. 3B shows a cutout area cut out from the image data of the captured image, “Wcrop” is the number of cutout pixels in the horizontal direction, and “Hcrop” is the number of cutout pixels in the vertical direction.

FIG. 3C is a diagram showing a cutout area for image data when performing panoramic imaging while moving the digital camera 100 in the horizontal direction indicated by an arrow. An area S1 hatched in FIG. 3C indicates a cutout area from the image data, and the following (Expression 1) and (Expression 2) are satisfied.
Wv> Wcrop ············ (Equation 1)
Hv = Hcrop ··········· (Equation 2)

Similarly, FIG. 3D is a view showing a cutout area for image data when performing panoramic imaging while moving the digital camera 100 in the vertical direction indicated by an arrow. An area S2 indicated by hatching in FIG. 3D indicates a cutout area of the image data, and the following (Expression 3) and (Expression 4) are satisfied.
Wv = Wcrop ············ (Equation 3)
Hv> Hcrop ············ (Equation 4)

  The cutout area of the image data of the captured image may be different for each image data. In addition, with regard to the image data at the start of panoramic imaging and at the end of panoramic imaging, the cutout region may be widened in order to widen the angle of view. The method of determining the cutout region of the image data can be determined, for example, by the difference between the angle of the digital camera 100 immediately after imaging and the angle of the digital camera 100 one frame before, or the like. The storage capacity of the memory 209 can be saved by cutting out and storing only the image data necessary for the synthesis process of the panoramic image.

  Next, a method of combining panoramic images will be described. The system control unit 210 reads out from the memory 209 the cutout area stored at the time of panoramic imaging, and performs panoramic combination on the read out image data.

  FIG. 4 is a diagram for explaining the flow of panoramic image synthesis processing using the conventional method. In FIG. 4, the dot-hatched area is an area schematically representing a row tree in the field, and the hatched area represents a cutout area of the image data. FIG. 4A shows a state in which the user presses the shutter button 121 and the first switch SW1 is turned on, and shows that the user performs focusing on the main subject. FIG. 4B shows the position where the second switch SW2 of the shutter button 121 is turned on, and the angle of view is set in accordance with one end of the panoramic image to be synthesized by the user. In FIG. 4B, the imaging unit 204 captures an image 410. FIGS. 4C to 4E schematically show a state in which panoramic imaging is performed while moving the digital camera 100 toward the other end of the panoramic image that the user is trying to combine. . FIG. 4E shows a state in which the user has stopped pressing the shutter button 121 and panoramic imaging has been completed. In FIGS. 4B to 4E, the imaging unit 204 captures a total of seven images 410 to 470, but the images 430, 450, and 460 are not shown. The image processing unit 206 performs cutout processing on the images 410 to 470 captured by the imaging unit 204, and generates cutout regions 411 to 471. In the system control unit 210, the width of the cutout region may be determined in advance, but may be changed according to the moving speed of the digital camera 100 during panoramic imaging and the like.

  FIG. 4F shows a panoramic image formed by combining the plurality of images captured by the imaging unit 204 by the image processing unit 206. Here, the system control unit 210 aligns the image before combining. In addition, since the upper side and the lower side of the cutout regions 411 to 471 do not coincide with each other because of camera shake or the like, the image processing unit 206 performs the cutout processing also in the vertical direction. As a result, the image processing unit 206 generates a panoramic image as indicated by the area 400.

  The system control unit 210 performs alignment based on the plurality of motion vectors detected by the image processing unit 206. As an example, the image processing unit 206 divides the cutout region into small blocks of an arbitrary size, and minimizes the sum of absolute differences of luminance (hereinafter referred to as SAD) for each small block. Calculate points. The system control unit 210 can calculate the motion vector from the corresponding point at which the calculated SAD is minimum. The system control unit 210 may use, in addition to SAD, a sum of squared difference (hereinafter referred to as SSD), a normalized cross correlation (hereinafter referred to as NCC), or the like.

  In FIG. 4, in order to facilitate the description, an example is shown in which there are no overlapping regions of the cutout regions 411 to 471 and they are adjacent to each other. If there is a region to be superimposed, the image processing unit 206 outputs pixel information of one cutout region on the left side and the other cutout region on the right side to the composite image, using the center of the superposition region as a boundary. Do. Alternatively, the image processing unit 206 outputs a value obtained by combining pixel information of both cutout regions by 50% on the boundary, and one cutout region is on the left side of the boundary as the distance from the boundary is longer, and on the right side of the boundary The synthesis is performed while increasing the ratio of the other cutout region.

  However, when the above-described panoramic image synthesis method is applied to self-shooting, the following problems occur. In the case of panning with self-portrait, in order to capture an image of the photographer, the imaging unit captures an image of the imaging unit toward itself in the entire process of panning. Therefore, in the image captured by the panning operation with self-portrait, unlike the conventional panorama, the image of the photographer's own is included in many images.

  FIG. 5 is a diagram for explaining self-portrait panoramic imaging. As shown in FIG. 5A, the image pickup person points the image pickup apparatus to himself / herself with the image pickup apparatus and performs imaging while panning (rotation). The panoramic image of the product expected by the photographer is an image in which only one person image of the photographer appears before the background image (lined tree) as shown in FIG. 5 (b). However, when the conventional panoramic imaging method is used, the image of the image capturing person is included in all the images captured during panning, and the image of the image capturing person is displayed a plurality of times.

  In order to solve the problems as described above, in the present embodiment, different from the prior art shown in FIG. 4, the foreground image and the background image are extracted during panorama imaging, the background image is synthesized, and the foreground image Select and compose. Hereinafter, the flow of the present embodiment will be described in detail with reference to the drawings.

  FIG. 6 is a flowchart for explaining the present embodiment. When the user selects the self-portrait panorama mode with the mode switching switch 104, the digital camera 100 executes the processing of FIG. Alternatively, the user selects the self-portrait panoramic mode from the menu screen of the digital camera 100 or the touch panel also serving as the display unit 101 by operating the button or the like provided on the operation unit 102.

  In step S601, the user performs setting for panoramic imaging. For example, the user sets the angle of view and panning direction of the panoramic composite image. If it is determined in step S601 that the user has not made settings, the digital camera 100 uses initial settings or uses settings from the previous panorama imaging in panorama imaging.

  Subsequently, in step S602, the system control unit 210 determines whether or not the first switch SW1 is pressed. If it is determined that the first switch SW1 is pressed, the process proceeds to step S603. If it is determined that the first switch SW1 has not been pressed, the process returns to the process of step S601.

  In step S603, the detection unit 215 performs posture detection processing. In the attitude detection process, the detection unit 215 determines the panning direction of the digital camera 100. If the user has set the panning direction of the digital camera 100 in step S601, then the system control unit 210 compares the panning direction set in the digital camera 100 with the actual panning direction, and if different, appropriately. Give a warning to

  In step S604, the digital camera 100 performs automatic exposure processing control (AE) processing and automatic focus detection (AF) processing.

  In step S605, the system control unit 210 determines imaging conditions such as imaging sensitivity based on the AE processing result in step S604.

  In step S606, the system control unit 210 determines whether the second switch SW2 has been pressed. If it is determined in step S606 that the second switch SW2 has not been pressed, the system control unit 210 returns to step S602, and whether or not the first switch SW1 is kept pressed (whether so-called half pressed or not) )judge. When it is determined in step S606 that the system control unit 210 determines that the second switch SW2 is pressed, the process proceeds to step S607, and the imaging unit 204 captures an image under the imaging condition determined in step S605.

  In step S608, the system control unit 210 determines whether imaging is continuing, and if so, the process returns to step S607, and if it is determined that imaging has ended, the process proceeds to step S609. For example, based on the user setting in step S601, the system control unit 210 determines whether the imaging is continued or ended. As an example, when the angle of view set in step S601 is reached, the system control unit 210 determines that the imaging has ended. In the case of a self-portrait panorama, as shown in FIG. 5A, the photographer often pans while rotating, and it is preferable that imaging ends when the rotation angle reaches 360 degrees. In this case, in step S608, the system control unit 210 may use information of a gyro provided in the detection unit 215 to determine the end of imaging. Alternatively, the system control unit 210 determines the end of imaging when the user cancels the pressing of the second switch SW2.

  In step S609, the system control unit 210 performs composition on the image captured by the imaging unit 204 in step S607. The processes in steps S601 to S608 are substantially the same as the conventional panoramic imaging, and the process in step S609 is a characteristic part of this embodiment. In the following, step S 609 will be described in detail using the drawings and the flowchart.

  FIG. 7 is a flowchart for explaining the process of image composition in step S609 in the present embodiment. First, in step S701, the system control unit 210 determines whether the digital camera 100 is set to the self-portrait panorama mode. If the digital camera 100 is not in the self-portrait panorama mode, the conventional panorama synthesis method as shown in FIG. 4 is used, and the flow proceeds to step S707. In the case of the self-portrait panoramic mode, the flow proceeds to step S702. In step S702, the system control unit 210 selects an image to be processed in the order of imaging.

  FIG. 8 is a diagram for explaining an area set for an image to be processed in the present embodiment. For the image to be processed selected by the system control unit 210 in step S702, it is necessary to specify the foreground area and the background area in order to prevent overlapping of human images in the process described later. At the same time, the system control unit 210 determines a motion vector search area for alignment. FIGS. 8A to 8C are examples of setting of the foreground area, the search area of the motion vector, and the background area. In FIG. 8A, the system control unit 210 extracts a human image area using known face detection processing or the like, and sets a rectangular area having a human image width and an image height as a foreground area. Is shown. Alternatively, before capturing a panoramic image, scan AF which repeats imaging while moving the focus lens in advance is performed to determine the subject distance for each area, and an area corresponding to distance information less than a predetermined value is an area of a human image It may be extracted as

  In FIG. 8B, the system control unit 210 sets a portion of the image other than the foreground area as a motion vector search area. FIG. 8C shows that the system control unit 210 sets rectangular portions on both sides of the human image as a background area. Note that FIG. 8 is only an example, and in FIG. 8 all images other than the foreground area are set as the search area for motion vectors in order to simplify the description, but the present invention is not limited to this. For example, the foreground area may be set along the contour of the human image. Note that the posture or expression of the person corresponding to the foreground area may greatly change depending on the image and is not suitable for detecting a motion vector, so it is desirable not to include it in the motion vector search area.

  The system control unit 210 sets a plurality of small blocks in all or part of the motion vector search area. The system control unit 210 selects one of a plurality of small blocks set in one image, and the same position as the selected small block in the search area of the motion vector of the other image, And set a search block wider than this small block. The system control unit 210 extracts, in the other image, an area where the correlation with the selected small block of one image is the highest (the value of SAD is the smallest). The system control unit 210 can calculate a motion vector corresponding to the small block from the coordinates of the corresponding point which is the center of the extracted area and the coordinates of the center position of the selected small block. The system control unit 210 performs this for all the small blocks, and calculates motion vectors corresponding to all the small blocks.

  In step S703, the system control unit 210 determines whether the foreground area of the image to be processed selected in step S702 is to be used for the composite image. For example, it is assumed that the user sets in advance on the menu screen or the like so that the first image, the last image, or the human image of any one image captured after a predetermined time from the start of shooting is used for composition. In step S703, the system control unit 210 determines whether to use the foreground area of the image to be processed for composition based on the settings of the user. If the foreground area of the image to be processed is used for composition, the process proceeds to step S704, and the system control unit 210 assigns a flag indicating that all of the image to be processed is used for composition. Conversely, when the foreground area of the image to be processed is not used for composition, the process proceeds to step S705, and the system control unit 210 assigns a flag indicating that only the background area to be processed is used for composition. Next, in step S706, the system control unit 210 determines whether all the images have been processed. If not, the process returns to step 702. If it is completed, the process proceeds to step S707. In step S 707, the system control unit 210 performs alignment and combination. Hereinafter, generation of a composite image will be described in detail with reference to the drawings.

  FIG. 9 is a diagram for describing the case of combining only the background area in the present embodiment. The images shown in FIG. 9A to FIG. 9D show each of the four images obtained by panoramic imaging. These four images do not use the foreground area for combining in step S 705 in FIG. 7, and are images provided with a flag that uses only the background for combining. Each of FIG. 9 (e) to FIG. 9 (h) shows a composite image after using each of FIG. 9 (a) to FIG. 9 (d) for composition. First, FIG. 9A shows that background areas a and b (gray parts) exist on both sides of a person image corresponding to a foreground area. FIG. 9E shows that only the background area of the image of FIG. 9A is used for composition. FIG. 9 (e) also shows the foreground area, but this foreground area is finally overwritten as described later. Thus, the foreground region of the image may be deleted after it is determined that it will not be used for compositing. FIG. 9B shows an image captured next to FIG. 9A. Similarly, background areas a and b (gray parts) exist on both sides of the foreground area of this image. In FIG. 9B, of the background areas located on both sides of the foreground area, areas not overlapping with the background area of FIG. 9A are set as background areas a and b. FIG. 9 (f) shows an image obtained by combining the background areas of the images shown in FIGS. 9 (a) and 9 (b). The system control unit 210 performs the same processing on the images shown in FIG. 9C and FIG. 9D, and finally generates a composite image as shown in FIG. In FIG. 9H, there is no foreground area in FIGS. 9A to 9D, and a composite image of a background area with natural joints is obtained. Since the gap between the background area on the left side of the fourth sheet and the background area on the right side of the first sheet disappears when the images up to the fourth image shown in FIG. In the image after the eye, only the background area on the right side of the foreground area may be used for composition.

  FIG. 10 is a diagram for describing the case where the foreground area of the first image is used for the composite image in the composition in the present embodiment. Similar to FIG. 9, the images shown in FIGS. 10 (a) to 10 (d) show four images obtained by imaging, and each of FIGS. 10 (e) to 10 (h) is a diagram. The composite image after synthesize | combining 10 (a) thru | or FIG. 10 (d) in order is shown. Here, it is assumed that the user sets the foreground area of the first image (the image shown in FIG. 10A) to be used for the composite image. In this case, as shown in FIG. 10E, the entire surface of the image of FIG. 10A is used as a composite image. Next, with respect to FIG. 10B, the system control unit 210 uses the gray background area b on the right side for synthesis. The reason is that the portions other than the background area b overlap the image shown in FIG. 10A, and the system control unit 210 preferentially uses the image shown in FIG. Similarly, the system control unit 210 performs the same process on FIGS. 10 (c) and 10 (d), and finally creates a composite image as shown in FIG. 10 (h). FIG. 10 (h) shows a panoramic image in which a composite image of the background is shown together with the foreground area of FIG. 10 (a).

  FIG. 11 is a diagram for describing an example of using the foreground area of the third image in the composition in the composition in the present embodiment. FIGS. 11A to 11D show four images obtained by imaging, which are similar to FIGS. However, FIG. 11 differs from FIG. 10 in that the foreground area of the third image (FIG. 11 (c)) is used for combining. The processes for FIGS. 11 (a) and 11 (b) are the same as in FIG. That is, the system control unit 210 cuts out the background areas of the images shown in FIGS. 11A and 11B and uses them in the synthesis to generate a synthesized image shown in FIG. Here, in order to use the foreground area of the third image shown in FIG. 11C for the composite image, the entire surface of the image shown in FIG. 11C is used for the composite image as shown in the flowchart in FIG. Do. Specifically, as shown in FIG. 11 (g), FIG. 11 (c) shows that the entire image shown in FIG. 11 (c) is used for the composite image, and the portion overlapping with the already synthesized portion is overwritten. Use images preferentially. As a result, after the third image is used for combining, a combined image as shown in FIG. 11 (g) can be generated. The image shown in FIG. 11 (g) consists of the background area on the left side of FIG. 11 (a) and the entire image of FIG. 11 (c). Next, after the image illustrated in FIG. 11D is captured, the system control unit 210 uses the background area b on the right side of the image illustrated in FIG. Create a composite image as shown in.

  However, the synthesis method as shown in the flowchart of FIG. 7 is merely an example, and various modifications can be made. For example, the following synthesis method may be used.

  FIG. 12 is a diagram for describing another example of using the foreground area of the third image in the composition in the composition in the present embodiment. The difference between the combining method shown in FIG. 12 and the method shown in FIG. 11 is that a range narrower than the entire surface of the third image is used for the combining image. Each of FIG. 12 (a) to FIG. 12 (d) is the same as each of FIG. 11 (a) to FIG. 11 (d). The system control unit 210 cuts out the background areas of the images shown in FIG. 12A and FIG. 12B and uses them for combining to generate a combined image as shown in FIG. 12 (f) is similar to FIG. 11 (f).

  Here, it can be seen from the composite image shown in FIG. 11 (h) that the background area a on the left side of the image shown in FIG. 11 (b) is not included in the composite image. This is because the image shown in FIG. 11C is overwritten when it is used for composition. In FIG. 12, the area used for combining the third image is reduced so that the background area a described above is not overwritten. In FIG. 12C, the area used for composition by the system control unit 210 is gray, and a portion corresponding to the background area a in FIG. 12B is missing compared to FIG. 11C. Recognize. According to the method shown in FIG. 12, the background area a of FIG. 12B can be reflected in the composite image of FIG.

  If it is determined in step S701 that the self-portrait panoramic mode is not set, in step S707, panoramic images may be synthesized by a conventional method. That is, it is not necessary to extract the foreground area and the background area from each of the continuously captured images, and as shown in FIG. 3 and FIG. 4, the area near the center of the image is cut out, aligned and synthesized. Just do it.

  Further, in FIG. 9 to FIG. 12 described above, although the example in which the entire image is divided into the foreground area and the background area has been described in order to facilitate the description, the present invention is not limited to this. As shown in FIG. 3 and FIG. 4, an area near the center of the image may be cut out in a strip shape, and the foreground area and the background area may be distinguished within this area. However, it is necessary to provide an area which is cut into a strip shape with a sufficient width so that the photographer does not go out.

  Further, the present invention can be applied not only to the case where the "foreground" is a photographer but to various scenes. For example, the system control unit 210 uses a trained convolutional neural network (CNN) as a foreground extraction unit, and uses a so-called general object recognition unit to extract a subject that can be the foreground. In that case, the system control unit 210 may set an object which the user wants to be recognized as a foreground in advance from among the learned general objects.

  Up to this point, the explanation has been made on the premise that a person other than the photographer has not entered an angle of view for photographing. When an image of a person other than the photographer enters the angle of view of panoramic imaging, the following problems may occur in the above-described implementation method. FIG. 13 is a diagram for describing the case where an image of a person other than the image capturer in the present embodiment exists at an angle of view of panoramic imaging.

  13A to 13D show four images continuously captured by the imaging unit 204 in panoramic imaging. Among them, it can be seen that the images of FIG. 13B to FIG. 13D contain one more person image other than the person image of the imaging person. In the foreground area determination method described above, the system control unit 210 identifies a human image by face recognition, and regards the human image as a foreground area. However, in the case as illustrated in FIG. 13, the system control unit 210 may recognize an image of a person who is not the image capturing person as the foreground area. If a human image originally existing in the background is erroneously recognized as being in the foreground by the system control unit 210, it is not used for synthesis, and the angle of view corresponding to the background area may be lost. For example, in the case shown in FIG. 13, it is assumed that the system control unit 210 misidentifies the rightmost area of the image shown in FIG. 13B as the foreground area. Since the rightmost area of the image shown in FIG. 13B is regarded as the foreground area, the system control unit 210 selects the foreground area of the image shown in FIG. The rightmost region of the image shown in FIG. 13 (b) is not used for composition. If the system control unit 210 does not use the area of the other image having the angle of view corresponding to the rightmost area of the image illustrated in FIG. 13B in the composite image, the angle of view is lost. For example, it is assumed that the system control unit 210 sets the first image, that is, the foreground area of the image illustrated in FIG. 13A to be used for composition. In this case, since the rightmost area of the image shown in FIG. 13B corresponds to the foreground area, the system control unit 210 does not use this area for the composite image. Similarly, even in the image shown in FIG. 13C, the system control unit 210 recognizes a person who is not the image pickup person as a foreground area and does not use it for the composite image. The same applies to FIG. As a result, none of the regions having the angle of view corresponding to the rightmost region of the image shown in FIG. 13B is used for the composite image, and a portion corresponding to the field angle is missing in the composite image.

  In order to solve the problems as described above, it is necessary to revise an implementation method in which the system control unit 210 simply recognizes a person and determines it as a foreground region. For example, when two or more human images appear, the system control unit 210 acquires the subject distance of the human image, and determines only the closest subject distance as the foreground area. Alternatively, the system control unit 210 determines that only the human image with the largest area is the foreground area. It can be expected to prevent the system control unit 210 from misidentifying a person who is not a photographer as a foreground area by the above method.

  According to the present embodiment, when capturing a panoramic image including a person's image of the foreground imager using an image captured and photographed by the imager by self-portrait, the image of the foreground does not appear many times, and it is uncomfortable Can create panoramic images.

(Other embodiments)
Although the above embodiment has been described based on a digital camera for individuals, if a panoramic imaging and combining function is installed, a portable device, a smartphone, a network camera connected to a server, etc. It is also possible to apply to

  Note that the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a recording medium, and one or more processors in a computer of the system or apparatus Can be realized by processing for reading out and operating. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

100 digital camera 102 operation unit 104 mode selector switch 130 recording medium 204 imaging unit 206 image processing unit 207 memory control unit 209 memory 210 system control unit 215 detection unit 216 I / F

Claims (13)

Recognition means for recognizing a foreground and a background from each of a plurality of images captured while performing a panning operation;
A foreground image, which is an image corresponding to the foreground recognized in any one of the plurality of images, and a plurality of background images corresponding to the background recognized in each of the plurality of images, are combined to form a panoramic image And generating means for generating
The image processing apparatus according to claim 1, wherein the combining unit combines the foreground image with priority in a region where the angle of view overlaps with the foreground image among the plurality of background images.   The combining unit may overwrite the background image on the background image or combine only the foreground image in a region where the angle of view overlaps the foreground image among the plurality of background images. The image processing apparatus according to claim 1, characterized in that   When the first mode is set, the combining means combines the foreground image and the plurality of background images using the recognition result of the recognition means, and the second mode is set. 3. The image processing apparatus according to claim 1, wherein the image is synthesized using the plurality of images without using the recognition result of the recognition unit.   The image processing apparatus according to claim 3, wherein in the first mode, the recognition unit recognizes a person's image of a photographer as a foreground.   The image processing apparatus according to claim 4, wherein in the first mode, the recognition unit recognizes an area not including the person image of the image pickup person as a background.   The image processing apparatus according to any one of claims 3 to 5, wherein the first mode is a mode in which a panoramic image is generated using an image obtained by capturing an image of a photographer.   The image processing apparatus according to any one of claims 1 to 6, wherein any one image is designated in advance before the plurality of images are captured.   The image processing apparatus according to any one of claims 1 to 7, wherein the combining unit combines the plurality of images in the order in which the plurality of images are captured.   The image processing according to any one of claims 1 to 8, wherein the combining means combines one whole of the image including the foreground image in preference to other background images used for combining. apparatus.   The combining unit preferentially combines, in one image including the foreground image, an image of a region present in the direction of the panning operation with respect to the foreground image over other background images used for combining. The image processing apparatus according to any one of claims 1 to 8, characterized in that: Imaging means,
A recognition unit that recognizes a foreground and a background from each of a plurality of images captured by the imaging unit while performing a panning operation;
A foreground image, which is an image corresponding to the foreground recognized in any one of the plurality of images, and a plurality of background images corresponding to the background recognized in each of the plurality of images, are combined to form a panoramic image And generating means for generating
The imaging device is characterized in that the combining means preferentially combines the foreground image in a region where the angle of view overlaps with the foreground image among the plurality of background images. Recognizing a foreground and a background from each of a plurality of images captured while panning;
A foreground image, which is an image corresponding to the foreground recognized in any one of the plurality of images, and a plurality of background images corresponding to the background recognized in each of the plurality of images, are combined to form a panoramic image Generating a synthesis step,
The image processing method according to claim 1, wherein, in the combining step, the foreground image is prioritized and combined in a region where the angle of view overlaps with the foreground image among the plurality of background images. A computer program that causes a computer of an image processing apparatus to operate,
Recognizing a foreground and a background from each of a plurality of images captured while panning;
A foreground image, which is an image corresponding to the foreground recognized in any one of the plurality of images, and a plurality of background images corresponding to the background recognized in each of the plurality of images, are combined to form a panoramic image Generating a synthesis step,
The program is characterized in that, in the combining step, in a region where the angle of view overlaps with the foreground image among the plurality of background images, the foreground image is combined preferentially.

Images