JP5230013B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus capable of performing image editing.

  In recent years, cameras are shifting from film cameras that use film to digital cameras that use image sensors. One of the big merits of digital cameras is that the built-in memory has a large capacity, so you can easily shoot anytime, anywhere without worrying about the remaining number of shots or failed shots. However, when shooting with such a handy digital camera, an object unnecessary for the photographer may enter the shooting range, and the advantage of being able to shoot anytime and anywhere may not be utilized. For example, when taking a memorial photo at a tourist spot, unnecessary objects may fall within the shooting range, such as other tourists entering the shooting range or a passing car at the time of shooting. There is. Therefore, it is necessary for the photographer to change the shooting location or to measure the shooting timing, and there is a problem that the photographer cannot take a photo at the desired location and timing.

  As a technique for deleting such an unnecessary object, several techniques for removing an object interrupted during photographing have been proposed. For example, when photographing a person with a landscape background, three images are continuously photographed at a predetermined time interval, and a matching portion is extracted from two or more images of the three images. A method has been proposed in which an image obtained by removing a moving third person who has entered a shooting range during shooting is synthesized by synthesizing images (see, for example, Patent Document 1). In addition, when shooting a landscape from a train window, a technique has been proposed in which continuous shooting is performed, and unnecessary obstacles in front are deleted from images before and after the obstacle is not reflected (for example, Patent Document 2).

Japanese Patent Laid-Open No. 2005-12660 JP 2007-305050 A

  However, the technique disclosed in Patent Document 1 is an unnecessary object that can be removed only by a moving object, and cannot be applied to an unnecessary object that is stationary. Further, since all moving objects are to be removed, they are removed even if the photographer wants to leave them when the subject is a moving object. For this reason, even if there are a plurality of moving bodies, it is not possible to select what should be removed and what should be left, and a desired image cannot be obtained. Furthermore, in the case of three consecutive images, there is a problem that if another moving object enters the same range during continuous shooting, the portion shielded by the removal target cannot be interpolated.

  On the other hand, the technique disclosed in Patent Document 2 is a limited environment in which a photographer is on a moving vehicle, and an unnecessary object cannot be removed under an environment where the photographer is stationary. Also, since moving objects relative to the photographer are removed, even if you do not want to erase a part, all the moving objects in front are removed, and the desired image is obtained. There is a problem that can not be.

  That is, the conventional technique removes only an object that moves relative to the photographer, and what can be removed is limited. In addition, an image obtained by deleting unnecessary objects from a plurality of photographed images is obtained. However, if the relative speed is low even if the object is moving relative to the photographer, There is no guarantee that a completely shielded part will enter the field of view, and in that case there is a problem that the removal cannot be performed well.

  The present invention has been made in view of such circumstances, and provides an imaging apparatus that can easily generate an image from which an object designated by a photographer is removed regardless of whether it is a moving body or a stationary body. The purpose is to do.

  The present invention displays an imaging unit that captures a subject, a first subject image acquisition unit that obtains first subject image data captured by the imaging unit, the first subject image data, and the first subject image data. An area designating unit for designating a removal target area on the image data; a second subject image obtaining unit for obtaining second subject image data captured by the imaging unit; and the removal target area from the second subject image data A partial image data corresponding to the image, and determining whether or not the partial image data can be combined with the removal target region on the first subject image data; and the image determination An image synthesizer that synthesizes the partial image data with the removal target area on the first subject image data when the determination result of the image can be synthesized.

  According to the present invention, the image composition unit uses the partial image data extracted from each of the plurality of second subject image data acquired by the second subject image acquisition unit, and uses the partial image data on the first subject image data. The image composition is performed on the removal target area.

  The present invention is characterized in that the image synthesizing unit deforms the partial image data and synthesizes the image with respect to the removal target region on the first subject image data.

  The present invention is characterized by further comprising an instruction display unit that displays a photographing instruction for photographing the second subject image again when the judgment result of the image judgment unit is not compositable.

  The present invention is characterized in that the instruction display unit displays a photographing instruction for photographing the second subject image at a photographing position different from the photographing position of the first subject image.

  The present invention is characterized in that when the plurality of second subject images are photographed, the instruction display unit displays photographing instructions so that the photographing positions of the second subject images are different from each other. To do.

  The present invention includes a moving body determination unit that determines whether or not the subject in the region specified by the region specifying unit is a moving body, and the instruction display unit includes the subject in the region as a moving body. In some cases, a shooting instruction for shooting the second subject image from the shooting position at which the first subject image was shot is displayed. If the subject in the region is not a moving object, the subject in the region is displayed. Moving to a position where no image is captured, and a shooting instruction for shooting the second subject image is displayed.

  The present invention includes a depth determination unit that can determine the positional relationship of the imaging region, and the instruction display unit displays the direction and distance of the second imaging position according to the removal target region. And

  According to the present invention, the imaging unit includes a two-lens imaging unit capable of obtaining two subject images 1 viewed from different viewpoints at the same time, and the depth determination unit acquires the depth from the parallax of the two subject images 1 It is characterized by.

  According to the present invention, it is possible to obtain an image obtained by removing an object unnecessary for a photographer regardless of whether it is a stationary body or a moving body when photographing.

1 is a block diagram illustrating a configuration of an imaging apparatus according to a first embodiment of the present invention. It is explanatory drawing which shows the processing operation which synthesize | combines two subject images. It is explanatory drawing which shows the positional relationship at the time of imaging | photography of an image. It is a flowchart which shows the processing operation of the imaging device by the 1st Embodiment of this invention. It is explanatory drawing which shows the positional relationship at the time of imaging | photography of an image. It is explanatory drawing which shows the positional relationship at the time of imaging | photography of an image. It is a flowchart which shows the processing operation of the imaging device by the 2nd Embodiment of this invention.

<First Embodiment>
Hereinafter, an imaging device according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a schematic block diagram illustrating the configuration of the imaging apparatus according to the first embodiment. This imaging apparatus can also be mounted on a digital camera such as a digital single lens reflex camera, a PDA (personal digital assistant), a mobile phone, or the like. The imaging apparatus includes an imaging unit 10, an image processing unit 11, an image recording unit 12, a display unit 13, an instruction unit 14, an image determination unit 15, and an imaging timing presentation unit 16. The imaging unit 10 includes an imaging device, an imaging optical system (lens optical system), an A / D conversion unit, and the like, and forms an image of a subject incident on the imaging device by the imaging optical system. The subject image is converted into an electrical signal by the image sensor, and converted from an analog signal to a digital image signal by the A / D converter, and output as image data. A solid-state electronic device such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor is used as the image sensor.

  The image processing unit 11 receives image data output from the imaging unit 10 and performs various image processing such as smoothing processing, contrast adjustment, color correction, and signal compression, and composition processing on the input image data. In the combination processing, the region selected by the instruction unit 14 from a plurality of images is replaced or interpolated in the captured image. That is, by replacing or interpolating a region where an unnecessary object is reflected in the captured image, an image from which the unnecessary object is removed is created. The area to be replaced or interpolated only needs to be larger than the unnecessary object, and as a minimum area, there is a method in which a person or an object reflected in a designated area is extracted and only the area of the object is processed. For the synthesis area, an optimum method is selected in consideration of the processing amount and the photographed image.

  The image recording unit 12 includes a buffer memory used as a work area for image processing performed by the image processing unit 11, a flash memory for recording captured images, and the like, and reads and writes image data as necessary. The display unit 13 is composed of, for example, a liquid crystal monitor, and displays a recorded image display, a captured image display instead of a viewfinder, or an operation screen such as a menu.

  The instruction unit 14 performs an operation for designating an area to be removed by the photographer with a button and a cursor on the image displayed on the display unit 13. The display unit 13 may be provided with a touch panel such as a resistive film type or a capacitance type as the instruction unit 14 so that the photographer can easily select the touched part by touching it. In this case, the display unit 13 and the instruction unit 14 are integrally configured to form a display instruction unit. Further, the number of designated areas is not limited, and a plurality of designated areas may be designated. When this imaging apparatus is mounted on a mobile phone or the like, existing switches and buttons may be used.

  The image determination unit 15 determines whether or not replacement or interpolation is possible in the area specified by the instruction unit 14. The image determination unit 15 determines whether to replace or interpolate the selected region by comparing with the image that is about to be captured or has been captured based on the captured image of the subject, and sends the determination result information to the image timing presentation unit 16. Output. Based on the determination result information output from the image determination unit 15, the image timing presentation unit 16 provides information such as a shooting location and timing to the photographer so that an image that can be replaced or interpolated can be captured. Present. The timing and place presentation method may be notified by the lamp whether the composition can be easily performed, or may be presented in which direction the photographing should be performed.

  Next, with reference to FIG. 2, a processing operation for generating a composite image will be described. In FIG. 2, reference numerals 1, 2, and 3 indicate images. Image 1 is the first captured image (referred to as subject image 1). Image 2 is a second captured image for obtaining a partial image for interpolation or replacement (this is referred to as subject image 2). For the image 3, the image processing unit 11 inputs the subject image 1 and the subject image 2, removes unnecessary portions from the subject image 1 by image processing, and removes unnecessary portions from the subject image 2 from the subject image 2. It is a composite image obtained by extracting partial images and combining the extracted partial images by performing processing such as geometric transformation.

  The composite image 3 is for selecting an unnecessary subject X in the photographed subject image 1 and obtaining an image from which the unnecessary subject X is removed by newly photographing at least one subject image 2. In the second and subsequent shootings performed for image composition, shooting is performed so that the area shielded in the subject image 1 is captured at a position that wraps around the unnecessary subject X with respect to the first shooting position. Do. By replacing or interpolating the subject image 1 using the subject image 2 thus obtained, a composite image 3 from which the unnecessary subject X is removed can be obtained. This operation can be easily performed by a photographer operating at the shooting site in accordance with an instruction from the imaging apparatus.

  Next, details of the image composition processing shown in FIG. 2 will be described with reference to FIGS. 2, 3, and 4. FIG. 3 is a diagram illustrating a relationship between the subject and the photographing position of the imaging device when the subject is photographed as viewed from above. FIG. 4 is a flowchart showing the processing operation of the imaging apparatus shown in FIG. First, the subject image 1 photographed at an angle that the photographer wants to photograph is stored in the image recording unit 12 (step S11). The subject image 1 is an image obtained by photographing a building 202 in front with a mountain 201 in the background in the distance as shown in FIG. Since the photographer has photographed at the position and timing at which the photographer wants to photograph, an unnecessary object (removal object) X is reflected between the building 202 and the photographing position 205. At this time, the shooting range of the camera is the range of the broken lines 206a and 206b shown in FIG. 3, and the area 208 surrounded by the solid lines 207a and 207b in the range is shielded by the unnecessary object X and the back cannot be seen. It is in a state. Therefore, a part of the door 203 is hidden by the unnecessary object X. In FIG. 3, a portion of the building 202 shielded by the unnecessary object X is indicated by a shielding area 209.

  Next, the photographer operates the instruction unit 14 to designate a region to be removed on the subject image 1 (step S12). The instruction unit 14 includes a touch panel provided on the display unit 13, and traces the polygonal region on the image while tracing the outline of the unwanted object X on the subject image 1 displayed on the display unit 13 with a finger or the like. specify. The image processing unit 11 extracts the coordinate information of the polygonal region on the image instructed by the instruction unit 14. The coordinate information of the area to be removed becomes the coordinate information of the synthesis area. In addition, the depth information acquisition unit that can acquire depth information is equipped in the camera to specify the removal target, and the depth information of the specified target is acquired, and an area with a depth close to the peripheral area of the specified part is automatically selected. Alternatively, a method of simply designating a removal target region by extracting automatically or a method of extracting a region based on color information may be used. For example, the depth information may be obtained by a method of calculating a depth by obtaining a parallax amount from a stereo image obtained from a multi-lens configuration in which the imaging unit has two or more eyes, and irradiating a certain pattern. A method of calculating the depth by combining lasers may be used. Although the designation operation using the touch panel is described here, the designation operation method is not limited to this.

  Further, the removal target is designated by allowing the camera to select and confirm the photographer, such as automatically selecting a person or object that is out of focus as the removal target based on the degree of focus of the subject image 1. Also good. A region having a certain size including the unnecessary object X is selected as the region to be combined, and the subject images 1 and 2 are processed according to the correspondence relationship in the peripheral region of the unnecessary object X. Of course, the unnecessary object X may be extracted from the image, and only the portion shielded by the unnecessary object X may be targeted. In this case, the smaller the area to be processed, the smaller the amount of processing for synthesis. In the present embodiment, a corresponding region in the subject image 2 is obtained from an area that includes the entire target region by pattern matching in the peripheral portion of the unnecessary object X and a correspondence relationship by feature points.

  Next, the image processing unit 11 determines whether or not the removal target specified in step S12 is a moving object (step S13). The image processing unit 11 has a function of discriminating a moving body on the subject image 1 and appropriately selects a processing method according to the moving speed of the removal target. In this embodiment, whether the unnecessary object X to be removed is a moving body or a stationary body is determined by the following process. When the subject image 1 captured by the imaging unit 10 is recorded in the image recording unit 12, the image processing unit 11 stores a plurality of images arranged in time series, and determines whether the object is a moving object based on the difference between the images. . For example, when the subject image 1 is captured, the back frame is recorded at the same time, and the positional relationship between the subject image 1 and the unnecessary object X in the back frame is compared to determine whether the object is a moving body or a stationary body. At this time, when the unnecessary object X is a moving object, the image processing unit 11 calculates the moving speed of the moving object based on the moving amount (shift amount on the image) of the moving object and the frame rate.

  Here, although it has been described that it is determined whether or not it is a moving body from a plurality of images, it is also possible to determine the speed and determine the speed of the moving body by providing a sensor for speed measurement. However, the processing method in the case of a moving body is performed for a moving speed exceeding a predetermined threshold. This is because it may take time to move out of the processing target area for a very slow target.

  Next, when the image processing unit 11 determines that the unnecessary object X is a moving body, the shooting timing presentation unit 16 displays a message for instructing the shooting position and timing to the photographer on the display unit 13. . If the unwanted object X is a moving object, the area shielded by the unwanted object X can be seen after a certain period of time, so if it is determined that the selected removal object is moving, shooting is performed. The timing presentation unit 16 displays a message on the display unit 13 instructing the user to shoot again on the spot. For example, when an unnecessary object X is selected from the subject image 1, a message “Please shoot again on the spot” is displayed on the display unit 13. When the photographer performs a shooting operation again on the spot in accordance with the instruction displayed on the display unit 13, the image processing unit 11 records the subject image 2 captured by the imaging unit 10 in the image recording unit 12 (step S14). ). As a result, the subject image 1 in which the unwanted object X is reflected and the subject image 2 in a state where the unwanted object X on the subject image 1 is moved to another position are recorded in the image recording unit 12.

  Next, the image processing unit 11 determines whether or not the subject image 1 and the subject image 2 recorded in the image recording unit 12 can be combined (step S15). For example, if at least a part of the removal target area is obstructed by another object at the time of re-shooting, since the compositing cannot be performed for the photographer, a message for instructing to shoot again is displayed on the display unit 13. The process is repeated until a subject image 2 that can be reliably combined is obtained. This makes it possible to check whether or not image composition is possible at the time of shooting, thereby eliminating image composition failure.

  Then, when the compositing subject image 2 is obtained, a compositing process is performed by interpolating or substituting the removal target area of the subject image 1 using the partial image of the subject image 2 (step S16). Since the subject image 2 is taken from almost the same position as the subject image 1, it is easy to match the subject image 1 without performing complicated processing. Therefore, the alignment of the subject image 1 and the subject image 2 and the size adjustment are minimized. Limit processing is performed to replace the area to be removed. At this time, since there is a time difference between the shooting times of the subject image 1 and the subject image 2, the brightness of the subject image 2 may change to the subject image 1 although the brightness due to sunlight may change when shooting outdoors. Process and match to match. Then, the image processing unit 11 stores the synthesized image in the image recording unit 12 (step S20).

  On the other hand, when the image processing unit 11 determines that the unnecessary object X is a stationary body (or an object that moves at a speed equal to or lower than the threshold), the shooting timing presentation unit 16 sets the shooting position and timing to the photographer. An instruction message is displayed on the display unit 13. If the unwanted object X is a stationary object, the region shielded by the unwanted object X cannot be photographed unless it moves from the photographing position where the subject image 1 was photographed. Therefore, the photographing timing presentation unit 16 determines that it is a stationary object. In response to the result, the photographer is instructed to move to the area between the unnecessary object X and the building 202 to shoot. The photographer changes the shooting position according to the instruction displayed on the display unit 13 to shoot the subject image 2, and the image processing unit 11 stores the shot subject image 2 in the image recording unit 12 (step S17). In this way, the subject image 2 shown in FIG. 2 is obtained. At this time, it approaches the door 203 which is the subject to be photographed, but it is necessary to photograph so that the entire area 209 where the unnecessary object X is shielded is in the frame.

  As shown in FIG. 3, the shooting position 210 indicates the shooting position of the subject to be shot (the door 203 of the building 202), and the shooting range at that time is from the broken line 211a to the broken line 211b and is shielded by the unnecessary object X. It can be seen that the region 209 is within the shooting range. The shooting position 210 is preferably on a line connecting the first shooting position 205 and the unnecessary object X. This is because the region 209 can be photographed at a photographing angle close to the subject image 1 and the composition process can be simplified.

  The image determination unit 15 determines whether or not the subject image 1 can be interpolated or replaced by the captured subject image 2 by determining whether or not the subject image 1 and the subject image 2 can be combined (step S18). ). This determination is performed based on whether or not the region 209 that is the interpolation target of the subject image 1 is completely within the shooting range in the subject image 2, for example, extracting feature points of the subject image 1 and the subject image 2, respectively. This is performed depending on whether a corresponding feature point is obtained. In the example shown in FIG. 2, feature points P1, P2, and P3 around the removal target area including the unnecessary object X designated on the subject image 1 are extracted, and feature points P1 ′, P2 ′, and P3 are extracted from the subject image 2. 'Is extracted, and the feature point P1 and the feature point P1', the feature point P2 and the feature point P2 ', and the feature point P3 and the feature point P3' are determined by correspondence. The feature points are extracted using a known image feature point extraction process.

  The image determination unit 15 cannot synthesize the display unit 13 when the area necessary for interpolation or replacement is not within the shooting range of the subject image 2 (no corresponding point). And a message instructing to change the position and take a picture is displayed. At this time, if the area 209 is larger than the shooting range, the zoom lens is driven in the wide direction, and if it still does not fit within the shooting range, the photographer is instructed to go down and take a picture. When the photographer takes a picture again according to the instruction, the image processing unit 11 stores the photographed subject image 2 in the image recording unit 12.

  Next, the image processing unit 11 combines the obtained subject image 2 and subject image 1 (step S19). The compositing process is performed by performing geometric transformation such as projective transformation or affine transformation on the subject image 2 based on the coordinates of the corresponding point points associated with the subject image 1 and the subject image 2, and the removal target object This is done by cutting out the region portion and interpolating or replacing the subject image 1. At this time, the brightness and the color tone are corrected so as to be matched with the subject image 1, and then synthesized. By doing so, it is possible to obtain a composite image obtained by removing unnecessary unnecessary objects X from the subject image 1 photographed at a position and timing that the photographer likes. Then, the image processing unit 11 stores the synthesized image in the image recording unit 12 (step S20).

  In the above description, the case where there is one unnecessary object X to be removed has been described. However, even when there are a plurality of unnecessary objects X on the subject image 1, each of the shielded regions is similarly described. Then, the composite image from which the unnecessary object X is removed can be obtained by photographing again and compositing. For example, when there are n unnecessary objects X to be removed, a shielding area that cannot be seen by each of the n unnecessary objects X is individually photographed, and a synthesis process is performed on each portion. At this time, a maximum of n shot images necessary for composition are required. Further, even when the moving object and the stationary object are mixed with the unnecessary object X, it can be dealt with by combining the re-photographing of the subject image 1 from the photographing location and the photographing after changing the photographing position.

  In some cases, a person or the like is photographed against a background of a building or landscape, but the person only needs to photograph when photographing the first subject image 1. This is because the entire image is not processed, but is divided and processed in the area up to the periphery of the removal target, and the subject image 1 is used as a reference except for the processing target area. Therefore, it is not necessary for the subject to stand by at the same position during the shooting of the second subject image.

  As described above, even when an unnecessary subject is captured, it is possible to easily obtain an image from which a designated target is removed and to determine whether or not such an image capable of interpolation has been captured. In addition, failure of image composition can be prevented.

<Second Embodiment>
Next, an imaging device according to a second embodiment of the present invention will be described with reference to FIGS. In the imaging apparatus according to the second embodiment, in order to synthesize the image of the subject image 1, the photographer moves to a position where the area shielded by the unnecessary object X can be photographed and photographs the subject image 2. An image of the shielding area is obtained and the image is synthesized. For this reason, it can be used when it is difficult to move to the other side of the unwanted object X and photograph it, such as when the unwanted object X is far away. The configuration of the imaging apparatus in the second embodiment is the same as the configuration shown in FIG. FIG. 5 and FIG. 6 are diagrams showing the relationship between the subject and the photographing position of the imaging device when the subject is photographed as viewed from above. FIG. 7 is a flowchart showing the processing operation of the imaging apparatus shown in FIG. 1 in the second embodiment.

  Here, as in the first embodiment, description will be made assuming that a composite image obtained by combining the subject image 1 and the subject image 2 shown in FIG. 2 by the image processing unit 11 is obtained. First, the subject image 1 photographed at an angle that the photographer wants to photograph is stored in the image recording unit 12 (step S21). The subject image 1 is an image obtained by photographing a building 202 in front with a mountain 201 in the background in the distance as shown in FIG. Since the photographer has photographed at the position and timing at which the photographer wants to photograph, an unnecessary object (removal object) X is reflected between the building 202 and the photographing position 205. At this time, the shooting range of the camera is the range of the broken lines 206a and 206b shown in FIG. 5, and the area 208 surrounded by the lines 207a and 207b in the range is shielded by the unnecessary object X and the back cannot be seen. It is in a state. Therefore, a part of the door 203 is hidden by the unnecessary object X. In FIG. 5, a portion of the building 202 shielded by the unnecessary object X is indicated by a shielding area 209.

  Next, the photographer operates the instruction unit 14 to designate a region to be removed on the subject image 1 (step S22). The instruction unit 14 includes a touch panel provided on the display unit 13, and traces the polygonal region on the image while tracing the outline of the unwanted object X on the subject image 1 displayed on the display unit 13 with a finger or the like. specify. The image processing unit 11 extracts the coordinate information of the polygonal region on the image instructed by the instruction unit 14. The coordinate information of the area to be removed becomes the coordinate information of the synthesis area. In addition, the depth information acquisition unit that can acquire depth information is equipped in the camera to specify the removal target, and the depth information of the specified target is acquired, and an area with a depth close to the peripheral area of the specified part is automatically selected. Alternatively, a method of simply designating a removal target region by extracting automatically or a method of extracting a region based on color information may be used. . For example, the depth information may be obtained by a method of calculating a depth by obtaining a parallax amount from a stereo image obtained from a multi-lens configuration in which the imaging unit has two or more eyes, and irradiating a certain pattern. A method of calculating the depth by combining lasers may be used.

  Although the designation operation using the touch panel is described here, the designation operation method is not limited to this. Further, the removal target is designated by allowing the camera to select and confirm the photographer, such as automatically selecting a person or object that is out of focus as the removal target based on the degree of focus of the subject image 1. Also good. As a region to be removed, an area having a certain size including the unnecessary object X is selected, and the subject images 1 and 2 are processed according to the correspondence relationship in the peripheral area of the unnecessary object X. Of course, the unnecessary object X may be extracted from the image, and only the portion shielded by the unnecessary object X may be targeted. In this case, the smaller the area to be processed, the smaller the amount of processing for synthesis. In the present embodiment, a corresponding region in the subject image 2 is obtained from an area that includes the entire target region by pattern matching in the peripheral portion of the unnecessary object X and a correspondence relationship by feature points.

  Next, the image processing unit 11 determines whether or not the removal target specified in step S22 is a moving object (step S23). The image processing unit 11 has a function of discriminating a moving body on the subject image 1 and appropriately selects a processing method according to the moving speed of the removal target. In this embodiment, whether the unnecessary object X to be removed is a moving body or a stationary body is determined by the following process. When the subject image 1 captured by the imaging unit 10 is recorded in the image recording unit 12, the image processing unit 11 stores a plurality of images arranged in time series, and determines whether the object is a moving object based on the difference between the images. . For example, when the subject image 1 is captured, the back frame is recorded at the same time, and the positional relationship between the subject image 1 and the unnecessary object X in the back frame is compared to determine whether the object is a moving body or a stationary body. At this time, when the unnecessary object X is a moving object, the image processing unit 11 calculates the moving speed of the moving object based on the moving amount (shift amount on the image) of the moving object and the frame rate.

  Here, although it has been described that it is determined whether or not it is a moving body from a plurality of images, the determination of the moving body and its speed may be obtained by providing a sensor for speed measurement. . However, the processing method in the case of a moving body is performed for a moving speed exceeding a predetermined threshold. This is because it may take time to move out of the processing target area for a very slow target.

  Next, when the image processing unit 11 determines that the unnecessary object X is a moving body, the shooting timing presentation unit 16 displays a message for instructing the shooting position and timing to the photographer on the display unit 13. . If the unwanted object X is a moving object, the area shielded by the unwanted object X can be seen after a certain period of time, so if it is determined that the selected removal object is moving, shooting is performed. The timing presentation unit 16 displays a message on the display unit 13 instructing the user to shoot again on the spot. For example, when an unnecessary object X is selected from the subject image 1, a message “Please shoot again on the spot” is displayed on the display unit 13. When the photographer performs a photographing operation again on the spot in accordance with the instruction displayed on the display unit 13, the image processing unit 11 stores the subject image 2 captured by the imaging unit 10 in the image recording unit 12 (step S24). ). As a result, the subject image 1 in which the unwanted object X is reflected and the subject image 2 in a state where the unwanted object X on the subject image 1 is moved to another position are recorded in the image recording unit 12.

  Next, the image processing unit 11 determines whether or not the subject image 1 and the subject image 2 recorded in the image recording unit 12 can be combined (step S25). For example, if at least a part of the removal target area is obstructed by another object at the time of re-shooting, since the compositing cannot be performed for the photographer, a message for instructing to shoot again is displayed on the display unit 13. The process is repeated until a subject image 2 that can be reliably combined is obtained. This makes it possible to check whether or not image composition is possible at the time of shooting, thereby eliminating image composition failure.

  Then, when the compositing subject image 2 is obtained, the compositing process is performed by interpolating or substituting the removal target area of the subject image 1 using the partial image of the subject image 2 (step S26). Since the subject image 2 is taken from almost the same position as the subject image 1, it is easy to match the subject image 1 without performing complicated processing. Therefore, the alignment of the subject image 1 and the subject image 2 and the size adjustment are minimized. Limit processing is performed to replace the area to be removed. At this time, since there is a time difference between the shooting times of the subject image 1 and the subject image 2, the brightness of the subject image 2 may change to the subject image 1 although the brightness due to sunlight may change when shooting outdoors. Process and match to match. Then, the image processing unit 11 stores the synthesized image in the image recording unit 12 (step S32).

  On the other hand, when the image processing unit 11 determines that the unnecessary object X is a stationary body (or an object that moves at a speed equal to or lower than the threshold), the shooting timing presentation unit 16 sets the shooting position and timing to the photographer. An instruction message is displayed on the display unit 13. If the unwanted object X is a stationary object, the region shielded by the unwanted object X cannot be photographed unless it moves from the photographing position where the subject image 1 was photographed. Therefore, the photographing timing presentation unit 16 determines that it is a stationary object. In response to the result, the photographer is instructed to move to a position where the area shielded by the unnecessary object X can be photographed. The photographer changes the shooting position according to the instruction displayed on the display unit 13 to shoot the subject image 2, and the image processing unit 11 stores the shot subject image 2 in the image recording unit 12 (step S27).

  As shown in FIG. 5, the shooting position 205 is the first shooting position, the shooting position 410 is the moved shooting position, and the shooting range when shooting from the shooting position 410 is a range indicated by alternate long and short dash lines 411a and 411b. In addition, the line-of-sight range corresponding to the shielding area 209 viewed from the imaging position 410 is a range surrounded by solid lines 412a and 412b. Since it is necessary to interpolate the area shielded by the unnecessary object X, the imaging timing presentation unit 16 instructs to move leftward or rightward to a position where the entire shielded area 209 can be seen. FIG. 5 shows an example in which the photographer moves leftward and moves to a position 410 where the area 209 is not completely shielded by the unnecessary object X.

  The image determination unit 15 determines whether or not the subject image 1 can be interpolated or replaced by the captured subject image 2 by determining whether or not the subject image 1 and the subject image 2 can be combined (step S28). ). This determination is performed based on whether or not the region 209 that is the interpolation target of the subject image 1 is completely within the shooting range in the subject image 2, for example, extracting feature points of the subject image 1 and the subject image 2, respectively. This is performed depending on whether a corresponding feature point is obtained. If the result of this determination is that composition is possible, the image processing unit 11 performs composition of the obtained subject image 2 and subject image 1 (step S29).

  The composition processing performs geometric transformation such as projective transformation or affine transformation on the subject image 2 based on the coordinates of the corresponding points associated with the subject image 1 and the subject image 2, and cuts out the region portion of the removal target object. Then, it is performed by interpolating or replacing the subject image 1. That is, since the subject image 2 is taken from an oblique direction compared to the photographing direction of the subject image 1, correction is performed so as to match the image from the photographing direction of the subject image 1 by a known projective transformation process or the like. . Further, the brightness and the color tone are corrected so as to match the subject image 1, and the subject image 1 is synthesized. Such a composition process does not need to be performed on the entire image, and the processing amount can be reduced by performing it only on a necessary area. In addition, it is necessary to secure an area for matching. Then, the image processing unit 11 stores the synthesized image in the image recording unit 12 (step S32).

  On the other hand, if the subject image 2 alone cannot interpolate the area 209 that is shielded by the unnecessary object X and the image composition cannot be performed, the image determination unit 15 notifies the display unit 13 that the composition cannot be performed, and further the shooting position. A message is displayed instructing you to shoot with different images. At this time, it is notified which direction to move further from the positional relationship between the unwanted object X and the region 209. When giving a specific instruction regarding the shooting position of the subject image 2, for example, the imaging device is configured with a multi-lens camera, and the positional relationship and size between the building 202 and the unnecessary object X are determined by the parallax between the camera images. By obtaining, it may be specifically shown how much distance should be moved and photographed. Although distance estimation using a multi-lens camera is presented here, other distance measurement methods such as distance estimation using a laser may be used without being limited to this method.

  Next, when the photographer moves according to the presented direction and distance and performs photographing, the image processing unit 11 stores the image data output from the imaging unit 10 in the image recording unit 12 (step S30). For example, when the shooting distance is some distance away from the building 202 as a subject, or when shooting a distant object such as a landscape, parallax with the subject image 1 does not occur so much, so that the composition by the subject image 2 is performed. It can be done relatively easily. Further, even when the shielding area 209 is planar, the processing is easy. However, if the region 209 has irregularities, occlusion occurs, and there are portions that cannot be partially combined. For this reason, it is necessary to interpolate that portion using captured images from different directions.

  As shown in FIG. 6, in the image taken from the photographing position 410, the entire shielding area 209 can be visually recognized, but if the shielding area 209 is uneven, interpolation cannot be performed completely. It is necessary to photograph a new subject image (referred to as subject image 3) different from the image 2. The shooting position 513 indicates a shooting position where the subject image 3 should be shot. A region surrounded by broken lines 514a and 514b indicates a shooting range when shooting from the shooting position 513, and a region surrounded by solid lines 515a and 515b indicates a range where the region 209 can be visually recognized. The shooting position 513 of the subject image 3 is preferably a position opposite to the shooting position 410 of the subject image 2 with respect to the line connecting the shooting position 205 of the subject image 1 and the unnecessary object X. This is because, when it is on the same side as the photographing position 410, there is a high possibility that the part that becomes a blind spot when viewed from the photographing position 410 does not change much. Therefore, the blind spot can be visually recognized by photographing from the opposite direction. In this way, the subject image 3 in which the region 209 can be visually recognized and photographed at a position different from the subject image 2 is stored in the image recording unit 12 and used for the composition process.

  Next, the image processing unit 11 performs composition processing using the subject image 1, the subject image 2, and the subject image 3 stored in the image recording unit 12 (step S31). The subject image 2 and the subject image 3 are each subjected to projective transformation or the like to create an image from the direction of the subject image 1 and are used to interpolate or replace the subject image 1 and the removal target region. At this time, the brightness and color tone of the subject image 1 are also adjusted. In addition, a free viewpoint image viewed from the direction of the subject image 1 may be created from the subject image 2 and the subject image 3 during the synthesis process, and the corresponding portion may be replaced or interpolated. Then, the image processing unit 11 stores the synthesized image in the image recording unit 12 (step S32).

  In the description of the second embodiment, the example in which the three subject images 1 to 3 are combined has been described. However, the number of subject images to be combined is not limited to three, and a plurality of subject images may be added as necessary. Images may be synthesized using these images. Moreover, although the case where there is one unnecessary object X to be removed has been described, the same applies to the case where there are a plurality of unnecessary objects X by performing the same process for each shielded area, when there are a plurality of unnecessary objects X Is possible. Even when unnecessary objects are mixed with a moving object or a stationary object, an image from which an unnecessary object is removed can be obtained by appropriately using the photographing methods.

  In some cases, a person or the like is photographed against the background of a building or the like, but the person only needs to photograph when the first subject image 1 is photographed. This is because the entire image is not processed, but is divided and processed in the area up to the periphery of the removal target, and the subject image 1 is used as a reference except for the processing target area. Therefore, it is not necessary to wait at the same position during shooting for composition.

  As described above, in order to generate an image in which an unnecessary subject to be specified is removed from the captured image, an unnecessary object to be removed is specified from the first captured subject image, and the specified unnecessary object is not captured. A new subject image was shot, and the unnecessary object specified by removing the target area on the first shot subject image by interpolating or substituting with the partial image of the newly shot subject image was removed. Since the image is obtained, it is possible to obtain an image in which an object unnecessary for the photographer is removed regardless of whether it is a stationary body or a moving body when photographing.

  In recent years, since the image data has been digitized, it has become possible to create a favorite photo by importing the captured image data into a computer and compositing on the computer, so it is not necessary for the first image taken When an object was captured, it was possible to create an image with the unnecessary object removed by taking a new image without the unnecessary object and compositing it on a personal computer at home. However, whether or not a newly taken image can be used to remove unnecessary objects is not known unless it is synthesized on a personal computer, and it is unnecessary if an appropriate image cannot be taken. There is a case where the synthesis for removing the object cannot be performed.

  When the imaging apparatus according to the present invention has been unable to obtain a subject image that can be combined by determining whether or not composition can be performed on the spot when a new subject image is captured Since the shooting instruction is displayed again, it is possible to prevent failure of image composition for removing unnecessary objects, and it is possible to reliably obtain an image from which unnecessary objects are removed.

  A program for realizing the functions of the image processing unit 11, the image determination unit 15, and the photographing timing presentation unit 16 in FIG. 1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is stored in the computer. The image composition processing may be performed by reading the system and executing it. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  By synthesizing digital image data, it can be applied to a purpose of newly generating an image from which unnecessary objects are removed.

  DESCRIPTION OF SYMBOLS 10 ... Imaging part, 11 ... Image processing part, 12 ... Image recording part, 13 ... Display part, 14 ... Instruction part, 15 ... Image determination part, 16 ... Photographing Timing presentation section

Claims (7)

An imaging unit for photographing a subject;
A first subject image acquisition unit for obtaining first subject image data captured by the imaging unit;
An area designating unit for displaying first object image data and designating a removal target area on the first object image data;
A second subject image acquisition unit that obtains second subject image data captured by the imaging unit;
Whether partial image data corresponding to the removal target area can be extracted from the second subject image data, and the partial image data can be combined with the removal target area on the first subject image data. An image determination unit for determining whether or not
An image compositing unit that composites the partial image data with the removal target region on the first subject image data when the determination result of the image determination unit is compositable ;
An instruction display unit that displays a shooting instruction for shooting the second subject image again when the determination result of the image determination unit cannot be combined;
The imaging apparatus , wherein the instruction display unit displays a shooting instruction for shooting the second subject image at a shooting position different from the shooting position of the first subject image . The instruction display unit, according to claim 1, wherein the second object image when a plurality of photography, each of the imaging position of the second object image and displaying the photographing instruction such that the different positions The imaging device described in 1. An imaging unit for photographing a subject;
A first subject image acquisition unit for obtaining first subject image data captured by the imaging unit;
An area designating unit for displaying first object image data and designating a removal target area on the first object image data;
A second subject image acquisition unit that obtains second subject image data captured by the imaging unit;
Whether partial image data corresponding to the removal target area can be extracted from the second subject image data, and the partial image data can be combined with the removal target area on the first subject image data. An image determination unit for determining whether or not
An image compositing unit that composites the partial image data with the removal target region on the first subject image data when the determination result of the image determination unit is compositable;
An instruction display unit that displays a shooting instruction for shooting the second subject image again when the determination result of the image determination unit cannot be combined;
And a moving object determining section for determining a subject in the specified the region whether a moving object by the area specifying unit,
The instruction display unit displays a shooting instruction for shooting the second subject image from a shooting position where the first subject image was shot when the subject in the area is a moving body,
Wherein when the object in the region is not a moving body is moved to a position where the subject in the field can not reflected, the second feature and to that imaging device to display a shooting instruction for shooting a subject image . An imaging unit for photographing a subject;
A first subject image acquisition unit for obtaining first subject image data captured by the imaging unit;
An area designating unit for displaying first object image data and designating a removal target area on the first object image data;
A second subject image acquisition unit that obtains second subject image data captured by the imaging unit;
Whether partial image data corresponding to the removal target area can be extracted from the second subject image data, and the partial image data can be combined with the removal target area on the first subject image data. An image determination unit for determining whether or not
An image compositing unit that composites the partial image data with the removal target region on the first subject image data when the determination result of the image determination unit is compositable;
An instruction display unit that displays a shooting instruction for shooting the second subject image again when the determination result of the image determination unit cannot be combined;
And a depth determination unit capable of determining the positional relationship between the photographing area, the instruction display unit, you and displaying the direction and distance of the second imaging position in response to the removal target area imaging device. The imaging unit includes an imaging unit of the 2 eyes can be obtained two object picture image data viewed from different viewpoints simultaneously,
The depth determination unit imaging apparatus according to claim 4, characterized in that to obtain the depth from the parallax of the two object picture image data. The image composition unit uses the partial image data extracted from each of a plurality of second subject image data acquired by the second subject image acquisition unit, and uses the partial image data extracted from the first subject image data. the imaging apparatus according to claim 1, characterized in that performing the image synthesis for region 4. The image synthesizing unit, the image pickup according to any one of claims 1 to 4, characterized in that the image synthesizing the partial image data by deforming to the removal target region on the first object image data apparatus.

Images