JP2012060299A - Image pickup device, control method thereof, program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain the right amount of images required when a dynamic body image is removed from continuously shot images.SOLUTION: Continuous shooting means (90) controls an image pickup part 22 to continuously perform image pickup at set time intervals and set times (the number of images) when a shutter button is depressed in a state where a continuous shooting function is set. Identification means (91) sequentially compares two continuous images picked-up by the continuous shooting means (90), identifies a coincident part and a non-coincident part, and generates a differential image after the removal of an area of the non-coincident part. The identification means (91) successively generates a synthetic image by synthesizing an image of the same area in a sequential differential image with the area of the non-coincident part of the differential image. A display part (28) displays the successively generated synthetic images. A system control part (50) terminates continuous shooting in accordance with an image pickup termination indication.

Description

  The present invention relates to an imaging apparatus capable of obtaining a necessary image without excess or deficiency when a moving body image is removed from continuously captured images, a control method and program thereof, and a recording medium.

  There is known a method for identifying a stationary part (usually a background) or a moving part by comparing successively captured images. There is known a method for identifying a background or a moving object by comparing an image captured with only a background and an image captured with a moving object added to the background.

  Also, by embedding a background image that matches this area in the area from which the moving object that becomes an obstacle is removed, an image of only the background can be generated. The background image can be obtained from other images obtained by continuous shooting.

  In Patent Document 1, when there is a moving body that crosses the front of a stationary body, an image composed of only a stationary portion is obtained by performing continuous shooting at the same angle of view by a single shutter operation by the user and erasing the moving body. The technology to generate is described.

  Patent Document 2 describes a technique for evaluating a static part / moving object part using a partial correlation between a plurality of series of images and generating a single still image. That is, each of a series of images is divided into a plurality of divided regions by the same dividing line on the screen, and the correlation between images is evaluated for each divided region. From the evaluation result, an image of a divided region having a high degree of correlation is extracted for each divided region, and a single still image is created by combining the extracted images.

JP 2000-13681 A JP 2002-288664 A

  However, in general, the user cannot recognize whether or not a background image that complements the area from which the recognized moving object has been removed has already been obtained, and therefore, the user performs shooting more than necessary. This leads to wasted time and memory.

  In the technique described in Patent Document 1, there is no guarantee that a necessary and sufficient image can be taken by continuous shooting by a user's one-shutter operation. That is, there is a problem that the moving object cannot be completely erased, or time and memory amount are wasted.

  In the technique described in Patent Document 2, there is no guarantee that a plurality of still images are necessary and sufficient for obtaining a composite still image. Similar to the technique described in Patent Document 1, there is a problem that a moving object cannot be completely erased, or time and a memory amount are wasted.

  As described above, in the related art, the user cannot know whether a continuous image or a divided image necessary for obtaining a desired still image has already been obtained.

  An object of the present invention is to provide an imaging apparatus, a control method and program thereof, and a recording medium that can obtain a necessary image without excess or deficiency when a moving body image is removed from continuously captured images.

  In order to solve the above-described problem, an imaging apparatus according to the present invention is imaged by an imaging unit, a continuous shooting unit that starts continuous shooting by the imaging unit upon receiving a continuous shooting start instruction, and the continuous shooting unit. The photographer identifies the matching part and the non-matching part identified by the identification means during sequential shooting by the continuous shooting means by identifying means for sequentially comparing a plurality of images and identifying a matching part and a non-matching part. And a display unit for displaying the image so as to be able to be displayed, and a control unit for controlling the continuous shooting by the continuous shooting unit to be ended when a continuous shooting end instruction is received during the display by the display unit. .

  The method for controlling an imaging apparatus according to the present invention sequentially compares a plurality of images captured by the imaging unit with a continuous shooting step that starts continuous shooting by the imaging unit when a continuous shooting start instruction is received, An identification step for identifying a non-matching part, a display step for displaying the matching part and the non-matching part identified in the identification step so that a photographer can identify them during continuous shooting, and a continuous shooting end instruction And a control step for controlling to end the continuous shooting by the continuous shooting step.

  The control program for an imaging apparatus according to the present invention is a control program for an imaging apparatus, wherein the imaging apparatus starts a continuous shooting by an imaging unit when receiving a continuous shooting start instruction, and the imaging unit A photographer can identify the matching part and the mismatching part identified by the identification function during sequential shooting by sequentially comparing a plurality of captured images and identifying a matching part and a mismatching part. A display function to be displayed on the display unit, and a control function for controlling to end the continuous shooting by the continuous shooting function when a continuous shooting end instruction is received.

  The above-described control program is stored in the recording medium according to the present invention.

  According to the present invention, it is possible to visually confirm whether or not the area from which the moving object has been removed is filled with the background on the composite image that is sequentially generated during continuous shooting. The continuous shooting can be terminated at an appropriate timing.

1 is an external view of an embodiment of an imaging apparatus according to the present invention. It is a schematic block diagram of a present Example. It is an example of the image obtained by carrying out continuous imaging | photography in a present Example. It is an example of the identification of the matched part and the mismatched part by the identification means. It is an example of a change of a synthetic picture generated one by one. It is an operation | movement flowchart of a present Example. It is a flowchart of another control operation regarding the end of continuous shooting. It is a figure which shows the example of a timing of the continuous shooting end instruction | indication by the user in the middle of continuous shooting.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is an external view of a digital camera as an embodiment of an imaging apparatus according to the present invention. The display unit 28 is a display unit that displays images and various types of information. The shutter button 61 is an operation unit for issuing a shooting instruction. The mode switch 60 is an operation unit for switching various modes. The connector 112 is a connector that connects the connection cable 111 to the digital camera 100. The operation unit 70 is an operation unit including operation members such as various switches, buttons, and a touch panel for receiving various operations by a user or a photographer. The controller wheel 73 is a rotatable operation member included in the operation unit 70. Reference numeral 72 denotes a power switch that switches between power on and power off. The recording medium 200 is a recording medium such as a memory card or a hard disk. The recording medium slot 201 is a slot for storing the recording medium 200. The recording medium 200 stored in the recording medium slot 201 can communicate with the digital camera 100. A lid 202 is a lid of the recording medium slot 201.

  FIG. 2 is a schematic configuration block diagram of the digital camera 100. Reference numeral 103 denotes a photographing lens including a focus lens. Reference numeral 101 denotes a shutter having an aperture function. An image pickup unit 22 includes a CCD or CMOS image pickup device that converts an optical image into an electric signal. Reference numeral 23 denotes an A / D converter that converts an analog signal output from the imaging unit 22 into a digital signal. The barrier 102 protects the imaging system including the imaging lens 103, the shutter 101, and the imaging unit 22 from dirt and damage by covering the imaging unit including the imaging lens 103 of the digital camera 100.

  An image processing unit 24 performs resize processing such as predetermined pixel interpolation and reduction and color conversion processing on the data from the A / D converter 23 or the data from the memory control unit 15. The image processing unit 24 performs predetermined calculation processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation result. As a result, TTL (through the lens) AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-emission) processing are performed. The image processing unit 24 further performs predetermined calculation processing using the captured image data, and also performs TTL AWB (auto white balance) processing based on the obtained calculation result.

  Output data from the A / D converter 23 is directly written into the memory 32 via the image processing unit 24 and the memory control unit 15 or via the memory control unit 15. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23 and image data to be displayed on the display unit 28. The memory 32 has a storage capacity sufficient to store a predetermined number of still images, a moving image and sound for a predetermined time.

  The memory 32 also serves as an image display memory (video memory). A D / A converter 13 converts image display data stored in the memory 32 into an analog signal and supplies the analog signal to the display unit 28. The display image data written in the memory 32 is displayed on the display unit 28 via the D / A converter 13. The display unit 28 displays an image corresponding to the analog signal from the D / A converter 13 on a display such as an LCD.

  The nonvolatile memory 56 is an electrically erasable / recordable memory, and for example, an EEPROM or the like is used. The nonvolatile memory 56 stores constants for operation of the system control unit 50, a control program, and the like. Here, the program includes a control program for executing various flowcharts to be described later.

  Reference numeral 50 denotes a system control unit that controls the entire digital camera 100. The system control unit 50 implements each process described later by executing a control program recorded in the nonvolatile memory 56. A system memory 52 is a RAM. In the system memory 52, constants and variables for operation of the system control unit 50, a control program read from the nonvolatile memory 56, and the like are expanded. The system control unit 50 also performs display control by controlling the memory 32, the D / A converter 13, the display unit 28, and the like.

  The mode change switch 60, the first shutter switch 62, the second shutter switch 64, and the operation unit 70 are operation means for inputting various operation instructions to the system control unit 50.

  The mode switch 60 switches the operation mode of the system control unit 50 to any one of a still image recording mode, a moving image recording mode, a reproduction mode, and the like. The digital camera 100 has a moving image removal mode. When the user selects the moving body image removal mode with the mode changeover switch 60, the system control unit 50 calls a shooting function necessary for removing the moving body image. The first shutter switch 62 is turned on when a shutter button 61 provided in the digital camera 100 is being operated, so-called half-press (shooting preparation instruction), and generates a first shutter switch signal SW1. In response to the first shutter switch signal SW1, the system control unit 50 starts operations such as AF processing, AE processing, AWB processing, and EF processing.

  The second shutter switch 64 is turned on when the operation of the shutter button 61 is completed, so-called full press (shooting instruction), and generates a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of imaging processing operations from reading a signal from the imaging unit 22 to writing image data to the recording medium 200.

  Each operation member of the operation unit 70 is assigned an appropriate function for each scene by selecting and operating various function icons displayed on the display unit 28, and acts as various function buttons. Examples of the function buttons include an end button, a return button, an image advance button, a jump button, a narrowing button, and an attribute change button. For example, when a menu button is pressed, various settable menu screens are displayed on the display unit 28. The user can make various settings intuitively using the menu screen displayed on the display unit 28, the four-way button, and the SET button.

  The controller wheel 73 is a rotatable operation member included in the operation unit 70, and is used when a selection item is instructed together with a direction button. When the controller wheel 73 is rotated, an electrical pulse signal is generated according to the operation amount. The system control unit 50 controls each unit of the digital camera 100 based on the pulse signal. The system control unit 50 can determine the rotation angle and the number of rotations of the controller wheel 73 based on the pulse signal.

  The controller wheel 73 may be any operation member that can detect a rotation operation. For example, the dial wheel may be a dial operation member that generates a pulse signal by rotating the controller wheel 73 itself in response to a user's rotation operation. Further, an operation member made of a touch sensor may be used to detect a rotation operation of a user's finger on the controller wheel 73 without rotating the controller wheel 73 itself. A so-called touch wheel corresponds to this.

  A power control unit 80 includes a battery detection circuit, a DC-DC converter, a switch circuit that switches a block to be energized, and the like, and detects whether a battery is installed, the type of battery, and the remaining battery level. Further, the power control unit 80 controls the DC-DC converter based on the detection result and an instruction from the system control unit 50, and supplies a necessary voltage to each unit including the recording medium 200 for a necessary period.

  A power supply unit 30 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery NiMH battery or a Li battery, or an AC adapter. Reference numeral 18 denotes an interface with the recording medium 200. The recording medium 200 is a recording medium such as a memory card, and includes a semiconductor memory or a magnetic disk.

  The digital camera 100 includes continuous shooting means 90 and identification means 91. The continuous shooting means 90 is a means for controlling the continuous shooting at the set time interval and the number of times (number of sheets) when the shutter button is pressed while the continuous shooting function is set. The captured image data is temporarily stored in the buffer area of the memory 32. The image data temporarily saved in the buffer area is used for previewing or various processing, and is stored in the storage means after the end.

  The identification unit 91 is a unit that sequentially compares two consecutive images captured by the continuous shooting unit 90 and performs a process of identifying a matching portion and a mismatching portion. Specifically, the system control unit 50 sends the image data stored in the buffer area of the memory 32 to the image processing unit 24. Under the control of the system control unit 50, the image processing unit 24 executes a process of sequentially comparing the matched portion and the unmatched portion for two consecutive images. As a method for discriminating a matching portion and a mismatching portion of two consecutive images, a method well known to those skilled in the information processing technical field can be used. For example, a difference between frames / fields is calculated, an image part having a difference greater than a certain value is identified as a mismatched part, and a part other than that is identified as a matched part.

  Note that the continuous shooting unit 90 and the identification unit 91 do not have to be hardware independent configurations, and the same function may be realized by the system control unit 50 controlling each unit. In that case, the continuous shooting function corresponding to the continuous shooting means 90 and the identification function corresponding to the identification means 91 are realized by a control program operating on the system control unit 50. In addition, a display function for displaying the above-described various images on the display unit 28 and a control function for realizing the above-described control are also realized by a control program that operates on the system control unit 50.

  FIG. 3 shows an example of images obtained by continuous shooting with the digital camera 100. 3A, 3B, 3C, and 3D respectively show the image 1, the image 2, the image 3, and the image 4 in the order of photographing. In the image example shown in FIG. 3, the camera is fixed to a tripod, and images are taken continuously at intervals of 30 seconds in the moving object image removal mode. The shooting interval can be arbitrarily set (for example, 300 milliseconds interval, 5 seconds interval, 1 minute interval, etc.).

  Images 1 to 4 shown in FIG. 3 are images that attempt to copy the Great Buddha in a sightseeing spot. In such a sightseeing spot, since tourists visit constantly, it is difficult to take a picture only with the object (the Great Buddha). That is, since photographing is performed at a tourist spot, each photographed image shown in FIG. 3 reflects the appearance of a tourist who is viewing, photographing or moving the object (the Great Buddha). In this embodiment, a plurality of times of shooting are continuously performed, a moving body image is recognized from the continuous shooting images, and the background image of the corresponding area of the other image is combined with the area from which the moving body image has been removed. Generate an object-only image.

  FIG. 4 shows how the identifying unit 91 identifies a matching part and a mismatching part in the photographed image example shown in FIG. The identification unit 91 first compares the data of the image 1 and the image 2 stored in the buffer area of the memory 32, and identifies the matching part and the mismatching part. Various methods can be adopted for identification. For example, identification is performed by comparing the values of both image data for each pixel. A pixel area where there is no change between the image 1 and the image 2, that is, there is no difference is defined as a matching portion. In addition, a portion having a change in both images, that is, a pixel region having a difference is identified as a mismatched portion.

  As a result of the difference calculation with respect to the image 2, the identification unit 91 erases the area identified as the coincident portion from the image 1 to obtain the difference image 1 shown in FIG. In the difference image 1, the area erased from the image 1 is displayed in a state where it can be identified by oblique lines. Similarly, based on the difference calculation between the image 2 and the image 3, the difference image 2 shown in FIG. 4B is obtained. Based on the difference calculation between the image 3 and the image 4, a difference image 3 shown in FIG. 4C is obtained. For the subsequent two images captured continuously, the discriminating means 91 calculates the difference between the two images captured in succession, and identifies the area identified as the inconsistent portion from the previously captured image. to erase.

  FIGS. 5A and 5B show examples of changes in the composite image sequentially generated by the system control unit 50.

  As described above, the calculation of the difference between two temporally continuous images may be performed for each block by dividing the image into m × n blocks instead of for each pixel. Further, in order to reduce the processing required for the difference calculation, for example, the thinning processing may be performed by a method of 3 pixels every 4 pixels.

  Each time a difference image is generated, the system control unit 50 combines the difference images. By synthesizing the difference image 1 and the difference image 2, a synthesized image 1 shown in FIG. 5A is obtained. The composite image 1 is created by synthesizing the image of the area of the differential image 2 created later with the area of the differential image 1 created earlier that has been erased and left blank. The created composite image 1 is displayed as a through image on the display unit 28. As a result of combining the area of the difference image 2 with the area from which the difference image 1 has been erased, there are fewer blank areas on the combined image 1. However, since there is a blank area in the area of the difference image 2, the blank area still remains in the composite image 1. Since the blank area is identifiable as a diagonal line, the user can know that the moving object removal is incomplete by looking at the through image on the display unit 28.

  The system control unit 50 continues the composite image update operation of sequentially combining the created difference images with the composite image. The display unit 28 displays the updated composite image as a through image. For example, by synthesizing the difference image 3 with the synthesized image 1, a synthesized image 2 shown in FIG. 5B is obtained. By synthesizing the area of the difference image 3 with the blank area of the composite image 1, a composite image 2 in which the blank area of the composite image 1 is completely filled is obtained. The user can know that the synthesized image from which the moving object has been removed is obtained by looking at the synthesized image 2 displayed on the display unit 28. At this stage, the user stops subsequent continuous shooting by pressing a button for canceling continuous shooting.

  The display method for identifying the blank area by the user is not limited to the above-described hatched display. For example, surrounding color components may be analyzed and displayed as complementary colors. Further, the blank area may be blinked so that the blank area can be identified more clearly. Furthermore, the display of the area other than the blank area may be changed, such as changing the saturation or contrast of the area other than the blank area.

  The operation of this embodiment for realizing the above function will be described. Processing performed by the digital camera will be described. FIG. 6 shows a flowchart of characteristic operations of the present embodiment. Each process in the flowchart shown in FIG. 6 is realized by the system control unit 50 developing and executing a program stored in the nonvolatile memory 56 in the system memory 52.

  When the user sets the mode switch to the moving image removal mode (S601), the system control unit 50 automatically sets each unit so that the continuous shooting function operates, and waits for a user's shooting start instruction (S602). When there is a shooting start instruction as a continuous shooting start instruction by the user (S602), the system control unit 50 starts continuous shooting (S603). The imaging start instruction is input to the system control unit 50 when the shutter button 61 is fully pressed, for example.

  As described above, the identification unit 91 sequentially compares two images captured by continuous shooting, identifies a matched portion and a mismatched portion (S604), and erases an area identified as a mismatched portion from each image. The difference image is generated (S605). Subsequently, the identification unit 91 combines the difference images to generate a combined image (S606). The system control unit 50 displays the generated composite image on the display unit 28 as a through image (S607).

  When the system control unit 50 detects a shooting end instruction as a continuous shooting end instruction by the user (S608), the system control unit 50 performs a process of ending the continuous shooting and recording the obtained composite image in the storage unit (S609). The shooting end instruction is input to the system control unit 50 by, for example, releasing the full press of the shutter button 61 or pressing the shutter button 61 again. In the former case, continuous shooting is performed while the shutter button 61 is fully pressed. In the latter case, continuous shooting continues even if the shutter button 61 is fully pressed and released once, and then when the shutter button 61 is fully pressed again, the continuous shooting ends.

  According to the present embodiment, the user can visually confirm the moving object removal image created from the comparison of the continuously captured images as a through image, so it is easy to determine how far the moving object has been deleted. Can be recognized. As a result, it is possible to perform photographing without excess or deficiency for moving object removal, and it is possible to reduce useless photographing time and recording area.

  In the first embodiment, the continuous shooting is ended by the user's operation. However, the continuous shooting may be automatically ended by detecting that the moving object is removed on the composite image.

  FIG. 7 shows a flowchart of a control operation incorporating automatic end of continuous shooting. Each process in the flowchart shown in FIG. 7 is realized by the system control unit 50 expanding and executing a program stored in the nonvolatile memory 56 in the system memory 52.

  The system control unit 50 determines whether or not the blank area is filled with the latest composite image (S701). If it is determined that the blank area is filled, the system control unit 50 controls to end the continuous shooting (S702). If the blank area is not filled, the system control unit 50 determines whether or not there is an instruction to stop continuous shooting by the user (S703). When it is determined that the user has instructed to stop continuous shooting (S703), the system control unit 50 controls to end the continuous shooting (S702). If the blank area is not filled (S701) and there is no instruction to end continuous shooting from the user (S703), it is determined whether or not a continuous shooting end condition is reached (S704). The continuous shooting end condition is, for example, a preset shooting time or the number of times of shooting, and can also be set by the capacity of the buffer area of the memory. When it is determined that the continuous shooting end condition has been reached (S704), the system control unit 50 controls to end the continuous shooting (S702).

  In this embodiment, when a blank area is filled in the composite image, continuous shooting is automatically terminated, so that waste of memory and time can be avoided without performing unnecessary shooting. Even when the blank area is not filled, the continuous shooting is ended based on the continuous shooting end operation by the user, so that the shooting can be stopped when the user determines that a desired image is obtained. For example, FIG. 8 shows an example of a composite image displayed on the display unit 28. There is a portion where the moving object region 801 is erased. However, if the user determines that a desired image can be obtained by trimming the area 802 excluding the area 801, the user can end the continuous shooting by instructing the end of shooting.

  Depending on the shooting situation, the moving object may not be absent for a long time, and in this case, continuous shooting is continued for a long time. On the other hand, in this embodiment, since continuous shooting is terminated when a certain shooting time or number of shootings is reached, it is possible to prevent continuous shooting from being continued unnecessarily for a long time.

  The control of the system control unit 50 may be performed by one piece of hardware, or the entire apparatus may be controlled by a plurality of pieces of hardware sharing the processing.

  Although the present invention has been described in detail based on the preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms without departing from the gist of the present invention are also included in the present invention. included. Furthermore, each embodiment mentioned above shows only one embodiment of this invention, and it is also possible to combine each embodiment suitably.

  Further, in the above-described embodiment, the case where the present invention is applied to a digital camera has been described as an example, but this is not limited to this example. That is, the present invention can be applied to any apparatus having an imaging means and a display unit, such as a digital single-lens reflex camera, a digital video camera, a mobile phone terminal having an imaging function, a PDA, a music player, a game machine, a personal computer, or an electronic book reader. Is possible.

  The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program code. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

Claims (5)

Imaging means;
Continuous shooting means for starting continuous shooting with the imaging means when receiving a continuous shooting start instruction;
An identification unit that sequentially compares a plurality of images captured by the continuous shooting unit and identifies a matching portion and a mismatching portion;
Display means for displaying a photographer so that a photographer can identify the matching part and the mismatching part identified by the identification means during continuous shooting by the continuous photographing means;
An image pickup apparatus comprising: a control unit configured to control so that continuous shooting by the continuous shooting unit is ended when a continuous shooting end instruction is received during display by the display unit.   When the control means determines that the non-matching portion has disappeared, receives a continuous shooting end instruction from the user, or determines that the continuous shooting end condition has been reached, the continuous shooting means The image pickup apparatus according to claim 1, wherein the image pickup apparatus is controlled to end continuous shooting. A continuous shooting step for starting continuous shooting by the imaging means upon receiving a continuous shooting start instruction;
An identification step of sequentially comparing a plurality of images captured by the imaging means and identifying a matching portion and a mismatching portion;
A display step for displaying the matched portion and the unmatched portion identified in the identifying step so that a photographer can identify them during continuous shooting;
A control method for an imaging apparatus, comprising: a control step of controlling to end the continuous shooting by the continuous shooting step when receiving a continuous shooting end instruction. An imaging device control program comprising:
A continuous shooting function that starts continuous shooting with an imaging means upon receipt of a continuous shooting start instruction;
An identification function for sequentially comparing a plurality of images captured by the imaging unit and identifying a matching portion and a mismatching portion;
A display function for displaying on the display unit so that a photographer can identify the matching part and the non-matching part identified by the identification function during continuous shooting;
A control program for an imaging apparatus, which realizes a control function for controlling to end continuous shooting by the continuous shooting function when a continuous shooting end instruction is received. A recording medium for storing a control program for an imaging apparatus, wherein the control program is stored in the imaging apparatus.
A continuous shooting function that starts continuous shooting with an imaging means upon receipt of a continuous shooting start instruction;
An identification function for sequentially comparing a plurality of images captured by the imaging unit and identifying a matching portion and a mismatching portion;
A display function for displaying on the display unit so that a photographer can identify the matching part and the non-matching part identified by the identification function during continuous shooting;
A recording medium, which realizes a control function for controlling to end continuous shooting by the continuous shooting function when a continuous shooting end instruction is received.

Images