JP2013012993A - Image processing device, imaging device, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To estimate a main subject.SOLUTION: An image processing device comprises: an image acquisition unit that acquires plural images captured in time series; a subject extraction unit that extracts plural mutually different subjects included in the plural images; an unnecessary subject estimation unit that estimates which of the plural subjects is an unnecessary subject on the basis of at least any image of the plural images; and a main subject estimation unit that estimates a main subject on the basis of subject candidates obtained by excluding the unnecessary subject from the plural subjects.

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

There is a digital camera that shoots under a shooting condition suitable for a main subject by excluding a subject that does not change included in a plurality of images acquired in time series (see Patent Document 1).
[Prior art documents]
[Patent Literature]
[Patent Document 1] Japanese Unexamined Patent Application Publication No. 2009-089174

  However, when the imaging interval is shortened, changes in the subject between the images are reduced. This makes it difficult to distinguish the main subject from the unnecessary subject.

  In order to solve the above problems, as a first aspect of the present invention, an image acquisition unit that acquires a plurality of images captured in time series, and a subject extraction unit that extracts a plurality of different subjects included in the plurality of images, An unnecessary subject estimation unit that estimates which one of a plurality of subjects is an unnecessary subject based on at least one of the plurality of images, and a subject candidate obtained by removing the unnecessary subject from the plurality of subjects. There is provided an image processing apparatus including a main subject estimation unit that estimates a main subject.

  According to a second aspect of the present invention, there is provided an imaging apparatus comprising the above image processing apparatus, a release button operated by a user, and an imaging unit that captures a plurality of images in response to a single operation of the release button. Is done.

  Furthermore, as a third aspect of the present invention, an image acquisition procedure for acquiring a plurality of images captured in time series, a subject extraction procedure for extracting a plurality of different subjects included in the plurality of images, Based on at least one of the images, an unnecessary subject estimation procedure for estimating which one of a plurality of subjects is an unnecessary subject, and a main subject is estimated based on subject candidates obtained by removing unnecessary subjects from the plurality of subjects. And an image processing program for causing a computer to execute a main subject estimation procedure.

  The above summary of the present invention does not enumerate all necessary features of the present invention. A sub-combination of these feature groups can also be an invention.

1 is a perspective view of a digital camera 100. FIG. 1 is a perspective view of a digital camera 100. FIG. 2 is a block diagram showing an internal circuit 200 of the digital camera 100. FIG. 5 is a flowchart illustrating an operation procedure of an unnecessary subject estimation unit 260. 4 is a schematic diagram illustrating a captured image group 410. FIG. FIG. 6 is a diagram schematically illustrating the operation of an unnecessary subject estimation unit 260. FIG. 6 is a diagram schematically illustrating the operation of an unnecessary subject estimation unit 260. FIG. 6 is a diagram schematically illustrating the operation of an unnecessary subject estimation unit 260. FIG. 6 is a diagram schematically illustrating the operation of an unnecessary subject estimation unit 260. 5 is a flowchart showing an operation procedure of a main subject estimation unit 270. FIG. 11 is a diagram schematically showing the operation of a main subject estimation unit 270. FIG. 11 is a diagram schematically showing the operation of a main subject estimation unit 270. FIG. 11 is a diagram schematically showing the operation of a main subject estimation unit 270. FIG. 11 is a diagram schematically showing the operation of a main subject estimation unit 270. 5 is a flowchart showing an operation procedure of an image selection unit 280. It is a figure which shows typically operation | movement of the image selection part 280. FIG. It is a figure which shows typically operation | movement of the image selection part 280. FIG. It is a figure which shows typically operation | movement of the image selection part 280. FIG. It is a figure which shows typically operation | movement of the image selection part 280. FIG. It is a figure which shows typically operation | movement of the image selection part 280. FIG. It is a figure which shows typically operation | movement of the image selection part 280. FIG. It is a schematic diagram of a personal computer that executes an image processing program.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 is a perspective view of a digital camera 100, which is a type of imaging device, as viewed obliquely from the front. The digital camera 100 includes a thin and thin cubic housing 110, a lens barrel 120 and a light emitting window 130 disposed on the front surface of the housing 110, a power switch 142 and a release button 144 disposed on the upper surface of the housing 110. A part of the operation unit 140 including the zoom lever 146 is provided.

  The lens barrel 120 holds a photographing lens 122 that forms a subject image on an image sensor disposed inside the housing 110. Light generated by a light emitting unit (not shown) disposed inside the housing 110 illuminates the subject through the light emitting window 130.

  Each time the power switch 142 is pressed, the power of the digital camera 100 is interrupted. The zoom lever 146 changes the magnification of the photographing lens held in the lens barrel 120.

  When the release button 144 is pressed halfway by the user, the automatic focusing unit, the photometric sensor, and the like are driven, and a through image capturing operation by the image sensor is executed. Thereby, the digital camera 100 prepares for the main photographing of the subject image following the through image photographing. When the release button 144 is fully pressed, the shutter is opened and the subject image is actually captured. When the brightness of the shooting range is dark, the light is projected from the light emission window 130 toward the subject in accordance with the timing of the main shooting.

  FIG. 2 is a perspective view of the digital camera 100 as viewed obliquely from the rear. Elements that are the same as those in FIG. 1 are given the same reference numerals, and redundant descriptions are omitted.

  A part of the operation unit 140 including a cross key 141 and a back button 143 and a back display unit 150 are arranged on the back surface of the housing 110. The cross key 141 and the back button 143 are operated by the user when inputting various settings to the digital camera 100, switching the operation mode of the digital camera 100, changing the magnification of the lens barrel in real time, and the like. The

  The rear display unit 150 is formed of a liquid crystal display panel or the like, and occupies many areas on the rear surface of the housing 110. When the digital camera 100 is in the shooting mode, a subject image incident on the lens barrel 120 is continuously photoelectrically converted by the imaging device and displayed on the rear display unit 150 as a through image. The user can know the effective shooting range by observing the through image displayed on the rear display unit 150.

  The rear display unit 150 also displays the status of the digital camera 100 such as the remaining battery level and the remaining capacity of the storage medium. Further, when the digital camera 100 operates in the playback mode, the playback image is displayed on the rear display unit 150.

  FIG. 3 is a block diagram schematically showing the internal circuit 200 of the digital camera 100. Elements common to those in FIGS. 1 and 2 are denoted by the same reference numerals, and redundant description is omitted. The internal circuit 200 includes a control unit 201, an image acquisition unit 202, and a captured image processing unit 203.

  The control unit 201 includes a CPU 210, a display driving unit 220, a program memory 230, and a main memory 240. The CPU 210 comprehensively controls the operation of the digital camera 100 according to the firmware read from the program memory 230 to the main memory 240. The display driving unit 220 generates a display image in accordance with an instruction from the CPU 210 and displays the generated image on the rear display unit 150.

  The image acquisition unit 202 includes an image sensor driving unit 310, an image sensor 312, an analog / digital conversion unit 320, an image processing unit 330, an automatic focusing unit 340, and a photometric sensor 350.

  The image sensor driving unit 310 drives the image sensor 312 to photoelectrically convert the subject image formed on the image pickup surface of the image sensor 312 by the photographing lens 122 into an image signal. As the imaging element 312, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like can be used.

  The image signal output from the image sensor 312 is discretized by the analog / digital conversion unit 320 and converted into captured image data by the image processing unit 330. The image processing unit 330 adjusts white balance, sharpness, gamma, gradation correction, and the like of the image in the process of generating the captured image data, and compresses the captured image data when it is stored in the secondary storage medium 332 described later. Set the rate, etc.

  The image data generated by the image processing unit 330 is stored and stored in the secondary storage medium 332. As the secondary storage medium 332, a medium including a nonvolatile storage element such as a flash memory card is used. Note that at least a part of the secondary storage medium 332 can be detached from the digital camera 100 and replaced.

  The automatic focusing unit 340 performs an automatic focusing operation by a known phase difference method using, for example, a phase difference type focus detection element disposed on the imaging surface of the imaging element 312. It should be noted that a contrast AF method that determines that the photographing lens 122 is in focus when the contrast of the image becomes the highest in the focus area of the through image may be used. The photometric sensor 350 measures the brightness of the subject and determines the shooting conditions of the digital camera 100. The magnification driving unit 360 moves a part of the photographing lens 122 in accordance with an instruction from the CPU 210. As a result, the magnification of the photographic lens 122 changes, and the angle of view of the photographic image changes.

  The input unit 370 receives an input from the operation unit 140 and holds setting values set in the digital camera 100. The CPU 210 refers to the input unit 370 to determine the operation condition.

  In the digital camera 100 having the internal circuit 200 as described above, the image acquisition unit (imaging unit) 202 receives a plurality of image data in response to a single shooting operation (full pressing operation) in which the user presses the release button 144. It has a shooting mode to acquire. When the shooting mode is set, the CPU 210 controls the image sensor 312 to perform continuous shooting using the image sensor driving unit 310. Thereby, time-series photographed image (video) data is acquired. The time-series captured image data acquired in this way is sequentially input to a FIFO (First In First Out) memory in the image processing unit 330.

  The FIFO memory has a predetermined storage capacity, and when input data sequentially input reaches the predetermined capacity, the captured image data is output in the input order. In the shooting mode, time-series shot image data is sequentially input to the FIFO memory until a predetermined time has elapsed since the user fully pressed the release button 144, and during this time, the data output from the FIFO memory is deleted. The When a predetermined time elapses after the release button 144 is fully pressed, writing of the captured image data to the FIFO memory is prohibited.

  As a result, a plurality of frames of time-series captured image data acquired before and after the release button 144 is fully pressed are held in the FIFO memory. That is, the image acquisition unit 202 acquires a plurality of images captured in time series for a single shooting operation (full press operation), so that shooting conditions (aperture opening, shutter speed, A captured image suitable for the imaging element sensitivity and the like and the photographing timing can be selected from the captured images of a plurality of frames acquired in the photographing mode. Thereby, the success rate of photographing can be improved.

  In recent years, the performance of the digital camera 100 has improved, and it has become possible to acquire several tens of image data in response to a single release button operation by the user. In such a case, it may be a burden on the user to select image data to be stored from a large amount of photographed image data. According to the present embodiment, such a burden on the user can be reduced as described later.

  A digital camera 100 shown in FIG. 3 includes a captured image processing unit 203. The captured image processing unit 203 includes a subject extraction unit 250, an unnecessary subject estimation unit 260, a main subject estimation unit 270, and an image selection unit 280. The captured image processing unit 203 is estimated to have a higher value as an image from a plurality of captured images. Select an image.

  Here, the value as an image includes not only the high quality as image data but also the evaluation of the captured content. High quality as image data means high value depending on signal quality as digital data, such as high image resolution, wide dynamic range, no overexposure / blackout, and the like. On the other hand, the evaluation of the shooting content is based on the subjective value of the user, such as whether the subject is highly interested in the video, whether the subject is in focus, or whether the exposure state of the subject is appropriate. Means dependent evaluation. Hereinafter, the operation of the captured image selection unit 280 will be described.

  FIG. 4 is a flowchart showing an operation procedure of the subject extraction unit 250 and the unnecessary subject estimation unit 260 in the captured image processing unit 203. FIGS. 5 to 9 are diagrams schematically showing processing executed by the subject extraction unit 250 and the unnecessary subject estimation unit 260 of the captured image processing unit 203, and are referred to as needed in the following description.

  As shown in FIG. 5, the captured image processing unit 203 includes a captured image group 410 including a plurality of captured images 41-1 to 41-n acquired by the image acquisition unit 202 by a single release operation. Obtained from the secondary storage medium 332 (step S101). Next, the captured image processing unit 203 causes the subject extraction unit 250 to extract all of the plurality of subjects 11 to 31 included in each of the plurality of captured images 41-1 to 41-n (step S102).

  Subsequently, the subject extraction unit 250 performs face recognition for each of the subjects 11 to 31 (step S103). As a result, as shown by the frame in FIG. 5, subjects 15 to 31 whose faces are recognized are set as subjects to be processed, and subjects 11 to 14 that are not other persons are processed by the captured image processing unit 203. It is removed from the target (step S104).

  By executing the face recognition, individual subjects are further identified for the subjects 15 to 31 to be processed, and the unnecessary subject estimation unit 260 evaluates the individual subjects 15 to 31 as unnecessary subjects. (Step S105).

  FIG. 6 illustrates one of the evaluation methods as an unnecessary subject by the unnecessary subject estimation unit 260. That is, the unnecessary subject estimation unit 260 extracts the line of sight for each of the subjects 15 to 31 whose faces have already been recognized. The unnecessary subject estimation unit 260 estimates the subjects 15, 16, 24, and 25 whose extracted lines of sight are deviated from the digital camera 100 as unnecessary subjects (step S106). The unnecessary subjects 15, 16, 24, and 25 are excluded from the processing targets of the unnecessary subject estimation unit 260.

  The unnecessary subject estimation unit 260 repeats the above step S105 and step S106 until there is no unevaluated subject (step S107: NO). If there are no unevaluated subjects (step S107: YES), the unnecessary subject estimation unit 260 ends the process.

  FIG. 7 illustrates another unnecessary subject evaluation method by the unnecessary subject estimation unit 260. That is, the unnecessary subject estimation unit 260 extracts a smile feature (smile level / smile level) from the recognized face (step S105). Since the evaluation of the smile level itself is a well-known technique, a description thereof is omitted here. Based on the extracted smile level, subjects 21, 23, and 28 whose facial expression evaluation is low (the smile level is lower than a predetermined smile level determined in advance) are estimated as unnecessary subjects (step S106). These subjects 21, 23, and 28 are excluded from the processing targets of the unnecessary subject estimation unit 260.

  The information obtained from the subject whose face is recognized includes not only the “line of sight” and “smile degree” as described above, but also information such as “face orientation” and “face size”. It is done. Unnecessary subject estimation may be performed based on these pieces of information.

  For example, regarding “face orientation”, a subject not facing the front may be estimated as an unnecessary subject. To determine whether or not the subject is facing the front, for example, using the above-described information of “line of sight”, a subject that does not have both eyes is estimated as a “subject that is not facing the front”.

  Regarding the “face size”, the ratio of the area occupied by the face of the subject relative to the shooting screen may be determined, and a subject whose ratio is less than a predetermined ratio may be estimated as an unnecessary subject. Alternatively, the face sizes of all the subjects may be relatively compared, and a relatively small face subject may be estimated as an unnecessary subject. As a relative comparison method, a method of determining all remaining subjects as unnecessary subjects except the largest face to the top five faces, or obtaining an average value of the face sizes of all subjects and calculating the average value Various methods such as a method of estimating a subject having a smaller face as an unnecessary subject can be considered.

  As another method, in step S105, the unnecessary subject estimation unit 260 uses a subject (a person's face) registered in advance in a RAM or the like in the digital camera 100 as comparison information, so that the subject shown in the image is displayed. However, it may be determined (evaluated) whether or not the subject is a face registered in advance. Thereby, evaluation as an unnecessary subject can be performed due to closeness with the user of the digital camera 100. That is, as shown in FIG. 8, the unnecessary subject estimation unit 260 estimates the subjects 22 and 29 as unnecessary subjects for the facial subjects (for example, the subjects 22 and 29) not registered in advance in the camera (step S106). . Therefore, these subjects 22 and 29 are also excluded from the processing target of the unnecessary subject estimation unit 260.

  FIG. 9 illustrates another unnecessary object evaluation method by the unnecessary object estimation unit 260. FIG. 9 is a schematic diagram showing which subject is photographed in how many frames in images (frames) 41-1 to 41-n photographed by a series of continuous photographing.

  In the case of FIG. 9, the subject 26 is shown in 28 frames, and then the subject 27 is shown in 25 frames. Hereinafter, the subject 30 is shown in 6 frames and the subject 31 is shown in 7 frames. It shows that it is reflected. Then, the unnecessary subject estimation unit 260 determines an unnecessary subject based on the number of frames that are captured.

  In other words, the unnecessary subject estimation unit 260 includes the subjects 26, 27, 30, 31 that are not yet recognized as unnecessary subjects (in other words, remain as main subject candidates) for the plurality of captured images 41-1 to 41-n. The individual (for each subject) shooting frequency (number of frames taken / number of frames taken) is extracted to evaluate the subject (step S105). As a result, the subjects 30 and 31 with low photographing frequency are estimated as unnecessary subjects by the unnecessary subject estimation unit 260 (step S106).

  Therefore, these subjects 30 and 31 are also excluded from the processing target of the unnecessary subject estimation unit 260. The determination of “whether or not the shooting frequency is low” may be made by determining a subject with a small number of frames as “low” in a relative comparison between subjects, or a number of frames smaller than a predetermined number of frames. May be determined to be “low”.

  As described above, the unnecessary subject estimation unit 260 estimates the unnecessary subject using various evaluation items and evaluation methods for estimating the unnecessary subject after individually evaluating the face of the subject. By excluding subjects that are estimated as unnecessary subjects based on the evaluation results, the processing load described later can be reduced.

  Of course, the evaluation items and the evaluation method as the unnecessary subject in the unnecessary subject estimation unit 260 are not limited to the above example. Moreover, not all the evaluation items described above are executed, and some of them may be executed.

  Next, the main subject estimation process will be described with reference to FIG. The CPU 210 causes the image selection unit 280 to execute the processing operation (FIG. 10) of the main subject estimation unit 270 after the processing operation (FIG. 4) of the unnecessary subject estimation unit 260 described above.

  FIG. 10 is a flowchart illustrating an operation procedure of the main subject estimation unit 270 in the captured image processing unit 203. 11 to 14 are diagrams schematically showing processing executed in the main subject estimation unit 270 of the captured image processing unit 203, and are referred to as needed in the following description.

  First, the main subject estimation unit 270 excludes the subjects 11 to 14, 21 to 25, and 28 to 31 that the above-described unnecessary subject estimation unit 260 has estimated to be unnecessary subjects from the main subject candidates (step S201). The remaining subject is the subject of the main subject estimation process. Thus, as shown in FIG. 11, candidate subjects 26 and 27 that are candidates for the main subject are identified (step S202).

  Next, the main subject estimation unit 270 individually evaluates the main subject candidate subjects 26 and 27 for each of the captured images 41-1 to 41-n (step S203). As a method for this evaluation, for example, a method based on the positions of the candidate subjects 26 and 27 in the individual captured images 41-1 to 41-n can be exemplified.

  FIG. 12 is a diagram schematically illustrating a method in which the main subject estimation unit 270 evaluates the candidate subjects 26 and 27 based on the subject position history in the object scene 421. It is not a figure which shows the content of a picked-up image.

  As illustrated, the main subject estimation unit 270 individually tracks each of the candidate subjects 26 and 27 in a series of continuous shot images (a plurality of shot images) 41-1 to 41-n taken by the digital camera 100. Then, it is examined how far the position of the candidate subjects 26 and 27 in each image is located from the center C of each image of the captured images 41-1 to 41-n.

More specifically, the main subject estimation unit 270 determines a position (existing position) where the candidate subject 26 is captured in the plurality of photographed images 41-1 to 41-n. Evaluation is based on an average value of distances d ₁ , d ₂ , d ₃ ,..., d _X (x can be arbitrarily set and MAX is n) from the center C of n to the candidate subject 26. Next, for the other candidate subjects 27, the main subject estimation unit 270 also determines the distances D ₁ , D ₂ , D ₃ ,... From the center C of the plurality of captured images 41-1 to 41-n to the candidate subject 27. .., D _X (x can be arbitrarily set and MAX is n) is evaluated based on an average value.
In FIG. 12, as an example, an average value of the positions of the candidate subjects is calculated from five images (5 images near the full press operation timing) in which the candidate subjects 26 and 27 are captured. Explains. The main subject estimation unit 270 determines the position where the candidate subject 26 appears in the five photographed images 41-1 to 41-5 from the center C of the five photographed images 41-1 to 41-5. Evaluation is based on an average value of distances d ₁ , d ₂ , d ₃ , d ₄ , and d ₅ up to 26. Next, for the candidate subject 27, the main subject estimation unit 270 also determines the distances D ₁ , D ₂ , D ₃ , D ₄ , from the center C of the plurality of captured images 41-1 to 41-5 to the candidate subject 27. It is evaluated by the average value of D _5.

  From the above evaluation, in the example illustrated in FIG. 12, it can be seen that the candidate subject 27 is often shown closer to the center C of the captured image than the candidate subject 26. Therefore, the main subject estimation unit 270 estimates that the candidate subject 27 is the main subject. Thus, the captured image processing unit 203 estimates the main subject 27 from among the many subjects 11 to 14 and 21 to 31 (step S202).

  Thereafter, the main subject estimation unit 270 repeats the above steps S203 and S204 until there are no unevaluated candidate subjects (step S203: NO). When there are no more unevaluated subjects (step S203: YES), the processing in the main subject estimation unit 270 ends.

  FIG. 13 schematically illustrates another method in which the main subject estimation unit 270 evaluates the candidate subjects 26 and 27 based on the position history in the object scene 422 (information on the position of each candidate subject in each frame). FIG. As shown in the figure, first, a center area A is set near the center of the object scene 422 of the digital camera 100. Next, the number of times the candidate subjects 26 and 27 have entered the central area A in the series of the plurality of captured images 41-1 to 41-n is counted.

  Accordingly, for example, in the example of FIG. 13, it can be understood that the candidate subject 27 is reflected in the central area A more frequently than the candidate subject 26. Therefore, the main subject estimation unit 270 estimates that the candidate subject 27 is the main subject.

As described above, the captured image processing unit 203 estimates the main subject 27 based on the history of the positions of the respective subjects (candidate subjects) between a series of continuous shot images (a plurality of images). However, it goes without saying that the evaluation method used when the main subject 27 is estimated based on the position history is not limited to the above method. For example, in the method shown in FIG. 12, when evaluating the distances D ₁ , D ₂ , D ₃ , D ₄ , and D ₅ with respect to the center C, the evaluation value is calculated by adding statistical processing instead of simple averaging. It may be calculated. Further, it may be evaluated that the subject 27 at a distance from the center C is approaching the center of the object scene 422 as time passes.

  FIG. 14 is a diagram schematically illustrating another method in which the main subject estimation unit 270 evaluates the candidate subjects 26 and 27. As already described, the image acquisition unit 202 of the digital camera 100 can acquire a plurality of images captured in time series for one imaging operation. Among the plurality of photographed images 41-1 to 41-n acquired in this way, the candidate subjects (26, 27) shown in the photographed image that is closer in time to the timing when the release button 144 is pressed down are photographers. Is likely to be a subject intended to be photographed.

  Therefore, when there are a plurality of candidate subjects (26, 27) as described above, when evaluating which is the main subject, the acquired timing is relative to the timing when the release button 144 is pressed. Thus, the evaluation may be performed with a weight applied to the candidate subject appearing in the photographed image close in time. In addition, in an image acquired at a timing closer to the release timing, a candidate subject that is located closer to the center C of the object scene 421 or a center of the object scene 421 in an image closer to the release timing. The candidate subjects that have entered the area A may be weighted. Thereby, the estimation accuracy of the main subject can be further improved.

  FIG. 15 is a flowchart showing an operation procedure of the image selection unit 280. First, the image selection unit 280 extracts a plurality of selection image candidates from the captured image group 410 (step S301). For example, the selected image candidate is extracted on the condition that the estimated main subject is already captured, and it is checked whether or not each of the captured images 41-1 to 41-n is a selected image candidate.

  Thereafter, the image selection unit 280 repeats step S301 until there is no captured image that can be a selection image candidate (step S302: NO). When there are no captured images that can be selected image candidates (step S302: YES), the image selection unit 280 evaluates the imaging states of the unnecessary subjects 11 to 14, 21 to 16, and 28 to 31 for each of the selected image candidates. The selected image is selected (step S303).

  FIG. 16 is a diagram schematically illustrating a method in which the image selection unit 280 evaluates the selected image candidates based on the shooting states of the unnecessary subjects 11 to 14, 21 to 16, and 28 to 31. The subjects 11 to 14 and 21 to 31 shown in the photographed image 41-2 are common to the first photographed image 41-1 in the photographed image group 410. However, in the captured image 41-2, the depth of field changes, and the contrast of the unnecessary subjects 11 to 14, 21 to 26, and 28 to 31 is lower than the contrast of the main subject 27.

  As described above, when the contrasts of the unnecessary subjects 11 to 14, 21 to 26, and 28 to 31 in the captured image 41-2 are lower, the image selection unit 280 emphasizes the main subject 27 relatively in the image. The photographed image 41-2 is selected. Since one unnecessary subject 26 in the photographed image 41-2 is located near the main subject 27, it is photographed with a high contrast equivalent to the main subject 27. However, when all the unnecessary subjects 11 to 14, 21 to 26, and 28 to 31 are evaluated as a whole, the contrast of the unnecessary subjects 11 to 14, 21 to 26, and 28 to 31 is lower than that of the main subject 27.

  FIG. 17 is a diagram schematically illustrating another method in which the image selection unit 280 evaluates the selected image candidate based on the shooting states of the unnecessary subjects 11 to 14, 21 to 16, and 28 to 31. The subjects 11 to 14 and 21 to 31 shown in the photographed image 41-3 are common to the first photographed image 41-1 in the photographed image group 410. However, in the captured image 41-3, the positions of the unnecessary subjects 11 to 14, 21 to 26, and 28 to 31 and the main subject 27 are changed.

  Accordingly, the area occupied by the unnecessary subjects 11 to 14, 21 to 26, and 28 to 31 in the captured image 41-3 is smaller than that of the captured image 41-1. As described above, when the area occupied by the unnecessary subjects 11 to 14, 21 to 26, and 28 to 31 in the captured image 41-3 is small, the image selection unit 280 causes the main subject 27 to be relative to the captured image 41-3. The photographed image 41-3 is selected as a selected image.

  FIG. 18 is a diagram schematically illustrating another method in which the image selection unit 280 evaluates the selected image candidates based on the shooting states of the unnecessary subjects 11 to 14, 21 to 16, and 28 to 31. The subjects 11 to 14 and 21 to 31 shown in the photographed image 41-4 are common to the first photographed image 41-1 in the photographed image group 410. However, in the captured image 41-4, the positions of unnecessary subjects 11 to 14, 21 to 26, and 28 to 31 are dispersed.

  Thereby, the positions of the unnecessary subjects 11 to 14, 21 to 26, and 28 to 31 in the captured image 41-4 are closer to the periphery of the captured image 41-4 than the captured image 41-1. As described above, when the positions of the unnecessary subjects 11 to 14, 21 to 26, and 28 to 31 in the captured image 41-4 are close to the peripheral portion, the image selection unit 280 performs the main processing in the captured image 41-4. It is evaluated that the subject 27 is relatively emphasized, and the captured image 41-4 is selected.

  As described above, the image selection unit 280 evaluates the shooting states of the unnecessary subjects 11 to 14, 21 to 26, and 28 to 31 in each of the shot images 41-1 to 41-n, and is a main subject that is more important for the user. 27, an image in which the shooting conditions of the unnecessary subjects 11 to 14, 21 to 26, and 28 to 31 are relatively inferior is selected. Of course, the evaluation items and the evaluation method when selecting the captured images 41-1 to 41-n are not limited to the above example. Moreover, not all the evaluation items described above are executed, and some of them may be executed.

  Referring to FIG. 15 again, the image selection unit 280 repeats step S303 until there is no captured image that can be a selected image candidate (step S304: NO). When the unnecessary subjects 11 to 14, 21 to 16, and 28 to 31 are evaluated for all the selected image candidates (step S304: YES), the image selection unit 280 sets the main subject 27 for each of the selected image candidates. The selected state is selected by evaluating the shooting state (step S305).

  FIG. 19 is a diagram schematically illustrating a method in which the image selection unit 280 evaluates the selected image candidate based on the main 27 shooting states. The subjects 11 to 14 and 21 to 31 shown in the photographed image 41-5 are common to the first photographed image 41-1 in the photographed image group 410. It is common. However, in the captured image 41-5, the main subject 27 is strongly illuminated, and the other subjects 11 to 14, 21 to 25, and 28 to 31 appear relatively dark.

  Thus, when the contrast of the main subject 27 in the captured image 41-5 is higher (large), the image selection unit 280 evaluates that the main subject 27 is captured relatively clearly in the image, The photographed image 41-5 is selected. Note that one unnecessary subject 26 in the photographed image 41-5 is photographed with a high contrast equivalent to the main subject 27. However, when all unnecessary subjects 11 to 14, 21 to 26, and 28 to 31 are evaluated as a whole, the contrast of the main subject 27 is higher than that of the main subject 27.

  FIG. 20 is a diagram schematically illustrating another method in which the image selection unit 280 evaluates the selected image candidate based on the shooting state of the main subject 27. The subjects 11 to 14 and 21 to 31 shown in the photographed image 41-2 are common to the first photographed image 41-1 in the photographed image group 410. However, in the captured image 41-5, the positional relationship between the main subject 27 and the unnecessary subjects 11 to 14, 21 to 26, and 28 to 31 has changed for some reason.

  Thus, the area occupied by the main subject 27 in the captured image 41-6 is larger than that of the captured image 41-1. As described above, when the area occupied by the main subject 27 is larger in the captured image 41-6, the image selection unit 280 evaluates that the main subject 27 is relatively emphasized in the captured image 41-6. The photographed image 41-6 is selected as a selected image.

  FIG. 21 is a diagram schematically illustrating another method in which the image selection unit 280 evaluates the selected image candidate based on the shooting state of the main subject 27. The subjects 11 to 14 and 21 to 31 shown in the photographed image 41-7 are common to the first photographed image 41-1 in the photographed image group 410. It is almost common. However, in the photographed image 41-7, the position of the main subject 27 has moved to the center of the object scene. As described above, when the position of the main subject 27 in the captured image 41-7 is approaching the center of the captured image 41-7, the image selection unit 280 makes the main subject 27 relatively in the captured image 41-7. It evaluates that it is emphasized, and the said picked-up image 41-7 is selected.

  In this way, the image selection unit 280 evaluates the shooting state of the main subject 27 in each of the shot images 41-1 to 41-n, and the shooting state of the main subject 27 that is more important to the user is the unnecessary subject 11 to 11. 14, 21-26, 28-31 images are selected. Of course, the evaluation items and the evaluation method when selecting the captured images 41-1 to 41-n are not limited to the above example. Moreover, not all the evaluation items described above are executed, and some of them may be executed.

  Referring to FIG. 15 again, the image selection unit 280 repeats step S305 until there is no captured image that can be a selected image candidate (step S306: NO). When the main subject 27 is evaluated for all the selected image candidates (step S306: YES), the image selection unit 280 ends the process.

  Note that the order of the selection based on the evaluation of the main subject (step S303) and the selection based on the unnecessary subject (step S305) may be interchanged. Further, both the unnecessary subject and the main subject may be evaluated in all selection candidate images, and the photographed image may be selected based on a comprehensive evaluation result.

  The captured images 41-2 to 41-7 selected by the image selection unit 280 as described above are preferentially presented to the user when the digital camera 100 is set to the playback mode, for example. Thereby, the user can shorten the selection work time when selecting a captured image from a large number of captured images. In addition, the digital camera 100 may automatically discard a photographed image that has a particularly low evaluation, or may not display the captured image until it is instructed to be displayed by the user.

  FIG. 22 is a diagram schematically illustrating a personal computer 500 that executes an imaging condition setting program. The personal computer 500 includes a display 520, a main body 530, and a keyboard 540.

  The main body 530 can acquire image data of a photographed image from the digital camera 100 through communication with the digital camera 100. The acquired image data can be stored in a storage medium of the personal computer 500. The personal computer 500 also includes an optical drive 532 used when loading a program to be executed.

  The personal computer 500 as described above operates as a photographed image processing apparatus that executes the procedure shown in FIGS. 4, 10, and 15 by reading a photographed image processing program. Further, the personal computer 500 can acquire photographed image data from the digital camera 100 via the cable 510 and make it a target of processing.

  That is, the captured image processing program stores at least one of an image acquisition procedure for acquiring a plurality of images captured in time series, a subject extraction procedure for extracting a plurality of subjects included in the plurality of images, and a plurality of images. Unnecessary subject estimation procedure for estimating which one of a plurality of subjects is an unnecessary subject based on an image, and in which position each of the subject candidates excluding the unnecessary subject is located in each of the plurality of images. A main subject estimation procedure for evaluating and estimating which of the subject candidates is the main subject based on a history of positions between a plurality of images. The photographed image processing program causes the personal computer 500 to execute a series of procedures.

  Thus, the user can easily operate with the larger display 520 and the keyboard 540. By using the personal computer 500, a larger amount of images can be processed at high speed. Further, the evaluation items in each of the subject extraction procedure, the unnecessary subject estimation procedure, the main subject estimation procedure, and the image selection procedure may be increased and the evaluation unit may be made finer. Thereby, image selection can be assisted while reflecting the user's intention in more detail.

  Note that the transfer of captured image data between the digital camera 100 and the personal computer 500 may be via a cable 510 as shown in the figure, or may be wireless communication. Alternatively, the captured image data may be acquired by delivering a secondary storage medium that stores the captured image data. The execution of the photographed image processing program is not limited to the personal computer 500, and may be executed at a storefront or an online print service facility.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  In addition, the execution order of each process such as operation, procedure, step, and stage in the apparatus, system, program, and method shown in the claims, the specification, and the drawings is particularly “before”, “ It should be noted that it can be realized in any order unless it is clearly indicated as “prior to” or the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it does not necessarily mean that implementation in this order is essential. Absent.

11 to 31 Subject, 100 Digital camera, 110 Case, 120 Lens barrel, 122 Shooting lens, 130 Light emission window, 140 Operation unit, 141 Cross key, 142 Power switch, 143 Rear button, 144 Release button, 146 Zoom lever, 150 Rear display unit, 200 internal circuit, 201 control unit, 202 image acquisition unit, 203 captured image processing unit, 210 CPU, 220 display drive unit, 230 program memory, 240 main memory, 250 subject extraction unit, 260 unnecessary subject estimation unit, 270 Main subject estimation unit, 280 Image selection unit, 310 Image sensor driving unit, 312 Image sensor, 320 Analog / digital conversion unit, 330 Image processing unit, 332 Secondary storage medium, 340 Automatic focusing unit, 350 Photometric sensor, 360 Magnification drive unit, 37 Input unit, 410 photographed image group, 41-1, 41-2, 41-3, 41-4, 41-5, 41-6, 41-7 ... 41-n photographed image, 421, 422 , 500 personal computer, 510 cable, 520 display, 530 main unit, 532 optical drive, 540 keyboard

Claims (13)

An image acquisition unit for acquiring a plurality of images captured in time series;
A subject extraction unit that extracts a plurality of different subjects included in the plurality of images;
An unnecessary subject estimation unit that estimates which of the plurality of subjects is an unnecessary subject based on at least one of the plurality of images;
An image processing apparatus comprising: a main subject estimation unit configured to estimate a main subject based on subject candidates obtained by removing the unnecessary subject from the plurality of subjects.   The main subject estimation unit determines where each of the subject candidates is present in each of the plurality of images when there are a plurality of the subject candidates, and The image processing apparatus according to claim 1, wherein which of the subject candidates is a main subject is estimated based on a history of existing positions.   Among the plurality of images, an image having a smaller contrast of the unnecessary subject estimated by the unnecessary subject estimation unit, an image having a smaller area occupied by the unnecessary subject, and an image in which the position of the unnecessary subject is closer to the periphery. The image processing apparatus according to claim 1, further comprising an image selection unit that selects at least one of them. The image selection unit includes an image having a smaller contrast of the unnecessary subject estimated by the unnecessary subject determination unit, an image having a smaller area occupied by the unnecessary subject, and a position of the unnecessary subject among the plurality of images. At least one of the images close to the periphery, and
The at least one of an image having a larger contrast of the main subject determined by the main subject determination unit, an image having a larger area occupied by the main subject, and an image in which the position of the main subject is closer to the center is selected. The image processing apparatus according to 3.   5. The main subject estimation unit weights and evaluates a subject appearing in an image closer to the timing at which imaging is started in the time series in the plurality of images. An image processing apparatus according to 1. The subject extraction unit detects a face for each of the plurality of subjects,
The image processing apparatus according to claim 1, wherein the unnecessary subject estimation unit estimates the unnecessary subject by evaluating the face individually.   The image processing apparatus according to claim 6, wherein the unnecessary subject estimation unit evaluates the subject by extracting a feature of a smile on the face.   The image processing apparatus according to claim 6, wherein when the subject is a person, the unnecessary subject estimation unit extracts a line of sight of the person and evaluates each of the subjects.   The image processing apparatus according to claim 7 or 8, wherein the unnecessary subject estimation unit evaluates the subject based on information indicating whether or not the subject is registered in advance.   The image processing apparatus according to claim 7, wherein the unnecessary subject estimation unit evaluates each of the subjects based on a photographing frequency for each subject in the plurality of images. The image processing apparatus according to any one of claims 1 to 10,
A release button operated by the user,
An imaging apparatus comprising: an imaging unit that captures the plurality of images for one operation of the release button. An image acquisition procedure for acquiring a plurality of images captured in time series;
A subject extraction procedure for extracting a plurality of different subjects included in the plurality of images;
An unnecessary subject estimation procedure for estimating which of the plurality of subjects is an unnecessary subject based on at least one of the plurality of images;
A main subject estimation procedure for estimating a main subject based on subject candidates obtained by removing the unnecessary subject from the plurality of subjects;
An image processing program that causes a computer to execute.   In the main subject estimation procedure, when there are a plurality of subject candidates, it is determined in which position each of the subject candidates is present in each of the plurality of images, and the main subject estimation procedure is performed between the plurality of images. The image processing program according to claim 12, wherein which of the subject candidates is a main subject is estimated based on a history of existing positions.

Images