JP2013012124A - Image processing device, imaging device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract an image in which a main subject is photographed in a preferable state.SOLUTION: An image processing device comprises: an image acquisition unit that acquires plural images; a main subject estimation unit that estimates a main subject in the plural images; a saliency degree calculation unit that calculates a visual saliency degree for each of the plural images; and an image selection unit that compares the saliency degrees of the main subject among the plural images, and selects any image of the plural images on the basis of the saliency degrees of the main subject. In the image processing device, the image selection unit may select the image when the saliency degree of the main subject in each of the plural images is greater.

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, and a program.

There has been proposed an image display device that automatically evaluates and selects individual qualities of a plurality of images, and leaves the selected images to user selection (see Patent Document 1).
[Patent Document 1] Japanese Patent Application Laid-Open No. 2006-311340

  However, when the evaluation is performed based on the quality of the entire image, how the subject focused on by the user is not evaluated, and the automatic evaluation does not always coincide with the user's selection.

  As a first aspect of the present invention, an image acquisition unit that acquires a plurality of images, a main subject estimation unit that estimates a main subject in the plurality of images, and a saliency that calculates visual saliency for each of the plurality of images An image processing apparatus comprising: a degree calculation unit; and an image selection unit that compares the saliency of the main subject between the plurality of images and selects any one of the plurality of images based on the saliency of the main subject Is provided.

  As a second aspect of the present invention, there is provided an imaging apparatus including the image processing apparatus and an imaging unit that captures image light of an object scene.

  As a third aspect of the present invention, a subject extraction stage for extracting a plurality of different subjects included in a plurality of images, and a main subject estimation stage for estimating which of the plurality of subjects is a main subject in the plurality of subjects. And a saliency calculation stage for calculating a visual saliency for each of the plurality of images, and comparing the saliency of the main subject between the plurality of images, and among the plurality of images based on the saliency of the main subject. An image processing program for causing a computer to execute an image selection step of selecting any one of the above images is provided.

  The above summary of the present invention does not enumerate all necessary features of the present invention. A sub-combination of these feature groups can also be an invention.

1 is a perspective view of a digital camera 100. FIG. 1 is a perspective view of a digital camera 100. FIG. 2 is a block diagram showing an internal circuit 200 of the digital camera 100. FIG. 5 is a flowchart illustrating an operation procedure of a captured image processing unit 203. 4 is a schematic diagram illustrating a captured image group 410. FIG. 6 is a diagram schematically showing the operation of a subject extraction unit 250. FIG. FIG. 6 is a diagram schematically illustrating the operation of a main subject estimation unit 260. FIG. 6 is a diagram schematically illustrating the operation of a main subject estimation unit 260. FIG. 6 is a diagram schematically illustrating the operation of a main subject estimation unit 260. 5 is a flowchart showing an operation procedure of an image selection unit 270. 5 is a block diagram of a saliency map generation unit 222. FIG. It is a flowchart which shows the procedure of saliency map generation. It is a figure which shows typically the production | generation process of a saliency map. 6 is a block diagram of a luminance information extraction unit 610. FIG. It is a flowchart which shows the procedure of luminance information extraction. 6 is a block diagram of a color information extraction unit 620. FIG. It is a flowchart which shows the procedure of color information extraction. 6 is a block diagram of an edge information extraction unit 630. FIG. It is a flowchart which shows the procedure of edge information extraction. It is a schematic diagram which shows the saliency of a main subject. FIG. 10 is a diagram schematically illustrating the operation of an image selection unit 270. FIG. 10 is a diagram schematically illustrating the operation of an image selection unit 270. FIG. 10 is a diagram schematically illustrating the operation of an image selection unit 270. FIG. 10 is a diagram schematically illustrating the operation of an image selection unit 270. FIG. 10 is a diagram schematically illustrating the operation of an image selection unit 270. It is a schematic diagram of a personal computer that executes an image processing program.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 is a perspective view of a digital camera 100 that is a type of imaging apparatus and includes a captured image processing apparatus as viewed obliquely from the front. The digital camera 100 includes a housing 110, a lens barrel 120 and a light emitting window 130 disposed on the front surface of the housing 110, a power switch 142, a release button 144, and a zoom lever 146 disposed on the top surface of the housing 110. Prepare.

  The lens barrel 120 holds a photographing lens 122 that forms an image light of a subject on an image sensor disposed inside the housing 110. Light generated by a light emitting unit (not shown) disposed inside the housing 110 illuminates the subject through the light emitting window 130. The power switch 142, the release button 144, and the zoom lever 146 form part of the operation unit 140 of the digital camera 100.

  Each time the power switch 142 is pressed, the power of the digital camera 100 is interrupted. The zoom lever 146 changes the magnification of the photographing lens held in the lens barrel 120. When the release button 144 is half-pressed by the user, the automatic focusing unit, the photometric sensor, and the like are driven, and a through image capturing operation by the image sensor is executed. Thereby, the digital camera 100 prepares for the main photographing of the subject image following the through image photographing.

  When the release button 144 is fully pressed, the shutter is opened and the subject image is actually captured. When the brightness of the shooting range is dark, the light is projected from the light emission window 130 toward the subject in accordance with the timing of the main shooting.

  FIG. 2 is a perspective view of the digital camera 100 as viewed obliquely from the rear. Elements that are the same as those in FIG. 1 are given the same reference numerals, and redundant descriptions are omitted.

  A part of the operation unit 140 including a cross key 141 and a back button 143 and a back display unit 150 are arranged on the back surface of the housing 110. The cross key 141 and the back button 143 are operated by the user when inputting various settings to the digital camera 100, switching the operation mode of the digital camera 100, changing the magnification of the lens barrel in real time, and the like. The

  The rear display unit 150 is formed of a liquid crystal display panel or the like, and occupies many areas on the rear surface of the housing 110. When the digital camera 100 is in the shooting mode, a subject image incident on the lens barrel 120 is continuously photoelectrically converted by the imaging device and displayed on the rear display unit 150 as a through image. The user can know the effective shooting range by observing the through image displayed on the rear display unit 150.

  The rear display unit 150 also displays the status of the digital camera 100 such as the remaining battery level and the remaining capacity of the storage medium. Further, when the digital camera 100 operates in the playback mode, image data is read from a secondary storage medium 332 described later, and the playback image is displayed on the rear display unit 150.

  FIG. 3 is a block diagram schematically showing the internal circuit 200 of the digital camera 100. Elements common to those in FIGS. 1 and 2 are denoted by the same reference numerals, and redundant description is omitted. The internal circuit 200 includes a control unit 201, an imaging unit 202, and a captured image processing unit 203.

  The control unit 201 includes a CPU 210, a display driving unit 220, a program memory 230, and a main memory 240. The CPU 210 comprehensively controls the operation of the digital camera 100 according to the firmware read from the program memory 230 to the main memory 240. The CPU 210 is also formed with a saliency map generation unit 222 that generates a saliency map of the input image. The input image to the saliency map generation unit 222 may be an image file supplied from the outside in addition to a captured image of the digital camera 100. The display driving unit 220 generates a display image in accordance with an instruction from the CPU 210 and displays the generated image on the rear display unit 150.

  The imaging unit 202 includes an imaging device driving unit 310, an imaging device 312, an analog / digital conversion unit 320, an image processing unit 330, an automatic focusing unit 340, and a photometric sensor 350. The image sensor driving unit 310 drives the image sensor 312 to photoelectrically convert the subject image formed on the image pickup surface of the image sensor 312 by the photographing lens 122 into an image signal. As the imaging element 312, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like can be used.

  The image signal output from the image sensor 312 is discretized by the analog / digital conversion unit 320 and converted into captured image data by the image processing unit 330. The image processing unit 330 adjusts white balance, sharpness, gamma, gradation correction, and the like of the image in the process of generating the captured image data, and compresses the captured image data when it is stored in the secondary storage medium 332 described later. Set the rate, etc.

  The image data generated by the image processing unit 330 is stored and stored in the secondary storage medium 332. As the secondary storage medium 332, a medium including a nonvolatile storage element such as a flash memory card is used. At least a part of the secondary storage medium 332 can be detached from the digital camera 100 and replaced.

  The automatic focusing unit 340 performs an automatic focusing operation by a known phase difference method using, for example, a phase difference type focus detection element disposed on the imaging surface of the imaging element 312. The photometric sensor 350 measures the brightness of the subject and determines the shooting conditions of the digital camera 100. The magnification driving unit 360 moves a part of the photographing lens 122 in accordance with an instruction from the CPU 210. As a result, the magnification of the photographic lens 122 changes, and the angle of view of the photographic image changes.

  The input receiving unit 370 receives an input from the operation unit 140 and inputs the input to the CPU 210. The CPU 210 generates a command to each unit according to the input signal. In addition, when generating a command, the CPU 210 refers to a set value held in the input receiving unit 370 to determine an operation condition.

  The digital camera 100 having the internal circuit 200 as described above has a shooting mode in which the imaging unit 202 acquires a plurality of image data in response to one shooting operation (full pressing operation) in which the user presses the release button 144. Have. When the shooting mode is set, the CPU 210 controls the image sensor 312 to perform continuous shooting using the image sensor driving unit 310. Thereby, time-series photographed image (video) data is acquired.

  The time-series captured image data acquired in this way is sequentially input to a FIFO (First In First Out) memory in the image processing unit 330. The FIFO memory has a predetermined storage capacity, and when input data sequentially input reaches the predetermined capacity, the captured image data is output in the input order.

  In the shooting mode, time-series shot image data is sequentially input to the FIFO memory until a predetermined time has elapsed since the user fully pressed the release button 144, and during this time, the data output from the FIFO memory is deleted. The When a predetermined time elapses after the release button 144 is fully pressed, writing of the captured image data to the FIFO memory is prohibited. As a result, a plurality of frames of time-series captured image data acquired before and after the release button 144 is fully pressed are held in the FIFO memory.

  That is, by acquiring a plurality of images captured in time series by the imaging unit 202 for a single shooting operation (full pressing operation), shooting conditions (aperture opening, shutter speed, imaging element) are acquired from the plurality of images. Sensitivity, etc.), shooting timing, and the like can be selected from a plurality of frames of shooting images acquired in the shooting mode. Thereby, the success rate of photographing can be improved.

  In recent years, the performance of the digital camera 100 has improved, and it has become possible to acquire image data of several tens of frames in response to a single release button operation by the user. In such a case, it may be a burden on the user to select image data to be stored from a large amount of photographed image data. According to the present embodiment, such a burden on the user can be reduced as described later.

  A digital camera 100 shown in FIG. 3 includes a captured image processing unit 203. The captured image processing unit 203 includes a subject extraction unit 250, a main subject estimation unit 260, and an image selection unit 270, and selects a captured image that is estimated to have a higher value as an image from a plurality of captured images.

  Here, the value as an image includes not only the high quality as image data but also the evaluation of the captured content. High quality as image data means high value depending on signal quality as digital data, such as high image resolution, wide dynamic range, no overexposure / blackout, and the like. On the other hand, the evaluation of the shooting content is based on the subjective value of the user, such as whether the subject is highly interested in the video, whether the subject is in focus, or whether the exposure state of the subject is appropriate. Means dependent evaluation.

  4 and 10 are flowcharts showing a processing procedure in the captured image processing unit 203 of the digital camera 100. FIG. FIG. 4 shows processing up to the subject extraction unit 250 and the main subject estimation unit 260. The captured image processing unit 203 acquires a plurality of captured image files by reading captured image data of a plurality of frames from the FIFO memory or the secondary storage medium 332 (step S101).

  FIG. 5 is a diagram conceptually showing a plurality of captured images acquired by the captured image processing unit 203. As illustrated, the plurality of captured images 41-1 to 41-n acquired are processed in the captured image processing unit 203 as a captured image group 410.

  Here, the captured images 41-1 to 41-n may be images captured continuously in time or time series by bracket shooting, continuous shooting, or the like. Further, the images may have a common part of shooting conditions such as the same place, the same day, the same season, and the like.

  However, the plurality of captured images 41-1 to 41-n have different contents from each other. The different contents referred to here include in-focus positions and exposure conditions even when the subjects included in the captured images 41-1 to 41-n are different, and even when the same subject is captured with the same composition. Including those that differ.

  Referring to FIG. 4 again, the photographed image processing unit 203 uses the subject extraction unit 250 to select subjects included in the photographed image from one frame of the obtained photographed images 41-1 to 41-n. Extract (step S102). Subject extraction from captured images includes various methods such as contour extraction using a Sobel filter, etc., well-known face recognition technology, eigenface using principal component analysis, linear discriminant analysis, elastic bunch graph matching, hidden Markov model, etc. Can be used.

  FIG. 6 is a diagram conceptually showing processing in the subject extraction unit 250. As illustrated, the subject extraction unit 250 extracts subjects 11 to 33 from each of the acquired plurality of captured images 41-1 to 41-n. The extracted subject may include subjects other than the person, such as a person including a face, a still life such as a tree, a car, an animal, and the like.

  Referring to FIG. 4 again, when the subject extraction unit 250 extracts all of the subjects 11 to 33 included in a certain frame of the captured image, for example, the captured image 41-1, the subject extraction unit 250 still extracts the subject in the acquired captured image group 410. It is checked whether there is a photographed image that has not been performed (step S103). If there are still captured images that have not been subject extracted (step S103: NO), the subject extraction unit 250 sequentially executes subject extraction from the captured images (step S102).

  If it is determined in step S103 that the subject has been extracted from all of the captured images 41-1 to 41-n included in the acquired captured image group 410 (step S103: YES), the subject extraction unit 250 completes the processing. Then, the process is passed to the main subject estimation unit 260.

  The subjects 11 to 33 extracted by the subject extraction unit 250 are recorded in the main memory 240 and become a unit of processing by the main subject estimation unit 260 described later. It should be noted that a subject cannot be extracted from an image having no target to be extracted as a main subject, such as a plain or sky. Therefore, when the acquired plurality of images include an image in which such a main subject cannot be detected, the captured image processing unit 203 excludes such a captured image from the processing target.

  The main subject estimation unit 260 targets a plurality of subjects 11 to 33 extracted by the subject extraction unit 250 in each of the captured images 41-1 to 41-n included in the captured image group 410, and takes a captured image from them. One main subject in 41-1 to 41-n is estimated (step S104). That is, the main subject estimation unit 260 estimates the subject that is estimated to be most important for the user from the plurality of subjects 11 to 33 included in the captured images 41-1 to 41-n as the main subject.

  In this way, the main subject in the series of captured images is estimated. When the main subject estimation unit 260 estimates which subject is the main subject from one frame of a captured image, for example, the subjects 11 to 33 included in the captured image 41-1, the region of the estimated main subject in each captured image. Is specified (step S105).

  Note that the process of estimating the main subject (S104) may include a process of specifying an area including the subject to be estimated. In such a case, after the main subject is estimated, it is possible to omit the process of respecifying the estimated main subject region (step S105).

  However, when the extraction of the main subject is executed using a part of the subject such as a face, the entire subject region including the face, for example, the whole body may be specified as the main subject again. Conversely, when the main subject is extracted using the entire subject as a target in step S104, the process of specifying the main subject region (step S105) can be omitted.

  In this way, the main subject estimation unit 260 specifies the main subject region for all the captured images 41-1 to 41-n included in the acquired captured image group 410. When the main subject area is identified (step S105) for all the captured images 41-1 to 41-n, the captured image processing unit 203 ends the processing in the main subject estimation unit 260.

  7 and 8 are diagrams schematically showing a method of estimating the main subject from the plurality of subjects 11 to 33 in step S104 of FIG. For example, as illustrated in FIG. 7, the main subject estimation unit 260 estimates the main subject based on the history of the position of each subject in the object scene 421. FIG. 7 is a schematic diagram showing a concept of a method for estimating a main subject, not a diagram showing the contents of a captured image.

  The main subject estimation unit 260 individually tracks the subjects 11 to 33 for the plurality of photographed images 41-1 to 41-n of the digital camera 100, and the positions of the subjects are the photographed images 41-1 to 41-n. It is examined how far away from a predetermined position (for example, center C). In FIG. 7, for the sake of simplicity of description, focusing only on the subjects 26 and 27, the positions of the subjects 26 and 27 are determined from predetermined positions (for example, the center C) of the captured images 41-1 to 41-n. The distant position is shown superimposed on the captured image of 5 frames.

The main subject estimation unit 260 determines the position of the subject 26 in the plurality of photographed images 41-1 to 41-5 from the center C of the plurality of photographed images 41-1 to 41-5 to the subject 26. The average value of d ₁ , d ₂ , d ₃ , d ₄ , d ₅ is calculated. In addition, the main subject estimation unit 260 also sets the distances D ₁ , D ₂ , D ₃ , D ₄ , and the distances from the center C of the plurality of captured images 41-1 to 41-5 to the complementary subject 27 for the other subjects 27. the average value of D ₅ is also calculated.

  The main subject estimation unit 260 compares the average values calculated as described above, and determines that the subject 27 is often shown closer to the center C of the captured image than the subject 26. . Thereby, the main subject estimation unit 260 estimates the subject 27 as the main subject. In the above example, an example in which the predetermined position in the captured image is the center C of the image has been described. For example, this predetermined position is preferably divided into three divided vertically and horizontally on the screen. A line or a predetermined point on the three-divided line may be used.

  In addition, as illustrated in FIG. 8, the main subject estimation unit 260 may evaluate the position in the object scene 422 by another method. FIG. 8 is also a schematic diagram showing a concept of a method for estimating a main subject, not a diagram showing the contents of a captured image.

  As shown in FIG. 8, the main subject estimation unit 260 sets the center area A near the center of the object scene 422 of the digital camera 100. Next, the number of times the subjects 26 and 27 have entered the central area A in the plurality of captured images 41-1 to 41-5 is counted. Thus, when the subject 27 is captured more frequently in the central area A than the subject 26, the subject 27 is estimated as the main subject.

As described above, the captured image processing unit 203 may estimate the main subject based on the position of the subject between a plurality of images. However, it goes without saying that the method for evaluating the position of the subject is not limited to the above example. For example, in the method shown in FIG. 7, when evaluating the distances D ₁ , D ₂ , D ₃ , D ₄ , and D ₅ with respect to the center C, the evaluation value is calculated by adding statistical processing instead of simple averaging. It may be calculated. Further, it may be evaluated that the subject 27 at a distance from the center C is approaching the center of the object scene 422 as time passes.

  Further, as shown in FIG. 9, the main subject estimation unit 260 weights the evaluation of the subject in the photographed image photographed at a timing close to the timing when the user performs the photographing operation (full-press operation of the release button 144). You may evaluate. That is, the image capturing unit 202 of the digital camera 100 can capture a plurality of images captured in time series for one capturing operation. Of the plurality of captured images 41-1 to 41-n acquired in this way, the subjects 26 and 27 that are reflected in the captured images that are close in time to the timing when the release button 144 is pressed must be photographed by the photographer. The probability that the subject is intended to be high.

  Therefore, when the subjects 26 and 27 are evaluated as described above, the subjects 26 and 27 appearing in the captured image that is close in time to the timing when the release button 144 is pressed are weighted. May be evaluated. Also, the subject 27 that is closer to the center C of the object scene 421 in the image closer to the release timing, or the subject 27 that is in the center area A of the object scene 421 in the image closer to the release timing may be weighted. . Thereby, the estimation accuracy of the main subject can be further improved. In particular, when shooting a moving subject while moving the camera, the photographer performs shooting while moving the camera so that the subject to be photographed is at a predetermined position on the shooting screen (for example, the center of the shooting screen). There are many. According to the above configuration, the main subject can be estimated appropriately.

  As described above, the main subject estimation unit 260 may estimate the main subject by identifying the face of the subject that is a person. However, it goes without saying that the main subject estimation method in the main subject estimation unit 260 is not limited to the above example. For example, when the subject is a person, a method is used such as estimating the subject with the highest smile level, which is the degree of smile, as the main subject, or estimating the subject with the largest detected face size as the main subject. It does not matter.

  Furthermore, a configuration may be adopted in which a main subject is estimated by performing a recognition process on each of a plurality of frames of captured image data for a unique subject that has been registered with priorities. Further, the accuracy of main subject estimation may be improved by combining or repeating some of the above methods.

  In the above example, the same main subject is estimated for a plurality of captured images. However, different main subjects may be estimated for the captured images. Thereby, it is possible to evaluate the shooting content, not the evaluation regarding the shooting state of the specific subject.

  FIG. 10 is a flowchart illustrating a processing procedure subsequent to the processing procedure illustrated in FIG. 4 in the captured image processing unit 203. In the photographed image processing unit 203, the image selection unit 270 including the saliency calculation unit 280 performs processing. For each of the captured images 41-1 to 41-n in which the main subject 27 is detected by the main subject estimation unit 260, the image selection unit 270 performs at least the region of the main subject 27 in the saliency calculation unit 280. A saliency map indicating the saliency distribution of the image is calculated (step S106).

  FIG. 11 is a block diagram schematically showing the internal structure of the saliency map generation unit 222 formed in the CPU 210. The saliency map generation unit 222 includes a luminance information extraction unit 610, a color information extraction unit 620, an edge information extraction unit 630, and a linear combination unit 640.

  The luminance information extraction unit 610, the color information extraction unit 620, and the edge information extraction unit 630 accept input images input to the saliency map generation unit 222 in parallel. Each of the luminance information extraction unit 610, the color information extraction unit 620, and the edge information extraction unit 630 that has received the input image extracts a feature amount related to a specific feature for each region of the image, and indicates information indicating the distribution of the extracted feature amount Generate a map.

  That is, the luminance information extraction unit 610 extracts the luminance for each area of the input image, for example, for each pixel, and generates a luminance information map indicating the luminance distribution in the input image. The color information extraction unit 620 generates a color information map indicating the color distribution in the input image. The edge information extraction unit 630 generates a directional edge information map indicating directional edge information indicating a directional edge distribution in the input image.

  The linear combination unit 640 linearly combines the luminance information map generated by the luminance information extraction unit 610, the color information map generated by the color information extraction unit 620, and the edge information map generated by the edge information extraction unit 630 to generate a saliency map. Generate. The generated saliency map shows the distribution of visual stimulus intensity in the input image.

  FIG. 12 is a flowchart outlining an example of processing in the saliency map generation unit 222. FIG. 13 is a diagram schematically showing a saliency map generation process, which is referred to in conjunction with FIG.

  For example, when the captured image 41-1 is supplied as an input image to be processed, the saliency map generation unit 222 first generates a luminance information map 46-1 based on the luminance information extracted from the input image. The process is executed by the luminance information extraction unit 610 (step S201). The luminance information extraction unit 610 transfers the generated luminance information map 46-1 to the linear combination unit 640.

  In addition, the saliency map generation unit 222 causes the color information extraction unit 620 to execute processing for generating the color information maps 47-1 and 47-2 based on the color information extracted from the input image (step S202). As color information to extract, the difference of R (red) -G (green) and the difference of B (blue) -Y (yellow) can be illustrated, for example. The color information extraction unit 620 transfers the generated color information map to the linear combination unit 640.

  Furthermore, the saliency map generation unit 222 causes the directional edge information extraction unit 630 to execute processing for generating the directional edge information maps 48-1 to 48-4 based on the directional edge information extracted from the input image ( Step S203). As the directional edge information to be extracted, directional edges of 0 degree, 45 degrees, 90 degrees, and 135 degrees can be exemplified. The directional edge information extraction unit 630 transfers the generated directional edge information maps 48-1 to 48-4 to the linear combination unit 640.

  In the following description, when “information map” is simply described, any one of the luminance information map 46-1, the color information maps 47-1 and 47-2, and the directional edge information maps 48-1 to 48-4 is selected. It is assumed that the entire information input to the linear combination unit 640 is pointed out without specifying. In addition, the order of the processes from step S201 to step S203 for generating each information map in FIG. 12 may be changed. Moreover, when there is a margin in the capacity of the processing device, parallel processing may be performed.

  Next, the saliency map generation unit 222 generates the luminance information map 46-1, the color information maps 47-1 and 47-2, and the directional edge information maps 48-1 to 48- generated from step S201 to step S203. The linear combination unit 640 is caused to execute a process of generating a saliency map 49-1 by linearly combining 4 (step S204).

  Here, the linear combination unit 640 may linearly combine the information maps using the information weight Wb that is predetermined for each information map. That is, the pixel value of the pixel on the target saliency map is obtained by multiplying the pixel value of the corresponding pixel in each information map by a weight Wb that is predetermined for each information map. Therefore, the sum of the pixel values multiplied by the information weight Wb becomes the pixel value of the pixel in the saliency map.

  Further, the pixel value of each pixel in the saliency map 49-1 is normalized so as to be distributed between 0 and 1. In this way, a saliency map is generated in which the feature amounts of the areas of the luminance information map, the color information map, and the edge information map are weighted and added for each area at the same position.

  In the above example, an information map is generated for each of the luminance, color, and directional edge, and these information maps are linearly combined to generate the saliency map 49-1. However, the types and number of information maps used for generating the saliency map are not limited to the above.

  For example, when an image acquired in time series, such as continuous shooting of a still image camera or bracket shooting, is input, it is also preferable to generate and use an information map related to the direction of motion in the image. Furthermore, an information map extracted by other image processing such as face detection or individual detection may be used.

  FIG. 14 is a block diagram of the luminance information extraction unit 610 in the saliency map generation unit 222. Here, FIG. 13 is also referred to. The luminance information extraction unit 610 includes a luminance image generation unit 612, a Gaussian pyramid generation unit 614, a difference image generation unit 616, and a luminance information map generation unit 618.

  For each pixel of the input image, the luminance image generation unit 612 generates a luminance image 42-1 having the pixel luminance as the pixel value. The Gaussian pyramid generation unit 614 generates a plurality of images whose resolutions are sequentially reduced from the luminance image acquired from the luminance image generation unit 612, and generates a Gaussian pyramid 43-1 having luminance as a feature amount.

  The difference image generation unit 616 calculates a plurality of luminance difference images 45-1 from images having different resolutions included in the Gaussian pyramid acquired from the Gaussian pyramid generation unit 614 and generates a luminance difference image 45-1. The luminance information map generation unit 618 generates a single luminance information map 46-1 from the plurality of luminance difference images 45-1 acquired from the difference image generation unit 616.

  The luminance information map shows the distribution of the intensity of visual stimulation due to a large luminance difference in the input image. The luminance information map 46-1 generated by the luminance information extraction unit 610 is provided to the linear combination unit 640 as described above.

  FIG. 15 is a flowchart showing processing executed in the luminance information extraction unit 610. Please refer to FIG.

  The luminance information extraction unit 610 receives the captured image 41-1 as an input image, and the luminance image generation unit 612 generates a luminance image 42-1 of the input image. The luminance image 42-1 uses the luminance value of each pixel of the input image as a pixel value.

  For example, the luminance image generation unit 612 generates a luminance image by converting the input image into a color difference-luminance signal and extracting the luminance component (step S301). The luminance image generation unit 612 may use an average value of R (red), G (green), and B (blue) components for each pixel as a pixel value for each pixel of the luminance image.

  Next, the luminance information extraction unit 610 causes the Gaussian pyramid generation unit 614 to generate the Gaussian pyramid 43-1 based on the luminance image 42-1 (step S302). The Gaussian pyramid 43-1 includes a plurality of images having different resolutions.

  That is, in the Gaussian pyramid generation unit 614, the initial luminance image 42-1 supplied from the luminance image generation unit 612 is the image of the level L1 with the highest resolution. Next, the Gaussian pyramid generation unit 614 performs a process of setting the average value of the pixel values of the four adjacent pixels as the pixel value of one pixel for each pixel of the luminance image 42-1 at the level L1, and thus the level having a lower resolution. An L2 image is generated.

  Hereinafter, the Gaussian pyramid generation unit 614 generates a plurality of images whose resolution gradually decreases to, for example, level L8 by repeating the same processing. Thereby, the Gaussian pyramid 43-1 hierarchized by 8 types of resolutions is produced | generated.

  Subsequently, the luminance information extraction unit 610 causes the difference image generation unit 616 to generate a luminance difference image 45-1 based on the Gaussian pyramid 43-1 (step S303). A plurality of luminance difference images 45-1 are generated as differences between a plurality of images included in the Gaussian pyramid 43-1.

  As an example, the difference image generation unit 616 generates five luminance difference images 45-1. The difference image generation unit 616 includes, for example, [level L6-level L3], [level L7-level L3], [level L7] among eight luminance images from level L1 to level L8 forming the Gaussian pyramid 43-1. The difference between the pixel values of the corresponding pixels is calculated by a combination of [Level L4], [Level L8-Level L4], and [Level L8-Level L5].

  However, in the Gaussian pyramid 43-1 acquired by the difference image generation unit 616, the plurality of images have different resolutions and therefore have different sizes (number of pixels). Therefore, the difference image generation unit 616 up-converts a low-resolution (small) image among the images to be processed, and matches the size to a high-resolution (large) luminance image.

  Next, the difference image generation unit 616 calculates a difference for each corresponding pixel with respect to the image 44-1 having the same size (number of pixels), and generates a luminance difference image 45-1 having luminance as a feature amount. Generate. The calculation of the difference is repeated for a plurality of luminance images until a predetermined number of luminance difference images 45-1 are generated. In this way, five luminance difference images 45-1 having the luminance difference as the pixel value are obtained.

  Further, the difference image generation unit 616 includes a luminance difference image 45-1 in each of the luminance difference images 45-1 such that the pixel value of each pixel is a value between 0 and 255, for example. Normalize pixel values. In this way, the difference image generation unit 616 generates five luminance difference images 45-1.

  Next, the luminance information extraction unit 610 causes the luminance information map generation unit 618 to generate the luminance information map 46-1 from the luminance difference image 45-1 (step S304). In other words, the luminance information map generation unit 618 generates a single luminance information map 46-1 by adding the plurality of luminance difference images 45-1.

  Here, each of the luminance difference images 45-1 may be weighted and added with a difference weight Wa that is a weight for each predetermined luminance difference image 45-1. As a result, the pixel value of the pixel in the luminance information map 46-1 is obtained by multiplying each pixel value of the pixel at the same position in the luminance difference image 45-1 by the difference weight Wa. It becomes.

  The pixel value in each pixel of the luminance information map 46-1 indicates the distribution of the difference between that pixel and the average luminance around that pixel. Since an area having a large luminance difference from the surroundings in the image strongly stimulates vision, the image of the area including the pixel is likely to be noticed. Therefore, when the difference in luminance of the area including the main subject is large, it can be estimated that attention is likely to be focused on the main subject.

  FIG. 16 is a block diagram of the color information extraction unit 620 in the saliency map generation unit 222. The color information extraction unit 620 includes an R (red) -G (green) difference image 621, a B (blue) -Y (yellow) difference image 622, a Gaussian pyramid generation unit 623, a difference image generation unit 624, and a color information map generation unit. 625.

  The RG difference image generation unit 621 uses the captured image 41-1 as an input image, and generates a color difference image having a pixel value as a difference between the red component and the green component of each pixel of the input image. The BY difference image generation unit 622 generates, for each pixel of the input image, a color difference image having a pixel value as a difference between the blue component and the yellow component of the pixel. The Gaussian pyramid generation unit 623 generates a plurality of images having different resolutions from the color difference images acquired from the RG difference image generation unit 621 or the BY difference image generation unit 622, and uses the color difference as a feature amount. Generate each Gaussian pyramid.

  The difference image generation unit 624 calculates a difference between images having different resolutions included in the Gaussian pyramid acquired from the Gaussian pyramid generation unit 623, and generates a color difference image. The color information map generation unit 625 generates single color information maps 47-1 and 47-2 based on a plurality of color difference images acquired from each of the difference image generation units 624. Therefore, the color information extraction unit 620, from a single input image, the color information map 47-1 derived from the RG difference image generation unit 621 and the color information map 47-2 derived from the BY difference image generation unit 622. And generate two.

  The color information maps 47-1 and 47-2 show the distribution of the intensity of visual stimulation due to a large change in hue in the input image. The color information map generated by the color information extraction unit 620 is provided to the linear combination unit 640 as described above.

  FIG. 17 is a flowchart showing processing executed in the color information extraction unit 620. First, the color information extraction unit 620 causes the RG difference image generation unit 621 to generate an RG difference image as a color information image of the input image (step S401).

  That is, the RG difference image generation unit 621 calculates the difference between the R (red) component and the G (green) component for each pixel of the input image, and uses the calculated difference as the pixel value of the pixel. -G Generate a difference image. The generated RG difference image is transferred to the Gaussian pyramid generation unit 623.

  Similarly, the color information extraction unit 620 causes the BY difference image generation unit 622 to generate a BY difference image of the input image (step S401). That is, the B-Y difference image generation unit 622 calculates the difference between the B (blue) component and the Y (yellow) component for each pixel of the input image, and uses the calculated difference as the pixel value of the pixel. -Y A difference image is generated. The generated BY difference image is transferred to the Gaussian pyramid generation unit 623.

  Next, the color information extraction unit 620 causes the Gaussian pyramid generation unit 623 to generate the Gaussian pyramids based on the RG difference image and the BY difference image, respectively (step S402). The Gaussian pyramid includes a plurality of images having different resolutions. In the following description, when simply described as “color difference image”, it means an RG difference image and a BY difference image comprehensively.

  In the Gaussian pyramid generation unit 623, the initial color difference image supplied from the RG difference image generation unit 621 or the BY difference image generation unit 622 is the image of the level L1 with the highest resolution. Next, the Gaussian pyramid generation unit 623 performs, for each pixel of the color difference image at the level L1, the average value of the pixel values of the four adjacent pixels as the pixel value of one pixel, thereby reducing the resolution of the level L2 with the lower resolution. Generate a color difference image.

  Thereafter, by repeating the same processing, for example, a plurality of color difference images whose resolution gradually decreases to level L8 are generated. Thereby, a Gaussian pyramid of the color difference image hierarchized with eight kinds of resolutions is generated.

  Subsequently, the color information extraction unit 620 causes the difference image generation unit 624 to generate a color difference image based on the acquired Gaussian pyramid (step S403). The color difference image is formed by a plurality of color difference images. The plurality of color difference images are generated as differences between the plurality of color difference images included in the Gaussian pyramid.

  A case where the difference image generation unit 624 generates five color difference images and forms a difference image having the hue as a feature amount will be described as an example. Of the eight color difference images from level L1 to level L8 forming the Gaussian pyramid, the difference image generation unit 624, for example, [level L6-level L3], [level L7-level L3], [level L7-level]. L4], [level L8-level L4] and [level L8-level L5] are combined to calculate the difference between the pixel values of the corresponding pixels.

  However, in the Gaussian pyramid acquired by the difference image generation unit 624, the plurality of color difference images have different resolutions and different sizes. Therefore, the difference image generation unit 624 up-converts a low resolution (small) image among color difference images to be processed, and sets the size (number of pixels) to a high resolution (large) color difference image. Match.

  Next, the difference image generation unit 624 calculates a difference for each corresponding pixel with respect to two color difference images having the same size (number of pixels), and generates a color difference image having a hue as a feature amount. . The difference calculation is repeated for a plurality of color difference images until a predetermined number of color difference images are generated. In this way, five color difference images having the pixel difference as the color difference difference are obtained.

  Further, the difference image generation unit 624 normalizes the pixel value in each color difference image so that the pixel value of each pixel becomes a value between 0 and 255, for example, for each color difference image. . In this way, the difference image generation unit 624 generates five color difference images. In addition, as the color difference image, a plurality of color difference images derived from the RG difference image and a plurality of color difference images derived from the BY difference image are generated individually.

  Next, the color information extraction unit 620 causes the color information map generation unit 625 to generate color information maps 47-1 and 47-2 from the color difference image (step S404). That is, the color information map generation unit 625 adds a plurality of color difference images forming the color difference image, and adds the color information map 47-1 derived from the RG difference image and the color information derived from the BY difference image. One map 47-2 is created.

  Here, each color difference image may be weighted and added with a difference weight Wa that is a weight for each color difference image determined in advance. Thus, the sum calculated by multiplying the pixel values of the pixels at the same position of each color difference image by the difference weight Wa becomes the pixel value of the pixel in the color information map.

  In the color information maps 47-1 and 47-2, the pixel value of each pixel indicates a distribution of an average hue difference between the pixel and its surroundings. In an image, an area having a large hue change with the surroundings strongly stimulates vision, and thus an image of an area including the pixel is likely to be noticed. Therefore, when the color difference of the area including the main subject is large, it can be estimated that attention is easily collected on the main subject.

  In the above example, the color difference image based on the difference between the R component and the G component and the difference between the B component and the Y component is used as the color information image extracted from the input image. However, for example, a color difference image may be generated based on the color difference component Cr that is the difference between the R component and the luminance component and the color difference component Cb that is the difference between the B component and the luminance component.

  FIG. 18 is a block diagram of the directional edge information extraction unit 630 in the saliency map generation unit 222. The directional edge information extraction unit 630 includes directional edge image generation units 631, 632, 633, and 634, a Gaussian pyramid generation unit 635, a difference image generation unit 636, and a directional edge information map generation unit 637.

  The directional edge image generation units 631, 632, 633, and 634 generate a directional edge image for each pixel of the input image, with the intensity of the directional edge at the pixel as a pixel value. The edge component in the input image can be extracted by an edge extraction operator such as a Gabor filter, Sobel filter, or Roberts filter.

  Each of the directional edge image generation units 631, 632, 633, and 634 has the edge strength evaluated according to the component of the angle specified in advance among the edge components extracted from the input image. A directional edge image based on the input image is generated as a pixel value. In the illustrated example, each of the directional edge image generation units 631, 632, 633, and 634 is designated in advance with an angle of 0 °, 45 °, 90 °, or 135 ° as a directional evaluation criterion. Therefore, the directional edge images generated by the individual directional edge image generating units 631, 632, 633, and 634 have inclination components of 0 °, 45 °, 90 °, or 135 ° included in the edges extracted from the input image. Show the distribution.

  The Gaussian pyramid generation unit 635 generates a plurality of directional edge images having different resolutions from the directional edge images acquired from the directional edge image generation units 631, 632, 633, and 634. As a result, a Gaussian pyramid having a directional edge corresponding to four types of designated angles as a feature quantity is generated.

  The difference image generation unit 636 calculates a plurality of directional edge difference images from the directional edge images having different resolutions included in the Gaussian pyramid acquired from the Gaussian pyramid 635, and generates a directional edge difference image. The directional edge information map generation unit 637 generates single directional edge information maps 48-1 to 48-4 from the directional edge difference image acquired from the difference image generation unit 636. Therefore, the directional edge information extraction unit 630 has four directional edge information maps 48 derived from directional edge images corresponding to four directions of 0 °, 45 °, 90 °, and 135 ° from a single input image. -1 to 48-4 are generated.

  The directional edge information map shows the distribution of the intensity of the visual stimulus due to a large change in the directional edge in the input image. The directional edge information map generated by the directional edge information extraction unit 630 is provided to the linear combination unit 640 as described above.

  FIG. 19 is a flowchart showing processing executed in the directional edge information extraction unit 630. The directional edge information extraction unit 630 first causes the directional edge image generation units 631, 632, 633, and 634 to generate a directional edge information image of the input image (step S501).

  That is, each of the directional edge image generation units 631, 632, 633, and 634 performs edge detection and directionality evaluation for each pixel of the input image, and generates a directional edge image. Thereby, for example, directional edge images corresponding to four types of directional characteristics of 0 °, 45 °, 90 °, and 135 ° are generated. The generated directional edge images are transferred to the Gaussian pyramid generation unit 635, respectively.

  Next, the directional edge information extraction unit 630 causes the Gaussian pyramid generation unit 635 to generate a Gaussian pyramid based on the directional edge image (step S502). The Gaussian pyramid includes a plurality of images having different resolutions. In the following description, when simply described as “edge difference image”, it means comprehensively directional edge images of 0 °, 45 °, 90 °, and 135 ° regardless of the angle of the directional edge.

  In the Gaussian pyramid generation unit 635, the initial directional edge image supplied from the directional edge image generation units 631, 632, 633, and 634 is an image of the level L1 with the highest resolution. Next, the Gaussian pyramid generation unit 635 performs the process of setting the average value of the pixel values of four adjacent pixels as the pixel value of one pixel for each pixel of the edge difference image of the level L1, and the level L2 having a lower resolution. An edge difference image is generated.

  Thereafter, by repeating the same processing, for example, a plurality of edge difference images whose resolution gradually decreases to level L8 are generated. Thereby, a Gaussian pyramid of the edge difference image hierarchized with eight kinds of resolutions is generated.

  Subsequently, the directional edge information extraction unit 630 causes the difference image generation unit 636 to generate a directional edge difference image based on the acquired Gaussian pyramid (step S503). The directional edge difference image is formed by a plurality of edge difference images. The plurality of edge difference images are generated as differences between the plurality of edge difference images included in the Gaussian pyramid.

  A case where the difference image generation unit 636 generates five edge difference images and forms a directional edge difference image having a directional edge as a feature amount will be described as an example. The difference image generation unit 636 includes, for example, [level L6−level L3], [level L7−level L3], [level L7−level] among the eight edge difference images from level L1 to level L8 forming the Gaussian pyramid. L4], [level L8-level L4] and [level L8-level L5] are combined to calculate the difference between the pixel values of the corresponding pixels.

  However, in the Gaussian pyramid acquired by the difference image generation unit 636, the plurality of edge difference images have different resolutions and therefore have different sizes (number of pixels). Therefore, the difference image generation unit 636 up-converts a low resolution (small) image among the edge difference images to be processed, and sets the size (number of pixels) to the high resolution (large) edge difference image. Match.

  Next, the difference image generation unit 636 calculates a difference for each corresponding pixel with respect to two edge difference images having the same size (number of pixels), and generates an edge difference image having a directional edge as a feature amount. Generate. The difference calculation is repeated for a plurality of edge difference images until a predetermined number of edge difference images are generated. In this way, five edge difference images having pixel values corresponding to differences in directional edge images are obtained.

  Further, the difference image generation unit 636 normalizes the pixel value in each of the edge difference images so that the pixel value of each pixel becomes a value between 0 and 255, for example, for each of the edge difference images. . In this way, the difference image generation unit 636 forms a directional edge difference image from the five edge difference images. The directional edge difference images are individually generated corresponding to the directional edge images of 0 °, 45 °, 90 °, and 135 °.

  Next, the directional edge information extraction unit 630 causes the directional edge information map generation unit 637 to generate a directional edge information map from the directional edge difference image (step S504). That is, the directional edge information map generating unit 637 adds a plurality of edge difference images forming the directional edge difference image to generate a single directional edge information map. Further, the directional edge information maps 48-1 to 48-4 are generated one by one for each direction of 0 °, 45 °, 90 °, and 135 °.

  Here, each edge difference image may be weighted and added with a difference weight Wa that is a weight for each edge difference image determined in advance. As a result, the sum calculated by multiplying the pixel values of the pixels at the same position of each edge difference image by the difference weight Wa is the sum of the pixels in the directional edge information maps 48-1 to 48-4. It becomes a pixel value.

  In the directional edge information maps 48-1 to 48-4 generated in this way, the pixel value of each pixel indicates the magnitude of the change of the directional edge with respect to the surrounding area. In the image, a region having a stronger edge strength than the surrounding region is a region that catches the eye of a person who views the image. Therefore, when the difference in luminance of the area including the main subject is large, it can be estimated that attention is likely to be focused on the main subject.

  Referring to FIG. 10 again, next, the saliency calculating unit 280 specifies the saliency for at least the area of the main subject 27 for each of the captured images 41-1 to 41-n (step S107). The saliency of the subject 27 here is generated by quantifying the saliency of the area occupied by the main subject 27 in each of the captured images 41-1 to 41-n.

  FIG. 20 is a schematic diagram conceptually showing the quantification of the saliency as described above. The image selection unit 270 calculates, for example, an average value of the saliency of the area (surrounded by a white frame in the drawing) occupied by the main subject 27 in each of the captured images 41-1 to 41-3. As a result, the saliency of the subject 27 in each saliency map 49-1, 49-2, 49-3 is digitized, and the relative evaluation of the captured images 41-1 to 41-3 is facilitated.

  Note that the saliency of the main subject 27 is not limited to the arithmetic average of the saliency in the area of the subject 27, and the saliency of the calculated subject 27 is calculated based on statistical processing. You may be more faithful to your senses.

  Further, the image selection unit 270 may represent the saliency of the subject 27 with the maximum value of the saliency in the pixels included in the region of the main subject 27. Thereby, it is possible to reduce the processing load for specifying the saliency of the subject 27 for each of the captured images 41-1 to 41-n.

  Referring to FIG. 10 again, the saliency calculating unit 280 includes the captured images 41-1 included in the captured image group 410 according to the saliency of the subject 27 quantified in each of the captured images 41-1 to 41-n. ˜41-n are ordered (step S108). That is, each time the saliency of the subject 27 is digitized in each of the captured images 41-1 to 41-n, the captured images 41-1 to 41-1 are compared with the quantified saliency of the captured image that has already been processed. 41-n is arranged.

  Further, each time the saliency calculating unit 280 calculates and ranks the saliency, the saliency calculating unit 280 checks whether or not the captured images that have not yet been ranked remain (step S109), and the saliency of the main subject 27 is still present. When the captured images 41-1 to 41-n that are not digitized remain (step S109: NO), the saliency is digitized and the captured images 41-1 to 41-n are sequentially arranged. (Step S108). In the case where all the captured images 41-1 to 41-n included in the acquired captured image group 410 have been digitized the saliency of the main subject 27 (step S109: YES), the saliency calculating unit 280 The processing is passed to the image selection unit 270.

  Note that the saliency calculating unit 280 calculates the saliency map by normalizing the saliency so that the maximum value of the saliency in each image frame of the captured images 41-1 to 41-n is 1, for example. . Then, an image having the highest saliency in the main subject area is selected. Here, the saliency calculator 280 may normalize and calculate the saliency so that the maximum saliency of the captured images 41-1 to 41-n included in the captured image group 410 is 1. .

  Further, in each captured image frame, the maximum saliency in the frame and the saliency of the main subject area (which may be the average saliency of the main subject area or the maximum saliency of the main subject area) (The saliency of the main subject / the maximum saliency in the frame) is calculated, and based on this value, a captured image having a large value may be selected. With such a configuration, it is possible to evaluate the ease of eye catching of the main subject for each captured image.

  Further, the maximum saliency in the series of captured images is set to 1, and the saliency of the main subject area in each captured image frame is calculated, and the captured image frame having the maximum saliency in the main subject area is selected. It is good.

  Furthermore, the type of characteristic map is not limited to the above example. For example, a characteristic map may be created using feature quantities such as brightness, saturation, hue, and motion. Further, as the characteristic map creation processing, multi-scale contrast, Center-Surround color histogram, color space distribution, or the like may be used.

  Furthermore, when the difference between the quantified saliency levels of the main subject 27 is small in step S108 and it is difficult to rank the captured images 41-1 to 41-n, the saliency map 49-1 is calculated. One of the luminance information map 46-1, the color information map 47-1 and the directional edge information map 48-1 used in the above is used, and the captured images 41-1 to 41-1 are based on the feature amount of the main subject 27 region. 41-n may be ordered. Accordingly, it is possible to assist the ordering of the captured images 41-1 to 41-n without burdening the process of calculating new feature values.

  Furthermore, when the difference between the feature amounts of the main subjects in the captured images 41-1 to 41-n is small and it is difficult to order the captured images 41-1 to 41-n, a scale different from the saliency is introduced. Then, the captured images 41-1 to 41-n may be ordered. As a result, even when it is difficult to rank the saliency, the captured images can be ordered, and the selection accuracy of the captured images can be improved. Each of FIGS. 21 to 25 is a schematic view illustrating a method of ordering captured images that can be executed by the image selection unit 270.

  As illustrated in FIG. 21, the image selection unit 270 may select a captured image using the shooting state of the main subject 27 as an evaluation scale. That is, the subjects 11 to 33 reflected in the captured image 41-4 shown in FIG. 21 are the same as the subjects 11 to 33 reflected in the captured image 41-1, and the arrangement in the captured image 41-4 is also similar. equal.

  However, the captured image 41-4 is captured with an aperture value different from that of the captured image 41-1, and the depth of field of the captured image changes. For this reason, when compared with the captured image 41-1, the contrast of the other subjects 11 to 14, 21 to 26, and 28 to 33 is lower than the contrast of the main subject 27.

  Thus, when the contrast of the main subject 27 in the captured image 41-4 is higher than the other subjects 11-14, 21-26, and 28-33 (when there are many high spatial frequency components), the image selection unit 270 Determines that the main subject 27 is more emphasized in the captured image 41-4, and in the captured image group 410, gives a higher rank to the captured image 41-4 than the captured image 41-1.

  Note that the saliency map 49-1 used in step S109 is calculated in consideration of the luminance and color distribution, and thus the contrast of the captured image 41-1 is also reflected. Therefore, when comparing with the captured image 41-4, the evaluation of the contrast is nothing but the evaluation with weighting the scale called contrast.

  As illustrated in FIG. 22, the image selection unit 270 may select a captured image using the arrangement of the subjects 11 to 33 as an evaluation scale. That is, the subject reflected in the captured image 41-2 is common to the subjects 11 to 33 reflected in the captured image 41-1. However, in the captured image 41-2, the relative positions of the main subject 27 and the other subjects 11 to 14, 21 to 26, and 28 to 33 are different from the captured image 41-1.

  For this reason, the area occupied by the subjects 11 to 14, 21 to 26, and 28 to 33 in the captured image 41-2 is smaller than that of the captured image 41-1. As described above, when the area occupied by the other subjects 11 to 14, 21 to 26, and 28 to 33 in the captured image 41-3 is small, the image selection unit 270 determines that the main subject 27 in the captured image 41-2 is the main subject 27. It is evaluated that the image is relatively emphasized, and a higher order than the captured image 41-1 is given to the captured image 41-2 in the captured image group 410.

  In addition, as illustrated in FIG. 23, the image selection unit 270 may select a photographed image on the basis of the size of the main subject 27 reflected in the photographed image. That is, the subject reflected in the captured image 41-3 is substantially the same as the subjects 11-33 reflected in the captured image 41-1. However, in the captured image 41-3, the main subject 27 is shown relatively large relative to the other subjects 11 to 26 and 28 to 33.

  Thereby, the area occupied by the main subject 27 in the captured image 41-3 is larger than that in the captured image 41-1. Therefore, the image selection unit 270 evaluates that the main subject 27 in the photographed image 41-3 is greatly captured, and the photographed image group 410 in the photographed image group 410 is more than the photographed image 41-1. Give high ranking.

  In the saliency map 433 used in the evaluation in step S109, when the saliency value is calculated and the size evaluation scale of the main subject is included, the size of the main subject 27 in the photographed image is included. Reassessing is nothing but evaluating the weight 27 on the scale of the size of the subject 27.

  In addition, as illustrated in FIG. 24, the image selection unit 270 may select a captured image using the shooting state of the subject 27 as an evaluation scale. In other words, the subjects 11 to 33 reflected in the captured image 41-5 are substantially the same as the subject reflected in the captured image 41-1. However, in the photographed image 41-5, the main subject 27 is strongly illuminated, and the other subjects 11 to 26 and 28 to 33 appear relatively dark.

  Therefore, the image selection unit 270 makes the main subject 27 in the captured image 41-5 appear brighter than the other subjects 11 to 26 and 28 to 33 (the main subject 27 is shown with appropriate brightness). In other words, the captured image 41-5 is given a higher rank than the captured image 41-1 in the captured image group 410.

  Note that the saliency map used in step S109 is calculated by reflecting the luminance distribution. Therefore, the evaluation of the brightness of the main subject 27 in the captured image is nothing but an evaluation with weighting the scale of brightness.

  Also, as shown in FIG. 25, the image selection unit 270 selects a captured image using an evaluation scale that the main subject 27 is located at a predetermined position (for example, the center of the screen) of the captured image. Also good. The subjects 11 to 33 shown in the photographed image 41-6 are common to the photographed image 41-1. However, in the captured image 41-6, the position of the main subject 27 is close to the center of the object scene.

  Thereby, the captured image 41-6 can be evaluated as the main subject 27 being relatively emphasized when compared with the captured image 41-1. Therefore, the image selection unit 270 evaluates that the subject 27 is close to a predetermined position on the screen (for example, the center of the screen) in the captured image 41-6, and in the captured image group 410 with respect to the captured image 41-6. A higher order than the photographed image 41-1 is given. With such a configuration, a captured image having a more appropriate composition can be extracted.

  In this way, the image selection unit 270 evaluates the shooting state of the main subject 27 in each of the shot images 41-1 to 41-n, and the shooting state of the main subject 27 that is more important to the user is the other subject 11. To 26 and 28 to 33, images that are in a better shooting state and as a result the main subject 27 is emphasized are selected, and the plurality of captured images 41-1 to 41-n are ranked and ordered. Can be

  The evaluation of the main subject 27 can be executed by all or part of the methods exemplified so far. Further, when the evaluation of the main subject 27 is executed by a plurality of methods, the execution order may not be the order described above. Furthermore, the above evaluation method is only a part of the example, and other evaluation items or evaluation methods may be used in combination.

  Refer to FIG. 10 again. The ordering of all the photographed images 41-1 to 41-n is completed (step S109: YES), and the image selection unit 270 to which the process has been passed selects the photographed images according to the order (step S110). Thereby, the image selection unit 270 compares the saliency of the main subject 27 between the plurality of photographic images 41-1 to 41-n, and the photographic images 41-1 to 41-1 are based on the saliency of the main subject 27. 41-n can be selected.

  In other words, the image selection unit 270 selects a photographed image based on the order of the ordered photographed image group 410 according to a predetermined selection rate, threshold value, number of frames, and the like. That is, when the selection rate is designated in advance, a certain ratio of the captured images 41-1 to 41-n forming the captured image group 410 is selected from the top of the order. Alternatively, the image selection unit 270 may select from the lower order of the order.

  If a threshold is set in advance, a captured image in which the saliency of the main subject 27 exceeds a given value is selected. Further, when the number of frames is determined in advance, a photographed image having a given number of frames is selected from the upper or lower order of the above order. In this way, the image selection unit 270 selects a captured image according to an order formed based on the saliency of the main subject, so that the user can select a captured image that reflects the shooting intention without taking time and effort.

  The captured images 41-2 to 41-6 selected by the image selection unit 270 as described above are preferentially presented to the user when the digital camera 100 is set to the playback mode, for example. Thereby, the user can shorten the selection work time when selecting a captured image from a large number of captured images. In addition, the digital camera 100 may automatically discard a photographed image that has a particularly low evaluation, or may not display the captured image until it is instructed to be displayed by the user.

  As described above, by using the saliency map, it is possible to quantify the user's subjective value as to whether or not a subject that is highly interested in the user is prominent in each captured image (whether it is conspicuous and easy to catch attention). Can be expressed. Therefore, the image selection unit 270 selects an image by evaluating whether or not the main subject is more prominent in each captured image.

  Thereby, the user can save the trouble of extracting the selected image from many photographed images. Further, instead of omitting all the selections by the user, the selection range by the image selection unit 270 may be widened to reduce the user's selection effort.

  FIG. 26 is a diagram schematically illustrating a personal computer 500 that can execute a captured image processing program equivalent to the processing in the captured image processing unit 203. The personal computer 500 includes a display 520, a main body 530, and a keyboard 540.

  The main body 530 can acquire image data of a photographed image from the digital camera 100 through communication with the digital camera 100. The acquired image data can be stored in a storage medium of the personal computer 500. The personal computer 500 also includes an optical drive 532 used when loading a program to be executed.

  The personal computer 500 as described above operates as a device that executes a captured image processing program that executes the procedure shown in FIGS. 4 and 10 by reading the captured image processing program. Further, the personal computer 500 can acquire photographed image data from the digital camera 100 via the cable 510 and make it a target of processing.

  That is, the captured image processing program is based on a subject extraction stage for extracting a plurality of different subjects included in a plurality of images, and on which position each of the plurality of subjects is in each of the plurality of images. A main subject estimation stage for estimating which of a plurality of subjects is a main subject of the plurality of subjects, and a saliency calculation for calculating a distribution of saliency according to the intensity of visual stimulus for each of the plurality of images The computer executes a step and an image selection step of comparing the saliency of the main subject between the plurality of images and selecting any one of the plurality of images based on the saliency of the main subject.

  Thus, the user can easily operate with the larger display 520 and the keyboard 540. By using the personal computer 500, a larger amount of images can be processed at high speed. In addition, the number of evaluation items in each of the subject extraction procedure, the main subject estimation procedure, and the image selection procedure may be increased, and the evaluation unit may be made finer. Thereby, image selection can be assisted while reflecting the user's intention in more detail.

  Note that the transfer of captured image data between the digital camera 100 and the personal computer 500 may be via a cable 510 as shown in the figure, or may be wireless communication. Further, the image data may be acquired by delivering a secondary storage medium in which the image data is stored. The execution of the photographed image processing program is not limited to the personal computer 500, and may be executed at a storefront or an online print service facility.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  In addition, the execution order of each process such as operation, procedure, step, and stage in the apparatus, system, program, and method shown in the claims, the specification, and the drawings is particularly “before”, “ It should be noted that it can be realized in any order unless it is clearly indicated as “prior to” or the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it does not necessarily mean that implementation in this order is essential. Absent.

11-33 Subject, 41-1, 41-2, 41-3, 41-4, 41-5, 41-6, 41-n Photographed image, 42-1 Luminance image, 43-1 Gaussian pyramid, 44-1 Normalized image, 45-1 Luminance difference image, 46-1 Luminance information map, 47-1, 47-2 Color information map, 48-1, 48-2, 48-3, 48-4 Directional edge information map, 49-1, 49-2, 49-3 Saliency map, 100 digital camera, 110 housing, 120 lens barrel, 122 taking lens, 130 light emitting window, 140 operation unit, 141 cross key, 142 power switch, 143 back button 144 Release button, 146 Zoom lever, 150 Rear display unit, 200 Internal circuit, 201 Control unit, 202 Imaging unit, 203 Captured image processing unit, 210 CPU 220 display drive unit, 222 saliency map generation unit, 230 program memory, 240 main memory, 250 subject extraction unit, 260 main subject estimation unit, 270 image selection unit, 280 saliency calculation unit, 310 image sensor drive unit, 312 imaging Element, 320 Analog / digital conversion unit, 330 Image processing unit, 332 Secondary storage medium, 340 Automatic focusing unit, 350 Photometric sensor, 360 Variable magnification drive unit, 370 Input reception unit, 410 Captured image group, 421, 422 Field, 500 personal computer, 510 cable, 520 display, 530 main unit, 532 optical drive, 540 keyboard, 610 luminance information extraction unit, 612 luminance image generation unit, 614, 623, 635 Gaussian pyramid generation unit, 616, 624, 636 difference Image generation unit, 618 luminance information map generation unit, 620 color information extraction unit, 621 RG differential image generation unit, 622 BY differential image generation unit, 623 Gaussian pyramid generation unit, 625 color information map generation unit, 630 edge Information extraction unit, 640 linear combination unit, 631, 632, 633, 634 Directional edge image generation unit, 637 Directional edge information map generation unit

Claims (10)

An image acquisition unit for acquiring a plurality of images;
A main subject estimation unit for estimating a main subject in the plurality of images;
A saliency calculator that calculates a visual saliency for each of the plurality of images;
An image processing apparatus comprising: an image selection unit that compares the saliency of the main subject between the plurality of images and selects any one of the plurality of images based on the saliency of the main subject. . A subject extraction unit that extracts a plurality of different subjects included in the plurality of images;
The image processing apparatus according to claim 1, wherein the main subject estimation unit estimates which of the plurality of subjects is a main subject.   The image processing apparatus according to claim 1, wherein the image selection unit selects the image when the saliency of the main subject in each of the plurality of images is larger.   The saliency calculating unit calculates the saliency distribution by combining the distributions of luminance values, colors, and directional edges in each of the plurality of images. The image processing apparatus according to item.   The image processing apparatus according to claim 4, wherein the saliency calculating unit calculates the saliency distribution by combining distributions individually normalized for each of the luminance value, color, and directional edge.   The image processing device according to claim 5, wherein the image selection unit selects any one of the plurality of images further based on any one of the distributions of the luminance value, the color, and the directional edge. .   The image processing apparatus according to claim 1, wherein the image acquisition unit acquires a plurality of images taken continuously in time as the plurality of images. In the image processing device according to any one of claims 1 to 7,
The image selection unit further calculates a ratio between the maximum value of the saliency of each of the plurality of images and the saliency of the main subject for each of the plurality of images, and the plurality of the plurality of images based on the ratio An image processing apparatus, wherein any one of the images is selected. An image processing apparatus according to any one of claims 1 to 8,
An imaging device comprising: an imaging unit that captures image light of an object scene. A subject extraction stage for extracting a plurality of different subjects included in a plurality of images;
A main subject estimation step for estimating a main subject in the plurality of images;
A saliency calculating step for calculating a visual saliency for each of the plurality of images;
Comparing the saliency of the main subject between the plurality of images, and causing a computer to execute an image selection step of selecting any one of the plurality of images based on the saliency of the main subject Image processing program.