JP2012049815A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect a subject characteristic to a photographing person without causing a large load on the photographing person.SOLUTION: An image processing device comprises: means for obtaining the local feature of an image whenever an image is obtained; means for registering or deleting the obtained local feature; means for determining, on the basis of comparison between the obtained local feature on the image and local feature registered in the past, a region in which a past image and the obtained image are similar; and means for obtaining a subject region on the basis of the frequency with which the regions of the combination of the similar regions have overlapping portions.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly to a technique suitable for use in detecting a subject area.

A method of detecting a subject at the time of shooting is widely used for controlling shooting parameters.
For example, Non-Patent Document 1 discloses a technique for detecting a subject area from an image. This Non-Patent Document 1 describes a method of preparing descriptors for all partial images, statistically acquiring their fluctuation ranges, and creating matching patterns. In Non-Patent Document 1, an arbitrary rectangular area in the image is used for the partial image, and the comparison result of the light and shade of the image of two areas determined in advance in the rectangular area is used for the descriptor. Further, a Boosting algorithm is used for obtaining a statistical fluctuation range.

US Pat. No. 6,711,293 JP 2009-69996 A JP 11-136568 A

Viola and Jones, "Rapid object detection using boosted cascade of simple features", Computer Vision and Pattern Recognition, 2001 M. A. Fischler and R. C. Bolles, "Randomsample consensus: A paradigm for model fitting with applications to image analysis and automated cartography," Commun. ACM, no.24, vol.6, pp.381-395, June 1981.

  When a subject area is detected in photography such as a photograph / moving picture, there are few cases where a photographer has a common subject to be detected. In many cases, the subject to be detected is unique to the photographer, such as several family members or acquaintances of the photographer, or a single pet kept by the photographer. Therefore, a technique for detecting a subject unique to a photographer without causing a large load on the photographer is desired. In detecting the subject area at the time of shooting, the method described in Non-Patent Document 1 requires many images in order to obtain the statistical fluctuation range, and the image collection imposes a burden on the photographer. ing. Further, the amount of calculation necessary for calculating the fluctuation range increases as the number of collected images increases.

  In view of the above-described problems, an object of the present invention is to make it possible to detect a subject unique to a photographer without causing a heavy load on the photographer.

  The image processing apparatus according to the present invention includes means for acquiring local features of an image every time an image is acquired, means for registering or deleting the acquired local features, and local features on the acquired image registered in the past. Means for obtaining a similar area between the past image and the acquired image based on comparison with the local feature, and means for acquiring a subject area based on a frequency at which each combination of the similar areas has a partial area overlapping. It is characterized by providing.

  According to the present invention, a subject unique to a photographer can be detected without causing a large load on the photographer.

It is a block diagram which shows the hardware structural example of the image processing apparatus which concerns on embodiment. It is a block diagram which shows the function structural example of the image processing apparatus which concerns on embodiment. It is a flowchart which shows an example of the process sequence which calculates | requires a local feature. It is a flowchart which shows an example of the detailed procedure of step S305 of FIG. It is a flowchart which shows an example of the detailed procedure of step S313 of FIG. 4 is a flowchart illustrating an example of a detailed procedure of step S315 in FIG. 3. 6 is a flowchart showing an example of detailed procedures of step S403 of FIG. 4, step S503 of FIG. 5, and step S603 of FIG. It is a figure which shows an example of the image image | photographed in the past. It is a figure which shows an example of a trial image. It is a figure which shows an example of a picked-up image. It is a figure which shows an example of imaging | photography composition.

Hereinafter, preferred embodiments to which the present invention is applied will be described.
FIG. 1 is a block diagram illustrating a configuration example of an image processing apparatus according to the present embodiment.
In FIG. 1, an imaging system 101 is composed of a lens, a CCD, a MOS, and the like, and converts a subject image from light to an electrical signal. Reference numeral 102 denotes a signal processing circuit that converts an electrical signal relating to a subject image obtained from the imaging system 101 into a digital signal, and sets imaging conditions and the like for the imaging system 101.

  A CPU 103 controls the entire apparatus by executing a control program stored in the ROM 104. A ROM 104 stores a control program executed by the CPU 103 and various parameter data. The control program is executed by the CPU 103, thereby causing the apparatus to function as various means for executing each process shown in a flowchart described below. Reference numeral 105 denotes a RAM which stores image data and various types of information. The RAM 105 functions as a work area for the CPU 103 and a temporary data saving area.

  Reference numeral 106 denotes a touch display, which includes, for example, an LCD or a CRT. Further, the touch display 106 presents information to the user, accepts the user's contact operation, and acquires the coordinate information. The touch display 106 is also used to display the video signal that has passed through the imaging system 101 and the signal processing unit 102 as it is. Hereinafter, the image displayed here will be referred to as a preview image.

  In the present embodiment, processing corresponding to each step of the flowchart described later is realized by software using the CPU 103, but part or all of the processing is realized by hardware such as an electronic circuit. It doesn't matter. In addition, the image processing apparatus of the present embodiment may be realized using a general-purpose PC by omitting the imaging system 101 and the signal processing unit 102, or may be realized as an image processing dedicated apparatus. Further, software (program) acquired via a network or various storage media may be executed by a processing device (CPU, processor) such as a personal computer.

FIG. 2 is a block diagram illustrating a functional configuration example of the image processing apparatus according to the present embodiment.
In FIG. 2, an image acquisition unit 201 corresponds to the imaging system 101, and acquires a trial image and a captured image. The trial image is used when deriving a shooting parameter for determining a shot image or determining a composition. An example of the captured image is shown in FIG. 10A, and examples of the trial image are shown in FIGS. 9A and 9B.

  The subject area instruction unit 202 corresponds to a part of the touch display 106, displays the sample image acquired by the image acquisition unit 201, and accepts a subject instruction for the image. When the instruction is received, a subject area 941 is shown as shown in FIG. 9B, for example. The local feature acquisition unit 203 acquires a local feature from the sample image or the captured image acquired by the image acquisition unit 201. Stars 921 to 929 and 951 to 958 in FIGS. 9A and 9B indicate local features of the trial image. Further, stars 1011 to 1017 in FIG. 10B indicate local features acquired from the captured image shown in FIG. These local features are composed of local features obtained from the coordinates and the periphery thereof, and are calculated by the method described in Patent Document 1, for example. In addition, this local feature may be implemented as long as it is robust to rotation, enlargement / reduction, and out-of-plane rotation.

  The local feature DB unit 204 registers and deletes local features in response to requests from the local feature learning unit 207. Further, the local feature is searched according to the request of the local feature comparison unit 205. In the present embodiment, it is not always necessary to store past images. Stars in FIG. 8A to FIG. 8C indicate local features registered when the images were captured in the past.

  The local feature comparison unit 205 calculates a similar region included in an arbitrary image pair based on comparison of local features of the images. The local features are the local features obtained by the local feature obtaining unit 203 and the local features of the past images registered in the local feature DB unit 204 with respect to the trial image and the photographed image obtained by the image obtaining unit 201. is there. As described in the local feature acquisition unit 203, the local feature is composed of its coordinates and feature amount. In the present embodiment, it is assumed that the similar region obtained by the local feature comparison unit 205 is a polygon including coordinates of local features determined to be similar as a result of the local feature comparison. Therefore, in the present embodiment, the expression of the similar area is substituted with a combination of local features constituting the similar area for convenience. A similar region obtained by comparing local features between the preview image shown in FIG. 9A and the images taken in the past shown in FIGS. 8A to 8C is, for example, a star in FIG. 9A. An area of marks 921 to 923, an area of stars 921, 922, and 925, and stars 931 to 934.

  The subject region acquisition unit 206 obtains a subject region on the image based on the similar region obtained by the local feature comparison unit 205 or the subject instruction by the subject region instruction unit 202. The subject region 911 in FIG. 9A is obtained based on the similar region obtained by the local feature comparison unit 205 based on the sample image shown in FIG. Also, the subject area 941 in FIG. 9B is obtained based on the instruction of the subject by the subject area instruction unit 202 based on the preview image shown in FIG. 9B. In the present embodiment, when the subject area is obtained based on the similar area, the subject area is equal to one of the similar areas. Therefore, as with the local feature comparison unit 205, the notation of the subject area is substituted with a combination of local features that constitute the subject area for convenience.

  The local feature learning unit 207 registers and deletes local features in the local feature DB unit 204. The local features to be registered / deleted change depending on whether or not the subject is instructed by the subject region instruction unit 202. When a subject is instructed, only local features in the subject area are registered for the captured image. In addition, local features not included in the subject area are deleted from past images including the subject. When the subject area 1002 shown in the photographed image of FIG. 10B is used, the local features to be registered are stars 1011 to 1013, and the local features to be deleted are the stars shown in FIGS. 8A and 8C. 801 to 803 and 841 to 844. If there is no instruction for the subject, all the local features of the photographed image are registered. In either case, old local features that satisfy a certain condition, such as local features that do not constitute a subject, are deleted from local features acquired in the past.

  FIG. 3 is a flowchart illustrating an example of a processing procedure for obtaining local features in the present embodiment. This process is roughly divided into a shooting condition adjustment process in steps S301 to S307, a main shooting process in steps S308 to S309, and a subject registration process in steps S310 to S317. In this process, a local feature group for each past image of the local feature DB unit 204 is input.

  First, in step S301, as the shooting condition adjustment process, steps S302 to S307 are repeated until there is a shooting instruction (recording instruction). Specifically, it is assumed that the shooting instruction is pressing the shutter button. In this shooting condition adjustment process, shooting conditions such as composition and shooting timing are adjusted and confirmed, and detection and instruction of the subject area are mainly used for adjusting the shooting conditions.

  In step S <b> 302, a trial image is acquired from the image acquisition unit 201 and displayed on the touch display 106. Next, in step S303, the subject area instruction unit 202 determines whether or not a subject has been designated by the user. If the result of this determination is that there is a designation, the process proceeds to step S306, and if there is no designation, the process proceeds to step S304.

  In step S304, the local feature acquisition unit 203 obtains local features indicated by stars 921 to 929 from, for example, a preview image shown in FIG. Next, in step S305, a region that is similar between the past image and the trial image is obtained, and a region that is most similar to the past image on the sample image is selected, and is set as a subject region. Details of step S305 will be described later.

  On the other hand, as a result of the determination in step S303, if the subject is designated by the user, the subject region is acquired from the user's operation on the subject region instruction unit 202 in step S306. In this embodiment, it is assumed that the user's operation on the subject area instruction unit 202 draws an area on the trial image as shown in FIG. 9B. Existing object detection technology and tracking technology can be used to acquire the subject area. For example, in this embodiment, a similar color histogram search is used. Specifically, a color histogram in the subject area 941 is obtained, and each time a trial image is updated, an area having a color histogram that is most similar to the obtained color histogram is set as the subject area. If no subject area is obtained, the subject area may not exist.

  In step S307, the obtained subject area is drawn on the image on the subject area instruction unit 202. If no subject area is obtained, the subject area is not drawn. Here, for example, even if the obtained subject area is a polygon, an ellipse or a rounded rectangle circumscribing the obtained polygon is displayed. Thereby, the amount of unnecessary information can be reduced and the display can be refined. Also, based on the type of method in which the subject area is detected, the color, shape, solid broken line, etc. are changed and displayed. This clearly indicates to the user the technique used for subject detection. In addition, if subject detection technology is implemented, colors and line types corresponding to the detection method are also used.

  After a shooting instruction is generated and the process exits the loop of steps S302 to S307, actual shooting is performed in steps S308 and S309. First, in step S308, focus, white balance, aperture, exposure time, and the like are obtained based on the subject area. The obtained shooting parameters and the like are set in the signal processing unit 102, and some of them are set in the imaging system 101. In step S309, an image is shot (main shooting). The captured image is stored in a memory card or the like and displayed on the touch display 106 or the like and presented to the user.

  Next, as subject registration processing, in steps S310 to S317, registration and deletion of local features in the local feature DB unit 204 are performed for subsequent subject region detection. First, in step S310, local features are obtained from the image captured in step S309. The local feature is acquired using the same method as in step S304. Step S310 is executed every time the image is taken in step S309, and the local feature obtained in this step is registered in the local feature DB unit 204 in step S312 or step S316. For this reason, the acquisition of the subject area in step S305 can always use the local feature obtained from the latest photographing.

  Next, in step S311, it is determined whether there is an explicit designation of the subject from the photographer in the loop of step S301. As a result of the determination, if there is no designation, the process proceeds to step S312, and if there is a designation, the process proceeds to step S314.

  In step S312, all local feature groups included in the captured image are added to the local feature DB unit 204. Thereby, when there is no explicit designation of the subject from the user, the subject included in an arbitrary position in the captured image can be detected in step S305 in the future. Next, in step S313, a local feature group that does not constitute the main subject is obtained from the local feature groups of the past image group. Details of step S313 will be described later. Note that stars 821 and 822 shown in FIG. 8B indicate local features determined not to constitute the main subject.

  On the other hand, if there is an explicit designation of the subject from the user as a result of the determination in step S311, first, in step S314, a subject region in the captured image is obtained. A subject area 1002 shown in FIG. 10B shows an area designated by the user. This calculation method is the same as in step S306.

  Next, in step S315, among the local features included in the past image having a region similar to the subject region of the photographed image, local features not included in the subject region are obtained. Details of this process will be described later. Note that stars 801 to 803 in FIG. 8A indicate local features determined not to be included in the subject area. These local features are deleted from the local feature DB unit 204 in step S317. As a result, the local feature determined not to be included in the subject is not used in future subject detection processing, and as a result, it is difficult to detect as a subject.

  Next, in step S316, a local feature group included in the subject area of the captured image is added to the local feature DB unit 204. Note that stars 1011 to 1013 in FIG. 10B indicate local feature groups included in the subject area 1002 of the captured image. By not registering local features that are not subject areas in this way, it is possible to prevent a decrease in the subject detection rate and an increase in the use memory and calculation amount in later photographing.

  Next, in step S317, local features that do not constitute a subject area and local features that have passed a specified time or a predetermined number of shots after registration are deleted. Moreover, it deletes from the oldest local feature until it falls below the upper limit of the number of local features set in advance. Here, the local features that do not constitute the subject region are the local feature groups obtained in step S313 or step S315. By deleting local features that have passed a specified time or a specified number of shots, it is possible to set an upper limit for the calculation amount and the required memory amount. Further, by deleting the local features that do not constitute the subject, it is possible to prevent the background region from being detected as the subject region in subsequent photographing.

  FIG. 4 is a flowchart illustrating an example of a detailed processing procedure in step S305. In this flowchart, a region similar to the sample image and the past image group is obtained, and a region having the largest number of times of overlapping on the sample image is selected. Each process shown in FIG. 4 is performed by the subject area acquisition unit 206. In this process, the local feature group of the trial image and the local feature group for each past image included in the local feature DB unit 204 are input.

  In steps S401 to S404, similar regions are obtained for all combinations of past images and sample images. First, in step S401, a list of similar regions and their appearance frequencies is initialized. Next, in step S <b> 402, the processes in steps S <b> 403 to S <b> 404 are repeated for each past image included in the local feature DB unit 204.

  In step S403, a similar area common to the past image of interest and the preview image is obtained. Details of step S403 will be described later. For example, the region surrounded by asterisks 921 to 923 shown in FIG. 9A is similar to the region surrounded by asterisks 811 to 813 shown in FIG. 9A is similar to the region surrounded by stars 831, 832, and 835 shown in FIG. 8C. The region surrounded by stars 921, 922, and 925 shown in FIG. Further, the area surrounded by the star marks 926 to 929 shown in FIG. 9A is similar to the area surrounded by the star marks 841 to 844 shown in FIG. In step S404, if a similar region exists in step S403, the similar region is added to the similar region list, and the appearance frequency is set to 1.

  Next, in step S405, it is determined whether or not there is at least one similar region in all combinations of past images and trial images. If there is no similar area as a result of this determination, it is determined that no subject is found on the trial image, and this process is terminated.

  On the other hand, as a result of the determination in step S405, if a similar region exists, in step S406, the processing in step S407 is repeated for all combinations of similar regions, and the number of times of overlapping with other similar regions is obtained. In step S407, an overlapping area is obtained between two similar areas, and if it exists, one is added to the appearance frequency of each similar area. This overlap may be obtained by, for example, expressing both similar regions in a bitmap and obtaining a logical product, or there is an intersecting side between two partial regions, or one partial region is included in the other. It is sufficient to test whether it is. For example, the region surrounded by the asterisks 921 to 923 and the region surrounded by the asterisks 921, 922, and 925 shown in FIG. 9A overlap each other, and the appearance frequency thereof is 2.

  Finally, in step S408, a similar region having the maximum appearance frequency is selected and set as a subject region. When a plurality of similar regions have the same number of appearance frequencies, for example, a union of similar regions is used. For example, in the case where the regions surrounded by stars 921 to 923 overlap with the regions surrounded by stars 921, 922, and 925, the union stars 921, 922, A region surrounded by 923 and 925 is set as a subject region. Also, a subject area is set based on the appearance frequency. For this reason, for example, even if a region surrounded by stars 921 to 923 and a region surrounded by stars 931 to 934 are obtained as similar regions, stars 921 to 923 having a high appearance frequency in the past image group are obtained. The enclosed area can be selected as a subject.

  Furthermore, the past image may include a past image that does not include the subject as shown in FIG. Furthermore, the frequency of local features constituting the subject can be improved with a minimum of two shootings without a zoom operation or a photographer's movement. For example, in the environment illustrated in FIG. 11, the appearance frequency of the areas included in both the frames 1101 and 1102 is improved by performing shooting with the compositions illustrated in the frames 1101 to 1102.

  FIG. 5 is a flowchart illustrating an example of a detailed processing procedure of step S313. In this process, a local feature group that does not constitute the main subject is obtained from the local feature groups of the past image group included in the local feature DB unit 204. Specifically, the number of times that all local features are included in a similar region between past images is obtained, and a local feature with a small number of times is selected.

First, in step S501, a list of appearance frequencies of local features is initialized.
Next, in step S502, steps S503 to S504 are repeated for all combinations of different past images <A, B> in the past image group. Here, as an example, the image shown in FIG. 8A is a past image A, and the image shown in FIG.

  In step S503, a similar region between the past image A and the past image B is obtained. This step S503 is the same processing as step S403 in FIG. 4, and details will be described later. For example, similar regions in the past image A and the past image B are a region surrounded by asterisks 811 to 813 and a region surrounded by asterisks 831 to 833.

  Next, in step S504, for each of the past image A and the past image B, the local features constituting the similar region and the appearance frequency of the local features included in each region are increased. For example, since the region surrounded by the stars 831 to 833 includes the local feature indicated by the star 834, not only the local features indicated by the stars 811 to 813 and 831 to 833 but also the local feature indicated by the star 834 is displayed. Increases the frequency of appearance of features. The local feature whose appearance frequency has increased is less likely to be deleted from the local feature DB unit 204 in step S317.

  Here, in general, a local feature included in a certain similar region is likely to constitute the region regardless of whether or not it is used for detection of the similar region. Even if the same subject is projected, the local features obtained from the image vary depending on the shooting situation. Therefore, even if local features that have not been deleted do not have similar local features in the past image, the possibility of having similar local features, i.e. subject detection possibility, in subject detection in future shooting of the same subject, Improve.

  In step S505, among all the local features, a local feature having a low appearance frequency added in step S504 and satisfying a predetermined condition is selected. Here, the predetermined condition is a condition set in order to prevent the deletion of the local features constituting the subject included in the image immediately after shooting. Specifically, it is assumed that a predetermined number of times or time must have passed after acquiring local features. For example, the local features with a low appearance frequency include stars 801 to 803, 821, 822, and 841 to 844. Assuming that two images have been taken as a default condition for selecting a local feature, the local features indicated by stars 801 to 803 are selected.

  FIG. 6 is a flowchart illustrating an example of a detailed procedure of step S315 in FIG. This process is to obtain local features that are included in a past image having a subject region and a similar region of a photographed image and that are not included in the subject region. Further, the local feature group of the subject area of the captured image and the local feature group for each past image are input.

  First, in step S601, a local feature deletion list is initialized. Next, in step S602, step S603 to step S605 are repeated for each past image in the past image group. In step S603, a similar partial area is obtained between the subject area of the captured image and the past image. This step S603 is the same processing as step S403 in FIG. 4, and details will be described later. In this process, for example, the subject area 1001 shown in FIG. 10B is taken as a captured image, and the area surrounded by stars 811 to 813 and the area surrounded by stars 831 to 833 are similar to past images. To do.

  Next, in step S604, it is determined whether a similar area exists in step S603. As a result of the determination, if a similar region exists, the process proceeds to step S605, and if it does not exist, the loop of steps S603 to S605 in step S602 is continued.

  In step S605, all the local features not included in the similar region among the local features included in the past image focused in step S602 are added to the deletion list. For example, when the region surrounded by the stars 831 to 833 in the past image shown in FIG. 8C is a similar region, the local features indicated by the stars 841 to 844 not included in the similar region are added to the deletion list. To do.

  Here, the local feature indicated by the asterisk 834 does not have a similar local feature, but is included in the area surrounded by the asterisks 831 to 833, and thus is not added to the deletion list. As described above, as in step S504 in FIG. 5, it is assumed that local features included in a certain area constitute a subject and are not actively deleted. Thereby, for example, when the subject indicated by the star 831 to 833 is photographed in the future and a local feature similar to the local feature indicated by the star 834 is obtained, the possibility of detecting the subject is increased. it can.

  FIG. 7 is a flowchart illustrating an example of a detailed procedure of step S403, step S503, and step S603. This process is a process for obtaining a similar region commonly included in the pair of images <A, B>, and is performed by the local feature comparison unit 205. Further, this processing is close to an algorithm known as RANSAC shown in Non-Patent Document 2. This flowchart includes similar local feature collection processing between images in steps S701 to S704, inter-image transformation matrix acquisition processing in steps S705 to S709, and similar region acquisition processing in steps S710 to S713.

  First, in steps S701 to S704, pairs of local features that are similar between images are collected. Therefore, in step S701, a list of corresponding points between the local features in the image A and the local features in the image B and their similarity frequencies are prepared and initialized. Next, the process of step S703 is repeated for all combinations of the local feature A in the image A and the local feature B in the image B from step S702. In step S703, if the difference between the local feature A in the image A and the local feature B in the image B is equal to or smaller than the threshold value, the local feature A and the local feature B are added to the corresponding point list, and the similarity frequency is set to zero. . The method for deriving the difference between the two local features is arbitrary, and may be Manhattan distance or Euclidean distance, or some distance function may be provided.

  Next, in step S704, it is determined whether the number of entries in the corresponding point list obtained in step S703 is three or more. As a result of the determination, if it is 3 or more, the process proceeds to step S705. If it is 2 or less, it is determined that the two images do not have a similar area, and the process ends.

  Next, in steps S705 to S709, a maximum likelihood conversion matrix from the image A to the image B is obtained. First, in step S705, steps S706 to S708 are repeated until the convergence condition is satisfied. Here, the convergence condition is that a sufficient number of times is statistically meaningful. Specifically, in the case of the present embodiment, if the number of entries in the corresponding point list is 5 or less, the number of entries is repeated.

  First, in step S706, two arbitrary pairs are selected from the corresponding point list, and a conversion matrix from image A to image B is obtained based on the two pairs of coordinates. This transformation matrix is an affine transformation matrix, which is obtained from an arbitrary pair of two points, and expresses enlargement / reduction, translation, and rotation collectively. Next, in steps S707 to S708, it is verified whether or not the two pairs set in step S706 are likely. Specifically, first, some pairs of points <a, b> included in the corresponding point list obtained in step S703 are selected. Next, for each of them, the point a on the image A is converted into the point a ′ on the image B by the matrix obtained in step S706. Thereafter, if the difference between b and a ′ is equal to or smaller than the threshold value, three pairs of similar frequencies in the corresponding point list are added. Here, the method for deriving the difference between the two local features is the same as in step S703.

  Next, in step S709, two pairs having the maximum similarity frequency in the corresponding point list are selected, and a conversion matrix from image A to image B is obtained based on the two pairs of coordinates.

  Next, in steps S710 to S713, a region similar between the two images is obtained. First, in step S710, step S711 is repeated for each pair <a, b> of point C in the corresponding point list. In step S711, the point a in the image A is converted to the point a ′ on the image B by the conversion matrix obtained in step S709. If the distance between a ′ and b is equal to or greater than the threshold, the pair <a, b> is associated. Delete from the point list. Here, the method for deriving the difference between the two local features is the same as in step S703.

  Next, in step S712, it is determined whether or not the image A and the image B have a similar region. At this time, even if an entry in the corresponding point list is deleted in steps S710 to S711, if the number of entries remains three or more, it is determined that the similar area is present. As a result of this determination, if the number of entries is 3 or more, the process proceeds to step S713. If it is 2 or less, the processing is terminated assuming that the image A and the image B do not have a similar area.

  Finally, in step S713, for each of the images A and B, a polygon that includes the points remaining in the corresponding point list is obtained. The polygons on the images A and B are similar regions that are included in both the images A and B in common.

  As described above, the present embodiment includes the step S317 of determining a local feature that has passed a predetermined number of shots after registration, determining a local feature that has passed a predetermined time after registration, and deleting the determined local feature. Thereby, the upper limit of the number of local features can be suppressed, and the upper limit of the calculation amount in the flowchart of FIG. 7 can be limited.

  Further, in the present embodiment, in step S313 for determining a local feature that does not constitute the main subject, a step S503 for obtaining a subject region for an arbitrary combination of past images is provided. Furthermore, step S504 which calculates | requires the frequency | count included in the to-be-photographed object area | region for every local feature, and step S505 which determines whether a main subject is comprised based on the said frequency | count are provided. This improves the subject detection possibility in shooting the same subject.

  Further, in the present embodiment, step S202 for receiving an instruction of a subject area on the acquired image, and step S604 for determining whether or not the registered local feature has an area similar to the subject area on the acquired image. Is provided. Furthermore, steps S605 and S317 are included for deleting local feature groups that are included in the image determined to have the similar region and are not included in the similar region. Thereby, it is possible to prevent a local feature once instructed not to be included in the subject from being detected in future imaging.

  Patent Document 2 discloses a technique for detecting a main subject in an image based on automatic measurement of local saliency, that is, how conspicuous a part is with respect to the background. However, in the method described in Patent Document 2, the result is undefined when a region having higher local saliency enters the background, for example, when a butterfly is photographed in a flower field. On the other hand, in the present embodiment, in the determination of the subject area based on step S408, the subject area is based on the detection of the similar area based on the local feature and the frequency thereof. That is, the local feature obtained from the background is not considered in the derivation of the frequency of the similar region, and is robust against changes in the background.

  Patent Document 3 discloses a subject detection method based on a user interface for designating a main subject area on a touch panel. The technique described in Patent Document 3 requires a user operation immediately before shooting, but it is difficult to perform a user operation in shooting a moving object such as a railway or an object with an undefined trajectory such as a flying butterfly. On the other hand, in the present embodiment, the operation of the touch panel is not necessary for the detection of the subject area in step S305.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

201 Image acquisition unit, 202 Subject area instruction unit, 203 Local feature acquisition unit, 204 Local feature DB unit, 205 Local feature comparison unit, 206 Subject region acquisition unit, 207 Local feature learning unit

Claims (3)

Means for acquiring local features of the image each time an image is acquired;
Means for registering or deleting the acquired local features;
Means for obtaining a similar region between the past image and the acquired image based on a comparison between the local feature on the acquired image and a local feature registered in the past;
An image processing apparatus comprising: means for acquiring a subject region based on a frequency at which each combination of similar regions has a partial region that overlaps. Acquiring local features of the image each time an image is acquired;
Registering or deleting the acquired local features;
Obtaining a similar region between the past image and the acquired image based on a comparison between the local feature on the acquired image and a local feature registered in the past;
And a step of acquiring a subject region based on a frequency at which each combination of similar regions has a partial region that overlaps.   A program for causing a computer to execute each step of the image processing method according to claim 2.

Images