JP2013165321A - Image processor and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To infer a subject which a user intends by learning and to automatically determine it as a main subject without troubling the user.SOLUTION: An image, a name, a number n, a feature amount h, the number of main subject times c and main subject time t of a face of a registered person are stored in a personal information list L1. A feature detection section 124 detects a position and a size of photographed image in a face part included in the image for each frame and they are stored as a detection result list L2. A number n and a center position of the detected face part, the size of the face part, an authentication result (a), main subject information m, the number of main subject times c2, main subject time t2 and the like are stored in a subject list L3. When the number I2 of elements where the authentication result (a) is not zero is two or more in the subject list L3, the main subject is determined on the basis of the number of main subject times c and the main subject time t in the respective elements [a] in the personal information list L1 having the authentication result (a) whose value is the same as the authentication results (a).

Description

  The present invention relates to an image processing apparatus and method for detecting a characteristic portion such as a face portion of a subject image, specifying a main subject from the detected feature portion, and performing camera control or the like.

  In recent years, a still camera that can detect a face part as a feature part from a photographed image and compare it with personal information registered and held in advance to identify who the face part is. Cameras such as surveillance cameras and video cameras have appeared. In such a photographing apparatus, information (feature amount) representing the position of the detected facial organ (eyes, nose, mouth, etc.) and appearance such as skin color is extracted. In addition, in the registered personal information, a feature amount and a name of a person (feature portion) are recorded. In addition, there is a still camera that can set a priority used when a registered person is selected as a main subject.

  In Patent Document 1, when a plurality of persons registered with personal information are simultaneously detected from a photographed image, a digital camera that focuses and exposures the person set with the highest priority set in the personal information. Is described.

  In addition, such a process mainly targeting a person set with the highest priority in personal information is not performed only in the above-described photographing apparatus, but also in an application on a personal computer. To be implemented. For example, brightness correction or white balance correction is performed on the entire image so that the person with the highest priority is set to an appropriate value, or the size or the size of the person with the highest priority is set. The coordinates may be stored in association with the image.

JP 2004-320287 A

  However, for the user, setting the priority for each registered person is a very laborious operation. Also, when the number of registered persons is increased or when the priority is reset, it is necessary to individually change the priority, which takes time for the user.

  The present invention has been made to solve the above-described problems of the prior art, and its purpose is to automatically determine a subject intended by the user as a main subject by estimating the subject intended by learning without taking time and effort. An object of the present invention is to provide an image processing apparatus and method.

  In order to achieve the above object, the present invention provides a registration means for registering a feature portion, a detection means for detecting a feature portion from an image, and a determination means for determining a main subject from the feature portions detected by the detection means. Storage means for storing at least one of the cumulative number determined as the main subject and the cumulative time determined as the main subject as individual information for each of the feature parts registered by the registration means. The determining means has the same feature stored by the storage means when the same feature part as the feature part registered by the registration means exists in the feature part detected by the detection means. The main subject is determined based on the individual information regarding the portion.

  According to the present invention, a subject intended by a user can be estimated by learning and automatically determined as a main subject without taking time and effort.

1 is a block diagram illustrating a configuration of a photographing apparatus according to a first embodiment as an example of an image processing apparatus of the present invention. It is a key map showing an example of a personal information list, a detection result list, and a subject list. It is a flowchart of a detection result list update process. It is a flowchart of a subject list update process. It is a flowchart of a list deletion process. It is a flowchart of a main subject selection process. It is an example of the picked-up image in the middle of photographing at the start of photographing. It is a flowchart of the main subject selection process in the second embodiment. It is a figure (figure (b)) which shows the priority at the time of not considering the image (figure (a)) at the time of imaging | photography start, and the detection number of each person. It is a figure which shows the priority at the time of considering the detection number of each person in a picked-up image. It is a figure (figure (b)) which shows an example (figure (a)) of a picked-up image in the middle of photography, and a priority when the number of people detected of each person is considered.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a configuration of a photographing apparatus according to the first embodiment as an example of an image processing apparatus of the present invention. Hereinafter, although the digital video camera 100 which records a moving image is illustrated as an imaging device, it is not restricted to this, A digital still camera, a surveillance camera, etc. may be sufficient.

  First, the configuration of the video camera 100 will be described.

  The video camera 100 includes control system blocks denoted by reference numerals 101 to 105 in the drawing. A CPU (determining means) 101 is connected to the RAM 102, the ROM 103, and other blocks via an internal bus 105, and controls each block based on a control program stored in the ROM 103. Here, the ROM 103 is a rewritable ROM such as FLASH. The CPU 101 uses the RAM 102 as a temporary data storage location during operation. Further, the operation unit 104 connected to the CPU 101 accepts input of various operation levers, buttons, and the like by the user, and the CPU 101 can perform various processes according to the user's instructions.

  The video camera 100 includes camera system blocks denoted by reference numerals 110 to 115 in the drawing. The light that has passed through the variable magnification lens 110 and the focus lens 111 forms an image on the photographing surface of the CCD image sensor 114 as photographing means, and the optical image on the photographing surface is converted into an electrical signal. The lens driver 113 drives the zoom lens 110, the focus lens 111, and the aperture 112 under the control of the CPU 101, and adjusts the optical zoom magnification, focus, and aperture amount. An AFE (analog front end) 115 drives the CCD image sensor 114 based on the shutter speed instructed from the CPU 101, and reads an image signal for one frame every 1/60 seconds. Then, the image signal is sampled and held, amplified, and converted into a digital signal.

  The video camera 100 also includes image processing blocks indicated by reference numerals 120 to 128 in the drawing. Under the control of the CPU 101, the image processing unit 120 receives an image signal from the AFE 115, performs color conversion processing, and outputs the image signal to the buffer memory 123. The exposure evaluation unit 121 and the focus evaluation unit 122 evaluate the exposure and focus states for a plurality of regions included in the image signal obtained from the image processing unit 120, respectively. The feature detection unit (detection unit) 124 reads an image signal from the buffer memory 123 every frame. Then, the number of human faces (face portions), which are characteristic parts included in the image, is detected, and the position and size in the captured image are detected as “subject information”. The detection period is 1/60 second.

  In addition, the CPU 101 stores the position and size of the face portion detected by the feature detection unit 124 in the RAM 102 as the detection result list L2. The detection result list L2 will be described later with reference to FIG. The feature extraction unit 125 reads an image signal from the buffer memory 123 and extracts a feature amount for the face portion detected by the feature detection unit 124 from the detection result list L2. Here, the feature amount is information representing the appearance of the facial organs (eyes, nose, mouth, etc.) of the person's face, such as the shape, relative arrangement, skin color, and the like.

  Here, a list of feature amounts for specifying an individual is called a personal information list L1 (described later with reference to FIG. 2A). This personal information list L1 is stored in the ROM 103, and is compared with the feature amount of the subject extracted by the feature extraction unit 125.

  The personal authentication unit 126 compares the feature amount extracted by the feature extraction unit 125 with the feature amount h (FIG. 2A) of each person registered in the personal information list L1. The personal authentication unit 126 authenticates whether there is a person similar to the detected face part (which can be determined to be the same). The CPU 101 updates the detection result list L2 with the authentication result. The drawing unit 127 draws a frame from the subject list L3 in accordance with the position of the face portion of the image, and outputs the frame to the synthesis unit 128. The subject list L3 will be described later with reference to FIG.

  The synthesizing unit 128 synthesizes the frame output from the drawing unit 127 while reading out the image signal from the buffer memory 123, and outputs it to the output unit 130. The video camera 100 also includes an output unit 130 and outputs the image signal synthesized by the synthesis unit 128.

  2A, 2B, and 2C are conceptual diagrams showing examples of the personal information list L1, the detection result list L2, and the subject list L3, respectively.

  The personal information list L1 shown in FIG. 2A is a list of feature parts registered by the user. In the present embodiment, information such as the feature amount of the face part of a person is stored in the ROM 103. The

  In the personal information list L1, in addition to the face image (registered image), name, number n, feature amount h registered by the user, the number of main subjects c and main subject time t as “personal information” (individual information) Is stored. Here, the main subject is the most important subject among subjects, and is a subject that is highly likely to be intended by the user. The video camera 100 performs camera control such as aperture and focus according to the main subject.

  In the personal information list L1, information such as a registered image, name, and number n regarding one element is arranged in one line, and elements are arranged in the column direction by the number registered. The number n is information indicating the element number in the list. Hereinafter, the n-th element in the list is described as an element [n].

  The main subject count c is the cumulative number set as the main subject during shooting since the person was registered in the personal information list L1, and the main subject time t is the person registered in the personal information list L1. To the cumulative time taken as the main subject during shooting. The ROM 103 functions as storage means for storing these as personal information (individual information).

  In order to additionally register personal information of a person in the personal information list L1, the user needs to register in advance in ROM 103 an image of the person's face and the name of the person. When the user inputs an image, the feature detection unit 124 performs feature detection from the input image, and the feature extraction unit 125 obtains the feature amount of the detected face part. The user inputs the name of the person by operating a predetermined button on the operation unit 104. When the input of the name and the extraction of the feature amount of the input image are completed, the CPU 101 registers them as new elements in the personal information list L1 (registration means). At this time, the CPU 101 sets a value obtained by adding 1 to the number of elements already registered for the number n of the new element, and sets 0 for both the main subject count c and the main subject time t. The user can also delete a desired element from the personal information list L1 by operating a predetermined button on the operation unit 104.

  Next, the detection result list L2 will be described. The detection result list L2 illustrated in FIG. 2B is a list for the CPU 101 to manage the detection result of the feature detection unit 124 and the authentication result of the personal authentication unit 126, and is stored in the RAM 102.

  As shown in FIG. 2B, the detection result list L2 stores the authentication result a corresponding to the list of subject information (position and size in the photographed image) of the characteristic part detected from the photographed image. . That is, the detection result list L2 includes the number (n) of the detected face part, the center position of the face part (horizontal position x and vertical position y), the size of the face part (width w), and the authentication result a. Stored. The number n in the detection result list L2 is information indicating the element number in the list.

  A procedure for updating the detection result list L2 will be described. The detection result list L2 is updated every time the detection result of the feature detection unit 124 is output. First, when a detection result is output, the CPU 101 deletes all elements in the detection result list L2. Then, the positions and sizes of all face parts detected as detection results are stored in the detection result list L2. At that time, a serial number is set to the number n of each element, and 0 is set to the authentication result a.

  Then, feature extraction by the feature extraction unit 125 and authentication processing by the personal authentication unit 126 are performed on each element of the set detection result list L2. The CPU 101 sets the authentication result of the detection result list L2. Here, if the individual authentication unit 126 is authenticated that each element is the same as any person registered in the personal information list L1, the personal authentication unit 126 uses the value of the number n in the personal information list L1 of the person as the authentication result a. Set as a value. If no person is authenticated, 0 is set in the authentication result a.

  If the authentication result a is a positive integer (a ≠ 0), it indicates that the person is registered in the personal information list L1, and has personal information such as the main subject count c and the main subject time t. Indicates that If the authentication result is 0, it indicates that the person is not registered in the personal information list L1.

  A process in which the feature extraction unit 125 and the personal authentication unit 126 perform feature extraction and authentication processing from the detection result list L2 and the CPU 101 updates the detection result list L2 from the authentication result will be described using the flowchart of FIG.

  FIG. 3 is a flowchart of the detection result list update process. First, in step S301, the CPU 101 acquires the number of elements I1 of the detection result list L2. In step S302, the CPU 101 initializes a counter i indicating the number n of the element being processed in the detection result list L2 to 1. In step S303, the CPU 101 determines whether there is an unprocessed element in the detection result list L2 (i ≦ I1). If there is no unprocessed element, the CPU 101 ends this process. On the other hand, if there is an unprocessed element, the CPU 101 advances the process to step S304, and the feature extraction unit 125 extracts a feature amount for the face portion of the element [i] in the detection result list L2.

  In step S305, the personal authentication unit 126 compares the feature amount extracted by the feature extraction unit 125 with the feature amount h of the personal information list L1 stored in the ROM 103, and performs personal authentication processing. That is, the personal authentication unit 126 authenticates that an element that can be determined to be the same as the element being processed is registered in the personal information list L1 by comparing feature amounts.

  In step S306, the CPU 101 updates the authentication result a of the element [i] in the detection result list L2 from the authentication result in step S305. That is, when authenticated, the value of the element number n registered in the personal information list L1 is set in the authentication result a of the element [i]. Otherwise, 0 is set in the authentication result a. . Next, in step S307, the CPU 101 adds 1 to the counter i and returns the process to step S303.

  According to the processing of FIG. 3, the detection result list L2 is updated every 1/60 seconds, and positive values are set in the authentication result a for elements that match the elements of the personal information list L1.

  As will be described later, when the detection result list update process of FIG. 3 is completed, the process proceeds to a subject list update process (FIG. 4) for updating the subject list L3 based on the updated detection result list L2. Thereafter, the process proceeds to a list deletion process (FIG. 5) for deleting information of the face portion that is no longer detected from the updated subject list L3, and further to the main subject selection process (FIG. 6) when the list deletion process is completed. Note that the values of the variables and counters are reset at the end of each process in FIGS.

  The subject list L3 shown in FIG. 2C is a list in which the CPU 101 correlates the face portions detected by the feature detection unit 124 for each frame and continuously holds the face portion information. Stored in the RAM 102.

  In the subject list L3, the number n for the face part detected by the feature detection unit 124, the center position (horizontal position x and vertical position y), the size of the face part (width w), the authentication result a, and the main subject information m The main subject count c2, the main subject time t2, and the number of lost frames f are stored. The main subject count c2 is the number of times that the face portion is set as the main subject during shooting after the face portion is added to the subject list L3, and the main subject time t2 is shot after the face portion is added to the subject list L3. This is information indicating the shooting time taken as the main subject.

  The CPU 101 updates the subject list L3 every time the detection result list L2 is updated. Whether or not they are the same subject is determined from the correlation between the position and size of the face portion stored in the detection result list L2 and the face portion stored in the subject list L3. For the face portions determined to be the same subject, the position and size thereof are acquired from the detection result list L2, and the subject list L3 is updated. For a face portion that is not determined to be the same subject, an element is added to the subject list L3 as a new face portion.

  The number of lost frames f is a value that increases by 1 when each element has no correlation with the detection result list L2 and the face part information is not updated. The number of lost frames f is information indicating how many consecutive frames the face portion has not been detected, and is also a value indicating how many frames have passed since frame out.

  When the number of lost frames f becomes equal to or greater than a predetermined number, it is recognized that the person is out of frame from the angle of view during shooting and is deleted from the subject list L3. In the present embodiment, as an example, it is assumed that the predetermined number is 3, and the number of lost frames f is 3 when 3 frames are not detected consecutively, and the corresponding element information is deleted from the subject list L3.

  The main subject information m is information indicating whether the face portion has the highest priority among the face portions in the subject list L3. m = 1 indicates that the element (face part) has the highest priority, and m = 0 indicates that it is not. The value of the main subject information m is updated by main subject selection processing (FIG. 6) by the CPU 101 after completion of the subject list update processing (FIG. 4) and list deletion processing (FIG. 5). The main subject information m of the face portion selected as the main subject is set to 1, and the main subject information m of the other face portions is set to 0.

  Here, a method of selecting a main subject, which will be described later with reference to FIG. 5, will be briefly described.

  The user can set the main subject by selecting a desired subject from the detected face parts by operating the buttons on the operation unit 104 (designating means). The CPU 101 sets the main subject information m of the face part element selected by the user in the subject list L3 to 1, and sets the subject instruction information d stored in the RAM 102 to 1.

  Here, the subject instruction information d is information indicating by “1” that there is an element designated as the main subject by the user among the elements in the subject list L3, and “0” indicating that there is no element. Further, while the face portion designated by the user continues to be detected, the subject instruction information d keeps its value, and when the face portion is not detected continuously for three frames, the subject instruction information d is 0. Reset to. On the other hand, when the user has not designated the main subject, the main subject is automatically selected and determined by the main subject selection process (FIG. 6).

  FIG. 4 is a flowchart of subject list update processing.

  First, in steps S401 to S405, the CPU 101 adds 1 to the number of lost frames f of all elements of the subject list L3. That is, in step S401, the number J of elements in the subject list L3 is acquired. Next, in step S402, the CPU 101 initializes a counter j indicating the number n of the element being processed in the subject list L3 to 1. In step S403, the CPU 101 determines whether j ≦ J is satisfied, that is, whether there is an unprocessed element in the subject list L3.

  If there is an unprocessed element as a result of the determination (j ≦ J), the CPU 101 adds 1 to the number of lost frames f of the element [j] in the subject list L3 in step S404, and sets the counter j in step S405. Add one. Thereafter, the CPU 101 returns the process to step S403. If there is no unprocessed element in step S403, the CPU 101 advances the process to step S406.

  Next, in steps S406 to S414, the CPU 101 updates the element information stored in the subject list L3 based on the detection result list L2, and adds a newly detected face portion, if any.

  First, in step S406, the CPU 101 acquires the number of elements I1 in the detection result list L2. In step S407, the CPU 101 initializes a counter i indicating the number n of the element being processed in the detection result list L2 to 1. In step S408, the CPU 101 determines whether i ≦ I1 is satisfied, that is, whether there is an unprocessed element in the detection result list L2.

  If there is an unprocessed element as a result of the determination (i ≦ I1), the CPU 101 executes the same subject determination in step S409. Here, as the same subject determination, it is determined whether any element of the subject list L3 has a positional and size correlation as compared with the element [i] of the detection result list L2. In step S410, if there is an element determined to be the same in step S409, the number n in the subject list L3 of the element is stored in the variable k1. On the other hand, if there is no element determined to be the same, 0 is stored in the variable k1.

  Next, in step S411, the CPU 101 determines whether or not the variable k1 ≠ 0 is established. If the variable k1 is not 0 as a result of the determination, the CPU 101 detects the center position and size of the face portion of the element of the number [k1] in the subject list L3 and further the value of the authentication result a in step S412. Update with each value in element [i] of result list L2. Since the element has been updated, in the subsequent step S413, the CPU 101 clears the number of lost frames f of the element [k1] in the subject list L3 to zero.

  On the other hand, if the variable k1 is 0 in step S411, since the element [i] being processed is a new element, the CPU 101 adds the element to the subject list L3 in step S414. At this time, each value of the element [i] in the detection result list L2 is stored in the center position and size of the face portion of the element, and further the value of the authentication result a.

  After the process of step S412 or step S414, the CPU 101 adds 1 to the counter i in step S413. When there is no unprocessed element in step S408, the CPU 101 ends this process.

  FIG. 5 is a flowchart of the list deletion process.

  First, in step S501, the CPU 101 acquires the number of elements J of the subject list L3. In step S502, the CPU 101 initializes a counter j indicating the number n of the element being processed in the subject list L3 to 1. In step S503, the CPU 101 determines whether j ≦ J is satisfied, that is, whether there is an unprocessed element in the subject list L3.

  If there is an unprocessed element (j ≦ J), the CPU 101 determines in step S504 whether or not the number of lost frames f of the element [j] in the subject list L3 is 3 (f = 3). Determine. As a result of the determination, if the number of lost frames is 3, that is, there is a face portion that is not detected continuously for three frames, the corresponding elements in the subject list L3 are deleted in steps S505 to S512.

  First, in step S505, the CPU 101 determines whether or not the main subject information m of the element [j] of the subject list L3 is 1. When m = 1, since the element [j] is set as the main subject, the CPU 101 determines whether the subject instruction information d is 1 in step S506. As a result of the determination, if the subject instruction information d is 0, the CPU 101 proceeds the process to step S508. If the subject instruction information d is 1, the CPU 101 clears the subject instruction information d to 0 in step S507. The process proceeds to step S508. Thereby, the user designation is canceled.

  In step S508, the CPU 101 determines whether or not the authentication result a ≠ 0 of the element [j] of the subject list L3 is established. As a result of the determination, if a ≠ 0, the element [j] to be deleted has personal information, so the information in the personal information list L1 is updated. That is, in step S509, the CPU 101 stores the authentication result a of the element [j] of the subject list L3 in the variable k2. In steps S510 and S511, the CPU 101 sets the main subject count c2 and main subject time t2 of the element [j] of the subject list L3 to the main subject count c and main subject time t of the element [k2] of the personal information list L1. The value is updated by adding each. As a result, personal information is accumulated. Thereafter, the CPU 101 advances the process to step S512.

  On the other hand, if the authentication result a = 0 in step S508, since the element [j] to be deleted does not have personal information, the CPU 101 performs processing without updating the information in the personal information list L1. Proceed to S512.

  In step S512, the CPU 101 deletes the element [j] from the subject list L3, adds 1 to the counter j in step S513, and returns the process to step S503. If there is no unprocessed element in step S503 (j> J), the CPU 101 ends this process. If the number of lost frames is not 3 in step S504, the CPU 101 advances the process to step S513 because there is no face portion that is not detected continuously for 3 frames. If the main subject information m is not 1 in step S505, the element [j] is not set as the main subject, and the CPU 101 advances the process to step S508.

  The processing in FIGS. 4 and 5 is briefly summarized as follows. According to the process of FIG. 4, the subject list L3 is updated every time the detection result list L2 is updated. Elements that are in the detection result list L2 but not in the subject list L3 are newly added to the subject list L3. For the same determination element in both the detection result list L2 and the subject list L3, the subject list L3 is updated with the value of the detection result list L2.

  Further, according to the process of FIG. 5, when the state that is in the subject list L3 but not in the detection result list L2 (not the same determination) continues three times (the number of lost frames is 3), the element is deleted from the subject list L3. Is done. At this time, if the user has designated, the designation is canceled. When the element to be deleted is an element registered in the personal information list L1, the main subject count c and the main subject time t in the personal information list L1, and the main subject count c2 and the main subject time in the subject list L3. Each t2 is added and accumulated.

  FIG. 6 is a flowchart of the main subject selection process.

  In step S601, the CPU 101 determines whether the subject instruction information d is 1 (d = 1). As a result of the determination, if d = 0, the main subject has not been designated by the user. Therefore, in steps S602 to S609, the CPU 101 executes a process of selecting a face part element to be set as the main subject.

  First, in step S602, the CPU 101 obtains the number of elements I2 of elements whose authentication result a is not 0 (a ≠ 0) in the subject list L3. In step S603, the CPU 101 determines whether or not the number of elements I2 ≧ 1 is satisfied. If the number of elements I2 ≧ 1, the CPU 101 determines whether or not the number of elements I2 ≧ 2 is satisfied in step S604. Determine. When the number of elements I2 = 0, the CPU 101 advances the process from step S603 to step S605.

  In step S605, since the face portion having personal information is not detected, the CPU 101 prioritizes each element in the subject list L3 according to the center position and size of the face portion. The priority is higher as the position of the face portion is closer to the center of the image, and higher as the size of the face portion is larger. For example, the priority is obtained from the subject information (the center position (x, y) of the face portion and the size (width w) of the face portion) by the following formula 1. Here, the center position of the image used for detection by the feature detection unit 124 is defined as (Xmid, Ymid).

[Equation 1]
Priority = face size + (center position Xmid− | center position Xmid−position x |) + (center position Ymid− | center position Ymid−position y |)
This is because, when there is no person registered in the personal information list L1, it is considered that a person who is superior in shooting angle of view (a large subject close to the center) is a person intended to be taken by the user. It is.

  Thereafter, in step S606, the CPU 101 stores the number n of the element having the highest priority in the subject list L3 in the variable k3.

  If the number of elements I2 = 1 (only one face portion having personal information is detected), the CPU 101 advances the process from step S604 to step S607. In step S607, the number n of one element whose authentication result a is not 0 in the subject list L3 is stored in the variable k3. This is to preferentially select a person registered in the personal information list L1.

  If the number of elements is I2 ≧ 2 (two or more face portions having personal information are detected), the CPU 101 advances the process from step S604 to step S608. In step S608, the CPU 101 prioritizes two or more elements whose authentication result a is not 0 in the subject list L3. That is, the CPU 101 determines that the number of main subjects c and the main subject time of each element [a] in the personal information list L1 having the same authentication result a as the authentication result a of each element whose authentication result a is not 0 in the subject list L3. Get t. Then, prioritization is performed so that the priority becomes higher as the main subject count c is larger and the main subject time t is longer. The priority is obtained by the following formula 2.

[Equation 2]
Priority = number of main subjects c × main subject time t
This is because when a plurality of persons registered in the personal information list L1 are displayed on the screen, the user wants to shoot a person who has been taken many times as a main subject and has taken a long time. This is because it is considered to be an intended person. In step S609, the CPU 101 stores the number n of the element in the subject list L3 having the highest priority in the variable k3 as a result of the prioritization in step S608.

  The element [k3] in the subject list L3 is an element to be the main subject from now on. After the processes in steps S606, S607, and S609, the CPU 101 advances the process to step S610.

  In step S610, the CPU 101 determines whether or not the main subject information m of the element [k3] of the subject list L3 is 1. If the main subject information m of the element [k3] is 1 as a result of the determination, the element [k3] has already been set as the main subject, and the CPU 101 advances the process to step S616.

  On the other hand, if the main subject information m of the element [k3] is not 1, the CPU 101 stores the number n of the element whose main subject information m is 1 in the subject list L3 in the variable k4 in step S611. Further, when the main subject information is not 1 for any element, 0 is stored in the variable k4.

  Next, in step S612, the CPU 101 determines whether or not the variable k4 = 0. As a result of the determination, if the variable k4 = 0, the CPU 101 advances the process to step S614. If not, the CPU 101 clears the main subject information m of the element [k4] in the subject list L3 to 0. Then, the process proceeds to step S614.

  In step S614, the CPU 101 sets the main subject information m of the element [k3] of the subject list L3 to 1. As a result, the element [k3] is newly selected and determined as the main subject. Therefore, in step S615, the CPU 101 adds 1 to the main subject count c2 of the element [k3] (c2 ← c2 + 1). In step S616, the CPU 101 adds a detection cycle to the main subject time t2 of the element [k3] of the subject list L3 (t2 ← t2 + detection cycle). Here, the detection cycle is a cycle in which the feature detection unit 124 performs detection processing, and is 1/60 seconds.

  If the subject instruction information d is 1 in step S601, the CPU 101 stores the element number n of the subject list L3 designated as the main subject in the variable k3 in step S617, and performs the process of step S616. Run. After the process of step S616, the main subject selection process ends.

  The main subject selection process in FIG. 6 is briefly described as follows. First, if there is a user designation, the main subject is selected with the designation as the highest priority (step S617). When there is no user designation and there is an element registered in the personal information list L1 in the subject list L3, if there is one element, it is determined as the main subject (step S607). However, when there are a plurality of such elements, the priority order is determined based on the personal information (the main subject count c and the main subject time t) regarding these elements, and the one with the highest priority order is determined as the main subject ( Steps S608 and S609).

  When there is no user designation, and there is no element registered in the personal information list L1 in the subject list L3, the main subject is determined from the elements in the subject list L3. At this time, since personal information cannot be used, the main subject is determined exclusively based on subject information (center position (x, y) of face portion and size (width w) of face portion) (step S605, S606).

  In Steps S610 to S616, if the element [k3] having the highest priority in the subject list L3 is already the main subject, the main subject count c2 is not added and the main subject time t2 is added. To do. If the element [k3] is not the main subject, the element [k3] is newly set as the main subject. If there is an element that has been the main subject at that time, the main subject information m is set to 0. Reset. For the element selected as the main subject, the main subject time t2 increases every time the processing of FIG. 6 is performed.

  Next, an example of an operation reflecting the above-described processes of FIGS. 3 to 6 when a plurality of subjects are photographed by the video camera 100 will be described.

  FIG. 7A shows an example of an image at the start of photographing. At this time, it is assumed that the user does not designate the main subject (d = 0) and the main subject is not automatically set by the CPU 101.

  In the image 900, persons 910, 920, and 930 are shown. The personal information list L1 stored in the ROM 103 is the same as that shown in FIG. In the image 900, the persons 910 and 920 are the persons of numbers 2 and 3 in the personal information list L1, and the person 930 is not registered in the personal information list L1.

  The feature detection unit 124 detects the face portions of the persons 910, 920, and 930, and the personal authentication unit 126 authenticates the persons 910 and 920 as the persons (elements) of numbers 2 and 3 in the personal information list L1. (Step S305 in FIG. 3). The person 930 is not authenticated as any person.

  The CPU 101 updates the detection result list L2 from the detection result and the authentication result (FIG. 4). First, the positions and sizes of the faces of the persons 910, 920, and 930 in the subject list L3 are updated from the detection result list L2 (step S412 in FIG. 4). Next, the authentication results a of the persons 910 and 920 in the subject list L3 are set to 2 and 3, respectively, and the authentication result a of the person 930 is set to 0.

  Next, the CPU 101 deletes the subject list L3 (FIG. 5). In this example, since the face portions of all of the persons 910, 920, and 930 are detected, there is no element in the subject list L3 to be deleted.

  Next, the CPU 101 performs main subject selection processing (FIG. 6). In this example, the main subject is not determined in the subject list L3. Further, since there are two persons 910 and 920 who have already been registered in the personal information list L1, prioritization is performed based on the number of main subjects c and the main subject time t according to Equation 2 (step in FIG. 6). S608). That is, the CPU 101 refers to the personal information list L1 shown in FIG. 2A and selects the person 910 (number 2) as the main subject. This is because the person 910 has a larger number of main subjects c and a longer main subject time t than the person 920.

  Then, the CPU 101 adds 1 to the main subject count c2 of the person 910 in the subject list L3 (step S615). Further, since the photographing is being performed, a detection cycle of 1/60 seconds is added to the main subject time t2 (step S616).

  FIG. 7B is a diagram illustrating an image of the third frame after the video camera is panned to the left from the shooting state illustrated in FIG.

  In the image 1000 shown in FIG. 7B, the number of lost frames f of the person 910 is 2. At this time, since the number of lost frames f is less than 3, it is not deleted from the subject list L3. Further, it is assumed that the faces of the persons 920 and 930 are continuously detected. It is assumed that the main subject time t2 of the person 910 in the subject list L3 is 1 minute.

  When the number of lost frames becomes 3, the person 910 is deleted from the subject list L3 (step S512 in FIG. 5). At this time, since the person 910 has 2 as the authentication result a, the value of the main subject count c2 in the subject list L3 of the person 910 (here, the main subject count c of the person whose number n is 2 in the personal information list L1) Then, 1) is added (step S510). Further, the value (in this case, 1 minute) of the main subject time t2 in the subject list L3 of the person 910 is added to the main subject time t of the person whose number n is 2 in the personal information list L1 (step S511).

  Next, the CPU 101 performs main subject selection processing (FIG. 6). The element of the person 910 is deleted, and there is no subject having the main subject information m. At this time, since only the person 920 is detected as the subject having the authentication result a, the CPU 101 selects the person 920 as the main subject (step S607 in FIG. 6).

  Then, the CPU 101 adds 1 to the number of main subjects c2 of the person 920 in the subject list L3, and adds 1/60 seconds of the detection cycle to the main subject time t2 because shooting is in progress (steps S615 and S616).

  Thus, when a new face portion appears on the screen (frame-in), the face portion is added as an element to the subject list L3 (step S414 in FIG. 4). While the face portion is located in the screen, the main subject time t2 is accumulated in the subject list L3 (step S616 in FIG. 6). When the face portion disappears from the screen for three frames in succession, the main subject count c2 and main subject time t2 in the subject list L3 at that time are added to the main subject count c and main subject time t in the personal information list L1. (Steps S510 and S511 in FIG. 5). The main subject count c2 of each element in the subject list L3 is counted up each time the element becomes a new main subject (step S615 in FIG. 6).

  With this operation and configuration, even if there is no user designation, the person who would like to shoot is automatically selected as the main subject, and in the process of shooting, the priority to become the main subject of the registered person is set. Can learn automatically. As a result, even when there are a plurality of subjects having personal information, it is possible to automatically prioritize a person who is likely to be photographed by the user and to perform camera control appropriate for taking the person. The user of the video camera does not have to set the priority to be the main subject for each person whose personal information is registered.

  Therefore, according to the present embodiment, the subject intended by the user can be estimated by learning and automatically determined as the main subject without taking time and effort.

  If there is no element in the subject list L3 whose authentication result a is not 0, each element in the subject list L3 is prioritized according to the center position and size of the face portion. Thereby, even when there is no registered face portion in the captured image, the main subject can be automatically determined from the subjects in the captured image.

(Second Embodiment)
The first embodiment has a configuration in which a person registered in the personal information list L1 is more preferentially used as a main subject, and is effective when the number of subjects is small. However, when a large number of people are photographed at the same time, only a specific person tends to be prioritized. When there is a user's intention to capture the entire group of people in the angle of view, it may be difficult for the video camera of the first embodiment to automatically determine the main subject appropriately and capture it. .

  Therefore, in the second embodiment of the present invention, a configuration will be described in which priorities are assigned to all subjects regardless of whether or not they are registered, and the priorities are changed according to the number of detected people. The second embodiment differs from the first embodiment in the method in which the CPU 101 automatically selects the main subject, and will be described with reference to FIG. 8 instead of FIG.

  FIG. 8 is a flowchart of the main subject selection process.

  The processes in steps S801, S811, and S810 are the same as the processes in steps S601, S617, and S616 in FIG. As a result of the determination in step S801, if the subject instruction information d is 0, the main subject has not been designated by the user, so in steps S802 and S803, the CPU 101 selects the face part element to be set as the main subject. Perform prioritization for

  In step S802, the CPU 101 detects the number of detected face parts (number of detected persons), subject information (face position position (x, y) and face part size (width w)), and personal information ( Prioritization is performed based on the main subject count c and the main subject time t). The priority is obtained by the following formula 3. However, as long as the main subject can be appropriately determined according to the number of subjects, the formula for obtaining the priority may be other than this.

[Equation 3]
Priority = (priority based on personal information) / number of detected persons + (priority based on face position and size) × number of detected persons Here, “priority based on personal information” indicates that the main subject count c increases and the main subject time increases. The longer t is, the higher the value is, and here, it is obtained by the same expression as Expression 2 used in the first embodiment. The “priority based on the position and size of the face” is higher as the position of the face portion is closer to the center of the image and the size of the face portion is larger. Here, the mathematical expression used in the first embodiment is used. It is obtained by the same formula as 1.

  “Priority based on personal information” is divided by “number of detected persons”, and “priority based on face position and size” is multiplied by “number of detected persons”. As a result, as the number of detected persons increases, the subject information (face position and size) in the subject list L3 is weighted more heavily than the personal information (main subject count c, main subject time t) in the personal information list L1. The subject is determined.

  The reason for this is that when there are a large number of subjects in the image, it is considered that not only a person with a large record of the number of times and time of being a main subject is intended. In other words, this is because as the number of detected people increases, the current shooting state is more weighted than the past results. Therefore, when the number of subjects is large, priority is given to the priority obtained from the subject list L3 rather than the priority obtained from the personal information list L1. On the contrary, when the number of subjects is small, priority is given to the priority obtained from the personal information list L1 rather than the priority obtained from the subject list L3.

  By the way, “priority based on personal information” and “priority based on face position and size” are adjusted to be appropriate values according to the number of subjects. For example, the priority value of each face part is represented by a ratio, and the priority value of each face part is determined so that the total number of detected persons is 1, and the higher the value, the higher the priority.

  In step S803, the number n of the element having the highest priority in the subject list L3 is stored in the variable k3. The subsequent steps S804 to S809 are the same as the steps S610 to S615 of FIG.

  Next, a specific operation example in which the processing of FIGS. 3 to 5 and FIG. 8 is reflected when a plurality of subjects are photographed by the video camera 100 will be described.

  FIG. 9A shows an example of an image at the start of shooting.

  At this time, it is assumed that the user has not designated the main subject and the main subject has not been automatically set by the CPU 101. The photographed image 1200 includes people 1210, 1220, 1230, 1240, and 1250. The personal information list L1 stored in the ROM 103 is the same as that shown in FIG. In the captured image 1200, it is assumed that the persons 1210 and 1220 are the persons of numbers 2 and 3 in the personal information list L1.

  FIG. 9B is a diagram showing the priority based on personal information and the priority based on the position and size of the face when the number of detected persons in the captured image 1200 is not considered. The longer the graph in the figure, the higher the priority. As shown in FIG. 9B, the priority based on the position and size of the face is indicated by P (P1 to P5). The priorities according to the positions and sizes of the faces of the persons 1210, 1220, 1230, 1240, and 1250 are P1, P2, P3, P4, and P5, respectively.

  In terms of the position and size of the face, the person 1240 is close to the center of the image and appears large, so the length of the priority P4 is the longest. On the contrary, the person 1220 and the person 1230 are close to the end of the shooting angle of view, and the face size is small, so the priorities P2 and P3 are short. Q (Q1 to Q5) is a priority based on personal information. Since personal information is registered for the persons 1210 and 1220, the personal information has priorities Q1 and Q2. As indicated by numbers 2 and 3 in FIG. 2A, since the person 1210 has a larger main subject count c and a longer main subject time t than the person 1220, the priority Q1 is higher than the priority Q2. Is also getting longer.

  Processing until the main subject is selected in this state will be described. First, the feature detection unit 124 detects the face portion of the person 1210, 1220, 1230, 1240, 1250. The personal authentication unit 126 authenticates the persons 1210 and 1220 as the persons of numbers 2 and 3 in the personal information list L1, and authenticates the persons 1230, 1240, and 1250 as having no personal information. The CPU 101 updates the detection result list L2 from the detection result and the authentication result (FIG. 3). Then, the positions and sizes of the faces of the persons 1210, 1220, 1230, 1240, and 1250 in the subject list L3 are updated from the detection result list L2 (FIG. 4). At that time, the authentication results a of the persons 1210 and 1220 are set to 2 and 3, respectively, and the authentication results a of the persons 1230, 1240 and 1250 are all set to 0.

  Next, the CPU 101 deletes the subject list L3 (FIG. 5). In this case, since faces of all of the persons 1210, 1220, 1230, 1240, and 1250 are detected, there is no face part to be deleted.

  Next, the CPU 101 performs a main subject selection process (FIG. 8). First, the priority of each detected person is obtained. Here, since the number of detected people is 5, when the priority of each person in FIG. 9B is obtained, the priority P based on the position and size of the face is multiplied by 5 by the above Equation 3 to give priority based on personal information. The degree Q is increased to 1/5 times. The result is shown in FIG.

  FIG. 10 is a diagram illustrating the priority P and the priority Q when the number of detected persons in the captured image 1200 is taken into consideration. As shown in FIG. 10, when the number of detected people is large, a person who has an advantage in the shooting angle of view is given priority over the presence or absence of personal information. Since the person 1240 has the highest priority, the CPU 101 selects the person 1240 as the main subject. Then, the CPU 101 adds 1 to the main subject count c2 of the person 1240 in the subject list L3. In addition, since shooting is in progress, a detection cycle of 1/60 seconds is added to the main subject time t2.

  FIG. 11A shows a photographed image of the third frame when the persons 1220 and 1230 are out of the frame at the same time from the state shown in FIG. 9A and the persons 1220 and 1230 cannot be detected. In this captured image 1500, the number of lost frames of the persons 1220 and 1230 is 2. Further, it is assumed that the persons 1210, 1240, and 1250 have not moved and are still being detected. At this time, the main subject remains the person 1240 and is not changed.

  The feature detection unit 124 detects the face portions of the persons 1210, 1240, and 1250. The personal authentication unit 126 authenticates the person 1210 as a person number 2 in the personal information list L1, and authenticates that the persons 1240 and 1250 have no personal information. The CPU 101 updates the detection result list L2 from the detection result and the authentication result (FIG. 3). Then, the positions and sizes of the persons 1210, 1240, 1250 in the subject list L3 are updated from the detection result list L2 (FIG. 4). At this time, the authentication result a of the person 1210 is set to 2, and the authentication results a of the persons 1240 and 1250 are set to 0.

  The persons 1220 and 1230 are deleted from the subject list L3 when the third frame passes and the number of lost frames reaches 3 (FIG. 5). At this time, since the person 1220 is not the main subject and the person 1230 has no personal information, the personal information list L1 is not updated.

  Next, the CPU 101 performs a main subject selection process (FIG. 8). First, the priority of each detected person is obtained. Here, since the number of detected persons is three, when the priority of each person in FIG. 9B is obtained, the priority P based on the position and size of the face is tripled by the above-described formula 3 to give priority based on personal information. The degree Q is increased to 1/3 times. The result is shown in FIG.

  FIG. 11B is a diagram illustrating the priority P and the priority Q when the number of detected persons in the captured image 1500 is taken into consideration.

  As shown in FIG. 11 (b), when the number of detected people is as small as three, the priority P based on the personal information is lowered, but the priority is kept higher than when the number of detected people is five, and the personal information is The degree of priority of registered persons is relatively high. Since the person 1210 has the highest priority, the CPU 101 selects the person 1210 as the main subject. Then, the CPU 101 adds 1 to the number of main subjects c2 of the person 1210 in the subject list L3, and adds 1/60 seconds of the detection cycle to the main subject time t2 because shooting is in progress.

  Next, consider a case where the persons 1240 and 1250 do not move from the state shown in FIG. When the person 1210 goes out of the frame and the face cannot be detected and the third frame passes, the CPU 101 deletes the person 1210 from the subject list L3. At this time, since the person 1210 is the main subject and the authentication result a is 2, the CPU 101 sets the value of the main subject count c2 in the subject list L3 of the person 1210 to the main subject count c of number 2 in the personal information list L1. Add 1 which is Similarly, the main subject time t2 in the subject list L3 of the person 1210 is added to the main subject time t of number 2 in the personal information list L1.

  Next, the CPU 101 performs main subject selection processing (FIG. 8). The person 1210 has already been deleted, and there is no subject having the main subject information m. At this time, the persons 1240 and 1250 are subjects. Since none of the persons 1240 and 1250 has personal information, the CPU 101 selects a person 1240 having a high priority P based on the position and size of the face as the main subject based on the above Equation 3. Then, the CPU 101 adds 1 to the number of main subjects c2 of the person 1240 in the subject list L3, and adds 1/60 seconds of the detection cycle to the main subject time t2 because shooting is in progress.

  According to the present embodiment, the main subject is determined based on the personal information (main subject count c, main subject time t), the number of detected people, and subject information (face position and size). In particular, even when there are a plurality of subjects, the main subject is determined by weighting subject information more heavily than personal information as the number of detected people increases. As a result, the same effect as that of the first embodiment can be achieved with respect to automatically deciding the subject intended by the user by learning and automatically determining it as the main subject without taking time and effort. In addition, the intended subject can be estimated more appropriately according to the number of characteristic portions.

  In each of the above embodiments, when obtaining the priority, information on both the main subject count c and the main subject time t is used as personal information (individual information), but only one of them is used. It may be. Similarly, both the face position and size information are used as the subject information, but only one of them may be used.

  Moreover, when calculating | requiring a priority, although the calculation formula illustrated to said Numerical formula 1-3 was used, it is not restricted to this, You may obtain | require a priority using a map or a table.

  In addition, although the face part of a person was illustrated as a characteristic part used as the object of registration or detection, it is not restricted to this, What is necessary is just a part which becomes a characteristic in a picked-up image. For example, it may be a specific character, figure, pattern, or the like, and can also be applied to a part inspection or an imaging apparatus that searches for a specific object from an image.

  Furthermore, although the description has been given by taking as an example a photographing apparatus that performs photographing control such as an aperture and a focus in accordance with the determined main subject, the present invention is not limited to this. For example, with a personal computer or network server, the brightness and saturation of the acquired image are corrected so that the determined main subject has an appropriate value, or the image is deformed so that the optical aberration in the main subject is minimized. Or may be executed. Alternatively, a process may be executed in which an image is cut out so that the main subject is at the center, or information related to the size and coordinates of the main subject is attached to the image and stored.

  As described above, when a plurality of persons are detected, any image processing apparatus that can determine the main subject using at least one of the main subject count c and the main subject time t may be used. The type of processing used is not limited.

  Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined.

101 CPU
103 ROM
104 Operation unit 114 CCD image sensor 124 Feature detection unit L1 Personal information list L2 Detection result list L3 Subject list c, c2 Number of main subjects t, t2 Main subject time

Claims (6)

A registration means for registering the characteristic part;
Detecting means for detecting a characteristic portion from the image;
Determining means for determining a main subject from the characteristic portions detected by the detecting means;
Storage means for storing, as individual information, at least one of the cumulative number determined as the main subject and the cumulative time determined as the main subject for each feature portion registered by the registration means,
The determination means includes the same feature portion stored by the storage means when the same feature portion as the feature portion registered by the registration means exists in the feature portion detected by the detection means. An image processing apparatus, wherein a main subject is determined based on individual information relating to an image.   The detection means further detects at least one of the size of the detected feature portion and the position of the detected feature portion in the image as subject information, and the determination means includes the feature portion detected by the detection means. 2. The image processing according to claim 1, wherein a main subject is determined based on subject information detected by the detection unit when a feature portion identical to the feature portion registered by the registration unit does not exist. apparatus. The detection means detects a feature portion from the image, and detects at least one of a size of the detected feature portion and a position of the detected feature portion in the image as subject information,
The determination means includes the same feature portion stored by the storage means when the same feature portion as the feature portion registered by the registration means exists in the feature portion detected by the detection means. 3. The image processing apparatus according to claim 1, wherein a main subject is determined based on the individual information relating to the information, the number of detected feature portions, and the detected subject information.   The image processing apparatus according to claim 3, wherein the determination unit determines a main subject by weighting the subject information more heavily than the individual information as the number of detected feature portions increases.   It has a designation means for designating a main subject in accordance with a user instruction from among the characteristic portions detected by the detection means, and the decision means decides the main subject with priority given to designation by the designation means. The image processing apparatus according to any one of claims 1 to 4. A registration process for registering the feature part;
A detection step of detecting a feature from the image;
A determination step of determining a main subject from the characteristic portions detected by the detection step;
An accumulation step of accumulating at least one of the cumulative number determined as the main subject and the cumulative time determined as the main subject as individual information for each of the feature parts registered in the registration step;
In the determination step, if the same feature portion as the feature portion registered by the registration step exists in the feature portion detected by the detection step, the same feature portion accumulated by the accumulation step An image processing method comprising: determining a main subject based on individual information relating to

Images