JP4952920B2 - Subject determination apparatus, subject determination method and program thereof - Google Patents

Description

The present invention relates to a subject determination device , a subject determination method, and a program thereof, and more particularly to a subject determination device , a subject determination method, and a program thereof for determining the type of a subject.

In recent years, in a subject determination device, for example, an imaging device, a technique for determining what a subject has moved to image data by performing image recognition has been developed (Patent Document 1).

JP Patent Publication No. 2007-068149

  However, in the case of recognizing a similar subject, even if image recognition is performed, the recognized subject is not always shown. In addition, when capturing a moving subject, the recognized subject may change depending on the position and angle of the subject when the image is captured, and more so.

Therefore, the present invention has been made in view of such conventional problems, and an object of the present invention is to provide a subject determination device , a subject determination method, and a program thereof that improve the subject determination accuracy.

In order to achieve the above object, the subject determination apparatus according to the present invention recognizes image recognition for an acquisition unit that sequentially acquires image data and a specific subject portion that exists in common in the image data sequentially acquired by the acquisition unit. by intended to make aggregates an image recognition means for sequentially recognizing the type of the object corresponding to the particular object portion, the kind of a subject by Ri sequentially recognizing the image recognition unit, more recognition Subject determination means for determining the type of the subject as the type of subject corresponding to the specific subject portion .

In addition, for example, in the subject determination device, the subject determination unit totals the types of subjects sequentially recognized by the image recognition unit every predetermined number of times, and determines the types of subjects recognized more at the time of the totalization. The type of subject corresponding to the specific subject portion may be determined .
It is provided with.

Further, for example, in the subject determination apparatus, the image recognition unit calculates a degree of coincidence at the time of each image recognition, and the subject determination unit periodically totals the degree of coincidence calculated by the image recognition unit. The type of subject that is determined when the collected degree of coincidence exceeds a predetermined value may be output as a final determination result .

In addition, for example, in the subject determination apparatus, the subject determination device includes subject tracking means that performs subject tracking with respect to the specific subject portion that moves in the image data sequentially acquired by the acquisition means, and the image recognition means includes the acquisition In the image data sequentially acquired by the means, the type of the subject may be sequentially recognized for the specific subject portion where the subject tracking is performed by the subject tracking means .

For example, in the subject determination device, the image recognition unit performs image recognition on the entire range of image data when the subject tracking unit does not track the specific subject portion, and When the specific subject portion is being tracked by the subject tracking means, the type of the subject corresponding to the specific subject portion is recognized by performing image recognition only on the specific subject portion. It may be.

In addition, for example, the subject determination apparatus includes a change amount detection unit that detects a change amount of image data sequentially acquired by the acquisition unit, and the image recognition unit includes the change amount detected by the change amount detection unit. May be image recognition for the entire range of image data .

In addition, for example, in the subject determination apparatus, when the same type of subject is continuously recognized a predetermined number of times by the image recognition unit, a portion corresponding to the type of subject that has been continuously recognized a predetermined number of times is selected. You may make it further provide the selection means to select as said specific subject part .

In addition, for example, in the subject determination device, the subject determination unit periodically totals the average value of the degree of coincidence calculated by the image recognition unit, and the average value of the total degree of coincidence becomes a predetermined value or more. The type of the subject that is determined at the time may be output as the final determination result .

Further, for example, in the subject determination device, the subject determination unit is common to the image data sequentially acquired by the acquisition unit when the average value of the degree of coincidence counted for each predetermined number is not equal to or greater than a predetermined value. It may be determined that the type of the existing subject is unknown .

Further, for example, the subject determination apparatus includes a feature information recording unit that records the feature information for each type of subject, and the image recognition unit calculates the feature information of the specific subject portion by image recognition, The type of subject corresponding to the feature information recorded in the feature information recording means that most closely matches the calculated feature information may be recognized as the type of subject corresponding to the specific subject portion .

Further, for example, in the subject determination device, the feature information recording unit may record feature information corresponding to a plurality of directions for each type of subject .

For example, in the subject determination apparatus, the image recognition by the image recognition unit is
It may be face recognition for recognizing who the subject is .

In addition, the subject determination method according to the present invention performs image recognition for an acquisition process that sequentially acquires image data and a specific subject portion that exists in common in the image data sequentially acquired by the acquisition process. Thus, the image recognition processing for sequentially recognizing the types of subjects corresponding to the specific subject portion and the types of subjects sequentially recognized by the image recognition processing are totaled, and the types of subjects recognized more , Subject determination processing for determining the type of subject existing in common in the image data sequentially acquired by the acquisition processing, and causing each of the above processes to be executed by a computer.

In addition, the program according to the present invention performs an image recognition process for acquiring a specific subject portion that exists in common in the image data sequentially acquired by the acquisition process and the image data sequentially acquired by the acquisition process. The image recognition processing for sequentially recognizing the types of subjects corresponding to the specific subject portion, and the types of subjects that are sequentially recognized by the image recognition processing are aggregated, and the types of subjects recognized more Subject determination processing for determining the type of subject existing in common in the image data sequentially acquired by the acquisition processing, and causing each of the above processes to be executed by a computer.

The subject determination device according to the present invention includes an acquisition unit that sequentially acquires image data, and subjecting the image data that is sequentially acquired by the acquisition unit to recognize a subject that exists in each image data. Image recognition means for recognizing the type of the subject, subject determination means for determining the type of subject existing in common in the image data sequentially acquired by the acquisition means based on the recognition result by the image recognition means, Based on the image data sequentially acquired by the acquisition means, subject tracking means for tracking the subject in the image data recognized by the image recognition means, and images sequentially acquired by the acquisition means Change amount detecting means for detecting a change amount of data, and the image recognition means performs subject tracking by the subject tracking means. If the amount of change detected by the change amount detecting means is equal to or greater than a predetermined amount, image recognition is performed only on the image data of the area where the subject that is following the subject is present. Subject tracking by means is interrupted, and image recognition is performed on the entire range of image data.

In addition, the subject determination method according to the present invention includes an acquisition process that sequentially acquires image data, and image recognition that is performed sequentially on the image data acquired by the acquisition process, thereby providing a subject that exists in each image data. Image recognition processing for recognizing the type of the subject, subject determination processing for determining the type of subject existing in common in the image data sequentially acquired by the acquisition processing based on the recognition result by the image recognition processing, Based on the image data sequentially acquired by the acquisition processing, subject tracking processing for tracking the subject in the image data recognized by the image recognition processing, and images sequentially acquired by the acquisition processing A change amount detection process for detecting a change amount of data, and a subject determination method for executing the change by a computer, wherein the image recognition process includes: When subject tracking is performed by the subject tracking process, image recognition is performed only on image data in a region where the subject being tracked is present, and the amount of change detected by the change amount detection process is determined. If it is equal to or greater than the fixed amount, subject tracking by the subject tracking process is interrupted and image recognition is performed on the entire range of image data.

In addition, the program according to the present invention exists in each image data by causing the computer to perform image recognition on the acquisition unit that sequentially acquires the image data and the image data sequentially acquired by the acquisition unit. Image recognition means for recognizing the type of the object, and object determination means for determining the type of the object existing in common in the image data sequentially acquired by the acquisition means based on the recognition result by the image recognition means; Based on the image data sequentially acquired by the acquisition means, subject tracking means for tracking the subject existing in the image data recognized by the image recognition means, and the acquisition means sequentially acquire A program that functions as a change amount detection unit that detects a change amount of image data, wherein the image recognition unit When subject tracking is performed by the subject tracking means, image recognition is performed only on the image data of the area where the subject being tracked is present, and the amount of change detected by the change amount detection means is determined. If it is equal to or greater than the fixed amount, subject tracking by the subject tracking means is interrupted and image recognition is performed on the entire range of image data.

  According to the present invention, it is possible to improve the accuracy of the determination of what the imaged subject is.

Hereinafter, the present embodiment will be described in detail with reference to the drawings as an example in which the subject determination device of the present invention is applied to a digital camera.
[First Embodiment]
A. Configuration of Digital Camera FIG. 1 is a block diagram showing a schematic electrical configuration of a digital camera 1 that implements the subject determination device of the present invention.
The digital camera 1 includes a photographic lens 2, a lens driving block 3, an aperture / shutter 4, a CCD 5, a vertical driver 6, a TG (timing generator) 7, a unit circuit 8, a DMA controller (hereinafter referred to as DMA) 9, a CPU 10, and a key input. 11, memory 12, DRAM 13, DMA 14, motion vector detection unit 15, DMA 16, image generation unit 17, DMA 18, DMA 19, display unit 20, DMA 21, compression / decompression unit 22, DMA 23, flash memory 24, and bus 25. .

  The photographic lens 2 includes a focus lens and a zoom lens that are composed of a plurality of lens groups (not shown). A lens driving block 3 is connected to the photographing lens 2. The lens driving block 3 moves the focus lens and the zoom lens in the optical axis direction in accordance with a focus motor and a zoom motor (not shown) that drive the focus lens and the zoom lens in the optical axis direction, respectively, and a control signal sent from the CPU 10. And a zoom motor driver (not shown).

The diaphragm / shutter 4 includes a drive circuit (not shown), and the drive circuit operates the diaphragm / shutter 4 in accordance with a control signal sent from the CPU 10. The aperture / shutter 4 functions as an aperture and shutter.
The diaphragm is a mechanism for controlling the amount of light incident on the CCD 5, and the shutter is a mechanism for controlling the time for which light is applied to the CCD 5, and the time for which light is applied to the CCD 5 (exposure time). Depends on the shutter speed.
The exposure amount is determined by the aperture value (aperture level) and the shutter speed.

  The CCD 5 is scanned and driven by the vertical driver 6, photoelectrically converts the intensity of light of each color of the RGB value of the subject image at a constant period, and outputs it to the unit circuit 8 as an imaging signal. The operation timing of the vertical driver 6 and the unit circuit 8 is controlled by the CPU 10 via the TG 7.

  A TG 7 is connected to the unit circuit 8, a CDS (Correlated Double Sampling) circuit that holds the imaged signal output from the CCD 5 by correlated double sampling, and an AGC that performs automatic gain adjustment of the imaged signal after the sampling. (Automatic Gain Control) circuit and an A / D converter that converts the analog signal after the automatic gain adjustment into a digital signal. The image pickup signal obtained by the CCD 5 passes through the unit circuit 8 and is then Bayered by the DMA 9 The data is stored in the buffer memory (DRAM 13).

The CPU 10 has a function of performing AE processing, AF processing, and the like, and is a one-chip microcomputer that controls each part of the digital camera 1.
In particular, the CPU 10 includes an image recognition unit 101 that performs image recognition processing, and a subject determination unit 102 that determines the type of subject based on the recognition result of the image recognition processing.

The key input unit 11 includes a plurality of operation keys such as a shutter button, a mode switching key, a cross key, and a SET key that can be pressed halfway down, and outputs an operation signal corresponding to a user key operation to the CPU 10.
In the memory 12, a control program (an image recognition process, a program necessary for subject determination, etc.) necessary for the CPU 10 to control each part of the digital camera 1 and necessary data (feature data of each type of subject) are stored. The CPU 10 operates according to the program.

  The DRAM 13 is used as a buffer memory for temporarily storing image data picked up by the CCD 5 and also as a working memory for the CPU 10.

The DMA 14 reads Bayer data or luminance / chrominance signal image data stored in the buffer memory and outputs the data to the motion vector detection unit 15.
The motion vector detection unit 15 detects a motion vector of a certain region in the image data, and detects a motion vector of the certain region using a representative point matching method, a block matching method, or the like. Further, in order to detect this motion vector, the motion vector is detected by detecting where the certain area is in the next image data captured based on the image data of the certain area and the image data captured thereafter. Since the vector is detected, a storage circuit that holds the captured image data for a predetermined time is also included. The detected motion vector is sent to the CPU 10 via the DMA 14.
The motion vector detection unit 15 performs subject tracking.

The DMA 16 reads image data of Bayer data stored in the buffer memory and outputs it to the image generation unit 17.
The image generation unit 17 performs processing such as pixel interpolation processing, γ correction processing, and white balance processing on the image data sent from the DMA 16, and also generates a luminance color difference signal (YUV data). That is, it is a portion that performs image processing.
The DMA 18 stores the image data (YUV data) of the luminance / color difference signal subjected to image processing by the image generation unit 17 in a buffer memory.

The DMA 19 outputs image data of YUV data stored in the buffer memory to the display unit 20.
The display unit 20 includes a color LCD and its drive circuit, and displays an image of the image data output from the DMA 19.

The DMA 21 outputs the YUV data image data and compressed image data stored in the buffer memory to the compression / decompression unit 22, and buffers the image data compressed by the compression / decompression unit 22 and the decompressed image data. It is stored in memory.
The compression / decompression unit 22 is a part that performs compression / decompression of image data (for example, compression / decompression in JPEG or MPEG format).
The DMA 23 reads compressed image data stored in the buffer memory and records it in the flash memory 24, or stores the compressed image data recorded in the flash memory 24 in the buffer memory.

B. Operation of Digital Camera 1 The operation of the digital camera 1 in the first embodiment will be described with reference to the flowchart of FIG.

  When the subject determination process is started, in step S1, the CPU 10 acquires image data (Bayer data or luminance / color difference signal image data) that has been imaged most recently by the CCD 5 and stored in the buffer memory (DRAM 13). At this time, it is assumed that images are sequentially picked up by the CCD 5 at a predetermined frame rate, and the image data of the luminance and color difference signals sequentially picked up are sequentially displayed on the display unit 20.

  Here, it is assumed that the digital camera 1 is capturing a moving image of the moving elliptical object 31, and FIG. 3A shows the state at that time. At this time, the captured image data of the subject 31 is displayed on the display unit 20. Note that the optical system such as the photographing lens 2 and the CCD 5 is provided on the surface opposite to the display unit 20.

Next, in step S2, the image recognition unit 101 of the CPU 10 performs image recognition processing for recognizing what subject has been imaged based on the acquired image data.
Here, this image recognition is based on the captured image data. The feature data of the imaged subject (recognition of the contours and feature points of the subject and their positional relationship etc. is also recognized and digitized. (Numerical data) is calculated and compared with the characteristic data of each type of subject registered in advance to specifically recognize “what is the captured subject?”. At this time, among the registered feature data of each type of subject, the type of subject corresponding to the feature data having the highest degree of coincidence with the feature data of the captured subject is recognized as the captured subject. .
That is, by performing image recognition processing on the acquired image data, it is possible to recognize which of the registered feature data of each type of subject is the most matched feature data. A type of subject corresponding to the recognized feature data is recognized as a captured subject. As a result, the type of subject can be accurately recognized.

In addition, the user can register each type of subject in a registration mode or the like. In this registration mode or the like, when the user designates the type of subject image that the user wants to register, the CPU 10 registers the feature data calculated by image recognition from the subject image. The registered feature data is recorded in the memory 12. Note that feature data of each predetermined type of subject may be recorded in advance.
Here, for the sake of convenience, the registered subject types will be described as circles, ellipses, and rhombuses, but the registered subject types are not limited to this. Good.

  Next, in step S3, the CPU 10 stores the recognition result of the image recognition process, that is, the type of subject recognized by the image recognition process and the degree of coincidence thereof in the buffer memory. That is, the type of subject recognized as having the highest degree of coincidence and the degree of coincidence are stored.

Next, in step S4, the CPU 10 determines whether or not the image recognition process has been executed a predetermined number of times (here, 6 times). That is, the image recognition process is performed on a predetermined number (six in this case) of image data.
If it is determined in step S4 that the image recognition process has not been executed a predetermined number of times, the process returns to step S1 to repeat the above operation. Accordingly, it is possible to perform processing for specifically recognizing “what is the subject” for the predetermined number of times of image data.

FIG. 3B is a diagram illustrating an example of captured image data and the state of the type of subject and the degree of coincidence stored based on the recognition result of the captured image data by the image recognition process. .
Referring to FIG. 3B, the image recognition result of the first image data (image data subjected to the first image recognition process), the image recognition result of the second image data, and the third image recognition. As a result, it can be seen that the image recognition results up to the sixth image data are stored.

By the image recognition processing of the first image data, the type of the recognized subject (the type of subject recognized as having the highest degree of matching) is “ellipse”, and the degree of matching is “0.7”. I understand. The degree of coincidence is highest at “1.0”, and “1.0” is obtained when the feature data of the photographed subject and the feature data of the registered subject completely coincide with each other. The degree of coincidence decreases as the time shifts. In other words, the higher the degree of coincidence, the higher the probability that the subject existing in the image data is the recognized subject type. Thereby, it is possible to recognize the probability of the type of the recognized object with high accuracy.
Also, it can be seen that the type of subject recognized by the image recognition processing of the second image data is “circle” and the degree of coincidence is “0.5”. It can be seen that the type of subject recognized by the processing is “ellipse” and the degree of coincidence is “0.4”. In this way, the recognition result by each image recognition process is stored in the buffer memory.

On the other hand, if it is determined in step S4 that the image recognition process has been executed a predetermined number of times, the process proceeds to step S5, where the subject determination unit 102 of the CPU 10 totals the recognition results and selects the type of subject that has been recognized most frequently. As can be seen from FIG. 3B, the most recognized object type is “ellipse”, and therefore an ellipse is selected as the object type.
Next, in step S6, the subject determination unit 102 of the CPU 10 calculates an average value of the degree of coincidence when the subject is recognized as the selected type of subject. Here, since the type of the selected subject is “ellipse”, the average value of the degree of coincidence when it is recognized as an ellipse is calculated. That is, the degree of coincidence when recognized as “ellipse” is “0.7”, “0.4”, “0.5”, and “0.8”, respectively, so the calculated average value is “0.6”.

Next, in step S7, the subject determination unit 102 of the CPU 10 determines whether or not the calculated average value is equal to or greater than a predetermined value.
If it is determined in step S7 that the calculated average value is greater than or equal to the predetermined value, the process proceeds to step S8, and the subject determination unit 102 of the CPU 10 outputs the type of subject selected in step S5 as a result of subject determination. That is, the imaged subject is determined as the type of subject selected in step S5.
On the other hand, if it is determined in step S7 that the calculated average value is not equal to or greater than the predetermined value, the process proceeds to step S9, and the subject determination unit 102 of the CPU 10 outputs that the subject cannot be determined as a result of the subject determination. That is, it is determined that the type of subject is unknown.

As described above, in the first embodiment, image recognition processing is performed on a plurality of image data obtained by moving image capturing, and the type of subject is determined based on the recognition result. The accuracy of the determination of what the subject is can be improved, and it is possible to determine what the subject being imaged reliably.
Further, based on the recognition result of the image recognition process, when the average value of the degree of coincidence with the most recognized type of subject is equal to or greater than a predetermined value, the most recognized type of subject is captured. Therefore, the determination accuracy can be improved, and it is possible to determine what the subject being imaged reliably.
Further, since the image recognition is performed only a predetermined number of times, the subject determination processing time does not become longer than necessary, and the subject determination can be performed quickly.

[Second Embodiment]
Next, a second embodiment will be described.
Also in the second embodiment, the subject determination apparatus of the present invention is realized by using the digital camera 1 having the same configuration as that shown in FIG. Here, it is assumed that the image recognition unit 101 performs face recognition processing which is also an aspect of image recognition processing. It is possible to recognize who is the subject being imaged by this face recognition (type of subject). That is, in the first embodiment, the types of subjects such as dogs, cats, cars, people, etc. are roughly recognized as the types of subjects. You will know who you are specifically in detail.

C. Operation of Digital Camera 1 The operation of the digital camera 1 according to the second embodiment will be described below with reference to the flowchart of FIG.

  When the subject processing determination process is started, in step S31, the CPU 10 obtains the latest image captured by the CCD 5 and stored in the buffer memory (Bayer data or luminance / chrominance signal image data). Also at this time, it is assumed that images are sequentially captured by the CCD 5 at a predetermined frame rate, and the image data of the luminance color difference signals sequentially captured are sequentially displayed on the display unit 20.

  Next, in step S32, the CPU 10 determines whether or not a predetermined period has arrived. The predetermined cycle is a cycle in which a predetermined time has elapsed from the time when the image recognition process is performed (however, a cycle longer than the imaging cycle of the CCD). At this time, if the image recognition process has not been performed once, it is determined that a predetermined period has arrived. The predetermined period may be a period in which a predetermined number of image data is captured from the image data subjected to the image recognition process.

If it is determined in step S32 that the predetermined period has not arrived, the process returns to step S31. If it is determined in step S32 that the predetermined period has arrived, the process proceeds to step S33, and the CPU 10 performs subject tracking on the face being imaged. Judge whether or not. That is, it is determined whether or not the motion vector detection unit 15 has detected the motion vector of the face (subject) being imaged.
If it is determined in step S33 that subject tracking is not performed on the face being imaged, the process proceeds to step S34, where the image recognition unit 101 of the CPU 10 is based on the entire range of image data acquired most recently in step S31. Then, face recognition processing for recognizing who has been imaged is performed, and the process proceeds to step S36. As a result, it is possible to recognize who is the person imaged (the type of subject).

Here, this face recognition processing is performed by the method described in the first embodiment. Specifically, based on the acquired image data, recognition of the position of the eyes, the position of the mouth, the position of the nose, and the positional relationship thereof are also recognized, and these are converted into numerical values so that the face numerical data (feature data) ) Is calculated and compared with the facial feature data of the registered person to recognize who the imaged person is. At this time, of the registered facial data of each type of person, the type of face corresponding to the feature data having the highest degree of coincidence with the facial data of the captured person's face is recognized as the captured person. It will be. In other words, by performing image recognition processing on the acquired image data, it is possible to recognize which of the registered face data of the person is the most matched feature data. The person corresponding to the feature data thus identified is recognized as the person who has been imaged. Thereby, the type of person can be recognized with high accuracy.
In addition, the face recognition process also identifies the position and size of the recognized face on the image data, and the face area is determined based on the position and size of the face.

  In addition, the user can register a person's face in the face registration mode or the like. In this face registration mode or the like, when the face image of a person to be registered is designated by the user, the CPU 10 registers numerical data (feature data) calculated by face recognition from the face image of the person. The registered facial feature data is recorded in the memory 12.

On the other hand, if it is determined in step S33 that the subject is being tracked with respect to the face being imaged, the process proceeds to step S35, and the CPU 10 determines that the following face is the most recent image data acquired in step S31. Based on the image data of the existing face area, face recognition processing for recognizing who has been imaged is performed, and the process proceeds to step S36. The face area where the following face is present is specified by the motion vector or face area information sent from the motion vector detecting unit 15.
In this way, when subject tracking is performed, the face recognition process is performed only on the image data of the face area where the following face is present, so the burden of the face recognition process can be reduced.

In step S36, the CPU 10 stores the recognition result of the face recognition process, that is, the type of person recognized by the image recognition process and the matching degree thereof in the buffer memory. That is, the person of the type recognized as having the highest degree of coincidence and the degree of coincidence are stored.
Next, in step S37, the CPU 10 determines whether or not subject tracking is performed on the recognized face.

If it is determined in step S37 that the subject tracking is not performed, the process proceeds to step S38, and the CPU 10 causes the motion detection block 15 to start the subject tracking process on the recognized face, and then proceeds to step S39. When the subject tracking is started, the CPU 10 sends the image data of the recognized face area to the motion vector detecting unit 15, and also sends the image data acquired in step S31 to the motion vector detecting unit 15 and then the motion vector detecting unit 15 Starts the process of detecting a motion vector by detecting where the recognized face is in the image data by the block matching method. The detected motion vectors are sequentially sent to the CPU 10. Note that information (face area information) indicating in which range of image data the detected face is located may be sequentially sent to the CPU 10.
As described above, the motion vector detection unit 15 sequentially sends motion vector information or face area information to the CPU 10, so that the CPU 10 specifies where the recognized face exists in the sequentially captured image data. Can do.

  The operation of the motion vector detection unit 15 will be briefly described. The recognition is performed by detecting where the image data of the recognized face area exists in the image data captured next by the block matching method or the like. Detect face motion vectors. Further, when the next image data is acquired, a motion vector is detected by detecting where the image data of the detected face area of the previous image data is in the acquired next image data. Is detected.

  Note that the CPU 10 may display a tracking frame on the image data based on the motion vector (or face area information) detected by the motion vector detection unit 15. As a result, the follow frame is displayed following the face recognized.

On the other hand, if it is determined in step S37 that the subject tracking is being performed, the process proceeds to step S39 as it is.
In step S39, the subject determination unit 102 of the CPU 10 totals the recognition results and selects the most recognized person type.
Next, in step S40, the subject determination unit 102 of the CPU 10 calculates an average value of the degree of coincidence when it is recognized as the selected type of person.

Next, in step S41, the subject determination unit 102 of the CPU 10 determines whether or not the calculated average value is greater than or equal to a predetermined value.
If it is determined in step S41 that the calculated average value is not equal to or greater than the predetermined value, the process returns to step S31 and the above-described operation is repeated. In other words, the subject is not determined unless the calculated average value exceeds a predetermined value.
On the other hand, if it is determined in step S41 that the calculated average value is greater than or equal to the predetermined value, the process proceeds to step S42, where the subject determination unit 102 of the CPU 10 determines the type of person most recently selected in step S39 as a result of subject determination. Output as. That is, the most recently selected type of person is determined to be the subject being imaged.

  As described above, in the second embodiment, when subject tracking is not performed on the face recognized by the face recognition process, the face recognition process is performed on all the image data, and the face recognition process is performed. When this is done, subject tracking is started for the face that has been recognized, and face recognition processing is performed only on the image data of the face area of the face that has been tracked in a predetermined cycle.

  Further, the face recognition process is performed until the average value of the degree of coincidence when the person is recognized as the most recognized type among the recognition results obtained by the face recognition process, that is, the subject is not determined. Therefore, it is possible to determine the type of person being imaged with high accuracy. In this case, regardless of the number of times of image recognition processing, the subject is determined based on whether or not the average value of the degree of coincidence when it is recognized as the most recognized type of person is greater than or equal to a predetermined value. If the degree of coincidence with the type of person recognized by the image recognition process is greater than or equal to a predetermined value, the recognized type of person is determined as the subject.

FIG. 5 shows a state of movement transition of a subject followed by subject tracking, a face area targeted for face recognition processing, and a person type and matching degree stored based on a recognition result by face recognition processing. It is a figure which shows an example.
FIG. 5 shows the movement of the person followed by the subject tracking. A square displayed over the face position indicates the face area that is the target of the face recognition process. That is, it can be seen that the face recognition process is not performed on the face following the subject each time an image is taken, but at a predetermined interval.
It can be seen that the type of person recognized by the face recognition process and the degree of coincidence are stored.

  It can be seen that the type of person recognized by the first face recognition process (the type of person recognized as having the highest degree of matching) is “person A”, and the degree of matching is “0.6”. It can also be seen that the type of person recognized by the second face recognition process is “person B” and the degree of coincidence is “0.4”. In this way, each time the face recognition process is performed, the recognition result by the face recognition process is stored in the buffer memory.

Here, for example, if the predetermined value is “0.7”, the degree of coincidence of the person A recognized by the first face recognition process is “0.6”, so the subject is not determined and the second face recognition is performed. The matching degree of the person B recognized by the processing is “0.4”, so the subject is not determined, and the person recognized by the third face recognition processing is the person A and the matching degree is “0.7”. However, since the average value of the degree of coincidence of the most recognized person A is “0.65”, the subject is not yet determined.
Then, the image recognition process is performed for the fourth time and the fifth time, and the average value of the degree of coincidence of the person A most recognized for the first time is “0.7”. Will be output as

  As described above, in the second embodiment, face recognition processing is performed on each of a plurality of image data obtained by moving image capturing, and who is captured based on the recognition result is determined. Since the determination is made, it is possible to improve the accuracy of determining who the person being imaged is, and it is possible to determine who is the person being imaged reliably.

In addition, the face recognition process is initially performed on the entire range of image data, and for image data captured thereafter, the area of the recognized face in the image data (face area) is included. Since the face recognition process is performed only on the image data, the processing burden due to the face recognition process can be reduced.
In addition, since the subject tracking is performed on the face by detecting the motion vector of the face recognized by the face recognition processing, it is possible to determine in which area of the image data captured thereafter there is a recognized face. It can be identified with high accuracy.

  Further, face recognition processing is performed until the average value of the degree of coincidence with the person of the type most recognized by the face recognition processing becomes equal to or greater than a predetermined value. Since the determined type of person is determined as the person being imaged, it is possible to improve the determination accuracy and determine who is the person being imaged reliably.

[Modification]
D. The above-described embodiment can be modified as follows.

  (01) In the first embodiment, image recognition is performed for the entire range of the imaged image data. However, as in the second embodiment, Initially, the image recognition process may be performed on the entire range of the image data, and thereafter, the image recognition process may be performed on the image data in the area where the recognized subject is present. At this time, the area where the recognized subject is present can be specified by performing subject tracking.

  (02) In the first embodiment, it is determined in step S7 whether or not the calculated average value is equal to or greater than a predetermined value. If the average value is equal to or greater than the predetermined value, the subject selected in step S5 is determined. Although the type is output as the subject determination result, the most recognized type of subject may be output as the subject determination result. In this case, if the type of subject most recognized in step S5 is selected, the process proceeds to step S8 as it is. This also improves the accuracy of determining who is being imaged.

(03) In the first embodiment, the image recognition process is performed a predetermined number of times. However, as in the second embodiment, the degree of coincidence with the most recognized type of subject is increased. Image recognition processing may be performed until the average value is equal to or greater than a predetermined value.
In this case, when the operation of step S3 is completed, the process proceeds to step S5. If it is determined in step S7 that the average value is not equal to or greater than the predetermined value, the process returns to step S1, and if it is determined that the average value is equal to or greater than the predetermined value, the process proceeds to step S8. Like that.

  In this case, if it is determined in step S7 that the average value is not equal to or greater than the predetermined value, the process returns to step S1 only when the number of times the image recognition process has been performed is smaller than the predetermined number, instead of returning to step S1. If the number of times that the image recognition processing has been performed is not smaller than a certain number, a message indicating that the subject cannot be determined is output. In other words, even if the image recognition process is performed many times, the average value of the degree of coincidence of the most recognized subject may not be equal to or greater than a predetermined value. It will be judged. Thus, subject determination processing time does not become longer than necessary, and subject determination can be performed quickly.

  (04) In the first embodiment, the image recognition process is performed for each captured image data. However, the image recognition process may be performed at a predetermined cycle (every predetermined number). .

(05) In the second embodiment, first, face recognition processing is performed on the entire range of captured image data, and thereafter, the recognized face is converted into image data of a face area. The face recognition process is performed on the image data, but the face recognition process may be performed on the entire range of the image data at a predetermined interval.
There is a possibility that a subject that is not in the image data that has undergone the initial face recognition process will be captured later, and that the subject that has been captured will once protrude from the angle of view and be captured again. In such a case, the newly captured subject and the type of the re-captured subject cannot be determined. Therefore, the face recognition process is performed again on the entire range of the image data to newly capture the image. The type of subject can also be determined.

Further, when the captured image changes greatly, face recognition processing may be performed on the entire range of the image data. If the image has changed significantly, it is highly likely that the person who is the subject tracking subject has been lost, and the subject tracking can be performed accurately by performing face recognition processing on the entire range of the image data again. And the accuracy of subject determination can be improved.
As a case where the change of the captured image is large, the motion vector of the subject being followed by subject tracking is larger than a predetermined amount, or the motion vectors of a plurality of regions are set in the motion vector detection unit 15 separately from subject tracking. There are cases where the motion vectors of the plurality of detected areas are any one or the average is larger than a predetermined amount.

  (06) In the second embodiment, when it is determined that the average value of the degree of coincidence of the most recognized person in step S41 is not equal to or greater than a predetermined value, the process returns to step S31 as it is. If it is determined that the average value of the degree of coincidence is not equal to or greater than a predetermined value, it is determined whether or not the number of face recognition processes is smaller than a certain number. If the number of face recognition processes is smaller than a certain number, the process returns to step S31. If the number of face recognition processes is not smaller than a certain number, it may be output that the subject cannot be determined. In other words, even if face recognition processing is performed many times, the average value of the degree of coincidence of the most recognized type of subject may not be equal to or greater than a predetermined value. It will be judged. Thus, subject determination processing time does not become longer than necessary, and subject determination can be performed quickly.

(07) In the second embodiment, when the average value of the degree of coincidence with the person most recognized by the face recognition process is equal to or greater than a predetermined value, the person most recognized. Is output as a result of subject determination, but if the degree of coincidence with the type of person most recently recognized by the face recognition process is equal to or greater than a predetermined value, the type of person recognized most recently is determined as subject determination. The result may be output. That is, if the degree of coincidence with the type of person recognized by the face recognition process is equal to or greater than a predetermined value, the recognized type of person is immediately output as a result of subject determination.
For example, in the case where the predetermined value is “0.7”, as shown in FIG. 5, the type of person recognized as having the highest degree of coincidence by the third face recognition process is “person A”. Since the degree of coincidence is “0.7”, the person A is output as a result of subject determination when the third face recognition process is performed.

(08) In the second embodiment, the person with the most recognized face is selected in step S39, and the average value of the matching degrees of the selected types of persons is greater than or equal to a predetermined value in step S41. If there is, the selected type of person is output as a determination result in step S42. However, the image recognition process is performed a predetermined number of times by image recognition until the same type of person is recognized. When a type of person is recognized, the type of person recognized a predetermined number of times may be output as a determination result. Thereby, it is possible to improve the accuracy of the determination of who the person being imaged is, and it is possible to determine who the person being imaged reliably.
In this case, when a person of the same type is recognized a predetermined number of times, if the average value of the degree of coincidence with the recognized person is equal to or greater than a predetermined value, the person of the type recognized the predetermined number of times May be output as a determination result, and if the average value of the degree of coincidence is not equal to or greater than a predetermined value, the determination result may indicate that the subject cannot be determined. Thereby, it is possible to improve the accuracy of the determination of who the person being imaged is, and it is possible to determine who the person being imaged reliably.

  Further, when a person of the same type is recognized continuously a predetermined number of times, the person of the type recognized continuously a predetermined number of times may be output as a determination result. In this case, if the average value of the degree of coincidence with a person who has been continuously recognized a predetermined number of times is not greater than or equal to the predetermined value, a determination result may be output indicating that the subject cannot be determined. Image recognition may be performed until the average value exceeds a predetermined value. Thereby, it is possible to improve the accuracy of the determination of who the person being imaged is, and it is possible to determine who the person being imaged reliably.

(09) In the second embodiment, face recognition is performed. However, image recognition as described in the first embodiment may be performed.
Further, face recognition is performed in a predetermined cycle (every predetermined number of sheets), but face recognition may be performed on all acquired image data (for each image data).
Further, instead of face recognition, other recognition (car recognition, bird recognition, animal recognition, etc.) may be used. For example, a person who performs birdwatching can recognize “what kind of bird is the bird being imaged” by registering characteristic data of a plurality of types of birds in advance.

  (10) Further, in each of the above embodiments, the case where there is a single subject to be imaged has been described, but of course there may be a plurality of cases. At this time, when there are a plurality of subjects to be imaged, subject recognition is performed on each recognized subject, and image recognition processing is simply performed on the region of the subject being followed. Rather than doing, it is possible to distinguish which subject is recognized as which type of subject.

(11) Further, in each of the above embodiments, the feature data of the subject viewed from multiple directions (a plurality of directions) is recorded as the feature data of the subject to be registered (including the face of a person). Furthermore, the accuracy of image recognition (face recognition) processing can be increased.
For example, if only the feature data of the face when viewed from the front is registered, when the face is seen from the digital camera 1 and turned sideways or obliquely (when not facing the front ), Even though the face of the registered person is being imaged, there is a high possibility that it will not be recognized as the face of the registered person, especially if the person is moving .
Therefore, the accuracy of subject tracking can be improved by recording the feature data viewed from all directions (front, diagonal right, horizontal, diagonal left, left, etc.) for one registered subject.

(12) In the above embodiments, the image recognition in step S2 in FIG. 2 and the face recognition in step S34 in FIG. 4 are performed on the entire range of image data. Among them, image recognition and face recognition may be performed on a predetermined region (for example, the center region of the angle of view). Thereby, the processing burden of image recognition and face recognition can be reduced. In particular, when image recognition and face recognition are performed on the central area of the angle of view, the processing burden can be reduced, and the type of the main subject can be recognized and determined effectively. This is because the user normally captures an image so that the subject (main subject) that he / she really wants to shoot is in the center region of the angle of view.
In addition, the case where image recognition and face recognition are performed on a predetermined area has been described only in step S2 and step S34, but the point is that the image data of the area where the subject being tracked exists exists. Any method other than image recognition and face recognition may be used.

  (13) In each of the above embodiments, image data captured by moving image capturing is acquired. However, image data obtained by continuously capturing still images may be acquired.

  (14) In the above embodiments, image data currently captured by the CCD 5 is acquired. However, image data captured in the past (for example, a frame image of recorded moving image data). May be obtained.

  (15) Moreover, the aspect which combined each said embodiment and each said modification arbitrarily may be sufficient.

(16) Each of the above-described embodiments and modifications of the present invention is merely an example as the best embodiment so that the principle and structure of the present invention can be better understood. Therefore, it is not intended to limit the scope of the appended claims.
Therefore, it should be understood that all the various variations and modifications that can be made to the above-described embodiments of the present invention are included in the scope of the present invention and protected by the appended claims.

  Finally, in each of the above embodiments, the case where the subject determination device of the present invention is applied to the digital camera 1 has been described. However, the present invention is not limited to the above embodiment, and the main point is what the subject is. Any device that can determine whether or not can be applied.

1 is a block diagram of a digital camera 1 according to an embodiment of the present invention. It is a flowchart which shows operation | movement of the digital camera 1 of 1st Embodiment. The state when the moving elliptical subject 31 is captured as a moving image, and the type and coincidence of the subject stored based on the captured image data and the recognition result of the image recognition processing of the captured image data It is a figure which shows the mode of a degree. It is a flowchart which shows operation | movement of the digital camera 1 of 2nd Embodiment. The figure which shows the mode of the movement of the subject tracked by subject tracking, the face area targeted for the face recognition process, and the type of person stored based on the recognition result by the face recognition process and the degree of coincidence is there.

Explanation of symbols

1 Digital Camera 2 Shooting Lens 3 Lens Drive Block 4 Shutter / Shutter 5 CCD
6 Vertical driver 7 TG
8 Unit circuit 9 DMA
10 CPU
11 Key input section 12 Memory 13 DRAM
14 DMA
15 motion vector detector 16 DMA
17 Image generator 18 DMA
19 DMA
20 Display unit 21 DMA
22 Compression / decompression unit 23 DMA
24 flash memory 25 bus

Claims (17)

Acquisition means for sequentially acquiring image data;
By performing image recognition on a specific subject portion that exists in common in the image data sequentially acquired by the acquisition means, the types of subjects corresponding to the specific subject portion are sequentially recognized. Image recognition means;
The aggregate type of a subject by Ri sequentially recognizing the image recognition unit, and determining subject determination means more recognized type of the subject as the type of the object corresponding to the specific object portion,
A subject determination apparatus comprising: The subject determination unit counts the types of subjects sequentially recognized by the image recognition unit every predetermined number of times, and determines the types of subjects recognized more frequently for the subject corresponding to the specific subject portion at the time of counting. The subject determination apparatus according to claim 1, wherein the subject determination device is determined as a type. The image recognition means calculates the degree of coincidence at the time of each image recognition,
The subject determination means periodically totals the degree of coincidence calculated by the image recognition means, and determines the type of subject that is determined when the total degree of coincidence exceeds a predetermined value. The subject determination apparatus according to claim 2, wherein Subject tracking means for performing subject tracking on the specific subject portion moving in the image data sequentially acquired by the acquisition means ;
The image recognizing unit sequentially recognizes the type of the subject in the image data sequentially acquired by the acquiring unit with respect to the specific subject portion where the subject tracking is performed by the subject tracking unit. The subject determination device according to claim 1, wherein the subject determination device is a device. The image recognition means includes
When the specific subject portion is not tracked by the subject tracking means, image recognition is performed on the entire range of image data, and the specific subject portion is tracked by the subject tracking means. If, by performing an image recognition only for this particular object portion, the object determining device according to claim 4, wherein the recognizing the type of the object corresponding to the specific object portion. A change amount detecting means for detecting a change amount of image data sequentially obtained by the obtaining means;
The image recognition means includes
The subject determination apparatus according to claim 5, wherein when the amount of change detected by the change amount detector is equal to or greater than a predetermined amount, image recognition is performed on the entire range of image data. Wherein the predetermined number of times by the image recognition unit, if the same type of subject continuously is recognized, selecting means for selecting a portion corresponding to the object of the type specified number recognized by the continuous as the specific object portion The subject determination apparatus according to claim 1 , further comprising: The subject determination means periodically sums up the average value of the degree of coincidence calculated by the image recognition means, and the type of subject that is determined when the average value of the degree of coincidence is equal to or greater than a predetermined value. 4 is output as a final determination result. If the average value of the degree of coincidence calculated for each predetermined number of images is not equal to or greater than a predetermined value, the subject determination unit does not know the type of subject existing in common in the image data sequentially acquired by the acquisition unit. 9. The subject determination apparatus according to claim 8, wherein Features feature recording means for recording feature information for each type of subject,
The image recognition means includes
The feature information of the specific subject portion is calculated by image recognition, and the type of subject corresponding to the feature information recorded in the feature information recording unit that most closely matches the calculated feature information is set as the specific subject portion. The subject determination apparatus according to claim 1 , wherein the subject determination device recognizes the type of the corresponding subject. The feature information recording means includes
11. The subject determination apparatus according to claim 10, wherein feature information corresponding to a plurality of directions is recorded for each type of subject. Image recognition by the image recognition means is:
The subject determination device according to claim 1 , wherein the subject recognition device recognizes who the subject is.   Acquisition processing to acquire image data sequentially;
  By performing image recognition on a specific subject portion that exists in common in the image data sequentially acquired by the acquisition processing, the types of subjects corresponding to the specific subject portion are sequentially recognized. Image recognition processing,
  Subject determination that aggregates the types of subjects that are sequentially recognized by the image recognition processing, and determines the types of subjects that are recognized more as the types of subjects that exist in common in the image data sequentially acquired by the acquisition processing Processing,
  And subjecting each of the above processes to be executed by a computer. Acquisition processing to acquire image data sequentially;
By performing image recognition on a specific subject portion that exists in common in the image data sequentially acquired by the acquisition processing, the types of subjects corresponding to the specific subject portion are sequentially recognized. Image recognition processing,
The aggregate type of a subject by Ri sequentially recognizing the image recognition processing, determines more recognized type of the subject, as the type of subject present in common to the image data sequentially acquired by the acquisition process Subject determination processing;
A program characterized by causing a computer to execute the processes described above.   Acquisition means for sequentially acquiring image data;
  Image recognition means for recognizing the type of subject existing in each image data by performing image recognition on the image data sequentially acquired by the acquisition means;
  Subject determination means for determining the type of subject existing in common in the image data sequentially acquired by the acquisition means based on the recognition result by the image recognition means;
  Subject tracking means for performing subject tracking on a subject existing in the image data recognized by the image recognition means based on the image data sequentially acquired by the acquisition means;
  Change amount detecting means for detecting a change amount of image data sequentially acquired by the acquiring means;
  With
  The image recognition means includes
  When subject tracking is performed by the subject tracking means, image recognition is performed only on image data in an area where the subject being tracked is present, and the amount of change detected by the change amount detection means is determined. An object determination device characterized in that, if it is equal to or greater than a fixed amount, the object tracking by the object tracking means is interrupted and image recognition is performed on the entire range of image data.   Acquisition processing to acquire image data sequentially;
  Image recognition processing for recognizing the type of subject existing in each image data by performing image recognition on the image data sequentially acquired by the acquisition processing;
  Subject determination processing for determining the type of subject existing in common in the image data sequentially acquired by the acquisition processing based on the recognition result by the image recognition processing;
  Subject tracking processing that performs subject tracking on subjects existing in the image data recognized by the image recognition processing based on the image data sequentially acquired by the acquisition processing;
  A change amount detection process for detecting a change amount of image data sequentially acquired by the acquisition process;
  Is a subject determination method that is executed by a computer,
  The image recognition process includes:
  When subject tracking is performed by the subject tracking process, image recognition is performed only on image data in a region where the subject being tracked is present, and the amount of change detected by the change amount detection process is determined. A subject determination method comprising: subjecting subject tracking by the subject tracking process to image recognition for the entire range of image data when the amount is equal to or greater than a fixed amount.   Computer
  Acquisition means for sequentially acquiring image data;
  Image recognition means for recognizing the type of subject existing in each image data by performing image recognition on the image data sequentially acquired by the acquisition means;
  Subject determination means for determining the type of subject existing in common in the image data sequentially acquired by the acquisition means based on the recognition result by the image recognition means;
  Subject tracking means for performing subject tracking on a subject existing in the image data recognized by the image recognition means based on the image data sequentially acquired by the acquisition means;
  Change amount detecting means for detecting a change amount of image data sequentially acquired by the acquiring means;
  A program that allows
  The image recognition means includes
  When subject tracking is performed by the subject tracking means, image recognition is performed only on image data in an area where the subject being tracked is present, and the amount of change detected by the change amount detection means is determined. A program characterized by interrupting subject tracking by the subject tracking means and performing image recognition for the entire range of image data when the amount is equal to or greater than a fixed amount.

Images