JP2016019118A - Image processing apparatus, imaging apparatus including the same, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus that improves accuracy of identifying a subject, an imaging apparatus including the same, and a program.SOLUTION: The image processing apparatus includes: a determination part for determining an identification level of a subject included in a through image photographed by an imaging part; and an association part for associating identification information for identifying a subject and a photographed image containing the photographed subject when it is determined by the determination part that an identification level satisfies predetermined standards.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus, an imaging apparatus including the image processing apparatus, and an image processing program.

  2. Description of the Related Art In recent years, digital cameras that detect the face of a person in an imaging range and can recognize a person individually are known.

JP 2006-72770 A

  For example, the classifier that identifies the subject is not accurate enough to identify the subject.

  An object of the present invention is to provide an image processing apparatus that improves the identification accuracy of a subject, an imaging apparatus including the image processing apparatus, and an image processing program.

In order to achieve the above object, an image processing apparatus according to an aspect of the present invention includes a determination unit that determines an identification level of a subject included in a through image captured by an imaging unit, and an identification level that satisfies a predetermined reference by the determination unit. When it determines with satisfy | filling, it is provided with the correlation part which associates the identification information for identifying a to-be-photographed object, and the picked-up image containing the to-be-photographed object imaged by the imaging part.
In order to achieve the above object, an image processing apparatus according to an aspect of the present invention includes a determination unit that determines an identification level of a subject that appears in a series of captured images that are continuously captured, and an identification level that is determined by the determination unit. When it is determined that the predetermined criterion is satisfied, the information processing apparatus includes an association unit that associates identification information for identifying a subject and a captured image selected from a series of captured images.
In order to achieve the above object, an imaging device according to one embodiment of the present invention images an object, outputs an image data, and displays a live view image based on the image data output by the imaging unit. A display unit; an input unit for capturing an image of the subject by the imaging unit; and an association unit that associates an image including the object imaged based on an input from the input unit and a live view image including the object. Features.
In order to achieve the above object, an image processing program according to an aspect of the present invention includes a process for causing a determination unit to determine the identification level of a subject in a through image, and the determination unit satisfies a predetermined level. When the determination is made, the computer is caused to execute processing for associating the identification information for identifying the subject and the captured image obtained by capturing the subject with the associating unit.

  According to the present invention, subject identification accuracy is improved.

1 is a system configuration diagram of a camera 10 as an imaging apparatus according to an embodiment of the present invention. It is a functional block diagram of the determination part 44 with which the camera 10 as an imaging device by one Embodiment of this invention was equipped. It is a flowchart which shows an example of the flow of operation | movement of the image processing apparatus by one Embodiment of this invention, and an imaging device provided with the same. It is a flowchart which shows an example of the flow of the score determination process among the operation | movement of the image processing apparatus by one Embodiment of this invention, and an imaging device provided with the same. It is a flowchart which shows an example of the flow of the object site | part correlation process among the operation | movement of the image processing apparatus by one Embodiment of this invention, and an imaging device provided with the same. FIG. 3 is a diagram (part 1) schematically illustrating an operation of the image processing device and the imaging device including the image processing device according to the embodiment of the present invention. FIG. 6 is a diagram (part 2) schematically illustrating the operation of the image processing device according to the embodiment of the present invention and the imaging device including the image processing device. FIG. 6 is a diagram (part 3) schematically illustrating the operation of the image processing device and the imaging device including the image processing device according to the embodiment of the present invention; It is FIG. (4) which shows typically operation | movement of the image processing apparatus by one Embodiment of this invention, and an imaging device provided with the same. It is a flowchart which shows an example of the flow of a score determination process among operation | movement of the image processing apparatus by the modification of one Embodiment of this invention, and an imaging device provided with the same. It is a flowchart which shows an example of the flow of the additional information relevant process of the score determination processes based on the through image in the modification of one Embodiment of this invention. It is a flowchart which shows an example of the flow of an object site | part correlation process among operation | movement of the image processing apparatus by the modification of one Embodiment of this invention, and an imaging device provided with the same.

An image processing apparatus, an imaging apparatus including the image processing apparatus, and a program according to an embodiment of the present invention will be described with reference to FIGS. In the present embodiment, a camera will be described as an example of the imaging apparatus. The imaging apparatus according to the present embodiment is not limited to a camera, and may be a portable electronic device such as a mobile phone, a smartphone, or a notebook personal computer as long as the imaging device is provided.
First, a schematic configuration of the camera 10 according to the present embodiment will be described with reference to FIGS. 1 and 2. Although details will be described later, the image processing apparatus according to the present embodiment, the imaging apparatus and the program provided with the image processing apparatus track identification information of the subject that is determined to have the highest identification level while tracking the subject in the through image. The subject is stored in association with an image file based on a captured image obtained by capturing the subject.

  FIG. 1 is a system configuration diagram of a camera 10 as an imaging apparatus according to the present embodiment. As shown in FIG. 1, the camera 10 is abbreviated as a system control unit 11, a system memory 12, an optical system 20, an image sensor (imaging unit) 31, and analog / digital (hereinafter, “A / D”). ) Converter 32, internal memory 33, image processing unit (image processing device) 34, display control unit 36, display unit 37, and recording medium interface 38 (hereinafter, the interface is abbreviated as “IF”). , An operation member 39, a lens control unit 41, an exposure control unit 42, a determination unit 44, and a recording medium 50. The system control unit 11, the A / D converter 32, the internal memory 33, the image processing unit 34, the display control unit 36, the recording medium IF 38, and the determination unit 44 are connected to the bus line 40, respectively. The predetermined control signal, the predetermined image data, and the like are sent via the bus line 40 to the system control unit 11, the A / D converter 32, the internal memory 33, the image processing unit 34, the display control unit 36, the recording medium IF 38, and the determination unit. 44 can be transmitted and received.

  The system control unit 11 directly or indirectly controls each element constituting the camera 10. The system control unit 11 communicates with the system memory 12. The system memory 12 is an electrically erasable / recordable nonvolatile memory. The system memory 12 is configured by, for example, Electrically Erasable Programmable Read-Only Memory (EEPROM (registered trademark)). The system memory 12 records constants, variables, setting values, programs, and the like necessary when the camera 10 is operating so that they are not lost even when the camera 10 is not operating.

  The optical system 20 includes a lens group 21 including a zoom lens, a focus lens, a camera shake correction lens, and the like, and a shutter 22. The subject image enters the optical system 20 along the optical axis indicated by the broken-line arrow in the drawing, and forms an image on the imaging surface of the image sensor 31 via the lens group 21 and the shutter 22.

  The imaging element 31 is an element that photoelectrically converts an optical image that is a subject image that is transmitted through the optical system 20 and is incident thereon. For example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor is used as the imaging element 31. The subject image photoelectrically converted by the image sensor 31 is converted from an analog signal to a digital signal by the A / D converter 32.

  The subject image converted into the digital signal is sequentially processed as image data. The image data converted into a digital signal by the A / D converter 32 is temporarily stored in the internal memory 33. The internal memory 33 is a random access memory that can be read and written at high speed. As the internal memory 33, for example, a DRAM (Dynamic Random Access Memory), an SRAM (Static Random Access Memory), or the like is used. The internal memory 33 serves as a buffer memory that waits for the order of image processing when image data is continuously generated at high speed in continuous shooting or moving image shooting.

  The internal memory 33 serves as a work memory that executes image processing and compression processing. Further, the internal memory 33 not only serves as a work memory, but also serves to temporarily store image data processed according to the purpose. Further, constants, variables, setting values, programs, and the like recorded in the system memory 12 are appropriately developed in the internal memory 33 and used for controlling the camera 10. Therefore, the internal memory 33 has a sufficient memory capacity corresponding to the role of the image processing unit 34. The system control unit 11 serves as a memory control unit, and controls how much memory capacity is allocated to the internal memory 33 for what kind of work.

  The image processing unit 34 converts the image data into an image file according to a specific image format in accordance with the set shooting mode and an instruction from the user. For example, when a JPEG (Joint Photographic Experts Group) file is generated as a still image, after performing image processing such as color conversion processing, gamma processing, and white balance processing, adaptive discrete cosine transformation is performed to perform compression processing. In addition, when generating a moving picture experts group (MPEG) file as a moving image, intra-frame encoding and inter-frame encoding are performed on frame images as continuous still images generated by reducing the number of pixels to a predetermined number of pixels. To perform compression processing.

  The still image file and the moving image file processed by the image processing unit 34 are recorded on the recording medium 50 from the internal memory 33 via the recording medium IF 38. The recording medium 50 is configured by a flash memory or the like. The recording medium 50 is a nonvolatile memory that can be attached to and detached from the camera 10. However, the recording medium 50 is not limited to a removable type, and may be a recording medium such as an SSD (Solid State Drive) built in the camera 10. When the recording medium 50 is built in the camera 10, the still image file and the moving image file recorded on the recording medium 50 are wired (for example, USB (Universal Serial Bus)) or wireless (for example, wireless LAN ( Local Area Network)).

The image processing unit 34 executes processes related to the entire image processing such as translucent processing, composition processing, crop processing, matching processing, and the like. Further, the image processing unit 34 generates display image data in parallel with or in advance of the image data to be processed for recording. The normal display image data is image data with a small number of pixels, which is obtained by copying and thinning image data sent from the image sensor.
As will be described in detail later, the image processing unit 34 plays a role as an associating unit for associating detected subject identification information with an image file.

  The display unit 37 is a liquid crystal display unit including a liquid crystal display panel, for example. The display unit 37 receives RGB (Red, Green, Blue) signals and luminance adjustment signals output from the display control unit 36, and generates display image data from the received RGB signals and luminance adjustment signals.

  Regardless of the presence or absence of recording, if the image processing unit 34 sequentially generates image data for display according to the output of the image sensor 31 and outputs it to the display unit 37, a live view display as a through image (details will be described later) is realized. Is done. Thereby, the display unit 37 can realize playback display after shooting a still image and a moving image, and live view display before shooting and during moving image shooting.

  Further, the display control unit 36 can also display the information accompanying the image data to be displayed and the menu screen recorded in the system memory 12 on the display unit 37 alone or superimposed on the display image. .

  Although details will be described later with reference to FIG. 2, the determination unit 44 determines the identification level at which the detected subject is detected as a desired target while tracking the subject in the through image.

  The operation member 39 (input unit) is adapted to receive an operation from the user. The system control unit 11 detects that the operation member 39 (input unit) is operated (input) by the user, and executes an operation corresponding to the operation. The operation member 39 includes a release switch 39a and a through image switch 39b. The user operates the release switch 39a to cause the system control unit 11 to execute a shooting preparation operation and a shooting operation, and shifts to a captured image acquisition operation. The through image switch 39b will be described later. When the release switch 39a is half-pressed, for example, autofocus is performed, and when the release switch 39a is fully pressed, an image is taken. The half-pressed state refers to a state where the release switch 39a is pressed down to about half of the normal stroke. In the present embodiment, an example in which an image is captured when the release switch 39a is completely pressed will be described, but the present invention is not limited to this. For example, imaging may be performed based on the input of the user's voice. The acquired user's voice may be analyzed, and imaging may be performed when the analyzed voice is a voice indicating an imaging instruction (for example, “Please take a picture”).

  The zoom lens, the focus lens, and the camera shake correction lens included in the lens group 21 are controlled by the lens control unit 41. The zoom lens, the focus lens, and the camera shake correction lens change the angle of view of the subject image, focus a specific area of the subject image on the light receiving surface of the image sensor 31, cancel out camera shake, and operate on the light receiving surface of the image sensor 31. A stable imaging state is maintained. The shutter 22 is controlled by the exposure control unit 42. The shutter 22 realizes the set aperture value and shutter speed. The exposure value including the aperture value, shutter speed, and ISO (International Organization for Standardization) sensitivity is calculated by the system control unit 11 based on the luminance distribution of the preliminarily acquired image data and a preset program diagram.

  Next, a schematic configuration of the determination unit 44 will be described with reference to FIG. FIG. 2 is a functional block diagram of the determination unit 44. FIG. 2 shows an image processing unit 34 that constitutes the image processing apparatus according to the present embodiment together with the determination unit 44. FIG. 2 also shows an internal memory 33 in which a through image is stored. The determination unit 44 determines the identification level of the subject that appears in the through image acquired from when the subject is framed into the imaging range until when the subject is out of frame.

Here, the through image will be described. The through image is not particularly limited. For example, the through image may be an image picked up by pressing the through image switch 39b. Alternatively, the through image may be an image to be displayed on a monitor acquired before the imaging instruction by the system control unit 11. Alternatively, the through image may be an image that is not stored in the recording medium 50 and stored only in the internal memory 33 unless there is a special process. Alternatively, the through image may be an image different from the captured image captured by pressing the release switch 39a. In this embodiment, the camera 10 is described as a through image. However, the present invention is not limited to this. In the case of a mobile phone or the like, the image is captured when the camera mode is switched based on the selection of an application or the like for starting the camera. The image may be a through image.
In the camera 10, the image sensor 31 starts driving control for a through image based on an instruction from the system control unit 11 and acquires image data of the through image. The camera 10 is configured to repeat acquisition of a through image based on pressing of the through image switch 39b and display of the through image on the display screen of the display unit 37 when driving control for the through image is started. The image data of the through image is stored in the internal memory 33 based on an instruction from the system control unit 11. For example, among the through images acquired at the frame rate of 30 frames / second, the data of the through images of the latest predetermined number of frames (for example, 3 frames) is overwritten and stored in the internal memory 33.

  2, the determination unit 44 includes a detection unit 44a, a target part determination unit 44b, a storage unit (an example of an identification level storage unit) 44c, a level comparison unit 44d, and an identification related information generation unit 44e. The moving object detection unit 44f and the bus line 44g are provided. The detection unit 44a, the target part determination unit 44b, the storage unit 44c, the level comparison unit 44d, the identification related information generation unit 44e, and the moving object detection unit 44f are connected to the bus line 44g. A predetermined control signal, a through image, and the like are transmitted and received between the detection unit 44a, the target part determination unit 44b, the storage unit 44c, the level comparison unit 44d, the identification related information generation unit 44e, and the moving object detection unit 44f via the bus line 44g. It has become so.

  The detection unit 44a determines the identification level of the subject appearing in the through image. More specifically, the detection unit 44a detects at least a part of the target portion of the subject that appears in the through image. In the present embodiment, the detection unit 44a detects, for example, a person as a subject, and detects a face that is a part of the person as a target part. The detection unit 44a transmits the detected image data of the target part to the target part determination unit 44b and the moving object detection unit 44f.

  The target part determination unit 44b acquires the image data of the target part transmitted from the detection unit 44a. The target part determination unit 44b is configured to determine an identification level as to whether the target part detected by the detection unit 44a is identified as a desired target. The desired target refers to a detection target set in advance in the target part determination unit 44b. The storage unit 44c, which will be described later, stores determination parameters that are extracted from a desired target and used to determine the identification level of the transmitted target part. In the present embodiment, the storage unit 44c stores determination parameters extracted from image data of a person's front face as a desired target. The determination parameter is a parameter indicating the front face. The determination parameter and the comparison parameter to be described later include, for example, luminance information and edge information, statistical data regarding the information, setting elements in an algorithm that highlights features for detecting a target part, and the like.

  When the image data of the target part is transmitted from the detection unit 44a, the target part determination unit 44b extracts a comparison parameter for comparison with the determination parameter from the target part. The target part determination unit 44b reads a determination parameter based on a desired target from the storage unit 44c, compares both parameters, and determines the degree of coincidence. The storage unit 44c stores a score that increases as the matching degree increases. The target part determination unit 44b determines the identification level based on the score value. The target part determination unit 44b transmits an identification level based on the comparison result of both parameters to the level comparison unit 44d described later.

  As described above, the storage unit 44c stores the determination parameter used for determining the identification level of the target part and the score used for determining the identification level. The storage unit 44c updates the stored identification level every time the level comparison unit 44d determines that the identification level of the subject is the highest in the past from when the subject is framed in to when the subject is out of frame. It has become. When the latest identification level is the highest level in the past, the identification related information generation unit 44e generates identification related information including the highest identification level and transmits it to the storage unit 44c (details will be described later). . For this reason, the highest identification level in the past is always stored in the storage unit 44c. The number of target parts detected from an acquired through image or captured image may change when a subject is newly framed in or a tracked subject is framed out. For this reason, the determination unit 44 stores the number of identification-related information stored after detection of the target part, identification level determination, and maximum level determination from the acquired through image or captured image in the storage unit 44c.

The level comparison unit 44d determines whether the subject identification level detected by the detection unit 44a satisfies a predetermined standard. In this embodiment, when the identification level is the highest level in the past, it is determined that the predetermined criterion is satisfied. The condition that satisfies the predetermined standard is not limited to this, and for example, it may be determined that the predetermined standard is satisfied when the identification level is equal to or higher than the predetermined level. Alternatively, even when the identification level stored in the storage unit 44c has a larger value than the newly acquired identification level, a predetermined time or more has elapsed from the time when the identification level stored in the storage unit 44c is stored. If the identification level stored in the storage unit 44c is weighted such that the identification level becomes smaller, the weighted identification level is compared with the newly acquired identification level. However, when the newly acquired identification level has a larger value, it may be determined that the predetermined criterion is satisfied. By setting such a predetermined standard, there is an effect that a new image is easily stored.
When the level comparison unit 44d receives the latest identification level transmitted from the target part determination unit 44b, the level comparison unit 44d reads a storage identification level that is an identification level included in the identification-related information stored in the storage unit 44c. Are to be compared. The latest identification level is the identification level of the target part most recently determined by the target part determination unit 44b. If the level comparison unit 44d determines that the latest identification level is higher than the storage identification level, the level comparison unit 44d determines that the latest identification level is the highest identification level ever. On the other hand, when the level comparison unit 44d determines that the latest identification level is lower than or the same as the storage identification level, the level comparison unit 44d determines that the identification level currently stored in the storage unit 44c is the highest identification level in the past. The level comparison unit 44d transmits the determination result to the identification related information generation unit 44e.

  When the identification-related information generating unit 44e receives information indicating that the latest identification level is the highest in the past from the level comparing unit 44d, the image data of the target part determined to be the latest identification level and the identification level (for example, Identification-related information in association with the score value) is generated. The identification related information generation unit 44e transmits the generated identification related information to the storage unit 44c. The identification-related information generation unit 44e transmits the generated identification-related information to the storage unit 44c in association with the identification-related information including the highest historical identification level compared with the latest identification level included in the identification-related information.

  The moving object detection unit 44f detects the direction of movement of the subject from the acquired through image and the image data included in the identification related information stored in the storage unit 44c, and tracks the movement of the subject. Specifically, the moving object detection unit 44f scans the through image, and the through image includes image data having the same or predetermined coincidence as the image data included in the identification-related information acquired from the storage unit 44c. The movement of the subject is tracked while determining whether or not it is. Although details will be described later, the moving object detection unit 44f obtains identification-related information from which the highest score has not been updated in the score-related processing from the storage unit 44c, and determines whether or not the subject is being tracked. It has become. In addition, the moving object detection unit 44f associates the determination result indicating non-tracking with the identification-related information including the image data determined to be not tracking the subject (not tracking), and transmits the identification result to the system control unit 11.

The image processing unit 34 plays a role as an association unit. The associating unit stores identification information of a subject having the highest identification level in association with an image file based on a captured image obtained by capturing the subject. The subject identification information is, for example, subject image data. More specifically, the subject identification information is image data of a target part that is at least a part of the subject. In the present embodiment, the subject is a person, and the target part is a person's face. For this reason, the subject identification information is image data of a person's face. In the following description, the target part is described as a person's face, but the target part is not limited to a person's face, and may be a hand, a foot, or a part of the face (eyes, nose, mouth). Of course, the subject identification information in this case is image data of a part of the hand, foot, or face (eyes, nose, mouth). In addition, a plurality of target parts may be detected, and identification information of a plurality of subjects corresponding to the plurality of target parts may be stored in association with an image file based on a captured image obtained by capturing the subject.
The image processing unit 34 reads an image file based on the image data of the latest captured image from the internal memory 33, and associates identification information of the subject with the image file. The storage unit 44c stores identification information of the subject with the highest identification level. Therefore, the image processing unit 34 can associate the identification information of the subject with the highest identification level at the present time with the image file.

When the file format of the image file is, for example, Exif (Exchangeable image file format), the identification information is stored in the header portion of the image file.
The image processing unit 34 is configured to transmit an image file stored in association with subject identification information to the internal memory 33 or the recording medium 50.

  Next, operations of the image processing apparatus according to the present embodiment and the imaging apparatus including the image processing apparatus will be described with reference to FIGS. 3 to 9 with reference to FIGS. FIG. 3 shows an example of the flow of a score determination process (hereinafter, referred to as “score determination process” or “score determination process based on a through image”) of a target part detected from a through image executed by the camera 10. It is a flowchart which shows. In the present embodiment, any one of the system control unit 11, the determination unit 44, the image processing unit 34, and the internal memory 33 performs the score determination process illustrated in FIG. 3 under the control of the system control unit 11 provided in the camera 10. Each step is to be executed. Moreover, the score determination process in this embodiment is started, for example, when the user of the camera 10 selects the execution of the main score determination process, and is ended by deselecting the execution of the main score determination process.

In the present embodiment, the score determination process shown in FIG. 3 is started when the user of the camera 10 selects execution of the score determination process based on the through image and when the through image switch 39b is pressed. When the score determination process starts, first, in step S1, a through image is read from the internal memory 33, and the process proceeds to step S3.
Specifically, in step S <b> 1, when the system control unit 11 determines that a detection signal indicating that the through image switch 39 b is pressed has been received from the operation member 39, the system control unit 11 takes the first of the through images stored in the internal memory 33. The determination unit 44 is instructed to read the stored through image. Based on an instruction from the system control unit 11, the detection unit 44 a reads the through image stored first from the internal memory 33.
When the system control unit 11 receives information indicating that the through image has been read from the internal memory 33 from the detection unit 44a, the system control unit 11 proceeds to the next step S3.

  In the present embodiment, the same through image is acquired and stored in the internal memory 33 when the through image switch 39b is pressed. However, the present invention is not limited to this. Two or more identical through images may be stored in the internal memory 33 based on the pressing of the through image switch 39b. Thereby, even if some trouble occurs in the camera 10 and one through image stored in the internal memory 33 is deleted, the remaining through image can be used as a backup image.

In step S3, the target part detection process is executed, and the process proceeds to step S5. Specifically, in step S3, the detection unit 44a scans the through image by shifting a detection area having a predetermined size from the internal memory 33 in step S1 to the right or below by, for example, one pixel at a time. When detecting the target part in the detection region, the detection unit 44a transmits image data of the detected target part to the target part determination unit 44b. In addition, when the detection unit 44a transmits the image data of the target part to the target part determination unit 44b, the detection unit 44a attaches a serial number to the image data in the order of detection.
The detection unit 44a detects the target site based on the same determination parameters as those used in the target site determination unit 44b. By using the same determination parameters as those used in the target part determination unit 44b, there is a contradiction between the detection of the target part in the detection unit 44a and the determination of the identification level in the target part determination unit 44b. Can be prevented. In addition, by setting the identification level to the lowest level, it is possible to detect a large number of subjects that may be target portions in the through image. Further, the target region may be detected by changing the size of the detection region.
When the system control unit 11 receives from the detection unit 44a information indicating that scanning of the through image read from the internal memory 33 has been completed, the system control unit 11 proceeds to the next step S5.

In step S5, score determination of the target part is executed, and the process proceeds to step S7. Specifically, in step S5, the target part determination unit 44b extracts comparison parameters sequentially from the image data of the target part transmitted from the detection unit 44a (for example, in the order of numbers attached to the image data). The target part determination unit 44b determines the identification level of the target part by comparing the extracted comparison parameter with the determination parameter read from the storage unit 44c. The target part determination unit 44b refers to the identification level determination score read from the storage unit 44c, and obtains a score value indicating the identification level from the comparison result between the determination parameter and the comparison parameter. The target part determination unit 44b transmits score information in which the image data and the comparison parameter for which the identification level has been determined are associated with the acquired score value to the level comparison unit 44d.
When the system control unit 11 receives information indicating that transmission of score information to the level comparison unit 44d has been completed from the target part determination unit 44b, the system control unit 11 proceeds to the next step S7.

  In step S7, it is determined whether or not the current score value is higher than the current highest score value. If it is determined that the current score value is higher (Yes), the process proceeds to step S9. If it is determined that the score value is lower (No), the process proceeds to step S11 without executing step S9. Specifically, in step S <b> 7, the level comparison unit 44 d receives score information including a score (current score) indicating the latest identification level transmitted from the target part determination unit 44 b, and from the system control unit 11. When receiving the instruction to read the highest score, all the identification related information stored in the storage unit 44c is read. Since the storage unit 44c stores only the past highest score value, the score value included in the identification-related information read from the storage unit 44c is the highest score value at the present time.

  The level comparison unit 44d extracts a comparison parameter from the image data included in the acquired identification related information. The level comparison unit 44d extracts comparison parameters from all the acquired identification related information. The level comparison unit 44d selects the identification related information from which the comparison parameter that is included in the score information and most closely matches the comparison parameter is extracted. The level comparison unit 44d compares the highest identification level score value included in the selected identification-related information with the score value transmitted from the target part determination unit 44b, and determines that the current score is the highest score. In such a case, the determination result information is transmitted to the identification related information generation unit 44e. The determination result information includes a comparison result of the values of both scores, and score information and identification related information that are comparison targets. The level comparison unit 44d performs the process for all score information transmitted from the target part determination unit 44b.

  When the system control unit 11 receives information indicating that the determination result information has been transmitted to the identification related information generation unit 44e from the level comparison unit 44d, the system control unit 11 proceeds to the process of the next step S9. On the other hand, when the system control unit 11 receives information indicating that the determination result information has not been transmitted to the identification-related information generation unit 44e from the level comparison unit 44d, the system control unit 11 performs the process of step S11 without executing the process of step S9. move on.

In step S9, the value of the highest score and the identification related information are updated, and the process proceeds to step S11. Specifically, in step S9, the identification related information generation unit 44e is based on the instruction from the system control unit 11, and the target part determined to be the highest identification level included in the determination result information transmitted from the level comparison unit 44d. The identification-related information in which the image data is associated with the score value indicating the identification level is generated as the latest identification-related information. The identification-related information generation unit 44e transmits the generated latest identification-related information to the storage unit 44c in association with the identification-related information included in the determination result information that is the basis of the latest identification-related information. The storage unit 44c overwrites the received recent identification-related information with the identification-related information that is the same as the identification-related information associated with the latest identification-related information and is already stored in the predetermined storage area. Update. The identification related information generation unit 44e and the storage unit 44c execute the processing for all the determination result information transmitted from the level comparison unit 44d.
When the system control unit 11 receives information indicating that the update of the identification related information has been completed from the storage unit 44c, the system control unit 11 proceeds to the next step S11.

  In step S11, a subject tracking process is executed, and the process returns to step S1. The subject to be tracked in step S11 is stored in the storage unit 44c, and is included in the identification-related information (hereinafter sometimes referred to as “tracking target information”) for which the highest score has not been updated in step S9. Data (hereinafter, sometimes referred to as “tracking target image data”). In the image data included in the identification-related information for which the highest score has been updated in step S9, the moving object is substantially detected by comparison with the image data included in the identification-related information stored in the storage unit 44c. . For this reason, the image data included in the identification-related information for which the highest score has been updated in step S9 is not a subject tracking target in step S11.

Specifically, in step S11, the moving object detection unit 44f tracks the subject by moving object detection that detects the movement of the subject from the target region. The moving object detection unit 44f reads the tracking target information from the storage unit 44c, compares the tracking target image data included in the tracking target information with the through image read from the internal memory 33 in step S1, and is tracking the subject. It is determined whether or not. Therefore, if the through image read in step S1 is the first through image after the start of the score determination process, the identification-related information is not stored in the storage unit 44c, and the subject cannot be tracked. The determination process ends and the process proceeds to step S13 (see FIG. 3). On the other hand, when the through image read from the internal memory 33 in step S1 is the second or later through image and the tracking target information is stored in the storage unit 44c, the moving object detection unit 44f performs the tracking. It is determined whether or not the subject is being tracked using the tracking target image data included in the target information. If the moving object detection unit 44f determines that the image data corresponding to the tracking target image data exists in the through image, the moving object detection unit 44f determines that the subject of the tracking target part is being tracked, and transmits the determination result to the system control unit 11. On the other hand, when determining that the image data corresponding to the image data of the tracking target part does not exist in the through image, the moving object detection unit 44f determines that the subject of the tracking target part is not being tracked (not tracking), and the determination result Is transmitted to the system control unit 11.
The system control unit 11 that has received the determination result determines that the subject corresponding to the image data determined to be non-tracking is out of the frame, and deletes the identification-related information including the image data from the storage unit 44c. When the system control unit 11 receives the information indicating the end of transmission of the determination result from the moving object detection unit 44f and erases all the identification related information including the image data determined to be non-tracking from the storage unit 44c, the system control unit 11 performs step S1. Return to processing.
The camera 10 repeatedly executes the processing from step S1 to step S11 until the user of the camera 10 does not execute the score determination processing or the release switch 39a is pressed.

  FIG. 4 is a flowchart illustrating an example of a target part association process for associating a target part with a captured image executed by the camera 10. In the present embodiment, under the control of the system control unit 11 provided in the camera 10, any of the system control unit 11, the determination unit 44, the image processing unit 34, and the internal memory 33 is the target part association process illustrated in FIG. 4. Each step is executed. The target part association process shown in FIG. 4 is started when the user of the camera 10 presses the release switch 39a.

  As shown in FIG. 4, when the target part association process starts when the release switch 39a is pressed, first, in step S31, the system control unit 11 performs the score determination process based on the through image, that is, the process shown in FIG. If it is determined whether or not the score determination process is being performed, the process proceeds to step S33. If it is determined that the determination process is not being performed, the process proceeds to step S35 without executing step S33. .

  In step S33, the system control unit 11 interrupts the score determination process based on the through image, and proceeds to step S35. Note that the score determination process shown in FIG. 3 is not interrupted immediately, but may be interrupted when the score determination process for all through images stored in the internal memory 33 is completed.

In step S35, the captured image is read from the internal memory 33, and the process proceeds to step S37. Specifically, in step S <b> 35, the system control unit 11 instructs the determination unit 44 to read the captured image acquired from the operation member 39 by pressing the release switch 39 a and stored in the internal memory 33. Based on an instruction from the system control unit 11, the detection unit 44 a reads a captured image from the internal memory 33.
Upon receiving information indicating that the captured image has been read from the internal memory 33 from the detection unit 44a, the system control unit 11 proceeds to the next step S37.

  In step S37, the target part detection process is executed, and the process proceeds to step S41. The process in step S37 is the same as the process in step S3 except that the processing target is a captured image instead of a through image, and thus description thereof is omitted.

  In step S39, the target site score is determined, and the process proceeds to step S41. Since the process in step S39 is the same as the process in step S5 except that the processing target is a captured image instead of a through image, description thereof is omitted.

  In step S41, it is determined whether or not the current score value is higher than the current highest score value. If it is determined that the current score value is higher (Yes), the process proceeds to step S43. If it is determined that the score value is lower (No), the process proceeds to step S45 without executing step S43. The processing in step S41 is the same as the processing in step S7 except that the processing target is a captured image instead of a through image, and thus description thereof is omitted.

  In step S43, the value of the highest score and the identification related information are updated, and the process proceeds to step S45. The processing in step S43 is the same as the processing in step S9 except that the processing target is a captured image instead of a through image, and thus description thereof is omitted.

  In step S45, a subject tracking process is executed, and the process proceeds to step S47. The processing in step S45 is the same as the processing in step S11 except that the processing target is a captured image instead of a through image and that the next processing does not return to the original (step S31 in this example). Description is omitted.

In step S47, association processing is executed. Here, the association process will be described with reference to FIG. FIG. 5 is a flowchart illustrating an example of the flow of association processing.
When the association process is started, first, initialization is executed (step S51), and the process proceeds to step S53. Specifically, in step S51, the system control unit 11 stores an initial value “1” as the identification-related information processing number x in a predetermined storage area (for example, the system memory 12). The number of identification-related information processes is set in order to determine the number of identification-related information among the identification-related information stored in the storage unit 44c and performing the association process (step S47). The
When the initial setting of the identification-related information processing number x is completed, the system control unit 11 proceeds to the process of step S53.

  In step S53, it is determined whether or not the highest score is greater than or equal to a threshold. If it is determined that the highest score is greater than or equal to the threshold (Yes), the process proceeds to step S55, and if it is determined that the highest score is lower than the threshold (No). ) The process proceeds to step S57 without executing step S55. Specifically, in step S53, the system control unit 11 reads out the current score value stored in the storage unit 44c, that is, the score value of the highest identification level at the present time, and the read score value and a predetermined value. Compare with the threshold. When a plurality of pieces of identification related information are stored in the storage unit 44c, the system control unit 11 reads out an identification level included in one piece of identification related information that has not been determined whether or not the highest score is equal to or greater than a threshold value. The predetermined threshold value is stored in the system memory 12, for example.

In step S55, the subject identification information is associated with the image file, and the process proceeds to step S57. Specifically, in step S55, the image processing unit 34 executes the following processing based on an instruction from the system control unit 11.
First, the image processing unit 34 receives from the system control unit 11 identification-related information that is determined in step S53 as to whether or not the highest score is equal to or greater than a threshold value. Next, the image processing unit 34 reads the image file from the internal memory 33. Next, the image processing unit 34 stores the image data of the target part included in the received identification-related information in association with the image file as identification information, for example, in the header portion of the image file. The image processing unit 34 transmits the image file storing the identification information to the internal memory 33 or the recording medium 50.

  When the system control unit 11 receives information indicating that the image file has been transmitted to the internal memory 33 or the recording medium 50, the system control unit 11 proceeds to the process of step S57.

In step S57, the identification-related information processing number x is incremented by “1”, and the process proceeds to step S59. Specifically, in step S57, the system control unit 11 stores a value obtained by increasing the current identification-related information processing number x by “1” in a predetermined storage area.
When the identification-related information processing number x is increased by “1”, the system control unit 11 proceeds to the process of step S59.

In step S59, it is determined whether the identification-related information processing number x is larger than the number n of identification-related information stored in the storage unit 44c. If it is determined that the identification-related information processing number x is larger than the number n (Yes) ) The process proceeds to step S49 (see FIG. 4), and if it is determined that the identification-related information processing number x is equal to or less than the number n (No), the process returns to step S51. Specifically, in step S59, the system control unit 11 compares the number x of identification-related information processing at the present time with the number n of identification-related information acquired from the storage unit 44c.
If the system control unit 11 determines that the identification-related information processing number x is greater than the number n, the system control unit 11 proceeds to the process of step S49. If the system control unit 11 determines that the identification-related information processing number x is equal to or less than the number n, the system control unit 11 proceeds to the process of step S51. Return. If the identification-related information processing number x is larger than the number n, the highest score that has not been compared with the threshold does not exist in the storage unit 44c, so the association process (step S47) ends, and the process proceeds to step S49. . On the other hand, when the identification-related information processing number x is less than or equal to the number n, the association processing (step S47) is continued because there is identification-related information that has not been compared with the threshold value in the storage unit 44c.

  Returning to FIG. 4, in step S <b> 49, the interruption of the score determination process is canceled, the through image acquisition is restarted, and the score determination process illustrated in FIG. 3 is restarted. In the restarted score determination process, after the release switch 39a is pressed, the first through image stored in the internal memory 33 is sequentially read out.

Next, the operation of the image processing apparatus according to the present embodiment and the image pickup apparatus including the image processing apparatus will be described with reference to FIGS. 9 will be used for explanation.
6 to 9 schematically show a state in which a human face is detected as a target part from a through image and a captured image displayed on the display screen 37 a of the display unit 37. 6 to 9, characters “t = 0” to “t = 11” representing the time are shown on the left side of the drawing. Note that, over time from FIG. 6 to FIG. 9, the passage of time is shown from the top to the bottom in the figure.

  A display screen 37 a of the display unit 37 is shown in the “LV image” column on the right side of the characters such as “t = 0” in the diagrams of FIGS. 6 to 9. “Acquired face”, “score”, and “saved face” are shown in this order on the right side of the display screen 37a in FIGS. “Acquired face” indicates the face of a person as a target portion detected by the detection unit 44a from each through image and captured image. “Score” indicates the score value of the identification level determined by the target part determination unit 44b. “Saved face” indicates the face image that is included in the identification-related information and is the basis of the face image data stored in the storage unit 44c.

As shown in the upper part of FIG. 6, when the pressing of the through image switch 39 b is detected at time “t = 0” and the score determination process based on the through image is started, it is stored in the internal memory 33 first. The through image is read (step S1), and the detection of the human face, that is, the target part is started (step S3). As shown in the upper part of FIG. 6, when a human face is detected by scanning the display screen 37a in the detection area 60 (only one in this example), score determination is started (step S5). In the score determination, for example, the score “60” is determined for the “acquired face”, that is, the detected face 62 (step S5).
Since the score is not stored at the time “t = 0”, the current score is determined to be the highest score (Yes in step S7). As a result, the face image data 64 detected at time “t = 0” and the score “60” are displayed as indicated by a horizontal arrow in the figure from the “acquired face” field to the “saved face” field. The identification related information including is stored in the storage unit 44c. Thereby, the highest score and identification related information memorize | stored in the memory | storage part 44c are updated (step S9).

  Since the through image read this time is the earliest through image stored in the internal memory 33 based on the pressing of the through image switch 39b, the image data 64 included in the identification-related information stored in the storage unit 44c is used. The corresponding target part is processed as a tracking target of the subject (step S11), and the process returns to step S1.

  As shown in the middle part of FIG. 6, when the second through image stored in the internal memory 33 is read at time “t = 1” (step S <b> 1), the human face, that is, the target part is detected. Is started (step S3). As shown in the middle part of FIG. 6, the display screen 37a is scanned in the detection area 60 to detect a human face (only one in this example), and score determination is started (step S5). In the score determination, the identification level is determined for the detected face 62 shown in the “acquired face” column (step S5). Since the face orientation of the person at time “t = 1” is higher than the face orientation of the person at time “t = 0”, the score of the detected face 62 is shown in the “score” column. Is determined to be lower than the previous time, for example, “50”.

  When the score determination is finished, it is determined whether or not the current score is higher than the highest score (step S7). Since the score “50” this time (at time “t = 1”) is lower than the current highest score “60” (No in step S7), a downward arrow from the upper stage in FIG. As shown in FIG. 8, the process proceeds to step S37 without updating the identification information (image data 64) or the like.

Next, subject tracking processing is executed (step S11). In step S7, the score of the image data of the target part detected from the through image read this time is compared with the score of the image data of the target part detected from the through image read at time “t = 0”. ing. As a result, it is determined that the subject is being tracked, and the process returns to step S1.

As shown in the lower part of FIG. 6, when the third through image stored in the internal memory 33 is read at time “t = 2” (step S <b> 1), the human face, that is, the target part is detected. Is started (step S3). In step S3, a person's face is detected in the detection area 60 of the display screen 37a shown in the "LV image" column in the lower part of FIG. 6, and score determination is started (step S5).
In the score determination, the identification level is determined for the detected face 62 shown in the “acquired face” column (step S5). Since the face orientation of the person at time “t = 3” is more in front than the face orientation of the person at time “t = 0”, the score of the detected face 62 is shown in the “Score” column. Is determined to be higher than the current highest score “60”, for example “80”.

  Next, it is determined whether or not the current score is higher than the highest score (step S7). The score “80” this time (at time “t = 2”) is determined to be higher than the current highest score “60” (Yes in step S7), and “saved” is stored from the “acquired face” column. The identification-related information including the face image data 64 detected from the through image read at time “t = 2” and the score “80”, as indicated by the horizontal arrow in FIG. The highest score and the identification related information are stored in the storage unit 44c (step S9).

  When the highest score and the identification-related information are updated, it is determined that the subject is being tracked (returning to the processing of step S11 and step S1).

  At the time “t = 3” shown in the upper part of FIG. 7, as shown in the “Score” column, the human face identification level score is “75”, which is higher than the current highest score “80”. Is also low. For this reason, the process at time “t = 3” is the same as the process at time “t = 1”, and thus the description thereof is omitted.

  When the fifth through image stored in the internal memory 33 is read at the time “t = 4” shown in the middle part of FIG. 7 (step S1), detection of a human face, that is, a target part is started. (Step S3). At time “t = 4”, since the person's face is sideways, the person's face is not detected, and the process proceeds to score determination (step S5). In FIG. 7 and FIGS. 8 and 9 to be described later, the fact that no human face has been detected in step S9 or step S19 is represented by leaving the “acquired face” field blank.

  Since the target part is not detected from the through image acquired this time, the score determination (step S5) and the score comparison process (step S7) are not executed, and the subject tracking process is executed (step S11). Since the highest score has not been updated, the moving object detection unit 44f acquires the identification-related information from the storage unit 44c, and the image data 64 (tracking target image data) included in the identification-related information is the current time (time “t” = 4 ") It is determined whether or not it is included in the read through image. The moving object detection unit 44f scans the through image, and for example, when the image data in the detection area 60 of the display screen 37a shown in the “LV image” column in the middle of FIG. 7 has a predetermined degree of coincidence with the tracking target image data. It is determined that the subject is being tracked. The moving object detection unit 44f transmits a determination result that the subject is being tracked to the system control unit 11. Upon receiving the determination result, the system control unit 11 determines that the subject is being tracked, and returns to the process of step S1 without deleting the identification-related information stored in the storage unit 44c.

  At time “t = 5” shown in the lower part of FIG. 7, as shown in the “Acquired face” column, although the face of a person is not detected in step S9, it is determined that the subject is being tracked. For this reason, the process at the time “t = 5” is the same as the process at the time “t = 4”, and thus the description thereof is omitted.

  When the first depression of the release switch 39a is detected after the score determination process based on the through image is started at time “t = 6” shown in the upper part of FIG. 8, the score determination process based on the through image is executed. It is determined that it is in the middle (Yes in step S31), and the score determination process based on the through image is interrupted (step S33). Next, the captured image acquired and stored by pressing the release switch 39a is read from the internal memory 33 (step S35), and detection of a human face, that is, a target part is executed from the read captured image. (Step S37). At time “t = 6”, since the face of the person is substantially sideways, the face of the person is not detected in step S37, and the process proceeds to score determination (step S39).

  Since the target part has not been detected from the captured image acquired this time, the score determination (step S39) and the score comparison process (step S41) are not executed, and the subject tracking process is executed (step S45). Since the highest score has not been updated, the moving object detection unit 44f acquires the identification-related information from the storage unit 44c, and the image data 64 (tracking target image data) included in the identification-related information is the current time (time “t” = 6 ") It is determined whether or not it is included in the read captured image. The subject tracking process executed this time is the same as the tracking process executed at time “t = 4” except that the through image is replaced with the captured image, and thus the description thereof is omitted.

  Subsequent to the subject tracking process, an association process is executed (step S47). In the association process, the identification-related information processing number x is initially set (x = 1) (step S51), and the current highest score is “80” and is equal to or greater than a threshold value (“60” in this example). It is determined that there is (Yes in step S53), and the image data of the person's face is associated with the image file as identification information. Specifically, the image data 64 shown in the “stored face” field in the upper part of FIG. 8 is an image file based on the image data displayed on the display screen 37a in the “LV image” field in the same part, for example. Stored in the header part.

  Subsequent to step S55, the identification-related information processing number x is incremented by “1” (step S57), and the identification-related information processing number x (= 2) is the number n of identification-related information stored in the storage unit 44c (= 1). ) (Yes in step S59), the process proceeds to step S49, and the determination process of the identification level of the target part detected from the through image is resumed.

  At times “t = 7” and “t = 8” shown in the middle and lower parts of FIG. 8, as shown in the “Acquired face” column, no human face is detected in step S9. For this reason, the processes at times “t = 7” and “t = 8” are the same as the processes at time “t = 4”, and thus the description thereof is omitted.

  At the time “t = 9” shown in the upper part of FIG. 9, as shown in the “Acquired face” column, a human face is detected in step S37 after detecting the second press of the release switch 39a. Not. For this reason, each process with respect to the captured image acquired at time “t = 9” at time “t = 9” is performed on the captured image acquired at time “t = 6” at time “t = 6”. Since it is the same as the process, the description is omitted.

  At the time “t = 10” shown in the middle part of FIG. 9, as shown in the “Acquired face” column, no human face is detected in step S3. For this reason, the process at the time “t = 10” is the same as the process at the time “t = 4”, and thus the description thereof is omitted.

As shown in the lower part of FIG. 9, at time “t = 11”, the orientation of the person's face in the through image read from the internal memory 33 is the orientation of the person's face at time “t = 2”. In the score determination (step S5), the score of the detected face 62 is determined to be higher than the current highest score “80”, for example “85” in the score determination (step S5). Is done.
The score “85” at this time (time “t = 11”) is determined to be higher than the current highest score “80” (Yes in step S7), and “saved” is stored from the “acquired face” column. The identification-related information including the face image data 64 detected at time “t = 11” and the score “85” is stored in the storage unit 44c, as indicated by the horizontal arrow in FIG. The highest score and the identification related information are updated (step S9). Further, it is determined that the subject is being tracked by updating the highest score and the identification related information (step S11), and the process returns to step S1.

As described above, the image processing apparatus according to the present embodiment and the imaging apparatus including the image processing apparatus include the determination unit 44 that determines the identification level of the subject in the through image, and the subject whose identification level satisfies the predetermined standard. An image processing unit that functions as an associating unit that associates the identification information with an image file based on a captured image of the subject.
For this reason, the image processing apparatus according to the present embodiment and the imaging apparatus including the image processing apparatus can identify a subject that is captured in a manner that is difficult to see in the captured image that is the source of the image file.

  The image processing apparatus according to the present embodiment and the imaging apparatus including the image processing apparatus can store identification information of a subject having a higher identification level in association with an image file as tag information. Thereby, the image processing apparatus according to the present embodiment and the imaging apparatus including the image processing apparatus can recognize the subject shown in the captured image based on the identification information stored in the image file based on the captured image. According to the image processing apparatus and the imaging apparatus including the image processing apparatus according to the present embodiment, the subject is not easily recognized in the captured image actually captured and stored in the internal memory 33 or the recording medium 50 instead of the through image. Even so, the subject can be recognized based on the identification information associated with the image file. Thereby, the image processing apparatus according to the present embodiment and the imaging apparatus including the image processing apparatus can improve subject identification accuracy.

  For example, a person shown in a captured image faces a side face, and it may be difficult to detect the person's face from the captured image. However, in the present embodiment, image data close to the front face of the person detected from the through image acquired before the release switch 39a is pressed can be associated with the image file as identification information of the person. For this reason, the image processing apparatus according to the present embodiment and the imaging apparatus including the image processing apparatus can identify the person based on the identification information associated with the image file even when the person is difficult to identify in the captured image. . As a result, the image processing apparatus according to the present embodiment and the imaging apparatus including the image processing apparatus can improve the identification accuracy of the subject.

  Since the through data of the through image is thinned out, the amount of data is small as compared with the image data of the captured image captured in the actual imaging. Since the target part is a part of the through image with a small data amount, the data amount is further reduced. For this reason, the data amount of the image file does not increase extremely even if the identification information of the subject is stored. As a result, the image processing apparatus according to the present embodiment and the image pickup apparatus including the image processing apparatus can improve subject identification accuracy without imposing a load on the internal memory or the recording medium.

  In the present embodiment, the target part is detected from the through image before the release switch 39a is detected and the identification level is determined (the flow from step S1 to step S11 before step S33), and the release switch 39a. The through image is acquired even after the pressing of is detected, and the identification level of the target part detected from the acquired through image is determined (the flow from step S1 to step S11 after step S49). As described above, the image processing apparatus according to the present embodiment and the image pickup apparatus including the image processing apparatus according to the present invention are based on a through image acquired before and after the release switch 39a is pressed as long as the subject is in the frame and then out of the frame. The site can be detected.

  Next, an image processing apparatus according to a modification of the present embodiment and an imaging apparatus including the image processing apparatus will be described with reference to FIGS. 1 and 2 and FIG. In the present modification, the detection unit 44a, together with the face of the person as the target portion, the color of the person's hair, the color of the person's clothes, the pattern of the person's clothes, and the through image or the captured image in the captured image At least one of the positions is detected as additional information. Further, if the target part determination unit 44b determines that the detected person's face is not a frontal face and the face cannot be detected, and if the specific person can be estimated based on the detected additional information, the frontal face of the specific person Alternatively, image data close to the front face is associated with the image file as identification information.

FIG. 10 is a flowchart illustrating an example of the operation of the image processing apparatus according to the present modification and the imaging apparatus including the image processing apparatus. Hereinafter, the description of the processing similar to the flowchart in the operation of the image processing apparatus shown in FIG. 3 and the imaging apparatus including the same will be omitted, and different points will be described.
As shown in FIG. 10, the processing from step S101, step S103, and step S105 to step S113 of the score determination process in the image processing apparatus according to this modification and the imaging apparatus equipped with the same detected the target part in step S103. This is the same as the processing from step S1 to step S11 shown in FIG.

  In step S103 following step S101, it is determined whether or not the target part has been detected. If it is determined that the target part has been detected (Yes), the process proceeds to step S105, and if it is determined that the target part has not been detected (No) ), The process proceeds to step S107. Specifically, in step S101, the detection unit 44a detects additional information such as the color of the person's hair as well as the face of the person as the target part.

In step S107, an additional information related process is executed, and when the additional information related process ends, the process proceeds to step S131. FIG. 11 is a flowchart showing an example of the flow of additional information related processing.
When the additional information related processing is started, first, additional information is acquired in step S131, and the process proceeds to step S133. Specifically, in step S131, the target part determination unit 44b also acquires additional information such as the color of the detected human hair along with the face of the person as the target detection part based on an instruction from the system control unit 11. To do. The additional information is stored in the storage unit 44c, for example.
Upon receiving information indicating that additional information has been acquired from the target part determination unit 44b, the system control unit 11 proceeds to the next step S133.

  In step S133, initial setting is executed, and the process proceeds to step S135. Specifically, in step S133, the system control unit 11 stores an initial value “1” as the additional information processing number y in a predetermined storage area (for example, the system memory 12). The number of additional information processings is the number of executions of whether or not a person can be specified based on the acquired additional information, with the total number of additional information acquired in step S131 as a maximum value.

In step S135, it is determined whether or not the person can be specified based on the additional information. If it is determined that the person can be specified (Yes), the process proceeds to step S137. If it is determined that the person cannot be specified (No), step S137 is performed. Without executing, the process proceeds to step S139. Specifically, in step S135, the target part determination unit 44b executes the following operation based on an instruction from the system control unit 11.
The detection unit 44a selects a through image in which a person's face can be detected in through images of several frames acquired before the through image read this time. Next, the target part determination unit 44b compares the additional information acquired in step S131 with the additional information read from the storage unit 44c. If the target part determination unit 44b determines that the degree of coincidence between both pieces of additional information is greater than or equal to a predetermined threshold, the target part determination unit 44b determines that the face of the person that could not be detected in step S103 is reflected in the through image read from the storage unit 44c. . The target part determination unit 44b specifies the face of a person in the vicinity of the compared additional information image as a target part in the through image read from the internal memory 33.
When the system control unit 11 receives information indicating that a person has been identified from the target part determination unit 44b, the system control unit 11 proceeds to the next step S137.

In step S137, the face image and score of the specific person are acquired, and the process proceeds to step S139. Specifically, in step S137, the target part determination unit 44b acquires the front face or the face image with the highest score of the specific person specified in step S135 based on the control of the system control unit 11. The target part determination unit 44b transmits the acquired score to the level comparison unit 44d.
When the system control unit 11 receives information indicating that the acquired score has been transmitted to the level comparison unit 44d from the target part determination unit 44b, the system control unit 11 proceeds to the next step S139.

In step S39, the additional information processing number y is increased by “1”, and the process proceeds to step S141. Specifically, in step S125, the system control unit 11 stores a value obtained by increasing the current additional information processing number y by “1” in a predetermined storage area.
If the additional information processing number y is increased by “1”, the system control unit 11 proceeds to the process of step S141.

In step S141, it is determined whether or not the additional information processing number y is larger than the total number m of the acquired additional information, and if it is determined that the additional information processing number y is larger than the total number m (Yes), step S113 (see FIG. 10). If it is determined that the additional information processing number y is less than the total number m (No), the process proceeds to step S135. Specifically, in step S141, the system control unit 11 compares the current number y of additional information processing with the total number m of additional information acquired in step S131.
If the system control unit 11 determines that the additional information processing number y is greater than the total number m, the system control unit 11 proceeds to step S103 (see FIG. 10), and determines that the additional information processing number y is equal to or less than the total number m, the process proceeds to step S135. Transition. When the number of additional information processing y is larger than the total number m, there is no additional information for which the person identification determination based on the additional information (step S135) has not been made, and therefore the person identification determination process based on the additional information is performed. The process ends, and the process proceeds to step S113. On the other hand, if the additional information processing number y is less than or equal to the total number m, there is additional information for which the person identification determination based on the additional information (step S139) has not been made, and therefore the person identification determination process based on the additional information is continued. Is done.

FIG. 12 is a flowchart illustrating an example of a flow of a target part association process for associating a target part with a captured image executed in the imaging apparatus according to the present modification.
As shown in FIG. 12, in the processes from step S151 to step S159 and from step S163 to step S171 in the target part association process in this modification, it is determined whether or not the target part has been detected in step S157. Except for this point, the process is the same as the process from step S31 to step S49 in the target part association process shown in FIG. The process of step S161 is the same as step S107 shown in FIG. 10, that is, step S131 to step S141 shown in FIG. 11, except that the processing target is not a through image but a captured image.

  As described above, the image processing apparatus according to this modification and the imaging apparatus including the same specify the target part using additional information such as information around the target part even if the target part itself cannot be detected. There are cases where it is possible.

  Next, an image processing device according to another modification of the present embodiment and an imaging device including the image processing device will be described. The image processing apparatus according to this modification and an imaging apparatus including the image processing apparatus are characterized in that instead of the through image, a series of captured images (so-called continuous captured images) are used. Yes. The image processing apparatus according to the present modification and an imaging apparatus including the image processing apparatus detect the subject in a series of captured images that are stored in the internal memory 33 or the recording medium 50 and are continuously captured, and determine the identification level. A determining unit and an associating unit for associating identification information of the subject determined by the determining unit with the identification level satisfying a predetermined criterion with an image file based on a captured image selected from the series of captured images And have. That is, in this modification, the determination unit 44 shown in FIG. 2 detects a subject in a series of captured images that are continuously captured, and determines the identification level. Also, the image processing unit 34 shown in FIG. 2 serves as an associating unit for associating with an image file based on a captured image selected from the series of captured images.

  The present invention can be embodied as a computer program. For example, the image processing unit as each unit of the determination unit 44 (the detection unit 44a, the target part determination unit 44b, the storage unit 44c, the level comparison unit 44d, the identification related information generation unit 44e, the moving object detection unit 44f) and the association unit. It can also be realized as an image processing program that functions as 34. Thus, some or all of the present invention can be incorporated into hardware or software (including firmware, resident software, microcode, state machines, gate arrays, etc.). Furthermore, the present invention may take the form of a computer program product on a computer-usable or computer-readable storage medium, which incorporates computer-usable or computer-readable program code. It is done. In the context of this specification, a computer usable or computer readable medium includes, stores, communicates, propagates, programs that are used by or in conjunction with an instruction execution system, apparatus or device, Alternatively, any medium that can be transported can be used.

The present invention is not limited to the above embodiment, and various modifications can be made.
In the above embodiment and the modification, the identification information associated with the image file is image data, but the present invention is not limited to this. For example, the image processing apparatus and the imaging apparatus including the image processing apparatus do not store the image data of the target part in association with the image file when the face of the captured image is equal to or higher than the set maximum level or the set threshold value. It may be configured to store only the coordinates and parameters of feature points (for example, parts such as the eyes, nose and mouth) in association with each other.

  In addition, the image processing apparatus and the image pickup apparatus equipped with the image processing apparatus may be inclined because the detected face is not front-facing but the neck is tilted, or the face is rotated by scolding or looking up. If it is possible to recognize the image, it may be configured to add face inclination information and rotation information and associate them with the image file. In addition, the image processing apparatus and the imaging apparatus including the same correct the detected face to be a front face based on the tilt information and the rotation information, and store the corrected face image in association with the image file. It may be configured to.

  Further, the image processing apparatus and the imaging apparatus including the image processing apparatus and the image capturing apparatus may store the feature point extracted result together with the algorithm name or identifier as the identification information associated with the image file in association with the image file. Good.

In the above embodiment, the identification information of the subject is associated with the image file of the captured image, but the present invention is not limited to this. For example, the image processing apparatus and the imaging apparatus including the image processing apparatus may associate the identification information of the subject with the image file in association with the subject that appears in the captured image.
For example, the image processing unit 34 compares the parameters for comparison extracted from the image data of the target part included in the identification related information read from the storage unit 44 c and all the target parts detected in the captured image read from the internal memory 33. The comparison parameter extracted from the image data is compared. The image processing unit 34 may be associated with the image file by associating the identification information with the subject including the target portion that is copied to the captured image and has the best match between the two comparison parameters. The image processing unit 34 executes this association process in, for example, step S55 shown in FIG.

  In this way, by associating the subject identification information with the subject shown in the captured image, it is possible to improve the identification accuracy of each subject.

  In the above embodiment, as described with reference to FIGS. 6 to 9, the process for associating the identification information of the subject with the captured image is executed every time the captured image is acquired based on the depression of the release switch 39a. However, the present invention is not limited to this. When the image processing apparatus and the imaging apparatus including the imaging apparatus complete a series of imaging, for example, the target part having the highest score stored in the storage unit 44c by the series of imaging is associated with the captured image captured by the series of imaging. Also good. Taking FIG. 6 to FIG. 9 as an example, the highest score at the time of imaging at time “t = 6” and time “t = 9” is “80” acquired at time “t = 2”. After the imaging is completed, the identification information of the subject associated with the captured images captured at time “t = 6” and time “t = 9” includes the highest score “85” acquired at time “t = 11”. Information. That is, identification-related information including image data of a target part acquired after the captured image is captured may be associated with the captured image. Here, as a series of imaging, for example, imaging from when tracking of a target part is started until tracking becomes impossible, imaging from when a subject is framed in to frame out, or within a predetermined time (for example, 20 Second, 60 seconds, etc.).

  In addition, the image processing apparatus according to the present embodiment and the imaging apparatus including the same first search the past shooting history when the score indicating the identification level does not reach the predetermined threshold when the subject is out of the frame. . Next, the image processing apparatus and the image pickup apparatus including the same search for face information (identification information) associated with the photographed image when it is likely that the same face can be found based on information such as the photographing time being close from the present time. Then, the position of the person in the captured image, the color of the person's hair, the color of the person's clothes, the pattern or the pattern are compared, and the matching faces are determined to be the same person. The face thus determined may be stored in association with the image file together with the additional information.

  Furthermore, if a face whose score indicating the identification level reaches the threshold value is not obtained even by the above method, the history taken after the release time is observed after a certain period of time after the release switch is pressed, and the same process as described above. The process may be executed. Thereby, the possibility of acquiring a front face can be improved.

  When a person is photographed against a background of a night view, the through image shows the face of the person brightly by a process such as exposure correction as compared to a captured image actually captured. Therefore, in the image processing apparatus according to the present embodiment and the imaging apparatus including the image processing apparatus, a relatively clear human face image in the through image is temporarily stored, and an image file based on the actual captured image is stored. The face image can be associated. Thereby, according to this embodiment, since it is possible to identify a person who is difficult to recognize in the captured image with the night view as the background based on the information associated with the image file, the detection accuracy of the subject is improved.

  When emotion detection (expression detection) is performed in addition to human face detection, no emotion is detected most as a human face among six emotions (joy, anger, sadness, comfort, love, hatred) and no emotion. Easy to do. For this reason, when performing emotion detection, “no emotion” is associated with a higher identification level than the six emotions. Therefore, as the detected face matches “no emotion”, the discrimination level, that is, the score increases. When the image processing apparatus according to the present embodiment and the imaging apparatus including the image processing apparatus are configured to be able to perform emotion detection, “no emotion” and “no emotion” compared to the image data of the six emotion facial images. A large number of facial image data having a facial expression close to "is stored in association with the image file as identification information.

  As a method of detecting emotions, for example, when a smile level is high in smile detection, it is detected that the subject is in a happy state (“joy”) or a happy state (“easy”), and shooting is performed. A person's body (heart rate or body temperature) is detected by a biosensor, and when the heart rate is high, the person is angry (“angry”) or hated (“hate”), and when the heart rate is low There is a method for detecting a sad state ("sorrow"). The biosensor is provided, for example, inside the grip portion of the imaging apparatus, and can be detected through the hand of the photographer holding the grip portion.

  In the above embodiment, a person is taken as an example of the subject and a person's face is taken as an example of the target part, but the present invention is not limited to this. The subject may be a non-human animal or landscape, the target part may be a part other than a human face, a non-human animal face or body, a building, a natural object such as a mountain, or the like. For example, in the case of an elephant, the shape of a face and a long nose extending from the face, and in the case of a giraffe, the shape of a long neck extending from a moving body and the shape of the face at the tip thereof, such as a building (such as a historical building) or a natural object ( For example, in a famous place such as Mt. Fuji, a characteristic portion such as a silhouette from a direction generally used for magazines or postcards is stored as a determination parameter in the storage unit 44c, and a target portion detected from a through image or a captured image is stored. And the identification level of the target part may be determined.

  For example, by embedding a telephoto image of an individual subject (for example, a natural object such as a flower or a historical building) that appears in the wide-angle image in an image file based on the wide-angle image in which the entire view is captured, the wide-angle image is visually recognized. Difficult subjects can be identified on the image data. Further, by acquiring the telephoto image from the through image, it is possible to embed subject identification information in the image file while suppressing an increase in image data.

  In the above embodiment and the modification, the acquired through image is stored in the internal memory 33, but the present invention is not limited to this. For example, in the present invention, the acquired through image may be stored in an external memory (recording memory). For example, the through image is transmitted to an external memory via a communication unit provided in the imaging apparatus and stored in the external memory. In addition, the external memory stores a target part (for example, a human face) detected from a through image and an identification level. The imaging apparatus reads out image data of a target portion (for example, a human face) of a subject stored in an external memory to the imaging apparatus, and uses the read image data as a captured image captured by actual shooting by pressing a release switch. The captured image associated with the association and image data may be stored in a memory (for example, an internal memory) provided in the imaging apparatus.

The present invention has the following uses.
For example, using the identification information associated with the image file, the image file is organized in the recording medium 50 (for example, organized by person), or the image file is organized in the storage unit of another device (for example, a personal computer). (For example, organized by person).
Further, for example, by using the identification information associated with the image file, a list display of each person in the thumbnail format can be displayed on the display unit 37.
Further, for example, using the identification information associated with the image file, the image file can be transmitted to the person specified by the identification information.

DESCRIPTION OF SYMBOLS 10 Camera, 11 System control part, 12 System memory, 20 Optical system, 21 Lens group, 22 Shutter, 31 Image sensor, 32 A / D converter, 33 Internal memory, 34 Image processing part, 36 Display control part, 37 Display Part, 37a display screen, 38 recording medium IF, 39 operation member, 39a release switch, 40, 44g bus line, 41 lens control part, 42 exposure control part, 44 detection part, 44a detection part, 44b target part determination part, 44c Storage unit, 44d level comparison unit, 44e identification related information generation unit, 44f moving object detection unit, 50 recording medium, 60 detection area, 62 detected face, 64 image data

Claims (17)

A determination unit that determines an identification level of a subject included in the through image captured by the imaging unit;
An association unit that associates identification information for identifying the subject and a captured image including the subject captured by the imaging unit when the determination unit determines that the identification level satisfies a predetermined criterion; An image processing apparatus comprising: The image processing apparatus according to claim 1, wherein the association unit stores the identification information in a header portion of an image file based on the captured image. The image processing apparatus according to claim 1, wherein the determination unit determines an identification level at which the face of the subject is identified as a front face. The determination unit is a detection unit that detects at least a part of the target portion of the subject while tracking the subject, and a target that determines an identification level at which the target portion detected by the detection unit is identified as a target. The image processing apparatus according to claim 1, further comprising: a part determination unit. The detection unit detects a human face as the target part,
The target part determination unit determines an identification level at which the face detected by the detection unit is identified as the front face of the target;
The image processing apparatus according to claim 4, wherein the association unit associates the face image data detected by the detection unit as the identification information. The determination unit determines that the predetermined criterion is satisfied when the identification level is a record high level,
The image processing apparatus according to claim 1, wherein the association unit associates the subject included in the captured image with the identification information. The detection unit includes at least one of the face of the person, the hair color of the person, the color of the clothes of the person, the pattern of the clothes of the person, and the position of the person in the through image. Detect as
The associating unit, when it is determined by the target part determination unit that the detected person's face is not the front face, when the target part determination unit can estimate a specific person based on the additional information, The image processing apparatus according to claim 6, wherein image data of a front face of a specific person is associated as the identification information. An identification level storage unit that updates a stored identification level when the determination unit determines that the identification level of the subject is a record high before the subject is out of frame. The image processing apparatus according to any one of claims 1 to 7. A determination unit that determines an identification level of a subject in a series of captured images that are continuously captured;
An association unit for associating identification information for identifying the subject and a captured image selected from the series of captured images when the determination unit determines that the identification level satisfies a predetermined criterion; An image processing apparatus comprising: An image pickup apparatus comprising the image processing apparatus according to any one of claims 1 to 9. An imaging unit for imaging a subject and outputting image data;
An input unit for imaging the subject by the imaging unit;
A display unit for displaying a live view image based on image data output by the imaging unit before input by the input unit or after imaging by the imaging unit is performed by input by the input unit;
An imaging apparatus comprising: an associating unit that associates an image including the subject imaged based on an input from the input unit and the live view image including the subject. An identification unit for identifying the subject,
The imaging apparatus according to claim 11, wherein the live view image including the subject associated by the associating unit is an image capable of identifying the subject by the identifying unit. The display unit indicates that the live view image including the subject identified by the identification unit and the image including the subject captured based on an input by the input unit are associated by the association unit. The imaging apparatus according to claim 12, wherein display is performed. The association unit associates a partial region of the subject included in the live view image with an image including the subject captured based on an input by the input unit. The imaging device according to any one of 13. The imaging apparatus according to claim 14, wherein the partial area of the subject is a face of the subject. An imaging unit for imaging a subject and outputting image data;
An input unit for imaging the subject by the imaging unit;
A display unit for displaying a live view image based on image data output by the imaging unit before input by the input unit or after imaging by the imaging unit is performed by input by the input unit;
An image pickup apparatus comprising: an association unit that associates an image including the subject imaged based on an input from the input unit with information indicating the subject included in the live view image. A process for causing the determination unit to determine the identification level of the subject appearing in the through image;
When the determination unit determines that the identification level satisfies a predetermined standard, the computer performs a process of associating the identification information for identifying the subject and the captured image obtained by capturing the subject with the association unit. An image processing program characterized by being executed.