JP2019185541A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To extract a subject from an image precisely within a short period of time to make it recognizable.SOLUTION: An image processing apparatus (40) detects a subject from a captured image to extract feature information, recognizes an individual of the subject whose feature information is extracted, and registers the feature information at a data base. Then, the image processing apparatus (40) primarily stores (46) the extracted feature information, collates, with the primarily stored feature information, feature information extracted from a subject detected after the subject disappears from the image, confirms whether or not the subject detected after disappearance of the subject from the image is the subject of the primarily stored feature information, and registers the feature information of the subject at the data base when the subject detected after disappearance from the image is the subject of the primarily stored feature information and an angle of view used for capturing the subject whose feature information is extracted is different from that used for capturing the subject of the primarily stored feature information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for detecting and identifying a subject from a captured image.

  Recent digital cameras have a function of detecting and tracking a subject from image data obtained from an image sensor and photographing the subject according to a suitable state of focus, brightness, and color. It has become common. In addition, there is a recognition technique as a method for specifying an individual from detected objects. In recognition of a specific object, feature information of each detected subject is extracted, and the extracted feature information is compared with reference feature information prepared in advance to check whether or not they match. Is done by. For reference feature information, for example, a subject assumed as a recognition target is photographed in advance by a camera, feature information is extracted, and the feature information is stored (that is, registered) in a nonvolatile memory or the like. Prepared by. However, in this case, a complicated work of preparing and registering reference feature information in advance is required. There are also cases where the subject itself assumed as a recognition target or its characteristic information cannot be prepared in advance.

  On the other hand, Patent Document 1 discloses a technique that makes the prior registration work substantially unnecessary. In the technique described in Patent Document 1, if the feature information of the subject detected from the photographed image is compared with the dictionary data, if the unregistered subject and the important subject condition are satisfied, the subject dictionary is temporarily stored as temporary dictionary data. As a condition for determining an important subject, a time length for tracking the subject, the number of times of selection as a main subject, and the like are used. Then, when the feature information of the subject detected from the photographed image acquired in the subsequent photographing action matches the feature information stored in the temporary dictionary data, the feature information of the subject is fully registered in the dictionary data.

JP 2014-44525 A

  As described above, in the technique described in Patent Document 1, the time length of tracking a subject and the number of main subject selections are conditions for determining whether or not the subject is an important subject. However, in the case where there are a plurality of subjects in the photographed image, for example, in the case where the important subject disappears from the image or is blocked by other subjects, the subject intended by the photographer is determined as the important subject in the first place. Often disappears. For this reason, the technique of Patent Document 1 has a problem that the important subject determination based on the tracking time length and the number of times selected as the main subject is not successful.

  Therefore, an object of the present invention is to make it possible to extract a subject accurately from a captured image in a short time and recognize it.

  An image processing apparatus according to the present invention includes a detection unit that detects a subject from a captured image, an extraction unit that extracts feature information from the detected subject, and recognition that recognizes an individual of the subject based on the feature information. Means for registering the feature information in a database, and storage means for temporarily storing the feature information. The recognition means is extracted from a subject detected after the subject disappears from the image. The registered feature information is compared with the primary stored feature information to confirm whether the subject detected after disappearing from the image is the subject of the primary stored feature information. The recognition means confirms that the subject detected after disappearing from the image is the subject of the feature information stored primarily, and the subject of the feature information stored primarily is copied. The feature information of the subject is registered in the database when the angle of view when the image is taken differs from the angle of view when the subject detected after disappearing from the image is taken. .

  According to the present invention, a subject can be accurately extracted in a short time from a captured image.

It is a figure which shows the structural example of the digital camera which is an example of an image processing apparatus. 1 is a schematic external view of a digital camera. It is a figure used for description of prior registration. It is explanatory drawing of the automatic registration at the time of panning. It is explanatory drawing of the automatic registration at the time of zooming. It is explanatory drawing of face ID numbering. It is a figure which shows an example of the management table of a face detection result. It is explanatory drawing of a dictionary database. It is explanatory drawing of primary storage. It is a flowchart which shows the flow of the process in a camera. It is a flowchart of a face ID numbering process. It is a flowchart of recognition status determination processing. It is a flowchart of a recognition process and an important subject registration process. It is explanatory drawing of reliability.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
FIG. 1 is a diagram illustrating a schematic configuration of an imaging apparatus (digital camera, hereinafter referred to as a camera 100) which is an application example of the image processing apparatus of the present embodiment.
The lens 10 collects external light and forms an optical image on the imaging unit 20.
The mechanical drive circuit 16 performs focus adjustment and field angle adjustment (zoom operation) by driving the lens 10 along the optical axis direction. The mechanical drive circuit 16 can also perform camera shake correction by driving the lens in a direction other than the optical axis direction according to camera shake. Note that the camera shake correction can be similarly realized by moving the imaging unit 20. When performing camera shake correction, the system control unit 42 detects a shake of the camera 100 based on detection outputs of an acceleration sensor and an angular velocity sensor (not shown), for example, and performs known correction control to cancel the shake of the camera 100. Do.

The aperture 13 changes the aperture. The ND filter 14 adjusts the light transmission amount. The mechanical shutter 12 is shielded by being fully closed. The diaphragm 13, the ND filter 14, and the mechanical shutter 12 are provided as a light quantity adjustment mechanism, and are used properly according to the application, and adjust the light quantity of the light that has passed through the lens 10.
The light emission control circuit 32 performs light emission driving and light emission control of the strobe unit 30 under the control of the system control unit 42.

  The light that has passed through the lens 10 and the light amount adjustment mechanism (12, 13, 14) is received by the imaging unit 20. The imaging unit 20 operates in response to a drive instruction from the imaging drive circuit 22 and exposes the light receiving element, adjusts the exposure time, reads the exposed imaging signal, amplifies or attenuates the readout imaging signal, and A / D of the imaging signal Perform conversions. The image data output from the imaging unit 20 is input to the image processing circuit 40 or temporarily stored in the RAM 46.

  The image processing circuit 40 performs various processes such as image processing and image analysis on the image data directly input from the imaging unit 20 or the image data input via the RAM 46. In exposure adjustment (AE: Auto Exposure) and focus adjustment (AF: Auto Focus) at the time of shooting, the image processing circuit 40 extracts a luminance component and a frequency component from image data sequentially output from the imaging unit 20. To the system control unit 42. The system control unit 42 uses the luminance component and frequency component from the image processing circuit 40 as evaluation values, and controls the AE and AF operations via the mechanical drive circuit 16 and the imaging drive circuit 22.

  The image processing circuit 40 can develop the image data acquired from the imaging unit 20 to adjust the image quality, and appropriately set the hue, gradation, brightness, etc. Generate data. The image processing circuit 40 can also perform various types of image processing such as clipping a part of the input image, rotating the image, and synthesizing the image. Further, the image processing circuit 40 can detect a subject area such as a person's face from the input image, and obtain information on the position, size, inclination, face likelihood, etc. of the person's face in the image. Can do. Further, the image processing circuit 40 can extract the feature information of the detected human face and recognize whether or not the person is a specific individual. When performing such recognition, the image processing circuit 40 reads the personal feature information stored in the ROM 48 in advance and compares it with the facial feature information of the person extracted from the image, thereby matching the registered individual. Whether to match or not is determined. Further, the image processing circuit 40 can perform detailed analysis of the characteristics of a person's face. For example, the image processing circuit 40 can also perform line-of-sight detection processing such as analyzing the characteristics of the eyes of the person and detecting the line-of-sight direction. it can.

The display device 50 includes a liquid crystal device (LCD) or the like, and can display an image developed by the image processing circuit 40 or display characters and icons, for example. Various information can be transmitted to the user (user) of the camera 100 by displaying characters and icons.
The operation unit 44 is operated by the user of the camera 100, and this user operation information is input to the system control unit 42. Based on user operation information from the operation unit 44, the system control unit 42 operates each unit of the camera 100, such as turning on power to each unit of the camera 100, switching a shooting mode, performing various settings, executing shooting, and reproducing an image. And control signal processing.

The external memory I / F 52 has the external memory 90 inserted through a memory socket (not shown) and connects the external memory 90 and the camera 100. The camera 100 can perform image exchange, program acquisition, and the like by connecting to the external memory 90 via the external memory I / F 52.
The external device I / F 54 connects the external device 92 and the camera 100 via a connection connector (not shown), wireless communication, or the like. By connecting to the external device 92 via the external device I / F 54, the camera 100 can exchange images, exchange command information for operating the devices, acquire a program, and the like.

The ROM 48 is a rewritable nonvolatile memory and stores various setting values and programs of the camera 100. The RAM 46 temporarily stores image data picked up by the image pickup unit 20, image data being processed by the image processing circuit 40, and processed image data. The RAM 46 is loaded with a program read from the ROM 48.
The system control unit 42 executes a program read from the ROM 48 and developed in the RAM 46, and controls the above-described units of the camera 100 and performs various calculations.

  2A and 2B are schematic external views of the camera 100 of the present embodiment. 2A shows the front side of the camera, and FIG. 2B shows the back side of the camera. As shown in FIG. 2A, a lens 10 is disposed on the front side of the camera, and the camera 100 can capture a subject image. In addition, a strobe unit 30 is disposed on the front side of the camera, and the camera 100 can obtain a sufficient amount of light by causing the strobe unit 30 to emit light when the main subject is dark, maintaining a fast shutter speed even in the dark, A suitable image can be obtained. The camera 100 is provided with operation members 200, 202, 210, 220, 222, 224, 226, and 228 in the operation unit 44 shown in FIG. Although not described in detail, each operation member includes an operation button, an operation switch, an operation lever, and the like. used. As an example, the operation member 200 is a power button, and the operation member 202 is a shutter button. In addition, the operation member may include a touch panel arranged on the LCD screen of the display device 50.

  Here, the camera 100 of the present embodiment detects a subject based on the image data obtained from the imaging unit 20, performs tracking, etc., and adjusts the focus, brightness, and color of the subject to a suitable state. Has a function to shoot. Examples of the subject to be detected include a human face, a human body, a specific animal such as a dog and cat, and the like. In addition, the camera 100 according to the present embodiment can identify and identify an individual from the detected subjects. That is, in the camera 100 of this embodiment, individual information can be identified by extracting feature information of a detected subject and comparing / collating it with feature information registered in a dictionary database.

  As an example of a method for preparing feature information of a subject to be recognized, a method as shown in FIG. 3 is known. FIG. 3 shows an example in which person A is assumed as an important subject to be recognized. In this example, the subject area 315 of the person A is detected from the photographed image of the camera 313, the feature information is extracted from the subject area 315, and the feature information is associated with the person A and registered in the dictionary database 330. Processing 310 is performed. Thereafter, for example, when a person A, a person B, a person C, etc. are photographed by the camera 353, a subject area 355 of the person A, a subject area 357 of the person B, and a subject area 359 of the person C are detected from the photographed images. To extract feature information. Then, by performing matching determination between the feature information extracted from each of the subject areas 355, 357, and 359 and the registered feature information in the dictionary database 330, a recognition process for determining whether or not the person A as an important subject is captured. 350 is performed. The camera 313 used in the registration process 310 and the camera 353 used in the subsequent recognition process 350 may be the same camera, or may be different cameras. Further, processing from detecting a subject from a captured image, extracting feature information, and registering the feature information in the dictionary database 330 may be performed by an image processing device other than the camera. Further, the dictionary database 330 used by the camera 353 for the recognition processing 350 may be acquired, for example, via a removable storage medium or via a network or the like. The dictionary database 330 may be set in the ROM 48 of the camera, may be set in the external memory 90, or may be set in an external storage device as an external device.

As described above, there are various methods for preparing the feature information of the subject to be recognized. In this embodiment, the camera 100 detects the subject from the captured image and extracts the feature information. The case of registration in the dictionary database will be described as an example.
Although details will be described later, the camera 100 according to the present embodiment automatically detects the subject from the captured image and automatically extracts feature information extracted from the important subject in the dictionary database when the detected subject is an important subject for the photographer. A function to register automatically.

Hereinafter, it is important to determine whether or not the subject is an important subject for the photographer by using FIGS. 4 (a) and 4 (b) and FIGS. 5 (a) and 5 (b). The outline of the process of automatically registering the feature information of the subject determined to be in the dictionary database will be described.
FIG. 4A and FIG. 4B show a case where an angle-of-view operation that pans the camera 100 and moves the angle of view is performed as an example of the angle-of-view operation of the camera 100 by the photographer. It is explanatory drawing.

  FIG. 4A shows an example in which a certain person is shown as a subject 403 in an angle of view 401 at the time of shooting by the camera 100. For example, the subject 403 is located near the center of the angle of view 401, and the size of the face in the angle of view 401 is greater than or equal to a certain level, and the face is facing the front with respect to the camera 100. To do. In this case, the person of the subject 403 is highly likely to be a person determined by the photographer (user) as a subject to be photographed, and therefore the subject 403 is considered to be highly likely to be an important subject. It should be noted that the degree of whether the person's subject 403 is located near the center of the angle of view 401, the size occupied by the face in the angle of view 401 is equal to or greater than a certain level, and the face is facing the front with respect to the camera 100. It can be calculated by a known face detection and recognition technique or the like.

  The dictionary database 415 has a large-capacity storage medium, and is capable of registering individual feature information groups to be recognized, that is, a plurality of feature information of subjects important to the photographer. Note that the dictionary database 415 may store not only a feature information group of one person as a recognition target but also a plurality of individual feature information groups as recognition targets. In the present embodiment, for example, if the subject 403 is an important subject for the photographer, the characteristic information of the subject 403 is registered in the dictionary database 415 in association with information related to the subject 403.

  Here, in this embodiment, in order to avoid the same feature information of the same subject 403 being registered repeatedly, the camera 100 first uses the feature information of the subject 403 and the feature information registered in the dictionary database 415. Then, recognition processing 413 for duplication confirmation is performed. If recognition is not established in the recognition process 413, that is, if it is confirmed that it is not registered in the dictionary database 415, the camera 100 primarily stores the feature information of the subject 403 in the primary storage memory 419 (for example, the RAM 46). Perform provisional registration processing. The camera 100 also temporarily stores the feature information of the subject 403 and stores the angle-of-view information when the subject 403 is photographed in the primary storage memory 419. As described above, the purpose of the provisional registration process is to realize automatic registration as will be described later, and in a situation where it is not yet known whether the subject 403 is an important subject or not, a limited capacity of the dictionary database 415 is wasted due to duplicate registration. It also includes avoiding consumption.

  In FIG. 4B, when the subject 403 moves in a direction in which the subject 403 disappears from the angle of view 401 (referred to as a frame-out direction 405), the photographer moves the angle of view by performing a panning operation 409 in the horizontal direction. This shows how the subject is again within the angle of view. A subject 407 shown in FIG. 4B represents the subject after the subject 403 has moved in the frame-out direction 405. Here, when the moved subject 407 is an important subject for the photographer, the photographer changes the angle of view by changing the direction of the camera 100 so that the subject 407 is within the angle of view. It is assumed that the operation is performed.

  An angle of view 411 illustrated in FIG. 4B represents a state in which the subject 407 is again within the angle of view by performing an angle of view operation such as the panning operation 409 of the camera 100 in the horizontal direction by the photographer. Yes. As shown in FIG. 4B, the subject 407 detected again after the view angle operation for moving the view angle by the panning operation 409 is performed after the subject 407 has disappeared temporarily from the view angle 401. The possibility of being an important subject for a person is thought to have further increased. However, even at this time, it has not yet been determined whether or not the subject 407 is an important subject.

  Therefore, when the subject 407 is re-detected from the photographed image acquired at the angle of view 411 after the angle of view movement operation by the photographer is performed, the subject 407 is an important subject for the photographer. A recognition process 433 is performed to check whether or not. In the recognition process 433, the camera 100 firstly uses the re-detected angle-of-view information at the time of photographing the subject 407 and the angle-of-view information at the time of photographing of the subject 403 temporarily registered in the primary storage memory 439 (RAM 46). Then, it is determined whether or not an operation for moving the angle of view has been performed. Further, when the camera 100 determines that an angle of view operation for moving the angle of view has been performed and detects the subject 407 after the angle of view is moved, the camera 100 extracts feature information of the subject 407. Further, the camera 100 compares the feature information extracted from the subject 407 detected after moving the angle of view with the feature information of the subject 403 temporarily registered in the primary storage memory 439, and the subject 407 and the subject 403 are the same. A recognition process 433 is performed to check whether or not. When the recognition process is established in the recognition process 433, that is, when the re-detected subject 407 is provisionally registered in the primary storage memory 439, the camera 100 determines that the subject 407 is an important subject. Thereafter, the camera 100 associates the feature information of the subject 407 or the primarily stored feature information of the subject 403 with the related information of the subject 407 or 403, and performs main registration in the dictionary database 435. Although details will be described later, in the case of the present embodiment, which of the feature information of the subject 407 and the feature information of the subject 403 is to be registered in the dictionary database 435 is determined based on the reliability of the feature information.

  As described above, the camera 100 according to the present embodiment performs a recognition process that combines the angle information together with the feature information of the subject when the subject is re-detected after the temporarily registered subject disappears. Check if it is an important subject. As a result, the camera 100 according to the present embodiment can estimate with high probability that the photographer intends to photograph the subject, and can automatically register the feature information of the important subject in the dictionary database.

  Note that when the amount of movement and the movement speed of the angle of view by the operation of moving the angle of view are very large, it can be considered that the subject 407 is unlikely to be an important subject. In this case, for confirmation The recognition process 433 may not be performed. In addition, here, as an example of the view angle operation for moving the view angle, the panning operation for moving the direction of the camera 100 in the horizontal direction (left-right direction) is described, but the operation for moving the view angle is not limited to this example. The operation for moving the angle of view may be, for example, a tilting operation for moving the direction of the camera 100 in the vertical direction (vertical direction). Further, the operation for moving the angle of view may be an operation such as a roll direction for rotating the camera 100 with respect to the optical axis of the lens 10. Moreover, the operation which combined those panning, tilting, and a roll may be sufficient. Whether or not panning, tilting, roll, or the like has been performed on the camera 100 can be determined by a detection technique based on the output of a known acceleration sensor, angular velocity sensor, azimuth sensor, or the like. That is, the camera 100 calculates the moving direction and moving amount of the camera before and after shooting based on the output of the acceleration sensor, angular velocity sensor, azimuth sensor, and the like, and an operation for moving the angle of view is performed based on the information. Can be determined. In addition, the determination can also be made by using a known technique for detecting a motion vector from a captured image.

  FIGS. 5A and 5B show an example of an angle of view operation of the camera 100 by the photographer, an operation of changing the angle of view by a zoom operation of the camera, or moving the camera in the perspective direction with respect to the subject. It is explanatory drawing which gave the case where operation which changes an angle of view was performed by this.

  FIG. 5A is a view similar to FIG. 4A, and shows an example in which a person is shown as a subject 503 in an angle of view 501 at the time of shooting. Similar to the subject 403 described above, the subject 503 is located near the center of the screen, and the face occupies a certain size or more in the angle of view, and the face is facing the front with respect to the camera 100. In the case of the example in FIG. 5A as well, it is considered that the subject 503 is highly likely to be an important subject, as in FIG. 4A. Also in this case, as described above, the camera 100 performs recognition processing 513 for duplication confirmation using the feature information of the subject 503 and the feature information registered in the dictionary database 515. Temporary registration for storing information in the primary storage memory 519 is performed. In addition, the camera 100 causes the primary storage memory 519 to store the angle-of-view information when the subject 503 is photographed together with provisional registration of the feature information of the subject 503.

  FIG. 5B shows a state in which the person of the subject 503 moves in the frame-out direction 505 where the person disappears from the angle of view 501, and the subject 507 moves in the frame-out direction 505, as in the example of FIG. It represents the subject after moving. In addition, the angle of view 511 is set so that the subject 507 can be accommodated when the photographer moves the camera away from the subject by zooming in the wide direction after the subject 507 is out of the frame. It represents the angle of view. In other words, the angle of view 511 is an angle of view that can capture a wider range than the angle of view 501. As shown in FIG. 5B, after the subject 507 is out of the frame, the subject 507 detected again after an operation for changing the angle of view by zooming or moving the camera is important for the photographer. It seems that the possibility of being a subject has further increased. However, even at this time, it has not yet been determined whether or not the subject 503 is an important subject.

  For this reason, the camera 100 is important for the photographer when the subject 507 is re-detected from the photographed image acquired at the angle of view 511 after the angle of view operation for changing the angle of view is performed by the photographer. Recognition processing 533 for confirming whether or not the subject is present is performed. In this recognition processing 533, the camera 100 first calculates the image angle from the angle of view information at the time of shooting the re-detected subject 507 and the angle of view information at the time of shooting of the subject 503 temporarily registered in the primary storage memory 539. It is determined whether or not an operation for changing a corner has been performed. Further, when the camera 100 determines that an angle-of-view operation for changing the angle of view has been performed and detects the subject 507 after the angle-of-view change, the camera 100 extracts feature information of the subject 507. Further, the camera 100 compares the feature information extracted from the subject 507 detected after the change in the angle of view with the feature information of the subject 503 temporarily registered in the primary storage memory 539, and the subject 507 and the subject 503 are the same. A recognition process 533 is performed to check whether or not. When the recognition is established in the recognition processing 533, that is, when the re-detected subject 507 is temporarily registered in the primary storage memory 539, the camera 100 determines that the subject 507 is an important subject. Thereafter, the camera 100 associates the feature information of the subject 507 or the subject 503 with the related information of the subject 507 or the subject 503 and performs main registration in the dictionary database 535. Also in this example, as will be described later, which of the feature information of the subject 507 and the feature information of the subject 503 is to be registered in the dictionary database 535 is determined based on the reliability of the feature information.

  In the example of FIG. 5A and FIG. 5B as well, the camera 100 recognizes again using the field angle information together with the feature information of the subject that is re-detected after the provisionally registered subject disappears. To confirm whether the subject is an important subject. As a result, the camera 100 can estimate with high probability that the photographer intends to photograph the subject, and can automatically register the feature information of the subject in the dictionary database.

  Note that if the amount of change in the angle of view and the speed of change due to the operation of changing the angle of view are very large, it can be considered that the subject 507 is unlikely to be an important subject. The recognition process 533 may not be performed. Further, whether or not an angle of view operation that changes the angle of view by a zoom operation can be determined based on control information of the mechanical drive circuit 16 that drives the lens 10. That is, in the camera 100, for example, the photographer intends the angle of view of the camera 100 based on the control value when the system control unit 42 controls the mechanical drive circuit 16 to change the focal length (zoom magnification) of the lens 10. It is possible to determine whether or not an angle of view operation to be changed is performed. Further, whether or not an operation for changing the angle of view by moving the camera in the perspective direction with respect to the subject can be determined by a detection technique based on the output of a known acceleration sensor, angular velocity sensor, azimuth sensor, or the like. is there. That is, the camera 100 calculates the movement direction and movement amount of the camera before and after shooting based on outputs from, for example, an acceleration sensor, an angular velocity sensor, and an orientation sensor, and moves the camera in the perspective direction based on the movement direction and movement amount. It can be determined whether or not an operation for changing the angle of view has been performed. In addition, the determination can also be made by using a known technique for detecting a motion vector from a captured image.

  In the above example, the example of the angle of view operation for moving the angle of view and the example of the angle of view operation for changing the range captured by the angle of view are described separately. The present embodiment can also be applied when corner operations are combined and performed substantially simultaneously.

  Here, in order to realize the processing described with reference to FIGS. 4A, 4B, 5A, and 5B, subject feature information is extracted and recognized. Before that, it is necessary to first detect the subject from the captured image. In addition, since it is assumed that there are many cases such as a case where a plurality of subjects exist in a captured image, a case where subjects cross each other, and a case where subjects disappear / return, it is possible to efficiently perform recognition processing. It is necessary to manage the recognition processing result.

6 (a) to 6 (c) manage a plurality of subjects by performing a numbering process for assigning identification information (referred to as face ID) to subjects detected from within an image acquired by photographing. It shows how it is. Assume that frame numbers n, n + 1, n + 2,.
FIG. 6A shows a case where three subjects can be detected from the image 600 of frame number n. The camera 100 detects the face ID (ID0, ID1, ID2) for each of these three subjects. ) Is issued (allocated), and each subject is managed by the face ID. In the example of FIG. 6A, it is assumed that the subject has moved between the image 601 with the frame number n + 1 and the image 602 with the frame number n + 2. In this case, in the camera 100, the subject before and after the movement is the same subject (the same person) based on the change in the relative positional relationship between the subjects and the size relationship between the subjects between the previous frame and the current frame. The same face ID is issued when the subject is the same. For example, when the feature information of the person with face ID 2 matches the feature information stored in the dictionary database at the time of frame number n + 1, the camera 100 recognizes the person with face ID 2 as a registered specific individual. After that, in order to keep the person with face ID 2 as a specific individual, it is necessary to continue the recognition process, but this need is eliminated by managing the face ID by issuing it. . If the recognition result determined at the time of the image 602 with the frame number n + 1 and the face ID 2 are managed in this way, the person with the face ID 2 is recognized in the past even at the time of the image 603 with the frame number n + 3. It turns out that it is a person who was done.

  FIG. 6B shows a state in which two persons (face ID 3 and face ID 4) can be detected from the image 630 with the frame number n, and then the image 631 with the frame number n + 1 to the image with the frame number n + 2. A case where two persons cross each other between 632 is shown. At the moment when the persons with face ID 3 and face ID 4 approach each other, the near-side subject can be detected continuously, but the back-side subject is blocked by the near-side subject. In the example of FIG. 6B, it is assumed that the person with face ID 3 is on the near side and the person with face ID 4 is on the back side. In this case, since the person with face ID 3 on the near side can be detected without being blocked, face ID 3 can be maintained, but the person with face ID 4 on the back side is blocked, and the continuity of ID is lost. Another face ID 5 is issued to the person detected after being blocked. It should be noted that face ID4 can be continued when a predetermined condition is satisfied, such as when the time to be blocked is short and the place where it reappeared after being blocked is close, but like image 632 with frame number n + 2 In some cases, another face ID is issued. In this case, since different face IDs are issued even if they are the same person, they are treated as different persons.

  FIG. 6C shows a state in which one person (face ID 6) can be detected from the image 650 having the frame number n. Then, a case is shown where a person with face ID 6 in the image 651 with the frame number n + 1 has gone out of the field of view and reappeared with the image 652 with the subsequent frame number n + 2. The person whose face ID 6 is assigned in the image 651 with the frame number n + 1 is subsequently assigned the face ID 7 when the person enters the frame 652 with the image 652 with the frame number n + 2. As shown in FIG. 6C, when a frame-in occurs after frame-out, both a case where a person with face ID 6 reappears and a case where another person appears are considered. May be right or wrong.

As shown in FIGS. 6A to 6C, there is an advantage that it becomes easy to realize information management of each individual by issuing the face ID.
However, in any of the cases shown in FIGS. 6 (a), 6 (b), and 6 (c), the face ID is not universal, and different face IDs are issued even for the same person. Can occur.

  FIG. 7 shows a management table of related information on the subject detected from the image, and the camera 100 holds this management table internally. In the management table, a face ID information column 703, a size information column 705, a position information column 707, a personal recognition information column 709, and a priority order information column 701 are prepared as areas for storing subject related information. The face ID information field 703 stores the face ID described above. The size information column 705 stores size information that represents the size of each subject by the number of pixels. The position information column 707 stores coordinate information representing the position of each subject in the image. The personal recognition information column 709 stores individual personal recognition information of the subject person. In the priority order information column 701, information indicating the importance of the subject with the face ID is stored. In the management table, by sorting in order of importance in the priority order information column 701, the size information of the subject corresponding to the face ID is divided into rows 730, 731, 732, 733,. Location information and personal recognition information can be managed. For example, it is most desirable that the degree of importance is the same as the intention of the photographer. The intention of the photographer is set based on factors such as the size of the subject, the position, and the success or failure of personal recognition. be able to. Of the information managed in the management table, the face ID, size information, and position information are information indicating the presence of the subject in the image, and are used, for example, for focusing control during AF and exposure adjustment during AE. It is done. The personal recognition information is information indicating an individual who can be identified and identified as a result of recognition processing using information in the dictionary database, and is managed by the management table together with the face ID. For this reason, it is possible to easily know on the management table which subject has been recognized or has not been recognized.

FIG. 8 is a conceptual diagram showing an example of the structure in the dictionary database.
In FIG. 8, the dictionary database 801 uses, for example, a non-volatile memory that continues to be stored even when power supply is stopped as a storage medium. In the example of the camera 100 in FIG. 1, an area for a dictionary database is prepared in the ROM 48. Yes. Note that the dictionary database 801 may have a configuration in the external memory 90.

  Management information 803 is stored in the dictionary database 801. The management information 803 stores information indicating the overall configuration, such as the number of registered subjects (number of subjects) and their storage locations, the number of feature information registered for each subject and their storage locations. . Further, as a concept, the management information 803 stores individual information 811, 821, 831,... In a tree shape below the management information 803, and further includes feature information 813, 815, 817,. 823, 825,..., 833, 835,.

  In these personal information, arbitrary information such as name information 863, birthday information 865, group information 867, and other information 869 can be stored as content information 861 of each personal information. As the use of personal information, for example, a person registered in the dictionary database is preferentially subjected to AF, an optimum shooting parameter is applied to the individual, and, for example, individual A and person B are subject to AF. For example, a difference in priority may be given. Other uses of personal information include, for example, a playback function application that creates various shooting applications based on personal information, or narrows down images using personal names as search keys when browsing shot images. You can also create.

  Each feature information includes, as content information 881 of the feature information, actual feature information 833 of the subject, image information 885 used for generating the feature information, information 887 such as a shooting time of the image where the feature information is generated, and the like. Arbitrary information such as information 889 can be stored. Depending on the collation algorithm for personal recognition, the performance of recognition can be improved by photographing and registering the same person under different conditions, so that a plurality of pieces of feature information can be registered per person. Further, by storing the image information 885 used when the feature information is generated, the registered feature information can be visually confirmed by the user. In addition, since the feature information of a person changes with the growth and aging of the person, the feature information is updated regularly. For this reason, by storing the information 887 indicating when the image used for generating the feature information is obtained, it can be used as a guideline for updating.

FIG. 9 is a conceptual diagram showing an example of the structure in the primary storage memory described above. The primary storage memory 901 uses, for example, a volatile memory that depends on power supply to the apparatus as a medium, and corresponds to the RAM 46 in FIG. 1 in the present embodiment.
As shown in FIG. 9, management information 903 is stored in the primary storage memory 901. The management information 903 in the primary storage memory 901 stores the number of feature information stored in the primary storage and information on the storage locations thereof. As a concept, the feature information 911, 913, and 915 are arranged in a tree form below the feature information. , 917,... Are stored.

  The feature information in the primary storage memory 901 includes the feature information content information 931, the actual feature information 933, the face ID 932, the image information 935 used when generating the feature information, and the shooting time of the image used for generating the feature information. The angle of view information 937 and time information 939 are stored. As the angle of view information, information obtained from a mechanical mechanism such as the zoom position of the lens 10, information obtained by electronic clipping such as electronic zoom and aspect setting, and panning change based on output from an acceleration sensor or the like are used. It is possible to store relatively expressed information.

  10, 11, 12, and 13 are flowcharts of each process from system activation of the camera 100 according to the present embodiment, detection of a subject from a captured image, determination of a recognition status, and recognition processing to registration. The processing of each of these flowcharts may be executed by a hardware configuration, or may be realized by a software configuration based on a program executed by a CPU or the like, partly realized by a hardware configuration and the rest by a software configuration. Also good. The program executed by the CPU or the like may be stored in the ROM 48 or the like, for example, may be acquired from a recording medium such as the external memory 90, or may be acquired via a network (not shown) or the like. In the following description, steps S101 to S465 of each process are abbreviated as S101 to S465.

FIG. 10 is a flowchart showing the overall processing flow after system startup in the camera 100 of this embodiment.
When the power button (200) is pressed and the main power is turned on, the camera 100 starts the process of the flowchart of FIG. 10 in S101. In step S103, the camera 100 performs system activation processing. Here, the basic system is started, such as power supply and clock supply to the CPU and LSI necessary for the operation of the camera system, and initialization of the memory and OS.

  Next, in S105, the system control unit 42 activates an imaging element such as a CCD or CMOS in the imaging unit 20, and activates a lens barrel system device such as a focus lens or a zoom lens of the lens 10 via the mechanical drive circuit 16. Do. As a result, the mechanical shutter 12 and the diaphragm 13 are operated, and external light is guided to the imaging device of the imaging unit 20. In addition, the system control unit 42 starts driving an imaging system device such as an imaging device, an amplifier, and an A / D converter of the imaging unit 20 via the imaging drive circuit 22.

  In this state, the system control unit 42 starts AE driving in S113, starts AF driving in S111, and continues to control the subject to be appropriately bright and focused. At this time, in S201, the system control unit 42 also starts subject detection and recognition processing by the image processing circuit 40. Further, the system control unit 42 can adjust the focus and brightness of the subject to be more appropriate by using the subject information detected by the image processing circuit 40 for AF and AE. At the same time, the image processing circuit 40 also starts development processing of the live image output to the display device 50, so that the photographer can check the live view video on the screen of the display device 50. Become. In other words, this is a state that satisfies the requirements for the viewfinder application, and the photographer can capture the object to be photographed and perform framing work such as angle adjustment.

  In step S121, the system control unit 42 determines whether the photographer has performed a so-called half-press operation (SW1 is turned on) of the shutter button (202) of the operation unit 44. The system control unit 42 returns the processing to S111, S113, and S201 when SW1 is not turned on (off), and when SW1 is turned on, the system control unit 42 uses AF for exposure suitable for AF in S123. AE control is performed. Then, after the exposure suitable for AF is made, the system control unit 42 performs so-called one shot AF (One Shot AF) control for still images, for example, in S125.

  Next, after completion of shooting preparation by turning on SW1, the system control unit 42 determines in S131 whether or not the shutter button is fully pressed (SW2 is turned on). If SW2 is not turned on and SW1 is also turned off, the system control unit 42 returns the processing to S111, S113, and S201, and if SW2 is turned on, causes the camera 100 to execute still image shooting in S151. .

11, FIG. 12, and FIG. 13 are flowcharts showing the details of the operation of subject detection and recognition processing in S201 of FIG.
FIG. 11 is a flowchart showing the flow of the face ID numbering process described with reference to FIGS. 6 (a) to 6 (c). When an image is captured in a state where the face detection and face recognition functions of the camera are set to be effective, the system control unit 42 performs face ID numbering processing shown in the flowchart of FIG. To start.

  When the face ID numbering process is started, the image processing circuit 40 first performs face detection in S203 by using an image generated in the drive cycle of the image sensor as an input. The image processing circuit 40 can detect a plurality of faces from the image by a single face detection process. The image processing circuit 40 then repeats the subsequent face ID numbering process as many times as the number of faces detected in S203. Therefore, in step S205, the image processing circuit 40 compares the number of detected faces with the number of issued face IDs, and determines whether the number of issued face IDs count is less than the detected number of faces. judge. If the number of issued face IDs count is less than the detected number of faces, the image processing circuit 40 proceeds to S207, and the number of issued face IDs count exceeds the detected number of faces. If so, the process proceeds to S215.

  In step S207, the image processing circuit 40 compares the previous face detection result old with the target face of the current face detection result, and evaluates the continuity using the previous face position and face size information. Do. In step S209, the image processing circuit 40 determines whether the target face of the current face detection result is a face in the previous face detection result old or a face newly found this time. If the image processing circuit 40 determines in S209 that the face has been newly found, the process proceeds to S211. If the image processing circuit 40 determines that a known face that is not a new face has been found, the image processing circuit 40 proceeds to S213.

In step S211, the image processing circuit 40 issues a new face ID to the new face, increments the value of several counts, and returns the process to step S203. The face ID issued here is stored in the face ID information column 703 of the management table shown in FIG. On the other hand, when the process proceeds to S213, the image processing circuit 40 continues the face ID issued in the previous face detection, increments the value of several counts, and returns the process to S203.
The above processing is repeated for each detected face, and when the issued face ID value count exceeds the number of detected faces, the process proceeds from S203 to S215, where the image processing circuit 40 performs the next face detection. The current face detection result is stored in the face detection result old. Thereafter, the face ID numbering process is temporarily ended, and when face detection and face ID numbering are performed for the next photographed image, the process of the flowchart of FIG. 11 is started again.

  FIG. 12 is a flowchart showing the flow of recognition status determination processing for determining whether or not recognition processing should be performed on a detected face. When an image is shot with the face detection and face recognition functions set to effective in the camera, the system control unit 42 performs recognition state determination processing shown in the flowchart of FIG. 12 on the image processing circuit 40 in S301. Let it begin.

  Similarly to the face ID numbering process described above, in the recognition status determination process, if a plurality of faces are detected, the process is repeated as many times as the number of detected faces. For this reason, when the recognition state determination process is started, first, in S303, the image processing circuit 40 compares the number of detected faces with the number of face IDs issued, and counts the issued face in the same manner as S205 described above. It is determined whether or not the ID count is less than the detected number of faces. The image processing circuit 40 advances the process to S305 when the number of issued face IDs count is less than the detected number of faces.

  In step S305, the image processing circuit 40 determines whether the detected face is a newly found face or a known face already found based on the management table and face ID shown in FIG. Determine if it exists. If it is determined that the face is a new face, the image processing circuit 40 treats the new face as a recognition face that has not yet been performed in step S307, and sets it as a target for recognition processing in FIG. Is incremented and the process returns to S303. On the other hand, if it is determined that the face is a known face, the image processing circuit 40 treats the known face as already recognized in S309, and advances the process to S311.

  Here, in the case of a known face, it may be better to repeat the recognition process. Therefore, when the process proceeds to S <b> 311, the image processing circuit 40 performs re-execution to determine whether the recognition process should be performed again for the known face. Judgment processing is performed. For example, as in the case of FIG. 6B described above, there is a concern that an error may occur in the issued face ID, such as when people approach each other and cross each other. The recognition information managed in this way will also have errors. For this reason, the image processing circuit 40 determines that it is better to repeat the recognition process for the known face even if the person is known as the face ID in S311 when an approach or intersection occurs between the persons. , The process proceeds to S313.

In step S313, the image processing circuit 40 sets the known face as a target for re-performing the later-described recognition process in FIG. 13, and then increments the value of several counts and returns the process to step S303.
On the other hand, if it is determined in S311 that the recognition process is not to be performed again, the image processing circuit 40 increments the value of the number count and returns the process to S303.
When the number of issued face IDs count is equal to or greater than the detected number of faces, the image processing circuit 40 once ends the recognition state determination process of FIG. 12 and the recognition state determination is performed for the next photographed image. Then, the process of the flowchart of FIG. 12 is started again.

  FIG. 13 is a flowchart showing the flow of the recognition process for the face that is the target of the recognition process in the flowchart of FIG. 12 and the important subject registration process using the information temporarily registered in the primary storage memory. When the face detection and face recognition functions in the camera are set to be effective and there is a recognition process target, the system control unit 42 recognizes the image processing circuit 40 in the recognition process shown in the flowchart of FIG. 13 in S401 and S451. The important subject registration process is started. These recognition processing and important subject registration processing are performed in parallel.

The flowchart of the recognition process will be described.
In S403, the image processing circuit 40 generates feature information for each detected face. The feature information generation processing is performed in steps of face detection, detection of organs such as eyes and nose and mouth, and generation of feature information based on the organs. However, when the detection accuracy in the middle is low, feature information cannot be generated, or only feature information with low accuracy can be generated. It is also desirable to avoid wasting the capacity of the dictionary database due to duplicate registration.

  For this reason, in S405, the image processing circuit 40 performs collation by comparing the generated feature information with the feature information registered in the dictionary database. For example, as shown in FIG. 8 described above, since the dictionary database can store a plurality of pieces of feature information and a plurality of pieces of feature information per person, the image processing circuit 40 sequentially collates with the pieces of information in S405. Do.

  The image processing circuit 40 represents the collation result as a numerical value as a similarity, and determines that the recognition is established when the similarity exceeds a predetermined threshold value in S407. If it is determined in S407 that the recognition is established, the image processing circuit 40 determines the recognition result in S409. The image processing circuit 40 employs the person with the highest similarity when there are a plurality of recognized persons.

On the other hand, if it is determined in S407 that the degree of similarity is equal to or less than the threshold value, that is, if any information registered in the dictionary database is not verified, the image processing circuit 40 determines that recognition is not established, and proceeds to S411. Proceed with the process. In step S411, the image processing circuit 40 temporarily registers the feature information of the face to be recognized in the primary storage memory (RAM 46). At this time, the image processing circuit 40 provisionally registers the face ID, the angle of view information when the original image that generated the feature information is captured, and the time information as well as the feature information.
Then, after these S409 and S413, the image processing circuit 40 once ends the recognition processing of FIG. 13, and when the recognition processing is performed for the next photographed image, the recognition processing of FIG. 13 is started again.

Next, a flowchart of important subject registration processing will be described. The important subject registration process is a process for determining whether or not the feature information provisionally registered in the primary storage memory is the feature information of the important subject and, if it is an important subject, performing the main registration in the dictionary database.
In step S453, the image processing circuit 40 first generates feature information from the subject. In S453, along with the feature information, face ID numbering, view angle information when the original image that generated the feature information is captured, and time information are also acquired.

  Next, the image processing circuit 40 uses S455, S457, S459, S461, and S463 by using the information generated in S453 and the information generated in the recognition process described above and temporarily registered in the primary storage memory. The determination process based on each comparison process is sequentially performed.

  In S455, the image processing circuit 40 compares the face ID issued in S453 with the temporarily registered face ID. For example, as shown in FIGS. 6 (b) and 6 (c), when a face once disappears due to blockage or out of frame and reappears, another face ID is assigned before and after disappearing. . Therefore, using the face ID numbering mechanism, if the face IDs are the same, they are excluded from the important subject determination as the same person. That is, the image processing circuit 40 proceeds to S457 when the face IDs are different, and ends the important subject registration process when the face IDs are the same.

  In step S457, the image processing circuit 40 compares the view angle information acquired in step S453 with the temporarily registered view angle information, and performs a determination process using the difference between the view angle information. Here, when the difference in the angle of view information is less than the predetermined difference, the face person who generated the feature information in S453 has not performed the angle of view operation such as the movement of the angle of view or the change of the angle of view described above. In addition, it is presumed that this is a new person who has entered the angle of view. For this reason, if the image processing circuit 40 determines in S457 that the difference in the angle of view information is less than the predetermined difference, the image processing circuit 40 excludes it from the important subject determination. On the other hand, if it is determined that the face ID is different in S455, and the difference in the angle-of-view information is greater than or equal to the predetermined difference and is within the predetermined range in S457, the target person is obtained by performing the angle-of-view operation. It can be determined that there is a high possibility that the image is within the angle of view. For this reason, if the image processing circuit 40 determines in S457 that the difference in the angle of view information is greater than or equal to the predetermined difference and is within the predetermined range, the image processing circuit 40 proceeds to the next time difference determination process. That is, if the image processing circuit 40 determines that there is no difference in the angle of view information in S457, the image processing circuit 40 ends the important subject registration process, whereas if there is a difference in the angle of view information, the image processing circuit 40 proceeds to S459.

  In step S459, the image processing circuit 40 compares the time information acquired in step S453 with the temporarily registered time information, and performs a determination process using a difference between the time information. Even if the determination process of S455 and the determination process of S457 are passed and the process proceeds to S459, if a long time has elapsed since provisional registration in the primary storage memory, the photographer searches for the subject. It is highly probable that the act of For this reason, if the image processing circuit 40 determines in S459 that the time difference between the pieces of time information is outside the predetermined time range, the image processing circuit 40 ends the important subject registration process. On the other hand, if the image processing circuit 40 determines in S459 that the time difference of the time information is within the predetermined time range, the image processing circuit 40 proceeds to S461.

  In step S461, the image processing circuit 40 collates the feature information generated in step S453 with the temporarily registered feature information, and determines whether or not recognition is established. If the recognition is not established, the image processing circuit 40 ends the important subject registration process. If the recognition is established, the image processing circuit 40 proceeds to S463.

  In step S463, the image processing circuit 40 performs reliability determination on the feature information, and performs main registration processing in the dictionary database based on the determination result. That is, since the feature information that can be registered includes the latest information and the information that has been provisionally registered in the primary storage memory in the past, the image processing circuit 40 performs a main registration of those having higher reliability by comparing the two. To do.

  FIG. 14A to FIG. 14C are diagrams for explaining a tendency that the reliability varies depending on conditions. FIG. 14A shows an example in which the sizes of the faces of the subject A and the subject B are different in the image. The larger the face size, the higher the reliability (reliability), and the smaller the reliability. Is low. FIG. 14B shows an example in which the orientations of the faces of the subject A and the subject B are different in the image. The higher the degree that the subject's face is facing the front, the higher the reliability is; It is low. FIG. 14 (c) shows an example of whether the eyes, nose and mouth organs of the faces of the subject A and the subject B are blocked in the image, and the reliability is so high that the eyes and nose are not blocked. The obstruction of any organ of the eyes and nose mouth indicates that the reliability is low. For example, in the case of the panning shown in FIG. 4 described above, there is no great difference in the subject itself before and after panning, so it is assumed that it is better to use feature information generated after panning that is a newer subject for registration. The Further, for example, in the case of the zoom operation shown in FIG. 5 described above, there is a difference in the size of the subject before and after the zoom operation, and before the zoom operation with a larger subject, the feature information is more reliable and suitable for registration. ing. In the image processing circuit 40 of this embodiment, the reliability is determined as described above, and the feature information of the subject with the high reliability is fully registered in the dictionary database.

  As described above, in the present embodiment, an important subject is determined based on an operation in which the photographer tries to keep the subject within the angle of view. For this reason, according to the present embodiment, even under severe conditions such as a case where there are a plurality of subjects in the screen or a case where the subject is blocked or disappeared, it is important that the time is short and highly reliable. Subject extraction can be performed. That is, according to the present embodiment, it is possible to accurately extract an important subject and automatically register it by determining whether the subject is important based on a view angle operation by the photographer. It becomes.

  In the above description, an example in which subject detection and recognition processing is performed in the image processing circuit 40 has been described. However, these processing may be realized by the system control unit 42 executing the program according to the present embodiment. Good. Further, a part may be executed by the image processing circuit 40 and the rest may be executed by the system control unit 42 based on a program.

  In the embodiment described above, a digital camera or the like has been described as an application example of the image processing apparatus. However, the present invention is not limited to this example, and can be applied to other imaging apparatuses. For example, this embodiment can be applied not only to digital cameras but also to various portable terminals such as smartphones and tablet terminals having camera functions, various monitoring cameras, industrial cameras, in-vehicle cameras, medical cameras, and the like.

  Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined. Also, when a software program that realizes the functions of the above-described embodiments is supplied from a recording medium directly to a system or apparatus having a computer that can execute the program using wired / wireless communication, and the program is executed Are also included in the present invention. Accordingly, the program code itself supplied and installed in the computer in order to implement the functional processing of the present invention by the computer also realizes the present invention. That is, the computer program itself for realizing the functional processing of the present invention is also included in the present invention. In this case, the program may be in any form as long as it has a program function, such as an object code, a program executed by an interpreter, or script data supplied to the OS. As a recording medium for supplying the program, for example, a magnetic recording medium such as a hard disk or a magnetic tape, an optical / magneto-optical storage medium, or a nonvolatile semiconductor memory may be used. As a program supply method, a computer program that forms the present invention is stored in a server on a computer network, and a connected client computer downloads and programs the computer program.

  10: Lens, 20: Imaging unit, 40: Image processing unit, 42: System control unit, 44: Operation unit, 46: RAM, 48: ROM, 50: Display device

Claims (14)

Detection means for detecting a subject from the captured image;
Extracting means for extracting feature information from the detected subject;
Recognition means for recognizing an individual of the subject based on the feature information;
Registration means for registering the feature information in a database;
Storage means for primarily storing the feature information,
The recognizing unit compares the feature information extracted from the subject detected after the subject disappears from the image with the feature information stored in the primary storage, and detects the subject detected after disappearing from the image. Confirm whether or not the subject of the primary stored feature information,
The registration unit confirms that the subject detected after disappearing from the image is the subject of the feature information stored in the primary storage by the recognition unit, and the subject of the feature information stored in the primary storage is captured. An image processing apparatus, wherein when the angle of view is different from the angle of view when a subject detected after disappearing from the image is photographed, the feature information of the subject is registered in the database.   The image processing apparatus according to claim 1, wherein the storage unit performs the primary storage when feature information extracted from the subject is not registered in the database. The detection means attaches identification information to the detected subject,
The recognizing unit performs the matching and the confirmation when the identification information of the subject before disappearing from the image is different from the identification information of the subject detected after disappearing from the image. The image processing apparatus according to claim 1.   The recognizing means collates the detected feature information of the subject from the image taken with the angle of view moved after the subject disappears from the image, and the primary stored feature information, and The image processing apparatus according to claim 1, wherein confirmation is performed.   The image processing apparatus according to claim 4, wherein the movement of the angle of view includes movement by moving the imaging apparatus in at least one of a horizontal direction and a vertical direction.   The recognizing unit collates the feature information of the subject detected from the image taken with the angle of view changed after the subject disappears from the image, and the feature information stored in the primary storage to check the confirmation The image processing apparatus according to claim 1, wherein:   The image processing apparatus according to claim 6, wherein the change in the angle of view includes a change by a zoom operation of the imaging apparatus.   The image processing apparatus according to claim 6, wherein the change in the angle of view includes a change by an operation of moving the imaging apparatus in the perspective direction with respect to the subject.   The registration means is configured such that a difference between an angle of view when a subject detected after disappearing from the image is photographed and an angle of view when the subject of the primary stored characteristic information is photographed is a predetermined difference or more. The image processing apparatus according to claim 1, wherein the registration is performed when the value is within a predetermined range.   The said recognition means performs the said collation and confirmation, when the time until a to-be-detected object is detected within the predetermined time range after disappearing from the said image. The image processing apparatus according to item 1.   The registration unit is configured to register the feature information having higher reliability among the feature information extracted from the subject detected after disappearing from the image and the feature information primarily stored. The image processing apparatus according to any one of claims 1 to 10.   The registration unit determines the reliability based on at least one of a size of a subject in the image, a degree of the subject facing the front, and a detection result of an organ included in the subject. The image processing apparatus according to claim 11, wherein: An image processing method executed by an image processing apparatus,
A detection step of detecting a subject from the captured image;
An extraction step of extracting feature information from the detected subject;
A recognition step of recognizing an individual of the subject based on the feature information;
A registration step of registering the feature information in a database;
A storage step of temporarily storing the feature information,
In the recognition step, the feature information extracted from the subject detected after the subject disappears from the image is compared with the feature information stored in the primary storage, and the subject detected after disappearing from the image is detected. Confirm whether or not the subject of the primary stored feature information,
In the registration step, the recognition step confirms that the subject detected after disappearing from the image is the subject of the feature information stored in the primary storage, and the subject of the feature information stored in the primary storage is photographed. An image processing method comprising: registering feature information of the subject in the database when the angle of view is different from the angle of view when the subject detected after disappearing from the image is captured.   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 to 12.

Images