JP7479803B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to a device that detects a subject and displays the detected subject position.

In digital cameras that use an image sensor, it is common to detect and track a subject from image data obtained from the image sensor, and then photograph the subject by adjusting the focus, brightness, and color to optimal conditions. Typical subjects to be detected include a person's face or body, or specific animals such as dogs and cats. Furthermore, organ detection, such as the eyes (pupils), nose, and mouth in the face, is a technique for detecting specific parts of a detected subject. A typical use of organ detection is pupil AF, which sets the detected eye as the focus detection area and performs autofocus. Patent Document 1 discloses a method for estimating the pupil area from face contour information, using the relative position of the pupil area from the face contour area in statistical data.

JP 2009-175744 A

When the subject is far from the camera, i.e. small, the amount of information about the subject is reduced, making it difficult to achieve accurate detection. When performing pupil detection on such small subjects, existing methods pose problems.

Therefore, the present invention aims to provide a technology that can estimate and display the pupil position with high accuracy without compromising real-time performance for small subjects, compared to the prior art.

The image processing device of the present invention has a display control means for displaying a captured image and an icon on a display means, a detection means for detecting a face and eyes from the captured image, a tracking means for detecting and tracking a face area set in a first captured image from a second captured image using a detection method that is simpler than detection by the detection means, and a determination means for determining a relative position of the eyes to the face based on the eye position, face position and face size information detected by the detection means, the determination means determining an estimated position of the eyes by applying the relative positional relationship of the eyes to the face to the face position and face size of the face area detected by the tracking means in the second captured image, and the display control means superimposes and displays an icon indicating the determined estimated position of the eyes on the second captured image displayed on the display means.

The present invention makes it possible to accurately estimate and display the pupil position for small subjects without compromising real-time performance.

Configuration diagram for implementing the present invention Diagram of face and organ detection method Diagram of tracking method Timing diagram in the first embodiment FIG. 1 is a diagram showing a pupil position estimation method in the first embodiment. Flowchart for estimating pupil position when face is small in the first embodiment FIG. 13 is a diagram showing a method for correcting the display frame position based on the face angle in the second embodiment. FIG. 13 is a diagram showing a method for correcting the display frame size depending on the face angle in the second embodiment.

Various embodiments of the present invention will be described with reference to the accompanying drawings. In each embodiment, an imaging device having an pupil detection function is exemplified. Examples of imaging devices having pupil detection function include video cameras, digital cameras, and silver halide still cameras, as well as mobile devices such as smartphones equipped with a camera function, which also constitute one aspect of the present invention.

In this embodiment, to notify the user of the activation and activation position of eye AF, a mark such as a frame is displayed in the eye AF area, superimposed on the image data of the subject on the display device on the camera. Particularly for moving subjects, real-time performance is required to accurately inform the user of the activation position of eye AF. Also, it is preferable that eye AF be activated and a frame be displayed even for subjects as far away as possible from the camera, to allow the photographer ample time to prepare for shooting.

Known methods for face detection and organ detection include a method using Haar-like features (brightness difference between two areas) and deep learning. Another method is a method that uses general-purpose object tracking using a template matching method to create an image area containing the position of an organ at a certain point in time as a template, and performs pattern matching to search for areas similar to the template in the next frame and thereafter. Still another method is a method of simply estimating the position of an organ from other information.

In general, organ detection means are highly accurate but take a long time to perform detection. This impairs real-time performance. In particular, when face detection and organ detection are configured as separate means in the device, more time is required. For example, it is possible to detect organs by detecting multiple face candidates within the screen, selecting and determining faces that can be AF targets from among the multiple face candidates, and then inputting the face area into the organ detection means and analyzing it in more detail. In this way, when face detection and organ detection are configured separately, the order of execution is automatically determined, and furthermore, depending on the time it takes for organ detection, a time lag occurs when face detection is performed.

On the other hand, tracking methods using template matching methods require less time for detection and are easier to ensure in real time, but they have problems with accuracy. Template matching is a method for searching for similar patterns in an image, so it may hit a similar subject other than the one you are actually trying to detect. Smaller eyes in a small face do not provide sufficient resolution for tracking, and there is a problem that mistracking occurs due to insufficient matching accuracy. The method described in Patent Document 1, which is a method for simply estimating the position of organs from other information, has the advantage of requiring less time for detection, but estimates the pupil area based on the average shape of a human face, so deviations occur in subjects whose facial features are biased from the average.

Therefore, this embodiment is characterized by combining organ detection processing and pattern matching processing to achieve both real-time and accurate pupil detection.

Figure 1 shows an example of the configuration of an imaging device according to a first embodiment of the present invention, and illustrates the configuration of a mirrorless camera (hereinafter, camera) equipped with an eye AF function.

The interchangeable lens 100 is one of the optical devices that can be attached to the camera body 120. The interchangeable lens 100 has a photographing lens unit 101 that includes a main photographing optical system 102, an aperture 103 that adjusts the amount of light, and a focus lens group 104 that adjusts the focus.

The lens system control microcomputer (hereinafter referred to as the lens control unit) 111 includes an aperture control unit 112 that controls the operation of the aperture 103, and a focus lens control unit 113 that controls the operation (also referred to as driving) of the focus lens group 104. The focus lens control unit 113 drives the focus lens group 104 in the optical axis direction of the photographing lens unit 101 based on focus lens driving information obtained from the camera body unit 120, and adjusts the focus of the camera.

The focus lens group 104 may have multiple focus lenses or only one focus lens. Also, to simplify the diagram, a fixed focal length lens is shown as an example of an interchangeable lens, but a lens with a changeable focal length (zoom lens) may also be used. In the case of a zoom lens, the lens control unit 113 obtains focal length information from an encoder output that detects the zoom lens position. In the case of a lens equipped with an image stabilization function, the lens control unit 113 also controls a shift lens group for shake correction.

The camera body 120 includes a shutter 121 used for exposure control and an image sensor 122 such as a CMOS (complementary metal-oxide semiconductor) sensor. The image signal output by the image sensor 122 is processed by an analog signal processing circuit 123 and then sent to a camera signal processing circuit 124.

The camera system control microcomputer (hereafter referred to as the camera control unit) 131 controls the entire imaging device. For example, the camera control unit 131 drives and controls a motor for driving a shutter (not shown), and drives the shutter 121. The memory card 125 is a recording medium that records data on captured images. The pressed state of a release switch 181 operated by the photographer is sent to the camera control unit 131, and the captured image according to that state is stored on the memory card 125.

The image display unit 171 is equipped with a display device such as a liquid crystal panel (LCD) that monitors the image that the photographer is about to capture with the camera and displays the captured image. The touch panel 172 is an operation unit that allows the photographer to specify coordinates on the image display unit 171 with a finger or a touch pen, and can be configured integrally with the image display unit 171. For example, the touch panel can be configured so that the light transmittance does not interfere with the display of the image display unit 171, and can be built into the display surface of the image display unit 171 (in-cell type). Then, the input coordinates on the touch panel 172 are associated with the display coordinates on the image display unit 171. This makes it possible to configure a GUI (graphical user interface) that makes it seem as if the user can directly operate the screen displayed on the image display unit 171. The operation state of the operation unit 172 is managed by the camera control unit 131.

The camera body 120 has a mount contact 161, which is a communication terminal for communicating with the interchangeable lens 100, on its mount surface with the interchangeable lens 100. The interchangeable lens 100 also has a mount contact 114, which is a communication terminal for communicating with the camera body 120, on its mount surface with the camera body 120.

The lens control unit 105 and the camera control unit 131 control communication so that serial communication is performed at a predetermined timing via the mount contact units 114 and 161. Through this communication, focus lens driving information, aperture driving information, and the like are sent from the camera control unit 131 to the lens control unit 111, and optical information such as focal length is sent from the lens control unit 111 to the camera control unit 131.

The camera signal processing circuit 124 includes a face information detection unit 141 and an organ information detection unit 142, which detects organ information such as the pupils and mouth from the face information detected by the face information detection unit 141. The tracking unit 143 receives an imaging signal from the analog signal processing circuit 123 and detects the pupils and face using a detection method that is faster but less accurate than the face information detection unit 141 and the organ information detection unit 142. The calculation unit 144 calculates the relative position of the pupils with respect to the face using the detection results from the face information detection unit 141, the organ information detection unit 142, and the tracking unit 143. The detection results from the face information detection unit 141, the organ information detection unit 142, and the tracking unit 143 and the calculation results from the calculation unit 144 are sent to the camera control unit 131.

The camera control unit 131 has, as blocks related to the present invention, an automatic pupil selection unit 150 that automatically selects a target pupil from the detected face information, and a display frame setting unit 151 that sets a detection frame to be displayed on the display unit 171 in response to the detected face or pupil information. It also has a pupil designation/designation cancellation unit 152 that designates the pupil designated by the photographer as a pupil to be continuously detected or cancels the designation in response to the operation by the photographer. In addition to the pupil or face designated in response to the operation by the user, there is a memory unit 153 that stores the detection results of the face information detection unit 141, the organ information detection unit 142, and the tracking unit 143, and further an AF target subject setting unit 154 that notifies the focus detection unit 155 of the selected or designated pupil or face as a subject to be focused on (also called a target subject). These operate based on the outputs of the face information detection unit 141, the organ information detection unit 142, and the tracking unit 143. The focus detection unit 155 performs focus detection processing based on an image signal corresponding to the subject to be focused on notified by the AF target subject setting unit 154. The focus detection process is performed, for example, by a known phase difference detection method or contrast detection method. In the case of the phase difference detection method, the focus detection process involves calculating the amount of image shift calculated by performing a correlation operation on a pair of image signals having parallax, or converting the image shift amount into a defocus amount and calculating it. The defocus amount can be further converted into a focus lens drive amount by taking into account the sensitivity of the interchangeable lens 100 when driving the lens. The camera control unit 131 transmits the focus detection result (image shift amount or defocus amount) detected by the focus detection unit 155 or the focus lens drive amount calculated based on the focus detection result to the lens control unit 111. The focus lens control unit 113 controls the drive of the focus lens based on the focus lens drive information received from the camera control unit 131. In other words, the camera control unit 131 controls the drive of the focus lens via the focus lens control unit 113.

We will explain how to detect and track facial features in a camera with this configuration.

Figure 2 shows the organ detection process in which a detector detects faces and organs. In this embodiment, this is performed by the face information detection unit 141 or the organ information detection unit 142 therein. Figure 2(a) shows that multiple faces can be detected in a situation in which multiple faces exist in an image (face area 201, face area 211, face area 212). Figure 2(b) shows the result of applying the face area 201 to the organ detector (organ information detection unit 142). Organ detection detects feature points inside the face, and representative examples include the eyes (203, 205), nose 207, and mouth 209. Figure 2(c) shows the magnitude of the face detection score that changes depending on the face situation. The detection score is information that can be rephrased as the reliability or confidence of face detection, and it changes depending on the person's own facial expression and external factors such as the way the ambient light hits the area. It is possible to distinguish between the higher the score, the more accurate the position detection, and the lower the score, the less reliable the position information. In face area 201 in FIG. 2(c), the eyes are wide open and the mouth is clear (appears relatively large), so the face detection score is relatively high. On the other hand, in face areas 231 and 251, the eyes are small or close to being closed, the mouth is not clearly visible, and is small and puckered, so the face detection score is relatively low overall.

Figure 2(d) shows how the face angle is estimated from the positions of the detected face and organs. In a frontal view (angle 0 degrees), the pupils are positioned evenly on the left and right of the center line 262 of the face frame 261, and the centers of the nose and mouth are positioned on the center line. On the other hand, in a face rotated at a 45-degree angle, the eyes, nose, and mouth are closer to the center line 272 of the face frame 271, destroying the left-right evenness. The face angle can be estimated based on the degree of bias in the positioning of the eyes, nose, and mouth. The reliability of the face angle is calculated based on the detection score of the eyes, nose, and mouth.

Figure 3 shows the operation concept of tracking processing by a general-purpose tracking means. In this embodiment, this tracking processing is performed by the tracking unit 143 based on a signal from the image sensor 122. The image sensor is driven by a drive signal VD cycle 301, and performs repeated operations of exposure 303 and readout 305. The data obtained from the image sensor is developed and generated as a visible image (image generation 307). In Figure 3, a template matching (pattern matching) method is applied using this image. In image 1 generated by the image generation 307 process based on exposure 1, an area including the tracking target is set as a template 331 (tracking area) by user specification or automatically based on subject detection processing. After that, a search operation is performed for image 2 generated based on exposure 2 to obtain the image difference with the template image 331 while sequentially changing the position (search 333). The operation of obtaining the difference value is repeated, and the match area 335 with the smallest difference is determined as the position where the tracking target exists. After that, after tracking processing is established between image 1 and image 2, the tracking position determined in image 2 is updated as a template for the next time, and tracking using images 2 and 3, and tracking using images 3 and 4 are repeatedly performed to achieve continuous tracking processing. The frequency of updates may be controlled for each specified frame, or according to the shooting conditions and environment. The tracking means can obtain the position of the face and the positions of organs such as the eyes, but to obtain matching accuracy, it is preferable that the target be a specified size or larger.

To explain the effect of the present invention by executing the above-described organ detection process and tracking process in combination, FIG. 4 shows a timing diagram of continuous shooting from the viewpoint of processing time, using the detection shown in FIG. 2 and the tracking process described in FIG. 3. FIG. 4(a) shows a sequence of still image shooting (411, 431, 451), detection and LV exposure for AF (413, 433, 453) between still image shooting, and AF 415 and frame display 417 using the detection result. When the pupil position information used for frame display 417 is obtained from face/organ detection 421, it takes more time than tracking 441, so the time 429 until the position of the frame display is updated becomes longer and real-time performance is impaired. In addition, the detection processing time is a limiting factor in achieving the high frame rate required for continuous shooting.

Figure 4(b) shows a sequence for determining the AF and display frame target positions by tracking processing using images obtained during the LV exposure period. It is generally known that tracking processing using a simple matching process can be completed in a shorter time than detection processing using organ detection, which is generally more complex. By performing AF 435 and frame display 437 using the results of tracking 441, it is possible to update the frame display position in a shorter time 449 than the sequence of Figure 4(a) in which detection is performed, and it can be seen that a fast frame rate is also achieved. However, continuous operation by tracking alone raises concerns about erroneous tracking of similar subjects. When the subject is small, for example, in the case of a human face whose entire body is included in the image frame, the accuracy of the face position by tracking is maintained to a certain extent, but the accuracy of the eye position is often insufficient.

Figure 4(c) shows a sequence in which detection and tracking are used together, and Figure 5 shows an image diagram for explaining the combined use method. By applying information on the relative position of the eyes 515 from the face frame center 513 obtained from the high-precision detection process 465 of the previous frame (20% to the right in the X direction and 15% upward in the Y direction when the horizontal width of the face frame 511 is 100% in Figure 5) to the face information 521, 523 (position and size) of the current frame obtained by the high-speed tracking process 463, it is possible to estimate and display the estimated position 525 of the eyes based on a detection result with higher accuracy compared to when only tracking information is used, while improving the real-timeness of the frame display compared to when only organ detection is used.

The processing flow of the present invention will be described with reference to FIG. 6. This flow is executed by the camera control unit 131 or each unit such as the camera signal processing circuit 124 in response to an instruction from the camera control unit 131. First, when the LV exposure for AF is completed, the pupil frame setting process of S201 in FIG. 6B is started. When the pupil frame setting process is started in S201, the camera control unit 131 proceeds to S202 to obtain the result of the tracking process in the current shooting frame by the tracking unit 143, and proceeds to S203. In S203, it is determined whether the face was detected with a reliability equal to or higher than a certain threshold as a result of the tracking process. If it is determined that the face was detected with a reliability equal to or higher than a certain threshold in S203, the process proceeds to S204, and the camera control unit 131 next determines whether the size of the face detected using the tracking process is equal to or smaller than a predetermined threshold. The reason for performing S204 is that it is preferable for the target to be equal to or larger than a specified size in order to obtain detection accuracy by tracking. If the camera control unit 131 determines in S204 that the face size is not large enough to obtain accuracy, that is, that the face size is equal to or smaller than a predetermined threshold, the process proceeds to S205. In S205, the calculation unit 144 acquires the latest face and eye detection results obtained from the face information detection unit 141 and the organ information detection unit 142. After acquiring the detection result in S205, the camera control unit 131 determines in S206 whether the difference between the frame number (frame number) of the frame in which the acquired detection result was detected and the current frame number (frame number of the current frame) is equal to or smaller than a predetermined threshold. Any information associated with the time between the detection result and the current frame may be used for this determination. For example, the time itself may be recorded and compared instead of the frame number. The threshold is preferably set to be small when the movement amount of the subject is large and large when the movement amount is small. The movement amount is determined based on at least one of the change amount of the face position, face angle, and face size of the same subject in the previous frame and the frame before the previous frame. In S206, if the camera control unit 131 determines that the difference between the number of frames for which the detection results were obtained and the number of current frames is equal to or less than the threshold, it determines that positional deviation with the current frame is unlikely to occur even if the obtained detection results are used, and proceeds to S207. In S207, the calculation unit 144 calculates the relative position of the pupil with respect to the face from the pupil position, face position information, and face size information obtained from the face information detection unit 141 and the organ information detection unit 142. In S208, the calculation unit 144 applies the calculated relative position to the face position and face size information obtained by tracking the current frame, and calculates an estimated position of the pupil in the current frame.

When the estimated pupil position in the current frame is calculated in S208, the process proceeds to S209, where the camera control unit 131 displays a pupil frame at the estimated pupil position on the display unit 171 set by the display frame setting unit 151. If the face size is larger than a predetermined threshold in the first place, the process proceeds from S204 to S211 because using the pupil detection result of tracking can ensure higher accuracy and real-time performance than estimating the position from the detection result by the organ detection process in the previous frame or earlier. In S211, the camera control unit 131 obtains the pupil detection result by the tracking process, and then in S212 determines whether the pupil was detected with a certain level of reliability or higher. If it is determined in S212 that the pupil was detected with a certain level of reliability or higher, the process proceeds to S209, where the camera control unit 131 displays the pupil frame set by the display frame setting unit 151 using the pupil detection result of tracking on the display unit 171. If the camera control unit 131 determines in S203 that a face cannot be detected by the tracking process, that the difference between the number of frames acquired in the organ detection results and the current number of frames in S206 is equal to or less than a predetermined threshold, or that an eye cannot be detected by the tracking process in S212, the display of the eye frame is discontinued (S213).

The detection result of the organ detection by the face information detection unit 141 (organ information detection unit 142) acquired in S205 is updated for each frame according to the processing flow shown in FIG. 6A. Specifically, after the organ detection process is performed for each frame, the detection result information update process of S101 is started. Next, proceed to S102, where the camera control unit 131 acquires the detection result information of the processed face information detection unit 141 and the organ information detection unit 142. After acquiring the detection result in S102, proceed to S103 to determine whether the acquired face and eye have both been detected with a reliability equal to or higher than a certain threshold. If the face and eye have been detected with a reliability equal to or higher than a certain threshold, proceed to S104, where the number of frames and the detection position information of the face and eye stored in the memory unit 153 are updated to the acquired detection result. In S103, if the reliability of the face and eye is less than the threshold, the update process is terminated without performing the update process. In addition, the update frequency of the organ detection process does not necessarily have to be every frame, and may be every predetermined number of frames or every predetermined period. Additionally, the organ detection process may be interrupted by some user instruction, scene determination, etc. In either case, when the organ detection process is performed, the latest detection results are stored in the storage unit 153 and updated.

As described above, in this embodiment, the real-time frame display is improved compared to when pupil AF is performed using only organ detection processing, while the eyes can be estimated based on detection results with higher accuracy compared to when only tracking processing using pattern matching processing or the like is performed.

Next, a second embodiment will be shown in which the calculation unit 144 performs the pupil position estimation process based on the detection results of the face information detection unit 141 (organ information detection unit 142), and the pupil frame can be estimated at an accurate position even when the subject is rotating in the Pitch, Yaw, or Ro11 directions.

7A, in the method of the first embodiment, if the subject moves so that the face angle does not change with respect to the camera lens 100, or if the subject slowly rotates in the Pitch, Yaw, or Ro11 directions, the pupil position can be estimated with high accuracy. However, if the subject moves suddenly in any or all of the three rotation directions, the deviation between the actual pupil position and the frame position becomes large because the relative position of the pupil to the face frame in the previous frame is used for pupil position estimation. For example, if the subject's face rotates in the Pitch direction from 711 to 721 to 731, if the relative position 722 of the pupil in the previous frame is applied to the current frame, it is assumed that the pupil is located lower than the actual pupil, and the frame 732 is shifted downward from the actual pupil position. Therefore, in this modified example, the display frame setting unit 151 corrects the position of the frame 732 to the frame 733 in FIG. 7B based on the face angle information of the frame before the previous frame and the previous frame obtained from the face information detection unit 141. In addition, the rotation in the Yaw (741 to 763) direction is corrected in the same way as the Pitch. As a simple method for correcting Yaw and Pitch rotations, a method of using the movement distance between the frame before the previous frame and the frame before the previous frame as the correction amount can be considered. Since the face is approximately spherical, for example, the pupil movement amount decreases as it moves from the front facing direction to the back in the Yaw direction. Therefore, as a method for more accurate correction, the movement amount for this face angle may be stored as a lookup chart and referred to. When the face is rotated in the Roll (771 to 793) direction, the method of using the movement distance between the frame before the previous frame and the frame before the previous frame used in Yaw and Pitch rotation as the correction amount is not suitable. This is because the pupil makes a circular movement on the image, and a correction method based on an approximately linear movement may worsen the position deviation. Therefore, as shown in FIG. 7B, when rotating in the Roll direction, a correction is made to move the display frame from 792 to 793 according to the face angle on a circle 796 whose center is the center 795 of the face frame and whose radius is the distance from the center to the pupil.

Figure 8 (a) shows a camera 811 and a subject 812 seen from above. Figure 8 (b) shows the change in pupil size on the screen when the face rotates in the direction of the arrow, with the face angle at which the pupil 813 of the subject 812 faces the camera 811 being 0 degrees. From 0 degrees to 45 degrees, the pupil 813 approaches the camera and becomes larger relative to the face, and when it faces further back, it becomes smaller. If the frame size is set taking into account the change in pupil size due to the rotation of the face, a frame that is more realistic can be attached. Also, since visibility decreases if the frame becomes too small, the frame size may be set so that it does not fall below a certain threshold, or the display method may be changed, such as by indicating the pupil position with an arrow. Frame size correction is also performed in the same way for Yaw. Since the pupil size does not change with Roll, this size correction is not performed.

As a result, when the subject is rotating in the Pitch, Yaw, or Ro11 directions, the processing load increases, but the pupil frame can be displayed in a more accurate position than in Example 1.

Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-mentioned embodiments may be appropriately combined. In addition, the present invention also includes a case where a software program that realizes the functions of the above-mentioned embodiments is supplied to a system or device having a computer that can execute the program directly from a recording medium or using wired/wireless communication, and the program is executed. Therefore, the program code itself that is supplied and installed to the computer in order to realize the functional processing of the present invention by the computer also realizes the present invention. In other words, the computer program itself for realizing the functional processing of the present invention is also included in the present invention. In that case, as long as it has the function of the program, the form of the program does not matter, such as object code, a program executed by an interpreter, script data supplied to an OS, etc. Examples of recording media for supplying the program include magnetic recording media such as hard disks and magnetic tapes, optical/magneto-optical storage media, and non-volatile semiconductor memories. In addition, as a method of supplying the program, a method in which the computer program forming the present invention is stored in a server on a computer network, and a connected client computer downloads the computer program and programs it is also considered.

122 Image sensor 124 Camera signal processing circuit 131 Camera control unit 141 Face information detection unit 142 Organ information detection unit 143 Tracking unit 144 Calculation unit 150 Automatic pupil selection unit 151 Display frame setting unit 152 Pupil designation/designation cancellation unit 153 Designated pupil storage unit 171 Display unit 172 Touch panel

Claims (8)

a display control means for displaying a captured image and an icon on a display means;
A detection means for detecting a face and eyes from a captured image;
a tracking means for detecting and tracking a face area set in the first captured image from the second captured image using a detection method that is simpler than the detection by the detection means;
a determining means for determining a relative position of the eyes with respect to the face based on the position of the eyes, the position of the face and the size of the face detected by the detecting means,
the determining means determines an estimated position of the eyes by applying a relative positional relationship of the eyes to the face to a face position and a face size of the face area detected by the tracking means in the second captured image;
The image processing device according to claim 1, wherein the display control means displays an icon indicating the determined estimated position of the eye in a superimposed manner on the second captured image displayed on the display means. The image processing device according to claim 1 , wherein the icon is displayed based on the estimated position when it is determined that the face size of the subject obtained from the tracking means is equal to or smaller than a predetermined threshold value. The image processing device according to claim 1 or 2, characterized in that the face and eye information detected by the detection means used by the determination means is information from a predetermined number of frames or a predetermined time before the current frame. An image processing device according to any one of claims 1 to 3, characterized in that the face and eye information detected by the detection means used by the determination means is information within a predetermined time from the current frame, and the time is set to be small when the movement amount of the subject is large and set to be large when the movement amount is small. The image processing device according to any one of claims 1 to 4, characterized in that the method of determining the relative position of the eyes is to determine the horizontal and vertical movement distances from the center point of the face to the center point of the eyes as amounts relative to the face size. The image processing device according to any one of claims 1 to 5, characterized in that the determining means corrects the relative position of the eye using the amount of rotation of the subject or the amount of movement of the organ between the most recent frames. The image processing device according to any one of claims 1 to 6, further comprising an imaging means. a display control step of displaying a captured image and an icon on a display means;
a detection step of detecting a face and eyes from a captured image by a detection means;
a tracking step of detecting and tracking a face area set in the first captured image from the second captured image using a detection method that is simpler than the detection by the detection means;
a determining step of determining a relative position of the eyes with respect to the face based on the position of the eyes, the position of the face and the size of the face detected by the detecting means,
In the determining step, an estimated position of the eyes is determined by applying a relative positional relationship of the eyes to the face to a face position and a face size of the face region detected in the tracking step in the second captured image,
The image processing method according to the present invention, wherein the display control step displays an icon indicating the determined estimated position of the eye superimposed on the second captured image displayed on a display means.

Images