JP5888986B2 - Image processing apparatus and control method thereof - Google Patents

Description

  The present invention relates to an image processing apparatus having a subject tracking function and a control method thereof.

  In a conventional digital camera, an arbitrary subject is selected as a tracking target from any one of the frame images in the moving image, and the luminance signal and color signal of the tracking target subject are selected from the frame images after that. There is one that performs tracking by detecting a region having a high correlation (see Patent Document 1).

  A digital camera equipped with a face detection function has been known. In such a digital camera, an autofocus process, an automatic exposure process, an auto white balance process, and the like are performed based on the automatically recognized face position to provide a comfortable shooting environment for the user. By using this face detection function for tracking, even if a person's face moves, the person's face can be tracked. For example, when multiple faces are detected from an image, the faces of the same person are identified between the images based on the position and size of each face detected in the previous image or the accumulated moving direction. Tracking is performed (see Patent Document 2).

  Further, there is a technique for determining whether or not a person's face is included in the range of the AF frame by using both tracking and face detection techniques, and changing the tracking mode based on the determination result. Here, when the face of a person is included in the range of the AF frame, the start of AF frame automatic tracking in the face detection tracking mode is instructed. When the face of the person is not included, the object tracking mode is used. Instructs the start of AF frame automatic tracking. (See Patent Document 3).

  Furthermore, some digital cameras that employ a touch panel and have a face detection function can take a person's face designated by the user using the touch panel as a subject, and perform shooting based on that face (Patent Document 4). See).

  Using these technologies, when a user specifies an object other than a face on the touch panel, object tracking is performed to detect a tracking target and an area highly correlated with a luminance signal or color signal, and a face is specified. It is conceivable to perform face tracking using a face detection function.

JP 2010-124120 A JP 2008-271310 A JP 2011-27846 A JP 2009-10777 A

  When the tracking target position is directly set by the touch panel, a subject intended by the user can be tracked as a tracking area. However, when the tracking frame position is moved to specify the tracking target using the up / down / left / right buttons, etc., if a face is detected in the vicinity of the moving tracking frame position, the face detection / tracking mode is automatically switched. In this case, even if the subject intended by the user is not a face, the mode automatically switches to the face detection tracking mode, and therefore it may not be possible to set the tracking target position that the user wants to specify.

  An object of the present invention is to provide an image processing apparatus capable of tracking a subject as intended by a user even when a tracking frame is set with up / down / left / right buttons or the like.

In order to achieve the above object, the present invention provides an image processing apparatus that performs tracking by continuously detecting a tracking region that is a region of a subject to be tracked from a moving image composed of a plurality of images. A display means for displaying a display indicating the position of the set tracking area together with an image based on the signal, and an operation member capable of moving the display indicating the position of the tracking area in a plurality of directions. A first specifying means for a user to specify a position from an image displayed on the means, a face detecting means for detecting a face area from an image signal, and a setting means for setting a tracking area as an object to be tracked. Setting means capable of setting a tracking area based on the face area, and the setting means until a first predetermined time elapses after an operation through the first designation means is performed. Before And limits the configuration of the tracking area based on the face area.

  According to the present invention, it is possible to track a subject as intended by a user even when a tracking frame is set with up / down / left / right buttons.

It is a block diagram which shows the structure of the control system which concerns on embodiment of this invention. It is a flowchart figure which shows the procedure of the imaging | photography process which concerns on embodiment of this invention. It is a flowchart figure which shows the procedure of the tracking process which concerns on embodiment of this invention. It is a flowchart figure which shows the procedure of the face tracking determination process which concerns on embodiment of this invention. It is screen display explanatory drawing which shows the transition of a display screen when performing the face tracking process by the touch input from a user. FIG. 10 is a screen display explanatory diagram showing a transition of a display screen when shifting from an object tracking process to a face tracking process when a profile is designated by a user input in the digital camera according to the embodiment of the present invention. It is screen display explanatory drawing for demonstrating the subject at the time of the tracking frame designation | designated by a user, and the tracking process which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In this embodiment, a digital camera (image processing apparatus) having a function for tracking a subject designated by a user in a moving image and a face detection function for detecting a human face area from an image will be described as an example. .

  The digital camera shown in FIG. 1 includes a photographing lens 10, a shutter 12 having a diaphragm function, and an image sensor 14 that converts an optical image into an electrical signal (image signal). An analog signal output at the time of imaging from the image sensor 14 is converted into a digital signal (image signal) by the A / D converter 16.

  The memory control circuit 22 and the system control circuit 50 constituting the control unit control the timing generation circuit 18. The timing generation circuit 18 supplies a clock signal and a control signal to the image sensor 14, the A / D converter 16 and the D / A converter 26 for control.

  The memory control circuit 22 controls the A / D converter 16, the image processing circuit 20, the image display memory 24, the D / A converter 26, the memory 30, and the compression / decompression circuit 32. The data (image signal) output from the A / D converter 16 or the data (image signal) output from the memory control circuit 22 is subjected to predetermined pixel interpolation processing and color conversion processing by the image processing circuit 20, and the image display memory. 24 or the memory 30. The display image data written in the image display memory 24 is displayed on a display unit 28 (display means) comprising a TFT LCD or the like via a D / A converter 26. The display unit 28 functions as an electronic finder by displaying image data generated from signals continuously output from the image sensor 14 in real time as a moving image. The memory 30 is a memory for storing captured still images and moving images, and has a sufficient storage capacity for storing a predetermined number of still images and moving images for a predetermined time. The memory 30 is configured to be usable as a work area for the system control circuit 50.

  The image stored in the memory 30 is read into the compression / decompression circuit 32, and the image data is compressed or expanded by adaptive discrete cosine transform (ADCT) or the like. The data after the compression / decompression process is written into the memory 30.

  In addition, the digital camera of this embodiment includes a shutter switch SW1 62, a shutter switch SW2 64, and an operation unit 70 as operation means for inputting various operation instructions. The operation unit 70 includes, for example, a single or a combination of a switch, up / down / left / right key buttons, dial, touch panel, pointing by line-of-sight detection, a voice recognition device, and the like. Here, the touch panel included in the operation unit 70 is configured integrally with the display unit 28, and when the user touches the screen of the display unit 28, information indicating coordinates on the screen at the touched position is displayed in the system. It transmits to the control circuit 50.

  The timer 130 is used for generating the shooting time and measuring the elapsed time, and is controlled by the system control circuit 50. The tracking designation determination unit 131 monitors the operation unit 70 and determines whether or not a tracking frame is set directly on the touch panel.

  In addition, this digital camera performs AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-emission) processing. For this reason, the image processing circuit 20 of the digital camera performs predetermined calculation processing on the captured image data, and the system control circuit 50 performs the exposure control circuit 40 and the distance measurement control circuit 42 based on the obtained calculation result. To control. Further, the image processing circuit 20 performs predetermined calculation processing using the captured image data (image signal), and performs AWB (auto white balance) processing based on the obtained calculation result. The exposure control circuit 40 controls the shutter 12 having a diaphragm function, and the distance measurement control circuit 42 controls the focusing of the photographing lens 10. Further, the system control circuit 50 controls a zoom control circuit 44 that controls zooming of the taking lens 10.

  The system control circuit 50 analyzes the image data processed by the image processing circuit 20 and the image data stored in the image display memory 24 to detect a region that seems to be a face in the image data. (Face detection means) is connected. For example, the face detection unit 101 detects, as a face region, a region in which human eyes, nose, and mouth-like shapes are arranged in a predetermined positional relationship in the image data. When the face detection unit 101 detects a region that seems to be a face, the face detection unit 101 outputs a probability that it is considered to be a human face, a position and size in the input image data, and the like. Further, the face detection unit is configured to output the feature amount of each feature point of the detected face.

  The system control circuit 50 performs an image difference calculation on the two images recorded in the image display memory 24 and the memory 30 based on at least one of luminance information and color information, and selects a region having a high correlation between the images. An image comparison unit 102 to be detected is connected.

  The system control circuit 50 is connected to a nonvolatile memory 56 composed of an electrically erasable / recordable EEPROM or the like. The system control circuit 50 is connected to a card controller 90 that transmits / receives data to / from an external recording medium such as a memory card, and an external recording medium 120 such as a memory card is connected via the card controller 90.

  The system control circuit 50 executes each process described below according to a program stored in a ROM (not shown). The shutter switch SW1 62 connected to the system control circuit 50 outputs an ON instruction signal to the system control circuit 50 in a state in which a shutter button (not shown) is being operated. Upon receiving this ON signal, the system control circuit 50 starts operations such as AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and EF (flash pre-emission) processing.

  Further, the shutter switch SW2 64 outputs an ON instruction signal to the system control circuit 50 in a state where the operation of a shutter button (not shown) is completed. Upon receiving this ON instruction signal, the system control circuit 50 performs an exposure process operation of the main photographing in which the signal read from the image sensor 14 is written into the memory 30 via the A / D converter 16 and the memory control circuit 22. Instruct the start. Next, the system control circuit 50 instructs the start of development processing using the calculation results in the image processing circuit 20 and the memory control circuit 22. Furthermore, the system control circuit 50 reads out the image data from the memory 30, compresses it with the compression / decompression circuit 32, and instructs the start of the recording process for writing the image data into the external recording medium 120.

  Next, photographing processing in the digital camera configured as described above will be described with reference to the flowchart of FIG.

  In this digital camera photographing process, when the digital camera is turned on and ready for photographing, the memory control circuit 22 and the system control circuit 50 constituting the control unit start the photographing process. The system control circuit 50 starts exposure to the image sensor 14 and causes the display unit 28 to display a moving image in real time using continuously generated image data.

  In this shooting process, in step S201, the current shooting mode corresponds to a user input from the touch panel included in the operation unit 70 (hereinafter referred to as touch input) or a user input by operating the up / down / left / right key button (operation member) as a designation unit. To determine whether or not the tracking mode allows specification of a tracking frame. In the present embodiment, when the display unit 28 displays a moving image, the user touches the screen of the display unit 28 (a touch panel included in the operation unit 70) or a setting key (not shown) included in the operation unit 70. ) To move to the tracking process. The operation for shifting to the tracking process is not limited to these, and the process may be shifted to the tracking process by a predetermined key operation other than the setting key.

  Note that the user can set in advance whether or not the tracking process can be performed using a menu button included in the operation unit 70. In the present embodiment, when the “face + tracking priority AF mode” is set in advance with the menu button, the tracking process can be executed. If the system control circuit 50 determines that the “face + tracking priority AF mode” in which the tracking process can be executed (YES in step S201), the system control circuit 50 proceeds to step S202 and executes a tracking process sub-flow described later.

  In the tracking process (step S202), when a touch input and a setting key operation are received from the user, a process of converting the touch-input position or a predetermined position (here, the center of the screen) into coordinates within the screen as a tracking frame position. Run. At this time, the digital camera may be configured to display that the touch input has been received by displaying a frame (indicator) superimposed on the screen being displayed at a position corresponding to the user's touch input. good.

  Further, in this tracking process (step S202), subject determination at the tracking frame position is performed based on touch input or setting key operation, and tracking processing according to the subject is started according to the determination, and the process proceeds to the next step S203. move on. The tracking process will be described in detail later using the flowchart of FIG.

  After the tracking process in step S202, the system control circuit 50 proceeds to step S203, and returns to step S201 if the shutter switch SW162 is not pressed.

  On the other hand, if the system control circuit 50 determines that it is not the “face + tracking priority AF mode” in which the tracking process can be executed (NO in step S201), the system control circuit 50 selects the main subject in the set mode, and the shutter switch Wait until SW1 62 is pressed (step S203). As modes other than the “face + tracking priority AF mode”, a “one-point AF mode” based on a designated AF area, or a subject located closest to a subject in a plurality of preset AF areas is set as a main subject. There is a “multi-point automatic AF mode” to be selected.

  Next, when it is determined that the shutter switch SW1 62 has been pressed (YES in step S203), the system control circuit 50 performs AF / AE processing (step S204) in which a weight is set on the set main subject.

  Then, the system control circuit 50 continues the AF / AE process (step S204) and waits until the shutter switch SW2 64 is pressed (NO in step S205). In this standby state, when the pressing of the shutter switch SW1 62 is released, the process returns to step S201.

  If the system control circuit 50 determines that the shutter switch SW264 has been pressed, the system control circuit 50 proceeds to step S206, performs shooting, records the shot image file in the external recording medium 120, and ends the shooting process.

  Next, with reference to FIGS. 5 and 7, a problem when the tracking frame is designated by the user will be described.

  In a conventional digital camera that automatically sets the face position to the tracking frame position when the position of the face and the tracking frame position match (close to each other), in the case of touch input, the button assigned to the tracking process transition to the touched position When is pressed, a tracking frame is displayed at the center of the screen.

  For example, when a touch input is made as shown in FIG. 5A, the touched position is set as a tracking area. In this case, although the details will be described later, since the detected face position is close to the tracking area, the face position is set as the tracking area, and the tracking face area is framed (indexed) as shown in FIG. ). Further, when the face area of the person surrounded by the frame (index) moves within the screen, the display of the frame is updated in accordance with the movement of the person's face as shown in FIGS. When a touch input is made at the position of the flower at the lower right of the screen, the position of the flower is set as the tracking area if the position of the flower is sufficiently away from the position of the person's face. Even if the tracking area is set once, when the touch input is performed again by the user, the tracking area is reset based on the newly input position.

  Next, a case where the process is shifted to the tracking process by an operation of a setting key or the like is described. In the state of the display example in FIG. 7M, when a button assigned to the tracking process transition such as a setting key is pressed, a tracking frame is displayed at the center of the screen as shown in FIG. In FIG. 7N, the tracking frame position is far from the face position. When the right is selected with the up / down / left / right key buttons in this state, the tracking frame moves to the right as shown in FIG. Further, by operating the up / down / left / right key buttons to move the tracking frame to the lower right, the tracking frame can be moved to the position of the flower as shown in FIG.

  On the other hand, in the state of the display example in FIG. 7A, when the button assigned to the tracking process transition is pressed, a tracking frame is displayed at the center as shown in FIG. 7B. In this state, since the face exists at the tracking frame position, the tracking frame position is automatically set to the face position. Even if an attempt is made to move the tracking frame to the position of the flower with the up / down / left / right key buttons in this state, the position of the face is included in the tracking frame in the state of FIG. 7 (C). Thus, the face position is automatically changed. Therefore, when the user wants to set the tracking frame at the flower position, the tracking frame cannot be moved to the flower position.

  In addition, in the case of a camera equipped with a function (= face selection) for directly moving the tracking frame between a plurality of face positions, a plurality of faces are detected when the button assigned to the tracking mode transition is pressed. Then, the face select function is assigned to the up / down / left / right key buttons. In the state of the face selection function, the operation of the up / down / left / right key buttons is limited to the operation of selecting a face from a plurality of faces. For example, when left is selected with the up / down / left / right key buttons in the state of FIG. 7 (E), the tracking frame moves to the left face as shown in FIG. 7 (F). Here, in FIG. 7E, a triangular mark is displayed on the left side of the tracking frame, indicating that the tracking frame can be moved to the left face. Furthermore, when the right is selected with the up / down / left / right key button in the state of FIG. 7 (F), the tracking frame moves to the right face as shown in FIG. 7 (G), and when the left is selected with the up / down / left / right key button from that state. As shown in FIG. 7H, the tracking frame moves to the left face. In this case, even if the user tries to set a tracking frame on the flower on the right side of the person by operating the up / down / left / right key buttons, the tracking frame cannot be set to other than the face when the face selection function is executed.

  In order to solve these problems, a tracking process according to the embodiment of the present invention that is executed during the photographing process will be described with reference to the flowchart of FIG. There are two types of tracking processing performed by this digital camera: object tracking processing for tracking an object other than a face as a main subject, and face tracking processing for tracking the face of a specific person displayed on the screen as a main subject. It is divided roughly into.

  When the tracking process is started, the system control circuit 50 confirms whether or not there has been a tracking process transition operation from the user in step S300. Specifically, it is confirmed whether touch input or setting key operation has been performed. If the system control circuit 50 determines that a new tracking process transition operation has been performed, the system control circuit 50 proceeds to step S302. On the other hand, if it is determined that a new tracking process transition operation has not been performed, the process proceeds to step S301 to check whether the tracking process is in progress. When the setting key is operated, the timer starts counting.

  In step S301, when the system control circuit 50 determines that tracking is not in progress, the flowchart of FIG. 3 ends. If the system control circuit 50 determines that tracking is in progress, the system control circuit 50 proceeds to step S312, and determines whether or not the face tracking process has already been started.

  If it is determined in step S312 that the face tracking process is not in progress, the system control circuit 50 proceeds to step S309, which will be described later, and executes the face tracking determination process. On the other hand, if the system control circuit 50 determines in step S312 that the face tracking process has already been started, the system control circuit 50 proceeds to step S332 and determines whether or not it is the next candidate for face selection. Here, when the up / down / left / right key buttons are operated during face tracking in a state where a plurality of faces are detected, it is determined that the next candidate for face selection is selected. If it is determined that the next candidate for face selection is selected, the process proceeds to step S333, and the next candidate face position information is acquired. In step S303, the tracking frame region is changed using the face position of the next candidate. On the other hand, if it is determined in step S332 that the face selection is not the next candidate selection, the process proceeds to step S311 to be described later and a face tracking process is executed.

  In step S300, when the system control circuit 50 determines that there has been a tracking process transition operation from the user, the system control circuit 50 proceeds to step S302 and executes tracking coordinate acquisition processing. Here, the tracking process transition operation from the user can be restated as the input of the tracking frame designation from the user. That is, when the tracking process is performed by touch input, a tracking frame is designated at the touched position, and when the tracking process is performed by the operation of the setting key, the tracking frame is set at a predetermined position (the center of the screen in this embodiment). Is specified. In the tracking coordinate acquisition process in step S302, the tracking frame control unit provided in the operation unit 70 converts the touched position or the screen center position into coordinates on the screen, acquires the coordinates, and obtains the system control circuit. 50.

  Next, the system control circuit 50 proceeds to step S323, and determines whether or not the tracking frame is set by touch input. If it is determined in step S323 that the tracking frame is set by touch input, the process proceeds to step S303.

  On the other hand, if it is not determined in step S323 that the tracking frame is set by touch input, the process proceeds to step S325, and the system control circuit 50 acquires face coordinate position information obtained from face detection. Next, in step S326, the tracking frame coordinates and the face coordinates are compared, and it is determined whether or not the face position is included in the tracking frame (predetermined region = the center of the screen here). If the face position is not included in the tracking frame, the process proceeds to step S303. If the face position is included in the tracking frame, the process proceeds to step S327.

  In step S327, the system control circuit 50 updates the timer time for measuring the elapse of the predetermined time, and proceeds to step S328. In step S328, the system control circuit 50 sets a tracking frame area for display at the position where the coordinates are acquired in step S302 (the first time is the center of the screen).

  Proceeding to step S329, the system control circuit 50 updates the tracking frame display to the area set at step S328 for the image display memory 24. In step S330, the system control circuit 50 determines whether the up / down / left / right key button has been operated. If the up / down / left / right key button is operated, the timer is reset in step S334, and the process returns to step S327. On the other hand, if the up / down / left / right key button has not been operated, the process proceeds to step S331, and monitoring of whether the up / down / left / right key button has been operated is continued until a predetermined time elapses. If the predetermined time has passed without the up / down / left / right key buttons being operated (Yes in step S331), the process proceeds to step S303.

  When the process proceeds again to step S330 via step S334, the predetermined time of the timer may be set longer than the first time. This is because when the process passes through step S334, the up / down / left / right key button operation is performed, and thus the user is likely to move the tracking frame. Therefore, by setting a predetermined time (first predetermined time) longer than the first predetermined time (second predetermined time), it is possible to more reliably accept the tracking frame movement by the user's up / down / left / right key button operation. .

  As described above, the setting of the tracking area based on the face detection area is limited until the (first) predetermined time elapses after the up / down / left / right key button operation. That is, even if a face is detected in the tracking frame, the tracking frame is not automatically changed to the position of the face. Further, even if a face is detected in the tracking frame until the (second) predetermined time elapses after the tracking processing shift by the setting key operation, the tracking frame is not automatically changed to the position of the face. Therefore, when an operation with the up / down / left / right key buttons is performed before the predetermined time elapses, the tracking frame area is set according to the operation, and the tracking frame display is updated. For example, in a state where an image as shown in FIG. 7 (I) is displayed, when a tracking process is performed by a setting key operation, a tracking frame is displayed at the center as shown in FIG. 7 (J). In the state of FIG. 7 (J), when an operation is performed to move to the right by the up / down / left / right key button during the (second) predetermined time after the tracking process transition by the setting key operation, the position of the face is set. The tracking frame can be moved in the right direction without being automatically changed. Furthermore, by continuing to operate the up / down / left / right key button again within the (first) predetermined time from the operation of the up / down / left / right key button, the position of the flower as shown in FIG. 7 (L) through FIG. It becomes possible to move the tracking frame.

  If the system control circuit 50 determines in step S323 that the tracking frame is set by touch input, the system control circuit 50 proceeds to the process of setting the tracking area in step S303.

  When it is determined that the tracking process shifts by touch input, the subject position can be directly specified on the display screen, so that it is not necessary to consider that the tracking frame position is specified by operating the up / down / left / right key buttons. Therefore, in the case of the tracking process transition by touch input, the reactivity of the display screen by touch input can be improved by not waiting for the elapse of the predetermined time as described above. In the case of touch input, a predetermined time may be awaited as in the case of setting key operation.

  In step S303, the tracking area of the subject to be tracked is set based on the obtained tracking frame coordinates. Here, the system control circuit 50 functions as setting means for setting the tracking area. In the process of setting the tracking area, the position on the display image data is obtained from the display image data stored in the image display memory 24 and the touched coordinates or the position set in the tracking frame in step S328. Further, in the process of setting the tracking area, the size of the tracking area of the subject is obtained in consideration of the contrast obtained from the luminance information around the position on the display image data, the image feature amount, and the like. In other words, a temporary tracking area of a plurality of sizes is set around the touched coordinates or the position set as the tracking frame, and the tracking and the minimum size of the temporary tracking area that satisfies the predetermined conditions for the contrast and the feature amount are tracked. Set as area.

  Here, the reason why the contrast and the feature amount are compared with predetermined conditions is that if these values do not reach a certain level, it is difficult to distinguish them from other regions. In addition, the minimum size is selected as the size is set larger, the background other than the subject to be tracked is included in the set tracking area, and the tracking accuracy may decrease. This is because it becomes higher.

  In step S304, the system control circuit 50 determines whether or not the subject tracking area has been correctly obtained in step S303. If it is determined that the subject tracking area has been correctly obtained (YES in step S304), the system control circuit 50 proceeds to step S308. move on.

  On the other hand, when the brightness around the tracking frame coordinates is uniform and the contrast cannot be detected, or when the screen has a similar pattern, the system control circuit 50 may not be able to obtain the tracking area correctly. In such a case (No in step S304), the process proceeds to step S322, and as a process incapable of tracking in step S322, a process such as displaying that the subject cannot be tracked to the user is performed, and the flowchart of FIG. .

  In step S308, the system control circuit 50 stores the color information of the tracking area determined as described above. Proceeding to step S309, the system control circuit 50 determines whether or not the subject is capable of face tracking. The face tracking determination process will be described later with reference to FIG.

  Next, when it is determined in step S310 that the face tracking process is possible in the face tracking determination process in step S309, the system control circuit 50 proceeds to step S311 and performs the face tracking process.

  In this face tracking process, the system control circuit 50 uses the display image data in the image display memory 24 and the face detection unit 101 to obtain a face area existing in the screen being displayed.

  The system control circuit 50 performs the face tracking determination process in step S309 when the tracking area is set by touch input or when the tracking area is updated by the object tracking process and when the system shifts to the face tracking process for the first time. A face area determined to be trackable is set as a new tracking area.

  If the face tracking process is already in progress, the system control circuit 50 sets a face area to be the next tracking area. In this case, for example, the distance between the position of the previously detected face area being tracked and a plurality of face areas detected on the current display image data, the amount of movement and the direction of movement of each face area, etc. Therefore, a face area that is considered to be the same as the face area set as the previous tracking area may be set. Alternatively, the system control circuit 50 uses a control module such as personal authentication (not shown) to detect the relationship between the face area detected last time and the feature points and similarities between the current face area and the face area. . Then, the system control circuit 50 may set a face area to be the next tracking area by specifying a face area that seems to be the same as the previous face area from the relevance of the detected face area.

  Next, proceeding to step S320, the system control circuit 50 determines whether the user has issued an instruction to end tracking by operating a GUI button or the like on the touch panel. When the system control circuit 50 determines that there is an instruction to end tracking, the system control circuit 50 proceeds to step S321 and performs tracking end processing. In the tracking end process, the system control circuit 50 notifies the user of the end of tracking and then ends the display of the frame (index) indicating the tracking area.

  In step S320, if the system control circuit 50 determines that there is no instruction to end touch tracking, it ends the flowchart of FIG. 3 and proceeds to step S203 of FIG. If the shutter switch SW1 62 is not pressed in step S203, the system control circuit 50 executes the flowchart shown in FIG. 3 again for the new display image data via step S201. In this way, the tracking is continued by repeating these processes until the shutter switch SW1 62 is pressed.

  Next, when it is determined in step S310 described above with reference to FIG. 3 that face tracking processing is impossible, the process proceeds to step S313, and the system control circuit 50 determines whether or not object tracking based on luminance information is possible. . The determination of whether or not object tracking is possible based on this luminance information is, for example, whether the surrounding area including the tracking area has not changed significantly compared to when the tracking area is updated, or the luminance around the tracking area is uniform. It is based on the condition whether it is not. If it is determined that the accuracy of object tracking due to the correlation of luminance information is reduced due to these conditions, the object tracking is impossible.

  If the system control circuit 50 determines that object tracking based on the luminance information is possible based on the above condition, the system control circuit 50 proceeds to step S314 and performs object tracking processing using the luminance information. In the object tracking processing based on the luminance information, the image comparison unit 102 detects a correlation between the display image data in the current image display memory 24 and the image data in the tracking area in the display image data in which the previous tracking area is set. To do. That is, in this object tracking process, the image data that has been set as the tracking area in the display image data in which the previous tracking area is set is moved to the current display image data by correlation calculation using each luminance information. calculate. Then, among the current display image data, the area that is determined to have the highest correlation with the image data that was the tracking area in the display image data that was previously set as the tracking area is updated as a new tracking area in the current frame image To track the object.

  Next, the process proceeds to step S315, and the system control circuit 50 stores the color information of the area newly set as the tracking area, and proceeds to step S320. In step S320, the system control circuit 50 determines the presence / absence of a tracking end instruction from the user as described above. When it is determined that there is no tracking end instruction, the system control circuit 50 ends the flowchart of FIG. The process proceeds to step S203 of step 2. When the system control circuit 50 determines that there is an instruction to end touch tracking, the system control circuit 50 proceeds to step S321 and performs tracking end processing. In the tracking end process, the system control circuit 50 notifies the user of the tracking end, and then ends the display of the frame (index) indicating the tracking area.

  Next, when the system control circuit 50 determines in step S313 described above whether or not the object tracking based on the luminance information is possible, and if it is determined that the object tracking is not possible, the process proceeds to step S316. In step S316, the system control circuit 50 next attempts to use the color information for the tracking area in which it is determined that the object tracking with the luminance information is impossible. Therefore, the system control circuit 50 detects whether color information is stored in step S308 or step S315 described above. When color information is stored, the system control circuit 50 uses the color information to search the display image data for an area having a color distribution similar to that of the current display image data.

  Next, proceeding to step S317, the system control circuit 50 determines whether or not a new tracking area has been successfully searched by searching for a tracking area based on color information. If it is determined that the operation has succeeded, the tracking is enabled again, and the process proceeds to step S318. Specifically, the system control circuit 50 generates a hue histogram from the stored color information, and detects a region where the difference from the histogram is the smallest. If the difference is less than or equal to a predetermined threshold, it is determined that the tracking area search by the color information has succeeded, and if it cannot be detected, it is determined that the search has failed. In step S318, the system control circuit 50 sets a region that has been successfully searched as a new tracking region. In step S315, the system control circuit 50 stores the color information of the area newly set as the tracking area.

  On the other hand, when the system control circuit 50 determines in step S317 described above that the search for the new tracking area has failed in the search for the tracking area based on the color information, the process proceeds to step S321, and the tracking process ends.

  Next, the face tracking determination process in step S309 described above will be described with reference to FIG. In the face tracking determination process, the system control circuit 50 determines whether to switch to the face tracking process based on the face detection result for the current tracking area. In this face tracking determination process, as described above, when it is determined that the face tracking process is possible, tracking based on the face detection result is performed. If it is determined that the face tracking process is impossible in this face tracking determination process, the object tracking process based on the luminance information and the color information is performed for the current tracking area without using the face detection result by the face detection unit 101. To do.

  When the face tracking determination process is started and the process proceeds to step S400, the system control circuit 50 acquires the coordinates of the current tracking area.

  Proceeding to step S401, the system control circuit 50 controls the face detection unit 101, and from the currently displayed display image data in the image display memory 24, all areas that are considered to be human faces on the display image data. Is detected, and the coordinates of each region are obtained.

  In step S402, the system control circuit 50 determines whether a face area has been detected from the display image data as a result of face detection. If the face area cannot be detected (NO in step S402), the system control circuit 50 proceeds to step S420 and counts up the number of face detections, and then proceeds to step S421 to determine that face tracking is impossible.

  On the other hand, if the face area can be detected (YES in step S402), the system control circuit 50 proceeds to the next step S403. In step S403, the system control circuit 50 calculates the distance between the tracking area and all the detected face areas. Proceeding to step S404, the system control circuit 50 determines the closest face area located closest to the tracking area from the distances calculated in step S403.

  Next, proceeding to step S405, the system control circuit 50 determines whether the tracking elapsed time from the tracking frame designation is less than a predetermined time. This elapsed tracking time is a time during which tracking is continued after the tracking frame is specified, and is reset when tracking fails midway. If the system control circuit 50 determines that the tracking elapsed time from the tracking frame designation is less than the predetermined time (YES in step S405), the process proceeds to step S406. On the other hand, when it is determined that the elapsed tracking time from the tracking frame designation is equal to or longer than the predetermined time (NO in step S405), the system control circuit 50 proceeds to step S408 and determines that face tracking is impossible.

  In step S406, the system control circuit 50 determines whether the distance between the closest face area and the tracking area determined in step S404 is less than a predetermined threshold (predetermined distance). If the system control circuit 50 determines that it is less than the threshold (YES in step S406), the system control circuit 50 proceeds to step S407, determines that face tracking is possible in the closest face area, and ends the face tracking determination processing.

  On the other hand, if the system control circuit 50 determines that the distance to the closest face area is equal to or greater than the threshold (NO in step S406), it determines that face tracking is impossible in step S408, and ends this face tracking determination process. This is because if the position specified by the user and the face area are separated from each other, the user is considered to have an intention to specify an object other than the face as a tracking target by specifying the tracking frame.

  For example, as shown in FIG. 6A, when a tracking frame region is specified by touch input, a side face, a face facing backward, or the like may be selected, and face detection may not be performed. Even in such a case, the face of the person faces the front as shown in FIG. 6B, and the face can be detected stably thereafter. In this case, face tracking based on the face detection result can be performed as shown in FIGS. 6C and 6D by control as described in Japanese Patent Application Laid-Open No. 2011-35765.

  In the above embodiment, when the user designates the subject to be tracked, the touch panel and the up / down / left / right key buttons included in the operation unit 70 are used as an example. However, the present invention is not limited to this. The subject may be designated by a dial included in the operation unit 70 or an icon on the display. Alternatively, if the tracking process described above is performed on an application of a personal computer that has captured a moving image, the subject to be tracked is specified with the cursor, or the subject to be tracked with the keyboard is present on the screen. An area may be designated.

  In the above embodiment, as an example of the face detection unit 101, an example in which a face area is detected from display image data has been described. However, a display image is separated from display image data from a signal generated by the image sensor 14. Image data for face detection with a resolution lower than that of the data may be generated.

  In addition, as an example of object tracking processing, a configuration in which either luminance information or color information is selected and a tracking area is detected from new image data using the correlation of the selected information has been described as an example. It is not limited. A new tracking area may be obtained from the sum of correlations of both luminance information and color information, or a new tracking area may be obtained using only one of luminance information and color information. That is, any configuration may be used as long as the tracking region is updated by continuously detecting a region having a high correlation between images.

  Furthermore, the above-described embodiment is not limited to shooting, and the main subject tracking process described above may be performed on a moving image that has already been shot on an application of a personal computer. By tracking the main subject, it is possible to perform luminance adjustment and color adjustment around the main subject on the moving image. The same applies to the case of playing back a moving image shot with a digital camera or digital video camera.

  Note that the present invention is not limited to these specific embodiments described above, and it is needless to say that the present invention can take various forms without departing from the gist of the present invention. The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media storing software, and the computer of the system or apparatus (or CPU, MPU, or the like). Is a process of reading and executing the program.

DESCRIPTION OF SYMBOLS 18 Display part 20 Image processing circuit 50 System control circuit 70 Operation part 101 Face detection part 102 Image comparison part

Claims (10)

In an image processing apparatus that performs tracking by continuously detecting a tracking region that is a region of a subject to be tracked from a moving image composed of a plurality of images,
Display means for displaying a display indicating the position of the set tracking area together with an image based on the image signal;
A first designation unit configured to include an operation member capable of moving a display indicating the position of the tracking region in a plurality of directions, and for a user to designate a position from an image displayed on the display unit;
Face detection means for detecting a face area from an image signal;
Setting means for setting a tracking area as a target to be tracked, and setting means capable of setting the tracking area based on the face area;
The setting unit restricts the setting of the tracking region based on the face region from when the operation through the first specifying unit is performed until a first predetermined time elapses. Processing equipment.   The setting means receives an operation via the first designation means from when the operation via the first designation means is performed until the first predetermined time elapses. The image processing apparatus according to claim 1, wherein the tracking area is set based on a designated position. The setting means, when a distance between the set tracking area and the face area is smaller than a predetermined distance after the first predetermined time has elapsed since the operation of the first specifying means, the image processing apparatus according to claim 1 or 2, characterized in that to set the tracking area based on information of the face area. An instruction means for the user to instruct the start of tracking;
The setting means restricts the setting of the tracking area based on the face area from when the operation through the instruction means is performed until a second predetermined time elapses when a predetermined condition is satisfied. the image processing apparatus according to any one of claims 1 to 3, wherein. The image processing apparatus according to claim 4 , wherein the predetermined condition is that the face area is included in the predetermined area. The image processing apparatus according to claim 5 , wherein the predetermined area is a tracking area that is set when an operation is performed via the instruction unit. The first predetermined time, the image processing apparatus according to any one of claims 4 to 6, wherein longer than the second predetermined time. A second designation means for the user to designate a position by a touch operation from the image displayed on the display means;
The said setting means sets a tracking area | region based on this designated position, when a position is designated via the said 2nd designation means, The any one of Claim 1 thru | or 7 characterized by the above-mentioned. The image processing apparatus described. The setting means sets the tracking area based on the information of the face area when the distance between the tracking area based on the position designated via the second designation means and the face area is smaller than a predetermined distance. The image processing apparatus according to claim 8 . A display means for displaying a display indicating the position of the set tracking area together with an image based on the image signal, and an operation member capable of moving the display indicating the position of the tracking area in a plurality of directions, A first designation means for the user to designate a position from an image displayed on the display means, and a tracking area that is a subject area to be tracked is continuously detected from a moving image composed of a plurality of images. In the control method of the image processing apparatus that performs tracking by
An acquisition step of acquiring information of the face area detected from the image signal;
A setting step for setting a tracking area as a target to be tracked, and a setting step capable of setting a tracking area based on the face area;
In the setting step, the setting of the tracking area based on the face area is limited until the first predetermined time elapses after the operation through the first specifying means is performed. A method for controlling a processing apparatus.

Images