JP2023086274A - Image processing device and control method for the same - Google Patents

Abstract

To provide an imaging apparatus including a subject tracking function that achieves excellent performance while suppressing power consumption and to provide a control method for the same.SOLUTION: An imaging apparatus has first tracking means for tracking a subject using an image acquired by imaging means, second tracking means for tracking the subject using the image acquired by the imaging means, which has a smaller computational load than the first tracking means, and control means for switching between enabling both the first tracking means and the second tracking means or disabling one of the two tracking means on the basis of feature points detected in the image acquired by the imaging means.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing apparatus and a control method thereof for subject tracking processing.

2. Description of the Related Art Some imaging devices such as digital cameras have a function (subject tracking function) of tracking a characteristic region such as a face region by applying detection of the characteristic region over time. Also known is a device that tracks a subject using a trained neural network (Patent Document 1).

JP 2017-156886 A

Improving the accuracy of subject tracking by using machine learning (neural networks, deep learning) in technologies for subject recognition and subject tracking using images, compared to when correlations and similarities between image regions are used. may be possible. However, processing using a neural network requires a large amount of calculations, requires a high-speed processor and a large-scale circuit, and has a problem of high power consumption. For example, when subject tracking using a neural network is applied to moving images for live view display, battery consumption due to live view display becomes a problem. Moreover, even during subject tracking using a circuit of a machine-learned model, there may be differences in calculation load and power consumption depending on the learning model.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an image processing apparatus and an image processing method having a subject tracking function that achieves good performance while suppressing power consumption.

In order to solve the above problems, the image processing apparatus of the present invention includes first tracking means for tracking a subject using an image acquired by an imaging means, and tracking a subject using the image acquired by the imaging means. a second tracking means having a smaller computational load than the first tracking means; and the first tracking means and the second tracking means based on feature points detected from the image acquired by the imaging means. and a control means for switching between enabling both of and disabling one of them.

According to the present invention, it is possible to realize a subject tracking function that achieves good performance while suppressing power consumption.

1 is a block diagram showing a functional configuration example of an imaging device according to a first embodiment; FIG. FIG. 4 is an operation flow diagram of the tracking control unit 113 in the imaging apparatus according to the first embodiment; FIG. 4 is a diagram showing live view display in subject tracking processing according to the first embodiment; FIG. 11 is an operation flow chart of the control unit 102 in the imaging apparatus according to the second embodiment; 7 is a table showing the relationship between the shooting scene and the operation modes of the detection unit 110 and the tracking unit 105 according to the second embodiment; Table showing operation modes of the detection unit 110 and the tracking unit 105 according to the second embodiment Operation flow chart of the control unit 102 of the third embodiment FIG. 11 is a flowchart of feature point detection processing performed by the feature point detection unit 201 of the third embodiment;

Preferred embodiments of the present invention will now be described with reference to the accompanying drawings.

(First embodiment)
The invention will now be described in detail on the basis of its exemplary embodiments with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. In addition, although a plurality of features are described in the embodiments, not all of them are essential to the invention, and the plurality of features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

In the following embodiments, the case where the present invention is implemented in an imaging device such as a digital camera will be described. However, the present invention can be implemented in any electronic device that has an imaging function. Such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smart phones, game consoles, robots, drones, and drive recorders. These are examples, and the present invention can also be implemented in other electronic devices.

FIG. 1 is a block diagram showing a functional configuration example of an imaging device 100 as an example of an image processing device according to the first embodiment.

The optical system 101 has a plurality of lenses including a movable lens such as a focus lens, and forms a scientific image of a photographing range on the imaging plane of the image sensor 103 .

The control unit 102 has a CPU, for example, loads a program stored in the ROM 123 into the RAM 122 and executes the program. The control unit 102 implements the functions of the imaging apparatus 100 by controlling the operation of each functional block. The ROM 123 is, for example, a rewritable non-volatile memory, and stores programs executable by the CPU of the control unit 102, setting values, GUI data, and the like. The RAM 122 is a system memory used to read programs executed by the CPU of the control unit 102 and to store values required during execution of the programs. Although omitted in FIG. 1, the control unit 102 is communicably connected to each functional block.

The imaging device 103 may be, for example, a CMOS image sensor having color filters in a primary color Bayer arrangement. A plurality of pixels having photoelectric conversion regions are two-dimensionally arranged in the image sensor 103 . The imaging element 103 converts an optical image formed by the optical system 101 into an electrical signal group (analog image signal) using a plurality of pixels. The analog image signal is converted into a digital image signal (image data) by an A/D converter of the image sensor 103 and output. The A/D converter may be provided outside the imaging device 103 .

The evaluation value generation unit 124 generates signals and evaluation values used for automatic focus detection (AF) from image data obtained from the image sensor 103, and calculates evaluation values used for automatic exposure control (AE). Evaluation value generator 124 outputs the generated signal and evaluation value to control unit 102 . The control unit 102 controls the focus lens position of the optical system 101 and determines shooting conditions (exposure time, aperture value, ISO sensitivity, etc.) based on signals and evaluation values obtained from the evaluation value generation unit 124. do. The evaluation value generation unit 124 may generate a signal or an evaluation value from display image data generated by the post-processing unit 114, which will be described later.

A first preprocessing unit 104 applies color interpolation processing to image data obtained from the image sensor 103 . Color interpolation processing, which is also called demosaicing processing, is processing for making each pixel data constituting image data have values of R, G, and B components. Also, the first preprocessing unit 104 may apply reduction processing for reducing the number of pixels as necessary. The first preprocessing unit 104 stores the processed image data in the display memory 107 .

The first image correction unit 109 applies correction processing such as white balance correction processing and shading correction processing to the image data stored in the display memory 107, conversion processing from RGB format to YUV format, and the like. Note that the first image correction unit 109 may use image data of one or more frames different from the processing target frame among the image data stored in the display memory 107 when applying the correction processing. For example, the first image correction unit 109 can use the image data of the frames before and/or after the frame to be processed in the correction process. The first image correction unit 109 outputs the processed image data to the post-processing unit 114 .

The post-processing unit 114 generates recording image data and display image data from the image data supplied from the first image correction unit 109 . The post-processing unit 114 applies, for example, an encoding process to the image data, and generates a data file storing the encoded image data as recording image data. The post-processing unit 114 supplies the recording image data to the recording unit 118 .

Also, the post-processing unit 114 generates display image data to be displayed on the display unit 121 from the image data supplied from the first image correction unit 109 . The image data for display has a size corresponding to the display size on the display unit 121 . The post-processing unit 114 supplies the display image data to the information superimposing unit 120 .

A recording unit 118 records the recording image data converted by the post-processing unit 114 on a recording medium 119 . The recording medium 119 may be, for example, a semiconductor memory card, built-in non-volatile memory, or the like.

A second preprocessing unit 105 applies color interpolation processing to the image data output by the image sensor 103 . The second preprocessing unit 105 stores the processed image data in the tracking memory 108 . Tracking memory 108 and display memory 107 may be implemented as separate address spaces within the same memory space. In addition, the second preprocessing unit 105 may apply reduction processing for reducing the number of pixels as necessary in order to reduce the processing load. Although the first preprocessing unit 104 and the second preprocessing unit 105 are described here as separate functional blocks, a common preprocessing unit may be used.

The second image correction unit 106 applies correction processing such as white balance correction processing and shading correction processing to the image data stored in the tracking memory 108, conversion processing from RGB format to YUV format, and the like. Also, the second image correction unit 106 may apply image processing suitable for subject detection processing to the image data. For example, if the representative luminance of the image data (for example, the average luminance of all pixels) is equal to or less than a predetermined threshold, the second image correction unit 106 applies a constant coefficient to the entire image data so that the representative luminance becomes equal to or greater than the threshold. (gain) may be multiplied.

Note that the second image correction unit 106 may use image data of one or more frames different from the processing target frame among the image data stored in the tracking memory 108 when applying the correction processing. For example, the second image correction unit 106 can use image data of frames before and/or after the frame to be processed in the correction process. The second image correction unit 106 stores the processed image data in the tracking memory 108 .

Note that functional blocks related to the subject tracking function, such as the second preprocessing unit 105 and the second image correction unit 106, do not need to operate when the subject tracking function is not performed. Image data to which the subject tracking function is applied is moving image data captured for live view display or recording. Moving image data has predetermined frame rates such as 30 fps, 60 fps, and 120 fps.

The detection unit 110 detects one or more predetermined candidate subject areas (candidate areas) from one frame of image data. For each detected area, the detection unit 110 detects an object class indicating the position and size in the frame, and the type of candidate subject (automobile, airplane, bird, insect, human body, head, pupil, cat, dog, etc.). and its confidence. Also, the number of detected regions is counted for each object class.

The detection unit 110 can detect candidate regions using known techniques for detecting characteristic regions such as human or animal face regions. For example, the detection unit 110 may be configured as a class discriminator that has been trained using learning data. There are no particular restrictions on the identification (classification) algorithm. The detection unit 110 can be realized by learning a discriminator implemented with multi-class logistic regression, support vector machine, random forest, neural network, or the like. The detection unit 110 stores the detection result in the tracking memory 108 .

The target determination unit 111 determines a subject area (main subject area) to be tracked from the candidate area detected by the detection unit 110 or the tracking result of the tracking unit 115 (to be described later). The subject area to be tracked can be determined, for example, based on the priority assigned in advance to each item included in the detection result, such as object class and area size. Specifically, the total priority may be calculated for each candidate area, and the candidate area with the lowest total may be determined as the subject area to be tracked. Alternatively, among the candidate areas belonging to a specific object class, the candidate area closest to the center of the image or the focus detection area, or the largest candidate area may be determined as the subject area to be tracked. The target determining unit 111 stores information specifying the determined subject area in the tracking memory 108 .

The difficulty determination unit 112 calculates a difficulty score, which is an evaluation value indicating the difficulty of tracking, for the tracking target subject area determined by the target determination unit 111 . For example, the difficulty level determination unit 112 can calculate the difficulty level score considering one or more factors that affect the tracking difficulty level. Elements that affect the tracking difficulty include, but are not limited to, the size of the subject area, the object class (kind) of the subject, the total number of areas belonging to the same object class, and the position within the image. A specific example of how to calculate the difficulty score will be described later. The difficulty level determination unit 112 outputs the calculated difficulty level score to the tracking control unit 113 .

The tracking control unit 113 determines whether to enable or disable each of the plurality of tracking units included in the tracking unit 115 based on the difficulty score calculated by the difficulty determination unit 112 . In this embodiment, the tracking unit 115 has a plurality of tracking units with different calculation loads and tracking accuracies. Specifically, the tracking unit 115 has a DL tracking unit 116 that tracks the subject using deep learning (DL) and a non-DL tracking unit 117 that tracks the subject without using DL. It is assumed that the DL tracking unit 116 has higher processing accuracy than the non-DL tracking unit 117 but has a larger computational load than the non-DL tracking unit 117 .

In this case, the tracking control unit 113 determines whether to enable or disable each of the DL tracking unit 116 and the non-DL tracking unit 117 . The tracking control unit 113 also determines the operation frequency of the active tracking units. The operating frequency is the frequency (fps) at which the tracking process is applied.

The tracking unit 115 estimates a subject area to be tracked from the image data of the frame to be processed (current frame) stored in the tracking memory 108, and calculates the position and size of the estimated subject area within the frame as a tracking result. Ask as For example, the tracking unit 115 uses image data of the current frame and image data of a past frame captured before the current frame (for example, the previous frame) to determine the subject area to be tracked in the current frame. presume. Tracking section 115 outputs the tracking result to information superimposing section 120 .

Here, the tracking unit 115 estimates an area within the frame to be processed that corresponds to the subject area to be tracked in the past frame. That is, the tracking target subject area determined by the target determining unit 111 for the processing target frame is not the tracking target subject area in the tracking process for the processing target frame. The subject area to be tracked in the tracking process for the frame to be processed is the subject area to be tracked in the past frame. The tracking target subject area determined by the target determination unit 111 for the processing target frame is used for the tracking process of the next frame when the tracking target subject is switched to another subject.

The tracking unit 115 has a DL tracking unit 116 that tracks the subject using deep learning (DL) and a non-DL tracking unit 117 that tracks the subject without using DL. Then, the tracking unit enabled by the tracking control unit 113 outputs the tracking result at the operation frequency set by the tracking control unit 113 .

The DL tracking unit 116 estimates the position and size of the subject area to be tracked using a trained multi-layer neural network including convolution layers. More specifically, the DL tracking unit 116 has a function of extracting feature points and feature amounts included in the feature points of the subject area for each object class that can be a target, and a function of associating the extracted feature points between frames. and Therefore, the DL tracking unit 116 can estimate the position and size of the tracking target subject area in the current frame from the feature points of the current frame that are associated with the feature points of the tracking target subject area of the past frame. .

The DL tracking unit 116 outputs the position, size, and reliability score of the tracking target subject area estimated for the current frame. The reliability score indicates the reliability of the matching of feature points between frames, that is, the reliability of the estimation result of the tracking target subject area. If the reliability score indicates that the reliability of the matching of feature points between frames is low, it is possible that the subject area estimated in the current frame is an area related to a different subject from the tracking target subject area in the past frame. indicates that there is

On the other hand, the non-DL tracking unit 117 estimates the tracking target subject area in the current frame by a technique that does not use deep learning. Here, it is assumed that the non-DL tracking unit 117 estimates the tracking target subject area based on the similarity of color configuration. However, other methods such as pattern matching using a tracking target subject region in a past frame as a template may be used. The non-DL tracking unit 117 outputs the position, size, and reliability score of the tracking target subject area estimated for the current frame.

Here, the degree of similarity of color configuration will be described. Here, in order to facilitate explanation and understanding, it is assumed that the shape and size of the subject area to be tracked are the same between the past frame and the current frame. It is also assumed that the image data has a depth of 8 bits (values 0 to 255) for each RGB color component.

The non-DL tracking unit 117 divides the range of possible values (0 to 255) for a certain color component (for example, the R component) into a plurality of regions. Then, the non-DL tracking unit 117 classifies the pixels included in the tracking target subject region according to the region to which the R component value belongs (the frequency for each range of values) as the color configuration of the tracking target subject region. do.

As the simplest example, it is assumed that the possible value range (0 to 255) of the R component is divided into Red1 of 0 to 127 and Red2 of 128 to 255. Assume that the color configuration of the subject area to be tracked in the past frame is 50 pixels for Red1 and 70 pixels for Red2. It is also assumed that the color configuration of the subject area to be tracked in the current frame is 45 pixels for Red1 and 75 pixels for Red2.

In this case, the non-DL tracking unit 117 can calculate a score (similarity score) representing the degree of similarity of color configuration based on the difference in the number of pixels classified into the same value range as follows.
Similarity score=|50-45|+|70-75|=10
Assuming that the color configuration of the tracking target subject area in the current frame is 10 pixels for Red1 and 110 pixels for Red2, the similarity score is:
Similarity score=|50-10|+|70-110|=80
becomes. Thus, the lower the color configuration similarity, the higher the similarity score. Alternatively, a smaller similarity score indicates a higher similarity of color configurations.

The information superimposing unit 120 generates a tracking frame image based on the size of the subject area included in the tracking result output by the tracking unit 115 . For example, the tracking frame image may be a frame-shaped image representing the outline of a rectangle that circumscribes the subject area. Then, the information superimposing unit 120 superimposes and synthesizes the image of the tracking frame on the display image data output from the post-processing unit 114 so that the tracking frame is displayed at the position of the subject area included in the tracking result. Generate image data. The information superimposing unit 120 also generates images representing the current setting values and states of the imaging device 100, and displays the images for display output by the post-processing unit 114 so that these images are displayed at predetermined positions. It may be superimposed on the image data. Information superimposing section 120 outputs the synthesized image data to display section 121 .

The display unit 121 may be, for example, a liquid crystal display or an organic EL display. The display unit 121 displays an image based on the composite image data output by the information superimposing unit 120. FIG. Live view display for one frame is performed as described above.

The evaluation value generation unit 124 generates signals and evaluation values used for automatic focus detection (AF) from image data obtained from the image sensor 103, and calculates evaluation values (luminance information) used for automatic exposure control (AE). . Luminance information is generated by color conversion from integrated values (red, blue, green) obtained by integrating each color filter pixel (red, blue, green). Note that another method may be used to generate luminance information. Also, an evaluation value (integrated value for each color (red, blue, green)) used for automatic white balance (AWB) is calculated in the same manner as when generating luminance information. The control unit 102 identifies the light source from the integrated value for each color, and calculates the pixel correction value so that the white color becomes white. White balance is performed by multiplying each pixel by the correction value in the first image correction unit 109 and the second image correction unit 106, which will be described later. Also, an evaluation value (motion vector information) used for camera shake detection for camera shake correction is calculated from image data serving as a reference using two or more pieces of image data. Evaluation value generator 124 outputs the generated signal and evaluation value to control unit 102 . The control unit 102 controls the focus lens position of the optical system 101 and determines shooting conditions (exposure time, aperture value, ISO sensitivity, etc.) based on signals and evaluation values obtained from the evaluation value generation unit 124. do. The evaluation value generation unit 124 may generate a signal or an evaluation value from display image data generated by the post-processing unit 114, which will be described later.

The selection unit 125 adopts one of the tracking results of the DL tracking unit 116 and the non-DL tracking unit 117 based on the reliability score output by the DL tracking unit 116 and the similarity score output by the non-DL tracking unit 117. . For example, when the reliability score is less than or equal to a predetermined reliability score threshold and the similarity score is less than or equal to a predetermined similarity score threshold, the selection unit 125 adopts the tracking result of the non-DL tracking unit 117. Otherwise, the tracking result of the DL tracking unit 116 is adopted. Selecting section 125 outputs the adopted tracking result to information superimposing section 120 and control section 102 .

Here, which of the tracking results of the DL tracking unit 116 and the non-DL tracking unit 117 is adopted is determined based on the reliability score and the similarity score. However, other methods may be used. For example, using the fact that the accuracy of the DL tracking unit 116 tends to be higher than the accuracy of the non-DL tracking unit 117, the tracking result of the DL tracking unit 116 may be preferentially employed. Specifically, if the tracking result of the DL tracking unit 116 is obtained, the tracking result of the DL tracking unit 116 may be adopted, and if not, the tracking result of the non-DL tracking unit 117 may be adopted.

The imaging device motion detection unit 126 detects the motion of the imaging device 100 itself, and is composed of a gyro sensor or the like. The imaging device motion detection unit 126 outputs the detected motion information of the imaging device to the control unit 102 . The control unit 102 detects camera shake based on motion information of the imaging device, detects swinging of the imaging device in a certain direction, and determines panning shooting. Note that for panning determination, the result of the imaging device motion detection unit 126 and the motion vector of the evaluation value generation unit 124 are combined, and the imaging device is shaken in a certain direction, but there is almost no motion vector of the subject. , the panning determination accuracy can be improved.

Next, with reference to FIG. 2, an operational flow of subject tracking processing by the tracking control unit 113 when the imaging apparatus 100 performs an imaging operation will be described. In the present embodiment, DL tracking and non-DL tracking are controlled depending on whether the scene is a panning shooting scene and whether the brightness of the scene is low. However, only one of the scenes may be determined, or another scene may be determined, and DL tracking or non-DL tracking may be determined.

In S201, the tracking control unit 113 acquires motion information of the imaging device itself detected by the imaging device motion detection unit 126, and proceeds to S202.

In S202, the tracking control unit 113 determines whether or not the imaging device is moving in a certain direction based on the motion information of the imaging device itself, and determines whether or not the imaging device is moving in a certain direction. If it is determined that the image has not been taken, the process proceeds to S203.

In S203, the tracking control unit 113 acquires the luminance information generated by the evaluation value generation unit 124, and proceeds to S204.

In S204, the tracking control unit 113 compares the acquired luminance information with a threshold value, and proceeds to S205 if less than the threshold value, and proceeds to S206 if greater than or equal to the threshold value. Specifically, if the image data is dark, the process proceeds to S205, and if the image data is bright, the process proceeds to S206. In this embodiment, the determination is made based only on the luminance information of one frame. However, the luminance information may be compared with the threshold over a plurality of frames, and if the threshold is less than the threshold in the plurality of frames, the process may proceed to S205. good.

In S205, the tracking control unit 113 determines to disable the DL tracking unit 116 and enable the non-DL tracking unit 117, and ends the process. In panning, the camera does not track a moving subject, but the user captures the subject and moves the camera in a certain direction. The frequency of operation of the non-DL tracking unit 117 may be reduced. Similarly, when the image data is dark, that is, because night scene photography is assumed, the frequency of operation of the non-DL tracking unit 117 may be reduced by regarding this as a scene that does not require tracking performance.

In S206, the tracking control unit 113 determines to enable the DL tracking unit 116 and disable the non-DL tracking unit 117, and ends the process.

(Display processing by display unit 121)
FIG. 3 is a diagram showing an example of live view display. FIG. 3A shows an image 300 represented by the display image data output by the post-processing unit 114. FIG. FIG. 2B shows an image 302 represented by combined image data in which the image of the tracking frame 303 is superimposed on the image data for display. Since only one candidate subject 301 exists in the imaging range here, the candidate subject 301 is selected as the subject to be tracked. A tracking frame 303 is superimposed so as to surround the candidate subject 301 . In the example of FIG. 2B, the tracking frame 303 is composed of a combination of four hollow hook shapes. Other forms of tracking frame 303 may be used. Also, the form of the tracking frame 303 may be selectable by the user.

FIG. 4 is a flowchart regarding the operation of the subject tracking function in a series of imaging operations by the imaging apparatus 100. FIG. Each step is executed by each unit according to the control unit 102 or an instruction from the control unit 102 .

In S400, the control unit 102 controls the image sensor 103 to capture one frame of image, and acquires image data.

In S<b>401 , the first preprocessing unit 104 applies preprocessing to image data read from the image sensor 103 .

In S<b>402 , the control unit 102 stores the preprocessed image data in the display memory 107 .

In S<b>403 , the first image correction unit 109 starts applying predetermined image correction processing to the image data read from the display memory 107 .

In S404, the control unit 102 determines whether or not all the image correction processes to be applied have been completed. Proceed to S405. Further, the first image correction unit 109 continues the image correction processing unless it is determined that all the image correction processing is completed.

In S<b>405 , the post-processing unit 114 generates display image data from the image data to which the image correction processing has been applied by the first image correction unit 109 , and outputs the display image data to the information superimposing unit 120 .

In S406, the information superimposing unit 120 uses the image data for display generated by the post-processing unit 114, the image data of the tracking frame, and the image data representing other information to add the tracking frame and other information to the captured image. data of a composite image in which the images of are superimposed. Information superimposing section 120 outputs the synthesized image data to display section 121 .

In S<b>407 , the display unit 121 displays the composite image data generated by the information superimposing unit 120 . This completes the live view display for one frame.

As described above, in this embodiment, in the image processing apparatus using the first tracking means and the second tracking means having a smaller computational load than the first tracking means, the movement of the imaging device and the brightness of the image data The validity/invalidity of the first and second tracking means is controlled based on at least one of Therefore, power consumption can be suppressed by disabling the first tracking means in a scene where it is less necessary to obtain a good tracking result.

Further, in this embodiment, in controlling the validity/invalidity of the DL/non-DL tracking unit based on the movement of the imaging device itself and the brightness of the image data, when the DL tracking unit 116 is valid, the non-DL tracking unit 117 An example of exclusive control such as disabling is shown. However, not limited to this, both the DL tracking unit 116 and the non-DL tracking unit 117 may be enabled according to the panning speed and low brightness value in panning scenes and low brightness values that are more difficult. That is, at this time, control may be performed so that the tracking process is performed based on both tracking results. In addition, in the above-described embodiment, an example is shown in which the control of the DL tracking unit 116 and the non-DL tracking unit 117 is switched between valid and invalid in binary. However, it is not limited to this, and may be switched in multiple stages according to the brightness of the image or the movement of the subject. In other words, a plurality of levels of calculation load may be prepared for the effectiveness of the DL tracking unit 116 and the non-DL tracking unit 117, and switching may be performed to perform processing with a higher calculation load when more effective.

In the present embodiment, the DL tracking unit 116 and the non-DL tracking unit 117 are invalidated by omitting or not executing all the arithmetic processing performed by the L tracking unit 116 . However, the present invention is not limited to this, and may include omitting or not executing at least a part of tracking calculation processing and tracking result output processing performed when valid, such as preprocessing for tracking processing and calculation for main processing of tracking. .

(Second embodiment)
Next, a second embodiment of the invention will be described. Here, only parts different from the above-described first embodiment will be described, and detailed description of the same parts will be omitted by attaching the same reference numerals. In the second embodiment, the DL tracking unit 116, the non-DL tracking unit 116, and the non-DL tracking unit 116 use the result of the imaging apparatus automatically recognizing the shooting scene based on at least one of the captured image, the shooting parameter, the posture of the imaging apparatus, and the like. It controls the unit 117 and the detection unit 110 . Hereinafter, description will be made with reference to FIGS. 5 and 6. FIG.

FIG. 5 is an operation flow of the control unit 102 of the second embodiment.

In S501, the control unit 102 determines the shooting scene shown in FIG. 4 (which will be described later), and proceeds to S502. In determining the shooting scene in FIG. 4, whether the background is bright or dark is determined from the luminance information acquired by the evaluation value generation unit 124, and the background blue sky/twilight is obtained in the process of calculating the white balance correction value. It is determined from light source information and brightness information. Also, a person or non-person subject is determined from the result of the detection unit 110 , and a moving object or non-moving object is determined by the tracking unit 115 . Any known processing sequence that can determine a shooting scene from information obtained by an image, a gyro sensor, an infrared sensor, a ToF (time of flight) sensor, or the like is applicable without being limited to these determination methods. . Panning determination is performed by the same method as in the first embodiment.

In S502, the control unit 102 performs control so that the operation mode shown in FIG. 5 (described later) corresponding to the shooting scene shown in FIG. 4 is set, and the process ends. Specifically, the control unit 102 controls the detection unit 110 and notifies the tracking control unit 113 according to the operation mode in the shooting scene table of FIG. The tracking control unit 113 that has received the notification controls the tracking unit 116 .

FIG. 6A is a table showing the relationship between the shooting scene and the operation modes of the detection unit 110 and the tracking unit 105. FIG. The abscissa indicates subject determination, whether it is a person, non-person, moving or non-moving object, or whether it is a panning scene, and the ordinate indicates the background brightness, blue sky, or evening scene. In other words, it is a table that determines the operation mode by judging the subject and the background. Note that the shooting scene in FIG. 4 is an example, and the operation mode may be determined by adding other shooting scenes.

FIG. 6B is a table showing operation modes of the detection unit 110 and the tracking unit 105. As shown in FIG.

In operation mode 1, the DL tracking unit 116 is disabled, the non-DL tracking unit 117 is enabled, the detection unit 110 is operated to detect a person and a person other than the person, and the operation period of the detection unit 110 is set to, for example, the shooting frame rate. less than half of

In operation mode 2, the DL tracking unit 116 is disabled, the non-DL tracking unit 117 is enabled, and the objects to be detected by the detection unit 110 are people and non-animal objects such as buildings, roads, the sky, trees, etc., In addition, the operation cycle is set to, for example, half or less of the shooting frame rate. The recognition result of the non-moving object is used, for example, to specify the light source for white balance, and the correction processing of the first image correction unit 109 and the second image correction unit 106 performs image processing that distinguishes between artificial objects and non-animal objects. use for

In the operation mode 3, the DL tracking unit 116 is enabled, the non-DL tracking unit 117 is enabled, the detection unit 110 detects objects other than a person, and the operation cycle is set to the same frame rate. , to reduce the frame rate of shots other than people to less than half of the shooting frame rate.

In operation mode 4, the DL tracking unit 116 is turned on, the non-DL tracking unit 117 is enabled, and the object to be detected by the detection unit 110 is a person and a non-moving object other than a person. In addition, for example, the motion cycle is set to the same shooting frame rate for people, and to half or less of the shooting frame rate for non-moving objects other than people.

In operation mode 5, the DL tracking unit 116 is disabled, the non-DL tracking unit 117 is enabled, the detection unit 110 detects objects other than a person, and the operation period is set to the same frame rate. , to reduce the frame rate of shots other than people to less than half of the shooting frame rate.

In operation mode 6, the DL tracking unit 116 is enabled, the non-DL tracking unit 117 is enabled, the detection unit 110 operates with a person and a person other than the person, and the operation cycle is set to, for example, half the shooting frame rate for the person. In the following, the frame rate is set to be the same as the shooting frame rate except for the person.

In operation mode 7, the DL tracking unit 116 is enabled, the non-DL tracking unit 117 is enabled, and the detection unit 110 operates as a person and a non-moving object other than a person. Also, for example, the operation cycle is set to be half or less of the frame rate of photographing for people and non-moving objects, and to be the same as the frame rate of photographing for non-human objects.

In the operation mode 8, the DL tracking unit 116 is disabled, the non-DL tracking unit 117 is enabled, the detection unit 110 detects a person and other objects than the person, and the operation cycle is set to, for example, half the shooting frame rate for the person. In the following, the frame rate is set to be the same as the shooting frame rate except for the person.

The operation mode shown in FIG. 6(b) is an example of the operation mode corresponding to the scene shown in FIG. 6(a), and the operation mode may be changed. In this embodiment, since the non-DL tracking unit 116 is used to determine whether the subject is moving or non-moving as the shooting scene determination, the non-DL tracking unit 116 is enabled in any operation mode. However, the moving object determination may be performed by monitoring the position of the object detected by the detection unit 110 over a plurality of frames. In that case, when the subject is a non-moving object (when determined not to be a moving object), the non-DL tracking 116 may be disabled.

As described above, in the present embodiment, in the image processing apparatus using the first tracking means and the second tracking means having a smaller computational load than the first tracking means, based on the scene in which the image was captured, Validity/invalidity of the first and second tracking means is controlled. Furthermore, based on the scene in which the image was taken, we restricted the objects detected by the detector from the image and changed the operation cycle. Therefore, power consumption can be suppressed in a scene where it is less necessary to obtain a good tracking result.

(Third embodiment)
Next, a third embodiment of the invention will be described. In the third embodiment, in an image processing device that detects so-called "feature points" in a plurality of regions in a captured image, the DL tracking unit 116, the non-DL tracking unit 117, and the detection unit are based on the detection results of these feature points. control 110; Hereinafter, description will be made with reference to FIGS. 7 and 8. FIG.

FIG. 7 is an operation flow of the control unit 102 of the third embodiment. In this flow, an imaging mode is selected from a menu while the power of the imaging apparatus 100 is turned on, and a tracking subject for which tracking processing is to be performed on captured images sequentially acquired from the imaging device 103 is determined. Assume that it operates when performing tracking processing. Moreover, when there is a setting of ONOFF for tracking control, control may be performed so that this flow starts when tracking control is set to ON.

In S<b>701 , the control unit 102 acquires a captured image output from the image sensor 103 or stored in the detection/tracking memory 108 .

In S702, according to an instruction from the control unit 102, the evaluation value generation unit 124 analyzes the captured image obtained in S601 and performs detection processing for detecting feature points from the image. The details of the feature point detection processing will be described later.

In S703, the control unit 102 acquires information on the feature point intensity calculated when detecting each feature point in S602.

In step S<b>704 , the control unit 102 detects a tracked subject that has been determined by the DL tracking unit 116 , the non-DL tracking unit 117 , or other subject detection processing (such as face detection) as an area including the subject to be tracked by the previous frame. Judgment processing is performed on the feature points detected within the area. Specifically, it is determined whether or not the number of feature points having a feature point intensity greater than or equal to a first threshold in the tracking subject area is greater than or equal to a second threshold. If the number of feature points whose feature point strength is equal to or greater than the first threshold is equal to or greater than the second threshold, the process advances to S705; if the number of feature points whose feature point strength is equal to or greater than the first threshold is less than the second threshold Proceed to S706.

In S705, the control unit 102 performs determination processing on the feature points detected outside the area determined as the tracking subject area in the previous frame in the captured image. Specifically, it is determined whether or not the number of feature points outside the tracking subject region having feature point intensities greater than or equal to the third threshold is greater than or equal to the fourth threshold. If the number of feature points whose feature point strength is greater than or equal to the third threshold is equal to or greater than the fourth threshold, the process advances to S707; if the number of feature points whose feature point strength is greater than or equal to the third threshold is less than the fourth threshold Proceed to S708.

In S706, the control unit 102 performs determination processing on the feature points detected outside the area determined as the tracking subject area in the previous frame in the captured image. Specifically, it is determined whether or not the number of feature points outside the tracking subject region having feature point intensities greater than or equal to the third threshold is greater than or equal to the fourth threshold. If the number of feature points with feature point strengths equal to or greater than the third threshold is equal to or greater than the fourth threshold, the process advances to step S709; Proceed to S710.

In S707, the tracking control unit 113 enables both the DL tracking unit 116 and the non-DL tracking unit 117 according to the instruction from the control unit 102, and sets the operating rate of the DL tracking process higher than the operating rate of the non-DL tracking process. Since there are many subjects with complex textures inside and outside the tracking subject area, and tracking is highly difficult, tracking accuracy can be maintained by performing both tracking processes at a high rate.

In S<b>708 , the tracking control unit 113 disables the DL tracking unit 116 and enables the non-DL tracking unit 117 according to an instruction from the control unit 102 . Here, in the present embodiment, the operating rate of the non-DL tracking process at this time is higher than the operating rate of the non-DL tracking process set in S707. Since it is easy to distinguish between the inside and outside of the tracking subject area, it is possible to suppress power consumption while maintaining tracking accuracy by performing tracking processing only with non-DL tracking.

In S<b>709 , the tracking control unit 113 enables the DL tracking unit 116 and disables the non-DL tracking unit 117 according to an instruction from the control unit 102 . Here, in this embodiment, the operating rate of the DL tracking process at this time is assumed to be the highest among the operating rates set in the DL tracking unit 116 in S707 to S710. The fact that there are few feature points in the tracking subject area and many feature points outside the tracking subject area makes tracking difficult. The DL tracking process is more likely to output an erroneous result. Therefore, tracking is performed only by the DL tracking process, thereby suppressing deterioration in tracking accuracy.

In S710, the tracking control unit 113 activates both the DL tracking unit 116 and the non-DL tracking unit 117 according to an instruction from the control unit 102, and sets the operation rates of the DL tracking process and the non-DL tracking process in S707. Set lower than rate. In a situation where there are few feature points that can be detected both inside and outside the tracking subject area, it is difficult to achieve accuracy in both the DL tracking process and the non-DL tracking process. If they are reflected at a high rate, they cause flickering in the image. Therefore, by lowering the operation rate while enabling both tracking processes, the decrease in visibility due to the flickering of the tracking result is suppressed.

(Feature point detection processing)
FIG. 8 is a flowchart of feature point detection processing performed by the feature point detection unit 201 . In S800, the control unit 102 generates a horizontal first-order differential image by performing horizontal first-order differential filter processing on the region of the tracking subject. In S802, the control unit 102 further performs horizontal primary differential filter processing on the horizontal primary differential image obtained in S800 to generate a horizontal secondary differential image.

In S801, the control unit 102 generates a vertical primary differential image by performing vertical primary differential filter processing on the region of the tracking subject.

In S804, the control unit 102 further performs vertical primary differential filter processing on the vertical primary differential image obtained in S801 to generate a horizontal secondary differential image.

In S803, the control unit 102 further performs vertical primary differential filter processing on the horizontal primary differential image obtained in S800 to generate horizontal primary differential and vertical primary differential images.

In S805, the control unit 102 calculates the determinant Det of the Hessian matrix H of the differential values obtained in S802, S803, and S804. Let Lxx be the horizontal secondary differential value obtained in S802, Lyy be the vertical secondary differential value obtained in S804, and Lxy be the horizontal primary differential value and the vertical primary differential value obtained in S803. (1), and the determinant Det is represented by equation (2).

In S806, the control unit 102 determines whether the determinant Det obtained in S805 is 0 or more. When the determinant Det is 0 or more, the process proceeds to S807. When the determinant Det is less than 0, proceed to S808.

In S807, the control unit 102 detects points with a determinant Det of 0 or more as feature points.

In S808, if the control unit 102 determines that processing has been performed on all of the input subject regions, it ends the feature point detection processing. If all the processes have not been completed, the processes from S800 to S807 are repeated to continue the feature point detection process.

As described above, in the present embodiment, in the image processing apparatus using the first tracking means and the second tracking means having a smaller computational load than the first tracking means, the first and second Validity/invalidity of the second tracking means is controlled. Therefore, power consumption can be suppressed in a scene where it is less necessary to obtain a good tracking result.

Although the present invention has been specifically described above based on the embodiments, it goes without saying that the present invention is not limited to the above embodiments, and that various modifications are possible without departing from the gist of the invention.

102 control unit 110 detection unit 113 tracking control unit 115 tracking unit 116 DL tracking unit 117 non-DL tracking unit 124 evaluation value generation unit 126 imaging device motion detection unit

Claims (7)

a first tracking means for tracking a subject using an image acquired by the imaging means;
a second tracking means having a smaller computational load than the first tracking means for tracking an object using the image acquired by the imaging means;
a control means for switching between enabling both the first tracking means and the second tracking means or disabling one of them based on the feature points detected from the image acquired by the imaging means; have
An image processing apparatus characterized by: The control means controls the first tracking means based on the number of feature points within the tracking subject area to be tracked by the first tracking means or the second tracking means and the number of feature points outside the tracking subject area. 2. The image processing apparatus according to claim 1, wherein switching is made between enabling both of said tracking means and said second tracking means and disabling one of them. The control means is
In the tracking subject area, the number of feature points whose intensity is higher than a first threshold is larger than a second threshold,
When the number of feature points whose intensity is higher than a third threshold outside the tracking subject area is larger than a fourth threshold,
3. The image processing apparatus according to claim 1, wherein both said first tracking means and said second tracking means are enabled. The control means is
the number of feature points whose intensity is higher than a first threshold in the tracking subject area is less than a second threshold;
When the number of feature points whose intensity is higher than a third threshold outside the tracking subject area is larger than a fourth threshold,
4. The image processing apparatus according to claim 1, wherein said first tracking means is enabled and said second tracking means is disabled. The control means is
In the tracking subject area, the number of feature points whose intensity is higher than a first threshold is larger than a second threshold,
outside the tracking subject area, if the number of feature points whose intensity is higher than a third threshold is less than a fourth threshold,
5. The image processing apparatus according to claim 1, wherein said first tracking means is disabled and said second tracking means is enabled. The control means is
the number of feature points whose intensity is higher than a first threshold in the tracking subject area is less than a second threshold;
outside the tracking subject area, if the number of feature points whose intensity is higher than a third threshold is less than a fourth threshold,
6. The method according to any one of claims 1 to 5, wherein the operation rate is made lower than in other cases while enabling both the first tracking means and the second tracking means. Image processing device. a first tracking step of performing tracking by a first tracking means for performing subject tracking using an image acquired by the imaging means;
a second tracking step of performing tracking by a second tracking means having a smaller computational load than the first tracking means, in which the subject is tracked using the image acquired by the imaging means;
a control step of switching between enabling both the first tracking means and the second tracking means or disabling one of them based on the feature points detected from the image acquired by the imaging means; have
An image processing method characterized by:

Images