JP2022160333A - Image processing apparatus and control method for image processing apparatus - Google Patents

Abstract

To provide an image processing apparatus and a control method for an image processing apparatus in which when the number of feature points in an image is small and the intensity of the feature points is small, the accuracy of tracking is reduced.SOLUTION: An imaging apparatus 100 has: a tracked subject determination unit that detects a subject from an image picked up by an image pickup device 104 and a feature point detection unit that detects feature points from an area corresponding to the subject detected by the tracked subject determination unit; and an evaluation value calculation unit that calculates an evaluation value based on the number of the feature points or the intensity of the feature points detected by the feature point detection unit or both the number and the intensity. The imaging apparatus has: an FPM tracking unit that tracks the subject detected by the tracked subject determination unit by using feature point matching based on the feature points detected by the feature point detection unit; a TM tracking unit 122 that tracks a subject detected by detection means by using a tracking technique different from that of the FPM tracking unit; and a tracking control unit 119 that switches between the FPM tracking unit and the TM tracking unit based on the evaluation value calculated by the evaluation value calculation unit.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing apparatus having a plurality of tracking circuits, tracking an object detected from a video signal by the plurality of tracking circuits, and using the tracked area as an object area for image processing and the like, and a control method for the image processing apparatus.

2. Description of the Related Art Imaging devices such as digital cameras are provided with a so-called subject tracking function for tracking a subject included in an image.

For example, Patent Literature 1 discloses a technique for automatically tracking a specific subject using template matching. Note that template matching involves registering, as a template image, a partial image obtained by cutting out an image area that includes a specific subject to be tracked, and estimating an area within the image that has the highest degree of similarity or the lowest degree of difference from the template image. , is a technology for tracking a specific subject.

Japanese Patent Application Laid-Open No. 2001-060269

However, template matching is weak against changes in the shape and orientation of the subject, and is not suitable for tracking subjects that move (change) rapidly.

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image processing apparatus and an image processing apparatus control method that improve tracking performance by appropriately using a plurality of tracking methods.

In order to solve the above problems, the image processing apparatus of the present invention includes detection means for detecting a subject from an image captured by an image capturing means, and feature points from an area corresponding to the subject detected by the detection means. feature point detection means for detecting; evaluation value calculation means for calculating an evaluation value based on the number of feature points detected by the feature point detection means, the feature point intensity, or both; A first tracking means for tracking a subject using feature point matching based on the feature points detected by the feature point detection means; and a first tracking means for tracking the subject detected by the detection means. second tracking means for tracking using different tracking methods; and control means for switching between the first tracking means and the second tracking means based on the evaluation value calculated by the evaluation value calculation means. characterized by having

A control method for an image processing apparatus according to the present invention is characterized by: a detection step of detecting a subject from an image captured by an imaging device; and detecting a feature point from an area corresponding to the subject detected by the detection device. a point detection step; an evaluation value calculation step of calculating an evaluation value based on the number of feature points detected by the feature detection point means, the feature point intensity, or both; and the subject detected by the detection means, a first tracking step of tracking using feature point matching based on the feature points detected by the feature point detection means;
a second tracking step of tracking the subject detected by the detection means using a tracking method different from that of the first tracking means; and a control step of switching between one tracking means and the second tracking means.

According to the present invention, in an imaging device having a plurality of tracking circuits including feature point matching,
To improve tracking performance by appropriately using multiple tracking techniques.

It is a figure which shows the structure of the imaging device which concerns on Embodiment 1 of this invention. 4 is a diagram showing the configuration of a tracking control unit according to Embodiment 1 of the present invention; FIG. 4 is a flow chart showing processing according to Embodiment 1 of the present invention. 4 is a flowchart showing image acquisition processing according to Embodiment 1 of the present invention. 4 is a flowchart showing feature point detection/feature point intensity calculation processing according to the first embodiment of the present invention. It is a flow chart showing processing concerning Embodiment 2 of the present invention. 9 is a flow chart showing feature amount calculation processing according to Embodiment 2 of the present invention. It is a schematic diagram of the feature amount calculation process according to the second embodiment of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.

The imaging device 100, which is an example of an image processing device according to the present invention, superimposes a subject tracking frame generated from an image signal on a display image generated from the image signal, and displays the superimposed image.

FIG. 1 is a diagram schematically showing the internal structure of an imaging device 100 according to Embodiment 1. As shown in FIG. The imaging apparatus 100 includes an optical system 101 , an optical control section 102 , a shutter 103 , an imaging element 104 , an image processing section 110 , a recording medium 130 , a display section 131 and a system control section 140 . The imaging device 100 also includes a release button 150 , a mode switch 151 , an operation section 152 , a nonvolatile memory 153 and a system memory 154 .

The image processing unit 110 includes a first preprocessing unit 111 that processes an image to be displayed on the display unit 131, and a second preprocessing unit 111 that processes an image corresponding to the detected subject and used for tracking. It has a preprocessing unit 112 and a second correction unit 113 . The image processing unit 110 also has a first memory 114 for recording the display image, a second memory 115 for recording the tracking image, and a first correction unit 116 for correcting the display image. Further, a subject detection unit 117 , a subject determination unit 118 , a tracking control unit 119 , a tracking unit 120 , a first post-processing unit 123 that performs post-processing of images for display, a recording unit 124 and a tracking frame superimposing unit 125 are included.

The optical system 101 includes a plurality of lenses such as a zoom lens, and forms and acquires optical information from a subject. The optical control unit 102 executes focus control of the optical system 101 .

The imaging device 104 converts the optical information transmitted from the optical system 101 into electrical information, and outputs the electrical information to the first preprocessing unit 111 and the second preprocessing unit 112 as image signals. The imaging element 104 can be realized using a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

The first preprocessing unit 111 converts the image signal output from the image sensor 104 into an RGB signal, which is a signal of each color of RGB, and writes the RGB signal into the display memory 114 .

The first correction unit 116 performs white balance correction, shading correction, conversion of RGB signals to YUV signals, etc. for all pixel RGB signals for all lines when the image is regarded as a plane and the vertical direction is defined as lines. perform various image correction processes. The first correction unit 116 writes and reads RGB signals or converted YUV signals to and from the display memory 114 when performing image correction processing on the RGB signals, and uses a plurality of lines or a plurality of frames. It is possible to execute the image correction processing that has been used.

The first post-processing unit 123 converts the RGB signal subjected to the image correction processing by the first correction unit 116 or the YUV signal subjected to the conversion processing into a recording image signal for recording on the recording medium 130 . The image signal for recording is, for example, a compressed data signal adapted to JPEG compression. Further, the first post-processing unit 123 converts the RGB signal subjected to the image correction processing by the first correction unit 116 or the YUV signal subjected to the conversion processing into a display image signal to be displayed on the display unit 131. .

The recording unit 124 records the recording image signal converted by the first post-processing unit 123 on the recording medium 130 . As a recording medium, for example, a flash memory such as an SD memory card or compact flash (registered trademark) is used.

The second preprocessing unit 112 converts the image signal converted by the image sensor 104 into a tracking RGB signal, which is a signal of each color of RGB, and writes the signal in the detection/tracking memory 115 .

A second correction unit 113 performs image correction processing such as white balance correction and shading correction on the tracking RGB signal and conversion of the RGB signal into a YUV signal. The image correction processing is performed so that subject detection by the subject detection unit 117 can be performed more appropriately. For example, if the average luminance value of the all-pixel tracking RGB signals for all lines is equal to or less than a preset threshold value, the all-pixel tracking RGB signals for all lines are multiplied by a constant value. This improves the detection performance of the subject detection unit 117 . Including such gain processing. Like the first correction unit 116 , the second correction unit 113 can execute image processing using multiple lines and multiple frames through the detection/tracking memory 115 .

The subject detection unit 117 detects the position information and size information of one or more tracking candidate subjects from the tracking RGB signal or YUV signal subjected to conversion processing for each frame. The subject detection unit 117 may be implemented by a multi-class classifier using machine learning, or may be implemented by a method that does not use machine learning.

The tracking subject determination unit 118 determines the tracking subject, which is the main subject to be tracked by the tracking unit 120, based on the position information and size information of the tracking candidate subject detected by the subject detection unit 117. FIG. As a method of determining a tracking subject from one or more tracking candidate subjects, for example, there is a method of determining a tracking subject based on an arbitrary priority order prepared in advance. Arbitrary priorities include, for example, prioritizing a large human face and prioritizing a subject closest to the autofocus setting frame specified by the user.

The tracking unit 120 determines to which position the tracking object determined by the tracking object determining unit 118 based on the RGB signal or YUV signal of the frame to be processed has moved. infer. Then, it outputs tracking object position information and size information to the tracking frame superimposing unit 125 as a tracking result.

The tracking control unit 119 will be explained. FIG. 2 is a diagram schematically showing the internal structure of the tracking control unit 119. As shown in FIG. It includes a feature point detection unit 201 that detects feature points, an evaluation value calculation unit 202 that calculates evaluation values from the number of feature points and feature point strengths, and a tracking determination unit 203 that determines a tracking method from the evaluation values. In the present embodiment, feature point tracking (FMP tracking) processing for detecting feature points and tracking the feature points is used as object tracking processing. The feature point tracking process is easy to detect if the subject has a feature such as an edge, regardless of the shape or state of the subject, and is resistant to changes in the subject. In this embodiment, FMP tracking processing and known template matching or histogram matching processing are used separately to improve accuracy.

A feature point detection unit 201 detects a feature point from within an image and calculates its intensity. The evaluation value calculation unit 202 calculates an evaluation value based on one or more of the number of feature points and the intensity of feature points. The tracking determination unit 203 determines whether to perform FMP tracking or TM tracking, which will be described later, based on the evaluation value calculated by the evaluation value calculation unit 202 .

The tracking unit 120 includes an FPM tracking unit 121 that performs tracking processing using feature point matching, and a processing unit that performs tracking processing using a tracking method that does not use feature point matching. For example, a matching means (template matching: TM ) is included. Matching means for matching the histogram of the partial image showing the target subject with the histogram of the partial area of the supplied image, changing the partial area to be checked, and estimating the area with high similarity or low dissimilarity. (histogram matching) or the like may be used. The FPM tracking unit 121 and the TM tracking unit 122 are controlled by the tracking control unit 119 as to which tracking unit is used for tracking.

The tracking unit 120 selects which of the center position and the size of the tracking object calculated based on the similarity score output by the FPM tracking unit 121 or the TM tracking unit 122, according to the determination of the tracking control unit 119. . Then, the central position and size of the tracking subject on the adopted side are output to the tracking frame superimposing unit 125 and the optical control unit 102 .

The tracking frame superimposing unit 125 generates a tracking frame from the center position and size of the tracking subject output by the tracking unit 120, and superimposes the generated tracking frame on the display image output by the first post-processing unit 123. Then, the superimposed display image with the tracking frame is output to the display unit 131 .

The display unit 131 is configured by, for example, an LCD panel or an organic EL panel, and displays the tracking frame-attached display image acquired from the tracking frame superimposing unit 125 .

The nonvolatile memory 153 is an electrically erasable/recordable memory, and for example, an EEPROM or the like is used. The nonvolatile memory 153 stores constants for operation of the system control unit 140, programs, and the like. The program here is a program for executing various flowcharts described later in this embodiment.

The system control unit 140 controls the imaging device 100 as a whole. By executing the program recorded in the non-volatile memory 153 described above, each process of this embodiment, which will be described later, is realized. A system memory 154 uses a RAM. In the system memory 154, constants and variables for operation of the system control unit 140, programs read from the nonvolatile memory 153, and the like are developed.

A release button 150 , a mode switch 151 , and an operation unit 152 are operation means for inputting various operation instructions to the system control unit 140 .

The release button 150 is input to the system control unit 140 by distinguishing between the so-called half-pressed state (shooting preparation instruction) during operation of the release button 150 and the so-called full-pressed state (shooting instruction) after the release button 150 is operated. The system control unit 140 starts operations such as AF (auto focus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, EF (flash pre-emission) processing, etc. by half-pressing. Further, the entire system is controlled to start a series of photographing processing operations from signal reading from the image sensor 104 to writing of image data to the recording medium 130 when the button is fully pressed.

A mode switch 151 switches the operation mode of the system control unit 140 between a still image recording mode, a moving image shooting mode, a playback mode, and the like. Modes included in the still image recording mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), and a shutter speed priority mode (Tv mode). In addition, there are various scene modes, program AE modes, custom modes, etc., which are shooting settings for each shooting scene. A mode switch 140 switches directly to any of these modes contained in the menu button. Alternatively, after switching to the menu button once with the mode switching switch 140, another operation member may be used to switch to one of these modes included in the menu button. Similarly, the movie shooting mode may also include multiple modes.

The operation unit 152 is operation means for inputting various operation instructions to the system control unit 140 . Each operation member of the operation unit 152 is appropriately assigned a function for each scene by selecting and operating various function icons displayed on the display unit 131, and acts as various function buttons. The function buttons include, for example, an end button, a return button, an image forward button, a jump button, a refinement button, an attribute change button, and the like. For example, when the menu button is pressed, a menu screen on which various settings can be made is displayed on the display unit 131 . The user can intuitively perform various settings by using the menu screen displayed on the display unit 131, the up, down, left, and right four-direction buttons and the SET button.

[Embodiment 1]
Processing performed in the first embodiment will be described below with reference to flowcharts shown in FIGS. 3, 4, and 5. FIG.

FIG. 3 is a flowchart for explaining the flow of image signal processing of the imaging apparatus 100 according to the first embodiment. S300 shows a sub-process for image acquisition. The flow of processing for acquiring this image will be described with reference to the flowchart of FIG. S303 indicates a sub-process for performing feature point detection and feature point intensity calculation. The flow of processing for detecting feature points and calculating feature point strength will be described with reference to the flowchart of FIG. Each step of this flowchart is performed by the system control unit 140 or each unit according to an instruction from the system control unit 140 .

The imaging device 104 collects the optical signal emitted from the subject by the optical system 101 and converts it into an image signal, which is an electric signal (S400). The first preprocessing unit 111 converts the image signal converted by the image sensor 104 into signals of RGB colors (S401).

The first preprocessing unit 111 writes the RGB signals converted in S401, and the first correction unit 116 writes the RGB signals and YUV signals subjected to image correction processing in S403 to the display memory 114 (S402).

The first correction unit 116 reads out the RGB signals and YUV signals written in the display memory 114 in S402, calculates evaluation values, and performs image correction processing (S403). The number of lines to be read varies depending on the evaluation value calculation processing and image processing performed by the first correction unit 116. When reading one line of RGB signals for one line of image processing, multiple lines of RGB signals are read. When reading out, the image is divided into rectangular areas, and various reading processes are executed, such as reading out in units of rectangular areas.

If all necessary image correction processing has not been completed, the first correction unit 116 returns to the processing of S402 (No in S404). When all necessary image correction processing is completed, the RGB signal or YUV signal subjected to image correction processing is output to the first post-processing unit 123 (Yes in S404).

In this flowchart, the first preprocessing unit 111, the display memory 114, and the first correction unit 116 have been described, but the second preprocessing unit 112, the detection/tracking memory 115, and the second correction are described. The processing of the unit 113 is the same except that the memory is not the display memory 114 but the detection and tracking memory 115, and the image correction processing to be performed is image processing for detection and tracking that is different from image processing for display. Therefore, the explanation is omitted.

Next, the flow of processing after S301 in the flowchart of FIG. 3 will be described. In S<b>301 , position information and size information of one or more tracking candidate subjects are detected by the subject detection unit 117 using the image signal acquired in S<b>300 , and the detection results are written in the detection/tracking memory 115 . In S302, the tracking subject determination unit 118 reads tracking candidate subject information from the detection tracking memory 115, and determines one tracking subject based on a predetermined priority.

In S303, feature points and feature point strengths are calculated. FIG. 5 is a flowchart of feature point detection performed by the feature point detection unit 201 . In S500, a horizontal primary differential image is generated by performing horizontal primary differential filter processing on the area of the tracking subject. In S502, a horizontal secondary differential image is generated by further performing horizontal primary differential filter processing on the horizontal primary differential image obtained in S500.

In S501, a vertical primary differential image is generated by performing vertical primary differential filter processing on the area of the tracking subject.

In S504, a horizontal secondary differential image is generated by further performing vertical primary differential filter processing on the vertical primary differential image obtained in S501.

In S503, the horizontal primary differential image obtained in S500 is further subjected to vertical primary differential filter processing to generate horizontal primary differential and vertical primary differential images.

In S505, the determinant Det of the Hessian matrix H of the differential values obtained in S502, S503, and S504 is calculated. When the horizontal secondary differential value obtained in S502 is Lxx, the vertical secondary differential value obtained in S504 is Lyy, and the horizontal primary differential and vertical primary differential value obtained in S503 are Lxy, the Hessian matrix H is expressed by the formula (1), and the determinant Det is represented by equation (2).

In S506, it is determined whether the determinant Det obtained in S505 is 0 or more. When the determinant Det is 0 or more, the process proceeds to S507. When the determinant Det is less than 0, proceed to S508.

In S507, points whose determinant Det is 0 or more are detected as feature points.

If it is determined in S508 that the process has been performed on all of the input subject areas, the feature point detection process ends. If all the processes have not been completed, the processes from S500 to S507 are repeated to continue the feature point detection process.

In S304, an evaluation value is calculated from one or both of the number of feature points and the intensity of feature points obtained in S303. For example, the evaluation value is calculated from the ratio of the feature points to the subject area and the average value of the feature point intensities. For example, it is assumed that the larger the ratio of the feature points to the subject area and the higher the average value of the feature point intensities, the higher the evaluation value.

In S305, it is determined whether or not the evaluation value obtained in S304 is equal to or greater than a preset threshold value. If the evaluation value is equal to or greater than the threshold, the process proceeds to S306, and subject tracking is performed by feature point matching. The feature point matching can use the feature point information obtained in S303.

If the evaluation value is less than the threshold, the process proceeds to S307 and switches to subject tracking processing based on template matching. Note that histogram matching or the like may be used instead of template matching.

In S308, the subject frame updated to the position of the subject obtained in S306 or S307 is displayed superimposed on the first signal.

As described above, the imaging apparatus 100 according to the first embodiment performs subject tracking using an existing method that does not use feature point matching when it is difficult to track the subject according to the evaluation value based on the number of feature points and the feature point intensity. can suppress a decrease in tracking accuracy.

[Embodiment 2]
Processing performed in the second embodiment will be described below with reference to the flowcharts shown in FIGS. 6 and 7 and the schematic diagram of FIG. Each step of this flowchart is performed by the system control unit 140 or each unit according to an instruction from the system control unit 140 .

S600, S601, S602, and S603 are the same as S300, S301, S302, and S303 in the flowchart of FIG.

S604 indicates a sub-process for performing feature point detection and feature point intensity calculation. While the sub-processes of feature point detection and feature point intensity calculation in S603 are performed on the subject area, S604 is the same except that it is performed on the entire image, so description thereof will be omitted.

S606 is a sub-process for calculating feature amounts for the feature points detected in S603 and S604. FIG. 8 is a schematic diagram of feature amount calculation processing. When a random line segment pattern indicated by 802 is drawn around a feature point of interest indicated by 801, the magnitude relationship between luminance values at both ends of each line segment is represented by 1 and 0. FIG. A bit string representing the magnitude relationship between 1s and 0s of all line segment patterns is a feature amount.

Processing for calculating the feature amount will be described with reference to the flowchart shown in FIG. In S701, feature amount calculation is performed for the detected feature points. In S702, it is determined whether processing of all detected feature points has been completed. If the feature amount calculation for all feature points has not been completed, the feature amount calculation in S701 is repeated.

Since S607 is the same as S304 in the flowchart of FIG. 3 showing the processing performed in the first embodiment, description thereof is omitted.

S608 determines whether the evaluation value is near a preset threshold value. It is assumed that the range of whether the evaluation value is near a preset threshold value is appropriately determined. If it is determined that the evaluation value is not near the threshold, the process proceeds to S609. S609 and S611, S612, and S613 subsequent to S609 are the same as S305, S306, S307, and S308 in the flowchart of FIG. If it is determined that the evaluation value is near the threshold value, the process proceeds to S610. In S610, the feature amount for a certain feature point in the subject area and the feature amount for the feature point in the entire image (feature amounts of other areas) are compared, and it is determined whether there are a plurality of similar feature amounts. If there are a plurality of feature amounts similar to the feature amount for a certain feature point in the subject area in the feature amount for the feature points of the entire image, the process proceeds to step S612 to avoid erroneous matching due to feature point matching. Perform subject tracking.

The similarity (Hamming distance D) of the feature amount between the feature points is defined by A, the bit string of the feature amount of the feature point in the subject area, Ai being the element, B being the bit string of the feature amount of the feature point of the entire image, and the element being Bi is represented by Equation (3).

Whether or not feature amounts are similar is determined by comparing the degree of similarity between feature points with a threshold value. The degree of similarity is maximum when the Hamming distance D is 0.

If there are not a plurality of feature amounts similar to the feature amount for a certain feature point in the subject area in the feature amounts for the feature points of the entire image, it is determined that high-precision matching can be achieved by feature point matching, and the process proceeds to S611.

As described above, the image capturing apparatus 100 according to the second embodiment determines whether it is difficult to track a subject by feature point matching based on the evaluation value based on the number of feature points and the feature point intensity, and the similarity between the feature amount of the subject and the entire image. In this case, it is possible to suppress deterioration in tracking accuracy by performing subject tracking using an existing method that does not use feature point matching.

It can be widely used in general imaging apparatuses equipped with a tracking circuit based on feature point matching and other tracking circuits.

100 imaging device 110 image processing unit

Claims (6)

a detecting means for detecting a subject from an image captured by the imaging means;
feature point detection means for detecting feature points from the area corresponding to the subject detected by the detection means;
evaluation value calculation means for calculating an evaluation value based on the number of feature points detected by the feature point detection means, the feature point intensity, or both;
a first tracking unit that tracks the subject detected by the detection unit using feature point matching based on the feature points detected by the feature point detection unit;
a second tracking means for tracking the subject detected by the detection means using a tracking method different from that of the first tracking means;
and control means for switching between the first tracking means and the second tracking means based on the evaluation value calculated by the evaluation value calculation means. 2. The image processing apparatus according to claim 1, wherein said control means selects said first tracking means when said evaluation value is greater than a preset threshold. The control means is
When the evaluation value is within a preset range, the first tracking means and the second 2. The image processing apparatus according to claim 1, wherein said tracking means is switched. 4. The image processing apparatus according to claim 1, wherein said second tracking means performs tracking processing using template matching. 4. The image processing apparatus according to claim 1, wherein said second tracking means performs tracking processing using histogram matching. a detection step of detecting a subject from an image captured by the imaging means;
a feature point detection step of detecting feature points from the area corresponding to the subject detected by the detection means;
an evaluation value calculation step of calculating an evaluation value based on the number of feature points detected by the feature detection point means, the strength of the feature points, or both;
a first tracking step of tracking the subject detected by the detection means using feature point matching based on the feature points detected by the feature point detection means;
a second tracking step of tracking the subject detected by the detection means using a tracking method different from that of the first tracking means;
and a control step of switching between the first tracking means and the second tracking means based on the evaluation value calculated by the evaluation value calculating means.

Images