JP2024056441A - Image processing device, method and program for controlling the image processing device - Google Patents

Abstract

[Problem] To provide an image processing device, a control method for an image processing device, and a program capable of improving the accuracy of tracking a subject when the subject moves quickly. [Solution] An image processing device 101 for tracking a specific subject in successive input images includes a reference image registration circuit 202 for registering a reference image corresponding to the specific subject, a correlation degree calculation circuit 203 for calculating the degree of correlation between each of a plurality of partial regions 304 set in the input image and the reference image, a distance calculation circuit 204 for calculating the distance from a predetermined reference position in the input image for each of the partial regions 304, an evaluation value calculation circuit 205 for calculating an evaluation value for each of the partial regions 304 using the correlation degree and the distance, and a tracking processing control circuit 206 for determining an area including the specific subject among the plurality of partial regions 304 based on the evaluation value. The evaluation value calculation circuit 205 changes the degree to which the distance contributes to the evaluation value when calculating the evaluation value according to the frame rate when the correlation degree calculation circuit 203 calculates the correlation degree. [Selected Figure] FIG.

Description

The present invention relates to an image processing device, a control method for an image processing device, and a program.

Technology that detects a specific subject in a frame of image that is supplied sequentially in a time series and tracks the detected subject is very useful, and is used, for example, to identify human face areas and human body areas in moving images. Such technology can be used in many fields, such as telephone conferences, man-machine interfaces, security, monitor systems for tracking arbitrary subjects, and image compression.

In addition, in digital still cameras and digital video cameras, a technique is known for extracting and tracking an arbitrary subject included in a captured image to optimize the focus and exposure of the subject. For example, Patent Document 1 discloses an image processing device that detects (extracts) and tracks the position of a face included in a captured image, focuses on the face, and captures the image with optimal exposure. At this time, tracking the detected face enables stable control over time series. Patent Document 2 discloses a tracking process that detects a face detected in a certain frame in a subsequent frame. As a method for tracking a specific subject in subsequent frames, a method that uses a template matching technique, as disclosed in Patent Document 2, is known. Template matching is a technique in which a partial image obtained by cutting out an image area including a specific subject to be tracked is registered as a reference image (template image), and the area with the highest correlation with the reference image is calculated to track the specific subject.

JP 2005-318554 A JP 2001-060269 A

In a subject tracking method using template matching, a subject is tracked based on the degree of correlation between a frame image for tracking the subject and a reference image (template image). Here, if there is an area in a frame image that is similar to the reference image but different from the subject to be tracked, that area (hereinafter referred to as a "similar area") may be erroneously detected as the subject. This problem is particularly likely to occur when the appearance of the subject in the frame image has changed from the reference image. Therefore, it is possible to introduce an assumption that the position of the subject does not change significantly between two chronologically consecutive frame images, and to consider an area with a large moving distance between images among multiple areas with high correlation as not including the subject. Hereinafter, an area including the subject is referred to as a "subject area". This will reduce the possibility of erroneously detecting a similar area as the subject area when the subject moves slowly. However, when the subject moves quickly, the position of the subject area changes significantly between two chronologically consecutive frame images, so the introduction of the above assumption may actually increase the possibility of erroneous detection when the subject moves quickly.

The present invention has been made in consideration of the above problems. It is an object of the present invention to provide an image processing device, a control method for the image processing device, and a program that can improve the accuracy of subject tracking when the subject is moving quickly.

In order to achieve the above object, the image processing device of the present invention is an image processing device that tracks a specific subject across multiple input images that are sequentially supplied, and is characterized in that it comprises: a registration means for registering a reference image corresponding to the specific subject; a correlation calculation means for calculating a degree of correlation between each of multiple partial regions set in the input image and the reference image; a distance calculation means for calculating a distance from a predetermined reference position in the input image for each of the multiple partial regions; an evaluation calculation means for calculating an evaluation value for each of the multiple partial regions using the degree of correlation and the distance; a determination means for determining an area that includes the specific subject among the multiple partial regions based on the evaluation value; and a change means for changing the degree to which the distance contributes to the evaluation value when the evaluation calculation means calculates the evaluation value, depending on the frame rate, the type of the specific subject, or the speed of the specific subject when the correlation calculation means calculates the degree of correlation.

The present invention can improve the accuracy of tracking a subject when the subject is moving quickly.

1 is a block diagram showing a schematic configuration of an image processing apparatus according to a first embodiment. FIG. 2 is a block diagram showing a configuration of a subject tracking circuit in the first embodiment. FIG. 11 is a diagram for explaining template matching. FIG. 11 is a diagram showing the relationship between distance and gain (GAIN) in the first embodiment. 5 is a flowchart showing a subject tracking process in the first embodiment. FIG. 11 is a diagram showing the relationship between distance and gain (GAIN) in the second embodiment. 10 is a flowchart showing a subject tracking process in the second embodiment. FIG. 13 is a block diagram showing a configuration of a subject tracking circuit according to a third embodiment. 13 is a flowchart showing a subject tracking process in the third embodiment.

Each embodiment of the present invention will be described in detail below with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the embodiments. For example, each part constituting the present invention can be replaced with any configuration that can perform a similar function. In addition, any component may be added. In addition, any two or more configurations (features) of each embodiment can be combined. Note that the same components are given the same reference numerals throughout the drawings, and their descriptions may be simplified or omitted.

First Embodiment
The first embodiment will be described below with reference to Figs. 1 to 5. Fig. 1 is a block diagram showing a schematic configuration of an image processing device 101 according to the first embodiment. The image processing device 101 is embodied as a digital still camera or a digital video camera that captures an image of a subject. The image processing device 101 also functions as a subject tracking device that tracks a subject included in images that are sequentially supplied in a time series. The image processing device 101 has an optical system 102 such as a lens, an image sensor 103, an analog signal processing circuit 104, an A/D converter 105, a control circuit 106, an image processing circuit 107, a display 108, a recording medium 109, a subject designation unit 110, and a subject tracking circuit 111.

Light representing the image of the subject is collected by the optical system 102 and enters the image sensor 103, which is composed of a CCD image sensor, a CMOS image sensor, or the like. The image sensor 103 outputs an electrical signal corresponding to the intensity of the incident light on a pixel-by-pixel basis. That is, the image sensor 103 photoelectrically converts the image of the subject formed by the optical system 102. The electrical signal output from the image sensor 103 is an analog video signal representing the image of the subject captured by the image sensor 103. The video signal output from the image sensor 103 is subjected to analog signal processing such as correlated double sampling (CDS) in the analog signal processing circuit 104. The video signal output from the analog signal processing circuit 104 is converted into a digital data format by the A/D converter 105 and input to the control circuit 106 and the image processing circuit 107.

The control circuit 106 is a CPU (Central Processing Unit) or a microcontroller, and controls the operation of the image processing device 101. Specifically, the control circuit 106 controls each part of the image processing device 101 by expanding a program stored in a ROM (Read Only Memory) into a working area of a RAM (Random Access Memory) and executing the program sequentially.

The control circuit 106 controls the shooting conditions such as the focus and exposure conditions when capturing an image with the image sensor 103. Specifically, the control circuit 106 controls the focus control mechanism and exposure control mechanism (neither shown) of the optical system 102 based on the video signal output from the A/D converter 105. For example, the focus control mechanism is an actuator that drives a lens included in the optical system 102 in the optical axis direction, and the exposure control mechanism is an actuator that drives an aperture and a shutter. The control circuit 106 also controls the readout of the image sensor 103, such as the output timing and output pixels of the image sensor 103.

The image processing circuit 107 performs image processing such as gamma correction and white balance processing on the video signal output from the A/D converter 105. In addition to normal image processing, the image processing circuit 107 also has a function of performing image processing using information on the subject area in the image supplied from the subject tracking circuit 111 described later. The video signal output from the image processing circuit 107 is sent to the display 108. The display 108 is composed of, for example, an LCD or an organic EL display, and displays the video signal. Therefore, images captured sequentially in time series by the image sensor 103 are displayed sequentially on the display 108. This causes the display 108 to function as an electronic viewfinder (EVF). The display 108 also displays the subject area including the subject being tracked by the subject tracking circuit 111 as a rectangle or the like. The video signal output from the image processing circuit 107 is also recorded on the recording medium 109 (for example, a removable memory card, etc.). The video signal may be recorded in the built-in memory of the image processing device 101 or in an external device (not shown) connected to enable communication via a communication interface.

The subject designation unit 110 is an input interface, such as a touch panel provided on the display 108, or keys and buttons provided on the housing of the image processing device 101. The user (photographer) can designate the subject to be tracked by designating the area of the desired subject from the video signal displayed on the display 108, for example, using the subject designation unit 110. There are no particular limitations on the method of designating an arbitrary area from within an image using a touch panel, keys, buttons, etc., and any well-known method can be used.

The subject tracking circuit 111 tracks subjects included in images sequentially supplied in a time series (i.e., images captured at different times) from the image processing circuit 107. The subject tracking circuit 111 has a subject detection circuit that detects a specific subject, such as face detection, and tracks the detected subject. The subject tracking circuit 111 may also estimate a subject area from images sequentially supplied based on the pixel pattern of the subject specified by the subject specification unit 110. Details of the subject tracking circuit 111 will be described later. The control circuit 106 can use the subject area information supplied from the subject tracking circuit 111 to control the focus control mechanism and exposure control mechanism described above. Specifically, the control circuit 106 performs focus control using the contrast value of the subject area and exposure control using the luminance value of the subject area. This allows the image processing device 101 to perform imaging that takes into account a specific subject area in the captured image.

Here, the subject tracking circuit 111 will be described in detail. The subject tracking circuit 111 functions as a matching means. In other words, the subject tracking circuit 111 functions as a matching means that uses a partial image showing the subject to be tracked as a template, compares it with a partial area of the supplied image, changes the partial area to be compared, and estimates an area with a high degree of correlation. Hereinafter, such a matching means will be referred to as "template matching". FIG. 2 is a block diagram showing the configuration of the subject tracking circuit 111. The subject tracking circuit 111 is composed of a subject detection circuit 201, a reference image registration circuit 202, a correlation calculation circuit 203, a distance calculation circuit 204, an evaluation value calculation circuit 205, and a tracking processing control circuit 206. Each block from the subject detection circuit 201 to the tracking processing control circuit 206 is connected by a bus, and data can be exchanged.

The subject detection circuit 201 detects and identifies a subject to be tracked from the supplied image. A typical subject to be tracked is, for example, a person's face. In this case, the subject detection circuit 201 identifies the face area of the person as the subject area and tracks the face area of the person. For example, when the detection target is a person's face, a publicly known face detection method may be used as the subject detection method in the subject detection circuit 201. Publicly known face detection techniques include a method that uses knowledge about the face (skin color information, parts such as the eyes, nose, and mouth) and a method that uses a learning algorithm such as a neural network to configure a classifier for face detection. In addition, in face detection, it is common to combine these methods to perform face recognition in order to improve the recognition rate. Such face detection methods include, for example, a method of detecting faces using wavelet transformation and image features. The reference image registration circuit 202 (registration means) registers a partial image showing the subject to be tracked as a reference image (template). The correlation calculation circuit 203 (correlation calculation means) compares the template registered by the reference image registration circuit 202 with a partial area of the supplied image, and estimates areas with high correlation by changing the partial area to be compared (template matching).

Template matching will be described in detail with reference to Fig. 3. Fig. 3(a) is a diagram showing an example of a reference image in template matching. A template 301 is a partial image (reference image) showing a subject to be tracked, and a pixel pattern of this partial image is treated as a feature. A feature 302 expresses the feature of each coordinate of a plurality of regions in the template 301, and in the first embodiment, the luminance signal of the pixel data is used as the feature. The feature T(i,j) is expressed by Equation (1), where the coordinates in the region of the template 301 are (i,j), the number of horizontal pixels is W, and the number of vertical pixels is H.
T(i,j)={T(0,0), T(1,0), ..., T(W-1,H-1)} ... (1)

FIG. 3B is a diagram showing information of an image for searching for a tracking target. 303 is a search image within a range in which template matching processing is performed. In the following, the search image 303 may be referred to as an "input image". Coordinates in the search image 303 are expressed as (x, y). A partial region 304 is a region for acquiring an evaluation value of template matching. A feature amount 305 expresses the feature amount of the partial region 304, and similarly to the template 301, the luminance signal of the image data is used as the feature amount. The feature amount S(i, j) is expressed by Equation (2), where the coordinates in the partial region 304 are (i, j), the number of horizontal pixels is W, and the number of vertical pixels is H.
S(i,j)={S(0,0), S(1,0), ..., S(W-1,H-1)} ... (2)

In the first embodiment, a sum of absolute differences (SAD) is used as a calculation method for evaluating the degree of difference between the template 301 and the partial region 304. The SAD value is calculated by the following formula (3).

The correlation calculation circuit 203 calculates the SAD value V(x, y) while shifting the partial region 304 one pixel at a time from the top left of the search image 303. The coordinates (x, y) at which the calculated V(x, y) has the minimum value indicate the position most similar to the template 301. In other words, the position showing the minimum value is the position in the search image 303 where the target tracking target is likely to exist. In the first embodiment, one-dimensional information of the luminance signal is used as the feature amount, but three-dimensional information such as lightness, hue, and saturation signals may be treated as the feature amount. In addition, although the SAD value has been described as a method for calculating the evaluation value of template matching, a different calculation method such as normalized cross correlation (NCC) may be used.

Among the SAD values obtained by template matching, the smallest SAD value has the highest correlation, so in the first embodiment, the degree of correlation is calculated as the reciprocal of the SAD value as in equation (4).
Correlation degree (x, y) = 1 / V (x, y) (4)

Returning to FIG. 2, the distance calculation circuit 204 (distance calculation means) calculates the distance to a predetermined reference position in the input image. The predetermined reference position is the position of the subject area specified by the subject designation unit 110 or identified by the subject detection circuit 201 in the input image immediately before subject tracking was performed. Alternatively, the predetermined reference position may be the position of the subject area in the current input image predicted based on past movement information of the subject area by subject tracking. Here, the predetermined reference position, i.e., the position of the subject area, can be set arbitrarily, and may be, for example, the position of the center of gravity of the subject area or the position of one vertex of the subject area. Note that, hereinafter, the distance to the predetermined reference position in the input image may be referred to as the distance calculated by the distance calculation circuit 204, or simply as the distance.

The evaluation value calculation circuit 205 (evaluation calculation means) calculates an evaluation value for each partial region 304 in the search image 303 by using the evaluation value as a function value of the correlation calculated by the correlation calculation circuit 203 and the distance calculated by the distance calculation circuit 204. As described above, if the evaluation value is calculated based only on the correlation, when a region (similar region) with similar characteristics to the subject region (reference image) exists in addition to the subject region, the similar region may be erroneously detected as the subject region. In particular, if the appearance of the subject changes between the time when the subject region used as the reference image is registered and the time when the input image to be tracked is captured, the similar region may have a higher correlation with the reference image than the subject region. In other words, when there is a high possibility that a similar region exists, the reliability of the correlation calculated by the correlation calculation circuit 203 decreases.

Therefore, in the first embodiment, the degree to which distance contributes to the evaluation value is determined using the frame rate when the correlation calculation circuit 203 calculates the correlation. The frame rate when calculating the correlation may be the imaging rate output from the image sensor 103, or may be a rate at which the input images sequentially supplied from the image sensor 103 are periodically thinned out. This assumes a case in which the frame rate when calculating the correlation is switched depending on the shooting conditions. For example, during recording of a moving image or continuous shooting of still images, the imaging rate and the frame rate when calculating the correlation are the same, but in shooting conditions where no recording is performed and only the display image is displayed on the display 108, it is assumed that the imaging rate is thinned out in order to reduce power consumption.

The higher the frame rate when calculating the correlation, the smaller the amount of movement of the subject becomes relative to the object, and the greater the contribution of distance to the evaluation value. This makes it easier to determine the object area from an area that is close to the nearest object area, and reduces the possibility of erroneous detection even when there is a high possibility of a similar area existing. In contrast, when the frame rate when calculating the correlation is low, the amount of movement of the subject becomes relatively large, and the greater the contribution of distance to the evaluation value becomes. This makes it easier to determine an area with a high correlation as the object area even if it is far from the nearest object area, and allows the object area to be detected with high accuracy even when the object moves quickly.

In this way, in the first embodiment, the evaluation value of the partial region 304 is calculated taking into account both the degree of correlation and the distance from the nearest subject region, and the degree to which the distance from the nearest subject region is considered is dynamically changed depending on the frame rate when the degree of correlation is calculated. Therefore, when the frame rate is high, it is possible to track fast-moving subjects while suppressing false detections in locations where similar regions exist. Furthermore, when the frame rate is low, the degree to which distance contributes to the evaluation value is reduced, making it possible to track fast-moving subjects in the same way as when the frame rate is high.

The evaluation value calculation circuit 205 calculates an evaluation value for each partial region 304 as a function value of the correlation calculated by the correlation calculation circuit 203 and the distance calculated by the distance calculation circuit 204, for example, by the following equation (5).
Evaluation value = correlation degree × GAIN (5)

Here, GAIN is calculated under the following three conditions:
If distance≦α, GAIN=1.0
If α<distance≦β, GAIN=1.0-(distance from nearest subject area-α)×γ
If β<distance, then GAIN=0.1

Figure 4 is a diagram showing the relationship between distance and gain (GAIN) in the first embodiment. In Figure 4, the horizontal axis is distance, and the vertical axis is GAIN. Figure 4 (a) is a diagram showing the relationship between distance and gain (GAIN) in the initial state. In the initial state, the frame rate when calculating the correlation is assumed to be 60 fps. α, β, and γ are coefficients, and α = 20, β = 60, and γ = 0.0225. The distance is the difference value between the position (x, y) of the subject area and the reference position (bx, by). The unit of distance is [pix]. If distance ≦ α, GAIN is set to 1.0. In the initial state, it is considered that the subject may move up to a distance of 20 [pix] per frame. If the distance is α < distance ≦ β, the GAIN is linearly reduced to reduce the evaluation value of the distant subject. If the distance is β < distance, there is a possibility that the subject is moving unexpectedly at high speed, so GAIN is not set to 0 but has a lower limit of 0.1.

Figure 4(b) is a diagram showing the relationship between distance and gain (GAIN) when the frame rate when calculating the correlation is high. In Figure 4(b), the frame rate when calculating the correlation is assumed to be 120 fps. α = 10, β = 30, and γ = 0.045. By setting α = 10, the distance range where GAIN is 1.0 is halved compared to the initial state of Figure 4(a). Also, by setting β = 30, the distance range where GAIN is 0.1 is also halved compared to the initial state of Figure 4(a). This is because it is believed that the amount of movement of the subject will be relatively small due to the high frame rate.

Figure 4(c) is a diagram showing the relationship between distance and gain (GAIN) when the frame rate when calculating the correlation is low. In Figure 4(c), the frame rate when calculating the correlation is assumed to be 30 fps. α = 30, β = 90, and γ = 0.015. By setting α = 30, the distance range where GAIN is 1.0 is increased by 1.5 times compared to the initial state of Figure 4(a). Also, by setting β = 90, the distance range where GAIN is 0.1 is also increased by 1.5 times compared to the initial state of Figure 4(a). This is because it is considered that the amount of movement of the subject will be relatively large due to the low frame rate.

The specific relationship between the frame rate and the coefficients α, β, and γ when calculating the correlation degree can be experimentally determined in advance. The evaluation value calculation circuit 205 may have a table that associates the frame rate with the coefficients α, β, and γ, or may have a function formula that obtains the coefficients α, β, and γ by substituting the frame rate.

Returning to FIG. 2, the tracking process control circuit 206 is composed of a CPU, ROM, RAM, etc., and controls the subject tracking process. Specifically, the tracking process control circuit 206 controls the subject tracking process by expanding a program stored in the ROM into the working area of the RAM and executing it sequentially. As a result, processing is executed via the tracking process control circuit 206 from the subject detection circuit 201 to the evaluation value calculation circuit 205. The tracking process control circuit 206 (determination means) determines the partial area 304 with the highest evaluation value calculated by the evaluation value calculation circuit 205 as the subject area.

Figure 5 is a flowchart showing the subject tracking process in the first embodiment. The process in Figure 5 (control method of the image processing device) is realized by the CPU (computer) of the tracking process control circuit 206 expanding a program stored in ROM into RAM, executing it, and controlling the subject detection circuit 201 to the evaluation value calculation circuit 205. In step S501, the subject detection circuit 201 reads the input image in frame t = 0, performs subject detection processing such as face detection processing, extracts the subject area, and obtains the subject detection result. In step S502, the tracking process control circuit 206 generates an initial reference image from the subject detection result of step S501, and registers it in the reference image registration circuit 202 (registration process).

In step S503, the evaluation value calculation circuit 205 (changing means) determines the degree to which the distance contributes to the evaluation value according to the frame rate when the correlation degree is calculated (changing step). Specifically, the evaluation value calculation circuit 205 determines the coefficients α, β, and γ to calculate GAIN in equation (5). In step S504, the correlation degree calculation circuit 203 reads the input image in the next frame t=1. Furthermore, the correlation degree calculation circuit 203 performs template matching processing between the partial area 304 of the input image and the reference image registered in the input image of frame t=0, and calculates the correlation degree with the reference image (correlation calculation step). In step S505, the distance calculation circuit 204 calculates the distance between the position where the correlation degree is calculated and the reference position (distance calculation step). The reference position is the position of the most recently determined subject area (i.e., the position of the reference image in the input image from which the reference image is extracted).

In step S506, the evaluation value calculation circuit 205 uses the correlation calculated in step S504, the distance calculated in step S505, and the coefficients α, β, and γ determined in step S503 to calculate an evaluation value based on formula (5) (evaluation calculation step). In step S507, the tracking processing control circuit 206 determines and extracts the image corresponding to the partial area 304 with the maximum evaluation value from the input image as the subject area (determination step). The tracking processing control circuit 206 also outputs the extracted image to the reference image registration circuit 202. The tracking processing control circuit 206 also outputs information about the determined subject area to the control circuit 106, the image processing circuit 107, and the distance calculation circuit 204. In step S508, the reference image registration circuit 202 updates the reference image based on the subject area extracted in step S507. The updated reference image is used in the template matching process (step S504) of the following next frame.

In step S509, the tracking process control circuit 206 determines whether or not to end the subject tracking process. This determination is made based on whether the power supply of the image processing device 101 has been turned off. If the power supply of the image processing device 101 has not been turned off, that is, if the tracking process control circuit 206 has determined not to end the subject tracking process, the process returns to step S503. As a result, the process is executed from step S503, and the subject tracking process continues. On the other hand, if the power supply of the image processing device 101 has been turned off, that is, if the tracking process control circuit 206 has determined to end the subject tracking process, the flowchart in FIG. 5 ends.

In this way, the image processing device 101 can appropriately track the subject even when the subject's appearance changes over time, such as when the subject's orientation changes, by sequentially updating the reference image based on the extracted subject area. However, in cases where the image processing device 101 does not take into account changes in the subject's appearance over time, it may maintain the initially registered reference image without updating the reference image.

In the first embodiment, for ease of explanation and understanding, the processing steps of the subject tracking process are described as being executed serially, but processing steps that can be executed in parallel may be executed simultaneously. For example, the process of determining the coefficients α, β, and γ (step S503) and the processes of calculating the correlation degree and distance (steps S504 to S505) may be executed in parallel.

As described above, according to the first embodiment, the image processing device 101, which also functions as a subject tracking device, tracks the subject using a reference image representing the subject to be tracked and the degree of correlation with the input image. At that time, the image processing device 101 calculates an evaluation value for each partial region 304 that takes into account the degree of correlation as well as the distance from the most recently determined subject region. Furthermore, when the frame rate when calculating the degree of correlation is high, the image processing device 101 increases the proportion of distance that contributes to the evaluation value, making it easier to determine a partial region 304 with a short distance as a subject region. Furthermore, when the frame rate when calculating the degree of correlation is low, the image processing device 101 reduces the proportion of distance that contributes to the evaluation value, making it easier to determine a partial region 304 with a high degree of correlation as a subject region. Therefore, the image processing device 101 can accurately detect the subject even when the subject moves quickly, and can stably track the subject. In this way, the image processing device 101 improves the accuracy of subject tracking.

Second Embodiment
The second embodiment will be described below with reference to Fig. 6 and Fig. 7. Here, the differences from the first embodiment will be mainly described. Note that the schematic configuration of the image processing device 101 in Fig. 1 and the subject tracking circuit 111 in Fig. 2 described in the first embodiment are also the same in the image processing device 101 according to the second embodiment.

The subject detection circuit 201 in the second embodiment is a subject detection circuit that detects subjects using dictionary data acquired by known machine learning. When the subject detection circuit 201 learns and recognizes subjects in an image in order to detect subjects using dictionary data acquired by machine learning, a method called deep learning is used. A representative method of deep learning is a method called convolutional neural network (hereinafter referred to as CNN). A typical CNN consists of multi-stage calculations. In each stage, a CNN performs a convolution calculation to spatially integrate local features of an image and input them to the neurons in the intermediate layer of the next stage. Furthermore, a CNN performs an operation called pooling or subsampling to compress features in the spatial direction. A CNN can acquire complex feature expressions through such multi-stage feature transformation. Therefore, a CNN can perform highly accurate subject category recognition and subject detection of an image based on features. In machine learning represented by a CNN, an image signal and a teacher signal are learned as a set. As a result of learning, dictionary data, which is a processing parameter for subject detection, is generated. There are various types of dictionary data, such as people, vehicles such as cars and airplanes, and animals such as dogs and birds. CNN detects the target subject by switching the dictionary data to other dictionary data.

Figure 6 is a diagram showing the relationship between distance and gain (GAIN) in the second embodiment. Figure 6 (a) is a diagram showing the relationship between distance and gain (GAIN) in the initial state. In the initial state, it is assumed that the detected subject is a person. Since Figure 6 (a) is the same as Figure 4 (a), the description will be omitted. Figure 6 (b) is a diagram showing the relationship between distance and gain (GAIN) when the detected subject is a car or an airplane. α = 30, β = 90, γ = 0.015. By setting α = 30, the distance range where GAIN is 1.0 is 1.5 times larger than the initial state of Figure 6 (a). Also, by setting β = 90, the distance range where GAIN is 0.1 is 1.5 times larger than the initial state of Figure 6 (a). This is because it is considered that subjects such as cars and airplanes move relatively faster than people.

The specific relationship between the detected object and the coefficients α, β, and γ can be experimentally determined in advance. The evaluation value calculation circuit 205 may have a table that associates the detected object with the coefficients α, β, and γ, or may have a function formula that obtains the coefficients α, β, and γ by substituting values associated with the detected object.

Figure 7 is a flowchart showing the subject tracking process in the second embodiment. The process in Figure 7 (control method of the image processing device) is realized by the CPU (computer) of the tracking process control circuit 206 expanding a program stored in ROM to RAM, executing it, and controlling the subject detection circuit 201 to the evaluation value calculation circuit 205. In step S701, the subject detection circuit 201 (detection means) reads the input image in frame t = 0, detects the subject and its type using dictionary data based on machine learning as described above, extracts the subject area, and obtains the subject detection result. In step S702, the tracking process control circuit 206 generates an initial reference image from the subject detection result of step S701 and registers it in the reference image registration circuit 202 (registration process).

In step S703, the evaluation value calculation circuit 205 (changing means) determines the degree to which the distance contributes to the evaluation value according to the type of subject detected in step S701 (changing process). Specifically, the evaluation value calculation circuit 205 determines the coefficients α, β, and γ to calculate GAIN in equation (5). In step S704, the correlation calculation circuit 203 reads the input image in the next frame t=1. Furthermore, the correlation calculation circuit 203 performs template matching processing between the partial area 304 of the input image and the reference image registered in the input image of frame t=0, and calculates the correlation with the reference image (correlation calculation process). In step S705, the distance calculation circuit 204 calculates the distance between the position where the correlation was calculated and the reference position (distance calculation process). The reference position is the position of the subject area determined most recently (i.e., the position of the reference image in the input image from which the reference image was extracted).

In step S706, the evaluation value calculation circuit 205 uses the correlation calculated in step S704, the distance calculated in step S705, and the coefficients α, β, and γ determined in step S703 to calculate an evaluation value based on formula (5) (evaluation calculation step). In step S707, the tracking processing control circuit 206 determines and extracts the image corresponding to the partial area 304 with the maximum evaluation value from the input image as the subject area (determination step). The tracking processing control circuit 206 also outputs the extracted image to the reference image registration circuit 202. The tracking processing control circuit 206 also outputs information about the determined subject area to the control circuit 106, the image processing circuit 107, and the distance calculation circuit 204. In step S708, the reference image registration circuit 202 updates the reference image based on the subject area extracted in step S707. The updated reference image is used in the template matching process (step S704) of the following next frame.

In step S709, the tracking process control circuit 206 determines whether or not to end the subject tracking process. This determination is made based on whether the power supply of the image processing device 101 has been turned off. If the power supply of the image processing device 101 has not been turned off, that is, if the tracking process control circuit 206 has determined not to end the subject tracking process, the process returns to step S703. As a result, the process is executed from step S703, and the subject tracking process continues. On the other hand, if the power supply of the image processing device 101 has been turned off, that is, if the tracking process control circuit 206 has determined to end the subject tracking process, the flowchart in FIG. 7 ends.

In this way, the image processing device 101 can appropriately track the subject even when the subject's appearance changes over time, such as when the subject's orientation changes, by sequentially updating the reference image based on the extracted subject area. However, in cases where the image processing device 101 does not take into account changes in the subject's appearance over time, it may maintain the initially registered reference image without updating the reference image.

In the second embodiment, for ease of explanation and understanding, the processing steps of the subject tracking process are described as being executed serially, but processing steps that can be executed in parallel may be executed simultaneously. For example, the process of determining the coefficients α, β, and γ (step S703) and the processes of calculating the correlation degree and distance (steps S704 to S705) may be executed in parallel.

As described above, according to the second embodiment, the image processing device 101, which also functions as a subject tracking device, tracks the subject using a reference image representing the subject to be tracked and the degree of correlation with the input image. At that time, the image processing device 101 calculates an evaluation value for each partial region 304 that takes into account the degree of correlation as well as the distance from the most recently determined subject region. Furthermore, the image processing device 101 detects the subject and its type using dictionary data based on machine learning, and when the detected subject is, for example, a person, the image processing device 101 increases the proportion of distance that contributes to the evaluation value, making it easier to determine a partial region 304 with a short distance as the subject region. Furthermore, when the detected subject is, for example, a car or an airplane, the image processing device 101 reduces the proportion of distance that contributes to the evaluation value, making it easier to determine a partial region 304 with a high degree of correlation as the subject region. Therefore, the image processing device 101 can accurately detect the subject even when the subject moves quickly, and can stably track the subject. In this way, the image processing device 101 improves the accuracy of subject tracking.

Third Embodiment
The third embodiment will be described below with reference to Fig. 8 and Fig. 9. The differences from the first embodiment will be mainly described. The schematic configuration of the image processing device 101 in Fig. 1 described in the first embodiment is also the same in the image processing device 101 according to the third embodiment. In the third embodiment, the image processing device 101 stores the past speed of the subject, and determines the degree to which the distance contributes to the evaluation value according to the speed of the subject.

Figure 8 is a block diagram of the object tracking circuit 111. The object tracking circuit 111 is composed of an object detection circuit 801, a reference image registration circuit 802, a correlation calculation circuit 803, a distance calculation circuit 804, an evaluation value calculation circuit 805, a tracking processing control circuit 806, and a speed memory circuit 807. Here, 801 to 806 are the same as 201 to 206 in Figure 2, so the explanation will be omitted. The speed memory circuit 807 (speed memory means) is a circuit that stores the speed of the object determined immediately before. Note that the speed of the object can be calculated from the distance, which is the difference between the object area (reference position) of n-2 and the object area (reference position) of n-1, and the frame rate, for example, if the current frame is n.

The relationship between distance and gain (GAIN) in the second embodiment corresponds to the relationship between distance and gain (GAIN) in the third embodiment. That is, FIG. 6(a) corresponds to a diagram showing the relationship between distance and gain (GAIN) when the calculated speed of the subject is low. FIG. 6(b) corresponds to a diagram showing the relationship between distance and gain (GAIN) when the calculated speed of the subject is high.

The specific relationship between the calculated subject speed and the coefficients α, β, and γ can be experimentally determined in advance. The evaluation value calculation circuit 205 may have a table that associates the calculated subject speed with the coefficients α, β, and γ, or may have a function formula that obtains the coefficients α, β, and γ by substituting a value associated with the calculated subject speed.

Figure 9 is a flowchart showing the subject tracking process in the third embodiment. The process in Figure 9 (control method of the image processing device) is realized by the CPU (computer) of the tracking process control circuit 806 expanding a program stored in ROM into RAM and executing it, and controlling the evaluation value calculation circuit 805 and the speed storage circuit 807 from the subject detection circuit 801. In step S901, the subject detection circuit 801 reads the input image in frame t = 0, performs subject detection processing such as face detection processing, extracts the subject area, and obtains the subject detection result. In step S902, the tracking process control circuit 806 generates an initial reference image from the subject detection result of step S901 and registers it in the reference image registration circuit 802 (registration process).

In step S903, the evaluation value calculation circuit 805 (changing means) determines the degree to which the distance contributes to the evaluation value according to the speed of the subject stored in the speed memory circuit 807 (changing process). Specifically, the evaluation value calculation circuit 805 determines the coefficients α, β, and γ to calculate GAIN in equation (5). As described above, the speed of the subject is calculated from the distance, which is the difference between the n-2 subject area (reference position) and the n-1 subject area (reference position), and the frame rate, assuming that the current frame is n. However, the speed of the subject at t=1 is set to a predetermined value (for example, 0). In step S904, the correlation calculation circuit 803 reads the input image in the next frame t=1. Furthermore, the correlation calculation circuit 803 performs template matching processing between the partial area 304 of the input image and the reference image registered in the input image of frame t=0, and calculates the correlation with the reference image (correlation calculation process).

In step S905, the distance calculation circuit 804 calculates the distance between the position where the correlation degree is calculated and the reference position (distance calculation process). The reference position is the position of the most recently determined object area (i.e., the position of the reference image in the input image from which the reference image is extracted). In step S906, the evaluation value calculation circuit 805 uses the correlation degree calculated in step S904, the distance calculated in step S905, and the coefficients α, β, and γ determined in step S903 to calculate an evaluation value based on equation (5) (evaluation calculation process). In step S907, the tracking processing control circuit 806 determines and extracts the image corresponding to the partial area 304 with the maximum evaluation value from among the input images as the object area (determination process). The tracking processing control circuit 806 also outputs the extracted image to the reference image registration circuit 802. The tracking processing control circuit 806 also outputs information regarding the determined object area to the control circuit 106, the image processing circuit 107, and the distance calculation circuit 804.

In step S908, the tracking processing control circuit 806 calculates the speed of the subject region determined and extracted in step S907, i.e., the speed of the subject, and stores it in the speed memory circuit 807. The stored subject speed can be used in the processing of step S903 of the following frame. In step S909, the reference image registration circuit 802 updates the reference image using the subject region extracted in step S907 as a reference. The updated reference image is used in the template matching processing (step S904) of the following frame.

In step S910, the tracking process control circuit 806 determines whether or not to end the subject tracking process. This determination is made based on whether the power supply of the image processing device 101 has been turned off. If the power supply of the image processing device 101 has not been turned off, that is, if the tracking process control circuit 806 has determined not to end the subject tracking process, the process returns to step S903. As a result, the process is executed from step S903, and the subject tracking process continues. On the other hand, if the power supply of the image processing device 101 has been turned off, that is, if the tracking process control circuit 806 has determined to end the subject tracking process, the flowchart in FIG. 9 ends.

In this way, the image processing device 101 can appropriately track the subject even when the subject's appearance changes over time, such as when the subject's orientation changes, by sequentially updating the reference image based on the extracted subject area. However, in cases where the image processing device 101 does not take into account changes in the subject's appearance over time, it may maintain the initially registered reference image without updating the reference image.

In the third embodiment, for ease of explanation and understanding, the processing steps of the subject tracking process are described as being executed serially, but processing steps that can be executed in parallel may be executed simultaneously. For example, the process of determining the coefficients α, β, and γ (step S903) and the processes of calculating the correlation degree and distance (steps S904 to S905) may be executed in parallel.

As described above, according to the third embodiment, the image processing device 101, which also functions as a subject tracking device, tracks the subject using a reference image representing the subject to be tracked and the degree of correlation with the input image. At that time, the image processing device 101 calculates an evaluation value for each partial region 304 that takes into account the degree of correlation as well as the distance from the most recently determined subject region. Furthermore, the image processing device 101 stores the past speed of the subject, and when the speed is small, the image processing device 101 increases the proportion of the distance that contributes to the evaluation value, making it easier to determine a partial region 304 with a short distance as the subject region. Also, when the past speed is large, the image processing device 101 decreases the proportion of the distance that contributes to the evaluation value, making it easier to determine a partial region 304 with a high degree of correlation as the subject region. Therefore, the image processing device 101 can accurately detect the subject even when the subject moves quickly, and stable subject tracking is possible. In this way, the image processing device 101 improves the accuracy of subject tracking.

Note that if the proportion of the subject in the input image (hereinafter referred to as "subject area") is large, for example if the subject is photographed enlarged, the subject may move significantly between frames, and this can be considered similar to the subject speed in the third embodiment. In other words, the speed of the subject in the third embodiment can be treated as a concept that includes the area of the subject.

<Other embodiments>
Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or a recording medium, and having one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

Furthermore, the functional blocks in the diagram can be realized by hardware, software, or a combination of both, but there is no need for a one-to-one correspondence between the functional blocks and the configuration that realizes them. Multiple functional blocks may be realized by a single software or hardware module.

In each of the above-described embodiments, the image processing device 101 has been described as an example of subject tracking. However, as described above, the present invention can be applied to a variety of devices other than image processing devices. For example, when the present invention is applied to a device for reproducing and displaying image data, the device for reproducing and displaying image data can use information on the subject area in the image data (such as the position and size of the subject in the image) to set the reproduction conditions and display conditions of the image data. Specifically, the device for reproducing and displaying image data can superimpose information indicating the subject, such as a frame, on the position of the subject in the image, and control display conditions such as brightness and color so that the subject portion is appropriately displayed according to the brightness and color information of the subject portion.

The disclosure of each embodiment includes the following configurations, methods, and programs.
(Configuration 1) An image processing device for tracking a specific subject across a plurality of input images that are sequentially supplied, comprising:
a registration means for registering a reference image corresponding to the specific subject;
a correlation calculation means for calculating a degree of correlation between the input image and the reference image for each of a plurality of partial regions set in the input image;
a distance calculation means for calculating a distance from a predetermined reference position in the input image to each of the plurality of partial regions;
an evaluation calculation means for calculating an evaluation value for each of the plurality of partial regions using the degree of correlation and the distance;
a determining means for determining an area including the specific subject among the plurality of partial areas based on the evaluation value;
and a change means for changing the degree to which the distance contributes to the evaluation value when the evaluation calculation means calculates the evaluation value, depending on a frame rate, a type of the specific subject, or a speed of the specific subject when the correlation calculation means calculates the correlation degree.
(Configuration 2) The image processing apparatus according to configuration 1, wherein the changing means treats the rate of the input images successively supplied as a frame rate when the correlation calculation means calculates the degree of correlation.
(Configuration 3) The image processing device according to configuration 1, wherein the changing means treats a rate at which the successively supplied input images are periodically thinned out as a frame rate when the correlation calculation means calculates the degree of correlation.
(Configuration 4) The image processing device according to Configuration 1, further comprising a detection means for detecting the specific subject and a type of the specific subject from the initial input image by referring to dictionary data acquired by machine learning.
(Configuration 5) A speed storage means for storing the speed of the specific object in the input image,
The image processing device described in configuration 1, characterized in that the change means changes the degree to which the distance contributes to the evaluation value when the evaluation calculation means calculates the evaluation value for each of the multiple partial areas set in the subsequently input input image, in accordance with the speed of the specific subject stored in the speed storage means.
(Method 1) A method for controlling an image processing device that tracks a specific subject across a plurality of input images that are sequentially supplied, comprising the steps of:
a registration step of registering a reference image corresponding to the specific subject;
a correlation calculation step of calculating a correlation degree between the reference image and each of a plurality of partial regions set in the input image;
a distance calculation step of calculating a distance from a predetermined reference position in the input image to each of the plurality of partial regions;
an evaluation calculation step of calculating an evaluation value for each of the plurality of partial regions using the correlation degree and the distance;
a determining step of determining an area including the specific subject among each of the plurality of partial areas based on the evaluation value;
a change step of changing the degree to which the distance contributes to the evaluation value when the evaluation calculation step calculates the evaluation value, depending on the frame rate, the type of the specific subject, or the speed of the specific subject when the correlation degree is calculated in the correlation calculation step.
(Program 1) A program for causing a computer to execute each unit of the image processing device according to any one of configurations 1 to 5.

101 Image processing device 202, 802 Reference image registration circuit (registration means)
203, 803 Correlation degree calculation circuit (correlation calculation means)
204, 804 Distance calculation circuit (distance calculation means)
205, 805 Evaluation value calculation circuit (evaluation calculation means) (changing means)
206, 806 Tracking processing control circuit (determination means)
301 Template (reference image)
303 Search image (input image)
304 Partial Area

Claims (7)

1. An image processing device for tracking a particular subject across a plurality of input images provided sequentially, comprising:
a registration means for registering a reference image corresponding to the specific subject;
a correlation calculation means for calculating a degree of correlation between the input image and the reference image for each of a plurality of partial regions set in the input image;
a distance calculation means for calculating a distance from a predetermined reference position in the input image to each of the plurality of partial regions;
an evaluation calculation means for calculating an evaluation value for each of the plurality of partial regions using the degree of correlation and the distance;
a determining means for determining an area including the specific subject among the plurality of partial areas based on the evaluation value;
and a change means for changing the degree to which the distance contributes to the evaluation value when the evaluation calculation means calculates the evaluation value, depending on a frame rate, a type of the specific subject, or a speed of the specific subject when the correlation calculation means calculates the correlation degree. The image processing device according to claim 1, characterized in that the change means treats the rate of the input images that are sequentially supplied as a frame rate when the correlation calculation means calculates the correlation degree. The image processing device according to claim 1, characterized in that the change means treats the rate at which the sequentially supplied input images are periodically thinned as the frame rate at which the correlation calculation means calculates the degree of correlation. The image processing device according to claim 1, further comprising a detection means for detecting the specific subject and the type of the specific subject from the initial input image by referring to dictionary data acquired by machine learning. a velocity storage means for storing a velocity of the specific object in the input image;
The image processing device according to claim 1, characterized in that the change means changes the degree to which the distance contributes to the evaluation value when the evaluation calculation means calculates the evaluation value for each of the multiple partial regions set in the subsequently input input image, in accordance with the speed of the specific subject stored in the speed storage means. 1. A method for controlling an image processing device for tracking a specific subject across a plurality of input images that are sequentially supplied, comprising:
a registration step of registering a reference image corresponding to the specific subject;
a correlation calculation step of calculating a degree of correlation between the input image and the reference image for each of a plurality of partial regions set in the input image;
a distance calculation step of calculating a distance from a predetermined reference position in the input image to each of the plurality of partial regions;
an evaluation calculation step of calculating an evaluation value for each of the plurality of partial regions using the correlation degree and the distance;
a determining step of determining an area including the specific subject among each of the plurality of partial areas based on the evaluation value;
a change step of changing the degree to which the distance contributes to the evaluation value when the evaluation calculation step calculates the evaluation value, depending on the frame rate, the type of the specific subject, or the speed of the specific subject when the correlation degree is calculated in the correlation calculation step. 2. A program for causing a computer to execute each of the means of the image processing apparatus according to claim 1.

Images