JP2016224627A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device, method, and program which improve the accuracy of tracking processing even in the case of the occurrence of random deformation and random and non-uniform blur in images constituting a video.SOLUTION: A region setting unit 102 sets a tracking target region from a first image. A contrast calculation unit 103 calculates a contrast value of each of pixels in the tracking target region of the first image. An image smoothing unit 104 executes smoothing processing for each pixel on the basis of the contrast value of the pixel in the tracking target region of the first image. A tracking unit 105 performs tracking to find which position of a smoothed target image a selected region of the smoothed first image is located in.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for tracking an object in an image.

In a moving image obtained by continuously shooting the same scene, the position of the shooting target may change in the image.
In that case, a tracking technique is known that tracks (tracks) in which region the photographic object is located in each image constituting the moving image.

As a tracking technique, a block matching method is known. This is based on the premise that the luminance pattern of the shooting target does not change significantly in each image constituting the moving image. This is a method for obtaining the most consistent position.
As an image processing apparatus using a block matching method, for example, Patent Document 1 estimates how a tracking target point has moved between images constituting a moving image, and the accuracy of a motion vector obtained as a result of the estimation. Describes an image processing apparatus that determines that a tracking target point has moved on the condition that is equal to or greater than a predetermined threshold.

However, in the image processing apparatus described in Patent Document 1, it is assumed that the luminance does not change between images to be subjected to tracking processing when performing tracking processing.
Therefore, for example, in a moving image obtained by photographing a space in which atmospheric density fluctuations such as a hot flame as shown in FIG. 10 are taken, an object to be photographed (in FIG. 10, two lines with horizontal and uniform darkness). In the case of tracking (minute), random deformation and random and non-uniform blur occur in the image constituting the moving image, as shown in the elliptical area of FIG. 10B. As a result, the luminance changes between the images (that is, the luminance of the line and its surroundings changes in FIGS. 10B1 and 10B2), and thus accurate tracking cannot be performed.

  In other words, in the image processing apparatus described in Patent Document 1, when the density of the air is generated in the image constituting the captured moving image (that is, random deformation and random and non-uniform blur are generated. If this occurs, the accuracy of the tracking process cannot be guaranteed.

  The present invention has been made in view of the above-described conventional problems, and the object thereof is even when random deformation and random and non-uniform blur are generated in an image constituting a moving image. It is to improve the accuracy of the tracking process.

  The present invention comprises, from a first image constituting a moving image, an area setting means for setting a tracking target area, a contrast value of each pixel of the tracking target area set by the area setting means, and the moving image. A contrast calculating means for calculating a contrast value of each pixel of the second image to be smoothed, and a first smoothing of the tracking target area based on the contrast value of each pixel of the tracking target area calculated by the contrast calculating means. One smoothing means, second smoothing means for smoothing the second image based on the contrast value of each pixel of the second image calculated by the contrast calculating means, and the first smoothing An area similar to the tracking target area smoothed by the second smoothing means is smoothed by the second smoothing means. And tracking means for tracking the image, an image processing apparatus having a.

  According to the image processing apparatus of the present invention, it is possible to improve the accuracy of tracking processing even when random deformation and random and non-uniform blur occur in an image constituting a moving image.

1 is a schematic block diagram of an image processing apparatus according to an embodiment of the present invention. It is a hardware schematic block diagram of the image processing apparatus which concerns on embodiment of this invention. 1 is a functional block diagram of an image processing apparatus according to an embodiment of the present invention. It is a figure which shows the outline | summary of the whole process in the image processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows the procedure of the whole process in the image processing apparatus which concerns on embodiment of this invention. It is a figure which shows the process in the contrast calculation part of the image processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows the procedure of the process in the contrast calculation part of the image processing apparatus which concerns on embodiment of this invention. It is a figure which shows the process in the image smoothing part of the image processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows the procedure of the process in the image smoothing part of the image processing apparatus which concerns on embodiment of this invention. It is a figure which shows an image when density fluctuations of air | atmosphere, such as a hot flame, generate | occur | produced.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a schematic block diagram of an image processing apparatus 1 according to an embodiment of the present invention.
The image processing apparatus 1 is connected to an external storage device 2 as shown in FIG.

In the image processing apparatus 1, an area to be tracked is specified by a user in an image captured by the imaging unit 12 (FIG. 2) mounted on the image processing apparatus 1 or an image transferred from the external storage device 2. Then, the tracking process is executed.
The image processing apparatus 1 stores the tracking processing execution result in the image processing apparatus 1 and transfers it to the external storage device 2 as necessary.

FIG. 2 is a schematic hardware configuration diagram of the image processing apparatus 1 according to the embodiment of the present invention.
The image processing apparatus 1 includes a control unit 11, a photographing unit 12, a network interface (I / F) 13, a main storage unit 14, a secondary storage unit 15, an input unit 16, and a display unit 17.

The control unit 11 is a processor such as a CPU (Central Processing Unit), and controls each unit by executing a program stored in the main storage unit 14.
The photographing unit 12 is a photographing element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor, and outputs image data when photographing a subject.

The network interface 13 is connected to the image processing apparatus 1 and the external storage device 2 by a wireless communication unit such as a wired communication unit such as a LAN (Local Area Network) cable, Wi-Fi (registered trademark), or Bluetooth (registered trademark). Is an interface for connecting
The network interface 13 is used when the control unit 11 acquires an image from the external storage device 2 or when the tracking processing result is transmitted to the external storage device 2.

The main storage unit 14 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or the like.
The main storage unit 14 stores various images (for example, images taken by the shooting unit 12, images transferred from the external storage device 2, etc.), programs executed in the control unit 11, and results of tracking processing, etc. .

The secondary storage unit 15 is a volatile memory such as a RAM (Random Access Memory).
The secondary storage unit 15 is used when a program is loaded and when a calculation result of the program is temporarily stored.

The input unit 16 is an input device such as a keypad, and is used when a user operates software (that is, when executing a program), specifies a tracking area, or the like.
Note that as the input unit 16, an input device such as a keyboard and a mouse can be connected to the image processing apparatus 1.

The display unit 17 is a display device such as a liquid crystal display (LCD).
The display unit 17 displays various images and software operation screens.

Next, the functional configuration of the image processing apparatus 1 will be described with reference to the drawings.
FIG. 3 is a functional block diagram of the image processing apparatus 1 according to the embodiment of the present invention.
The image acquisition unit 101 acquires an image stored in the main storage unit 14 (that is, an image shot by the shooting unit 12 and an image transferred from the external storage device 2) as an image to be subjected to tracking processing.
The area setting unit 102 sets (selects) an area to be tracked.

The contrast calculation unit 103 calculates the contrast value of each pixel.
The processing in the contrast calculation unit 103 will be described later with reference to FIGS.
The image smoothing unit 104 performs smoothing based on the contrast value calculated by the contrast calculation unit 103.
The processing in the image smoothing unit 104 will be described later with reference to FIGS.

The tracking unit 105 tracks at which coordinate the region set by the region setting unit 102 is located in the other image to be tracked.
Specifically, for example, the histogram of the luminance value of the region selected by the region setting unit 102 and the partial region of the other image to be tracked (that is, the same shape as the region selected by the region setting unit 102) And the histogram of the luminance value of the same size) is calculated, and tracking is performed by block matching using the similarity.
The calculation of the similarity of histograms of luminance values will be described later.

The user can display the execution result of the tracking process stored in the main storage unit 14 on the display unit 17 or transfer it to the external storage device 2.
Note that the image acquisition unit 101, the region setting unit 102, the contrast calculation unit 103, the image smoothing unit 104, and the tracking unit 105 are function implementation means generated by causing the computer of the image processing apparatus 1 to read a program.

FIG. 4 is a diagram showing an overview of the entire processing in the image processing apparatus 1 according to the embodiment of the present invention.
Note that the process shown in FIG. 4 is a process that is started on the assumption that a plurality of images of the same size obtained by photographing the same scene are stored in the main storage unit 14.
Also, FIG. 4 shows that in the first acquired image (that is, the first image acquired), the region set by the user is the next acquired image (that is, the image acquired after the second image). This is an example of a process for tracking which region it has moved to.

Here, the first acquired image is, for example, an image positioned first in time, and this image is referred to as a first image (corresponding to the first image of the present invention). In FIG. 4, image processing relating to the first image is shown on the upper side of FIG.
The next acquired image is an image to be subjected to tracking processing, and this image is referred to as a target image. In FIG. 4, image processing relating to the target image is shown on the lower side of FIG.
The target image corresponds to the second image of the present invention.

When an area to be tracked is set in the first image, the image processing apparatus 1 executes a smoothing process for each pixel according to the contrast value of the pixel for the set area.
An image obtained by performing the smoothing process on the first image by the image smoothing unit 104 is referred to as a first smoothed image.
Moreover, the image smoothing part 104 here respond | corresponds to the 1st smoothing means of this invention.

The image processing apparatus 1 executes a smoothing process on the target image for each pixel according to the contrast value of the pixel.
Note that an image obtained by performing the smoothing process on the target image by the image smoothing unit 104 is referred to as a target smoothed image.
Moreover, the image smoothing part 104 here respond | corresponds to the 2nd smoothing means of this invention.

The image processing apparatus 1 tracks (searches) a region similar to the selected region of the first smoothed image from the target smoothed image.
As described above, by performing the smoothing process on the first image and the target image, the tracking target area in the first image and the area where the tracking target exists in the target image have the same degree of uniform blur. Since it is given (that is, has a similar luminance distribution), tracking robust to deformation of the tracking target image can be realized by searching the target image for regions having similar luminance values.

FIG. 5 is a flowchart showing a procedure of the entire processing in the image processing apparatus 1 according to the embodiment of the present invention.
FIG. 5 shows an example of a processing procedure for tracking which region set (selected) by the user in the first image has moved to which region in the target image.
The process shown in FIG. 5 is a process that is executed on the assumption that the target image, that is, the image to be tracked is stored in the main storage unit 14.

The image acquisition unit 101 acquires a first image (S101).
The area setting unit 102 sets an area selected by the user in the first image as a tracking target area (S102).
Here, the tracking target area can be selected by an arbitrary shape such as a rectangle or a circle, or by freehand.

The contrast calculation unit 103 calculates the contrast value of each pixel of the first image (S103).
The processing in the contrast calculation unit 103 will be described later with reference to FIGS.
Further, the contrast value calculation processing in step S103 is executed for the region selected in step S102.

The image smoothing unit 104 performs a smoothing process for each pixel on the first image based on the contrast value of each pixel of the first image (S104).
The processing in the image smoothing unit 104 will be described later with reference to FIGS.
In addition, the smoothing process in step S104 is executed for the region set in step S102.

The image acquisition unit 101 determines whether an image to be subjected to tracking processing is stored in the main storage unit 14 (S105).
If the image acquisition unit 101 determines that an image to be tracked is stored (S105 Yes), the image acquisition unit 101 acquires the image (that is, the target image) (S106).
If it is determined in step S105 that the target image is not stored (No in S105), the image processing apparatus 1 ends the process.

The contrast calculation unit 103 calculates the contrast value of each pixel of the target image (S107).
Note that the contrast value calculation processing in step S107 is executed for the entire image acquired in step S106.

The image smoothing unit 104 performs a smoothing process for each pixel on the target image based on the contrast value of each pixel of the target image (S108).
Note that the smoothing process in step S108 is executed for the entire image acquired in step S106.

The tracking unit 105 tracks where in the target smoothed image the selected region of the first smoothed image is located (S109).
When the tracking process ends in step S109, the image processing apparatus 1 returns the process to step S105.

FIG. 6 is a diagram showing processing in the contrast calculation unit 103 of the image processing apparatus 1 according to the embodiment of the present invention.
The contrast calculation unit 103 selects a pixel for which a contrast value is to be calculated, and calculates a contrast value around the target pixel.
The contrast calculation unit 103 stores the calculated contrast value at the same position as the target pixel of the contrast array created in advance.

FIG. 7 is a flowchart showing a processing procedure in the contrast calculation unit 103 of the image processing apparatus 1 according to the embodiment of the present invention.
The contrast calculation unit 103 secures a numerical data array having the same size as the pixel of the image whose contrast value is to be calculated on the memory, and inserts “−1” as an initial value (S201).

The contrast calculation unit 103 determines whether there is a pixel for which a contrast value is not calculated for an image for which a contrast value is to be calculated (S202).
If the contrast calculation unit 103 determines that there is a pixel for which a contrast value has not been calculated (S202 Yes), the contrast calculation unit 103 designates (selects) one of the pixels that has not been calculated (S203).

The contrast calculation unit 103 calculates a contrast value around the pixel selected in step S203 (S204).
Specifically, the contrast calculation unit 103 sets the periphery of the pixel as a rectangle having a side of 2 × l + 1 around the pixel (x, y), and sets the contrast value of the target pixel within the rectangle. Is calculated as the standard deviation of the luminance values of the pixels.
The standard deviation is calculated by the following formula. The contrast value can also be calculated from the distribution of luminance values and the total variation of luminance values.

Here, in the above equation, the function f (x + j, y + k) is a function representing the luminance value of the image at the coordinates (x + j, y + k).
Further, the variable μ represents an average of luminance values of an image area (here, a square area having one side of 2 × l + 1 centered on coordinates (x + j, y + k)).

The contrast calculation unit 103 writes the contrast value calculated in step S204 in the corresponding region of the contrast array (that is, the same position as the target pixel) (S205).
Note that the association between the pixel (x, y) and the position i of the contrast array is calculated by the following equation with the width of the image as w.

FIG. 8 is a diagram showing processing in the image smoothing unit 104 of the image processing apparatus 1 according to the embodiment of the present invention.
The image smoothing unit 104 creates a smoothing filter according to the contrast value of a pixel to be smoothed.
The image smoothing unit 104 reads the luminance information of the application target area of the smoothing process and executes the smoothing process.
The image smoothing unit 104 stores the brightness value calculated by executing the smoothing process at the same position as the target pixel of the smoothed image created in advance.

FIG. 9 is a flowchart showing a processing procedure in the image smoothing unit 104 of the image processing apparatus 1 according to the embodiment of the present invention.
In step S201 (FIG. 7), the image smoothing unit 104 generates an empty smoothed image having the same size as the image for which the contrast value is to be calculated (S301).

The image smoothing unit 104 determines whether there is a pixel that has not been subjected to the smoothing process with respect to the image to be smoothed (S302).
If the image smoothing unit 104 determines that there is a pixel that has not been subjected to the smoothing process (S302 Yes), the image smoothing unit 104 designates (selects) one of the pixels that has not been subjected to the process (S303).

The image smoothing unit 104 acquires the contrast value of the pixel selected in step S303 from the contrast array generated in step S205 (FIG. 7) (S304).
Note that the pixel (x, y) and the position i of the contrast array are associated with each other by the above-described formula 2.

The image smoothing unit 104 sets the periphery of the pixel as a 2 × l + 1 rectangle (that is, a Gaussian kernel) centered on the pixel (x, y), and a sigma value is a Gaussian filter having a contrast value (that is, a Gaussian filter). , A smoothing filter) is generated (S305).
The sigma value is a value indicating the degree of variation, and the Gaussian filter having a larger sigma value can blur the image after the filtering process.
Although a Gaussian filter is shown here as an example of the smoothing filter, a smoothing filter having a constant value having a size proportional to the contrast value may be generated as the smoothing filter.

The image smoothing unit 104 performs a smoothing process (filtering process) on the pixel specified in step S303 using the smoothing filter generated in step S305 (S306).
The image smoothing unit 104 writes the luminance value, which is the execution result of the smoothing process, at the pixel position specified in step S303 of the empty smoothed image generated in step S301 (S307).

  The image processing apparatus 1 uses the tracking unit 105 to compare the brightness value histogram of the selected region of the first smoothed image with the histogram of the brightness value of the region to be compared with the target smoothed image, and the similarity The tracking processing is executed by block matching using the similarity and the highest similarity is searched for.

Specifically, a function indicating a histogram of luminance values of a selected area of the first smoothed image is represented by HF, where H (k) is a function indicating a histogram of luminance values that returns the number of pixels having a luminance value k. (K) In addition, a function indicating a histogram of luminance values of the comparison target area of the target smoothed image is defined as HT (k), and HF (k) and HT (k when the luminance value k is changed ) To evaluate the similarity.
That is, the similarity of histograms of luminance values can be expressed by the following equation, and the region having the smallest variable r is evaluated as the region having the highest similarity.
Note that Lmin represents the minimum luminance value, and Lmax represents the maximum luminance value.

  As described above, according to the image processing apparatus of the present invention, even when random deformation and random and non-uniform blur occur in an image constituting a moving image, the first acquired image Smoothing processing is performed on the image (that is, the first image) and the image to be tracked (that is, the second image), and the degree of tracking processing is improved by applying the same degree of uniform blur. Can be made.

  DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 2 ... External storage device, 11 ... Control unit, 12 ... Shooting unit, 13 ... Network interface, 14 ... Main storage unit, 15 ... Secondary storage unit, 16 ... Input unit, 17 ... Display unit, 101 DESCRIPTION OF SYMBOLS ... Image acquisition part, 102 ... Area setting part, 103 ... Contrast calculation part, 104 ... Image smoothing part, 105 ... Tracking part.

JP 2005-303983 A

Claims (12)

A region setting means for setting a tracking target region from the first image constituting the moving image;
Contrast calculation means for calculating the contrast value of each pixel of the tracking target area set by the area setting means and the contrast value of each pixel of the second image constituting the moving image;
First smoothing means for smoothing the tracking target area based on the contrast value of each pixel of the tracking target area calculated by the contrast calculating means;
Second smoothing means for smoothing the second image based on the contrast value of each pixel of the second image calculated by the contrast calculating means;
Tracking means for tracking an area similar to the tracking target area smoothed by the first smoothing means from the second image smoothed by the second smoothing means;
An image processing apparatus. The image processing apparatus according to claim 1,
The tracking means tracks an area similar in luminance value to the tracking target area smoothed by the first smoothing means from the second image smoothed by the second smoothing means. . In the image processing apparatus according to claim 1 or 2,
An image processing apparatus, wherein the contrast value is a standard deviation of luminance values within a predetermined range around a pixel to be calculated by the contrast calculation means. In the image processing apparatus according to claim 1 or 2,
An image processing apparatus, wherein the contrast value is a variance of luminance values within a predetermined range around a pixel to be calculated by the contrast calculation means. In the image processing apparatus according to claim 1 or 2,
An image processing apparatus, wherein the contrast value is a total variation of luminance values within a predetermined range around a pixel to be calculated by the contrast calculation means. The image processing apparatus according to any one of claims 1 to 5,
An image processing apparatus in which the tracking means performs tracking by block matching based on similarity of histograms of luminance values. The image processing apparatus according to any one of claims 1 to 6,
The first smoothing means is a smoothing filter having a size corresponding to the contrast value of each pixel of the tracking target area, and the second smoothing means is a size corresponding to the contrast value of each pixel of the second image. An image processing apparatus that is a smoothing filter. The image processing apparatus according to any one of claims 1 to 6,
The first smoothing means is a Gaussian filter using a Gaussian kernel corresponding to the contrast value of each pixel of the tracking target area, and uses a Gaussian kernel corresponding to the contrast value of each pixel of the second image. An image processing apparatus that is a filter. The image processing apparatus according to any one of claims 1 to 8,
An image processing apparatus in which the tracking target area set by the area setting unit is an area set freehand or in a predetermined shape by a user. An area setting step for setting a tracking target area from the first image constituting the moving image by the area setting means;
A first contrast calculation step of calculating a contrast value of each pixel of the tracking target region set in the region setting step by a contrast calculation unit;
A first smoothing step of smoothing the tracking target region based on a contrast value of each pixel of the tracking target region calculated in the first contrast calculating step by the first smoothing unit;
A second contrast calculating step of calculating a contrast value of each pixel of the second image constituting the moving image by the contrast calculating means;
A second smoothing step of smoothing the second image based on the contrast value of each pixel of the second image calculated by the second smoothing means in the second contrast calculating step;
A tracking step for tracking a region similar to the tracking target region smoothed in the first smoothing step by the tracking unit from the second image smoothed in the second smoothing step;
An image processing method. The image processing method according to claim 10,
In the tracking step, an image processing method for tracking a region similar in luminance value to the tracking target region smoothed in the first smoothing step from the second image smoothed in the second smoothing step . The computer of the image processing device
A region setting means for setting a tracking target region from the first image constituting the moving image;
Contrast calculation means for calculating the contrast value of each pixel of the tracking target area set by the area setting means and the contrast value of each pixel of the second image constituting the moving image;
First smoothing means for smoothing the tracking target area based on the contrast value of each pixel of the tracking target area calculated by the contrast calculation means;
Second smoothing means for smoothing the second image based on the contrast value of each pixel of the second image calculated by the contrast calculating means, and smoothed by the first smoothing means A program for causing a region similar to the tracking target region to function as tracking means for tracking from the second image smoothed by the second smoothing means.

Images