JP2005250576A - Moving object extracting device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily extract a moving object whose image is picked up in an image. <P>SOLUTION: At the time of inputting moving image data, a moving object extracting device calculates a difference between the pixel values of the corresponding pixels of an extraction object frame and a comparison object frame, and specifies a region where the value is a predetermined threshold T1 or more as a differential region. Then, the moving extracting device calculates a relative frequency distribution Nz of the pixel values of the overall extraction object frame and a relative frequency distribution Nd of the pixel values in the differential region, and calculates a rate R of them. The moving object extracting device estimates the pixel values belonging to a class where the rate R is a predetermined threshold T2 or more as a moving object constituting color. Then, the pixels pertinent to the moving object constituting color are extracted from the extraction object frame so that the moving object can be extracted. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image processing technique for extracting a moving object from an image.

  Conventionally, various studies have been made on techniques for separating a background and a subject from an image. For example, Patent Document 1 discloses a method of extracting a subject from an image based on a specified color and texture, or extracting a subject from a moving image by obtaining a motion vector of each block after dividing the image into a plurality of blocks. How to do is described. Patent Document 2 describes a technique for separating a background and a subject by taking a difference between a still image in which both the background and the subject are captured and a still image in which only the background is captured.

JP 2002-24834 A JP 2002-327992 A

  However, in the technique described in Patent Document 1, it is necessary to specify the color and texture of the subject in advance in order to extract the subject. In the technique described in Patent Document 2, it is necessary to prepare an image in which only the background is captured in advance. That is, these techniques require troublesome preparations when extracting a subject from an image.

  The present invention has been made to solve such a problem, and an object of the present invention is to provide a technique that can easily extract a moving subject (moving object) captured in an image.

In order to solve the above problems, the present invention is configured as a moving object extraction apparatus as follows. That is, a moving object extraction device that extracts a moving object from an image,
An input unit for inputting two images in which pixel values are recorded for each pixel;
A difference area specifying unit that calculates a difference between pixel values for each corresponding pixel of the two images and specifies an area where the difference is equal to or greater than a first threshold as a difference area;
A first distribution is obtained with a predetermined class width for the pixel values of the entire image of at least one image of the two images, and further, the pixel values in the area corresponding to the difference area of the image are calculated with the class width. A statistical processing unit for obtaining a second distribution;
A ratio of the second distribution to the first distribution is obtained for each class, and a pixel value belonging to a class in which the ratio is equal to or greater than a second threshold is estimated to be a moving object constituent color constituting the moving object. The gist is to include a moving object configuration color estimation unit.

  When two images that are closely related in time series are compared, it is highly likely that the portion where the pixel value hardly changes between the two images is the background, and the moving object is captured in the portion where the pixel value changes greatly. It can be said that there is a high possibility of being a part. Therefore, it can be considered that the difference area includes many pixels constituting the moving object. The moving object extraction apparatus of the present invention obtains a ratio of the distribution of pixel values in the difference area, which is considered to include a large number of pixels constituting the moving object, and the distribution of pixel values in the entire image. Is estimated.

  According to such a moving object extraction apparatus, it is possible to estimate the pixel values constituting the moving object simply by preparing two images. Therefore, the moving object can be easily extracted from the image without making troublesome preparation as in the prior art. It becomes possible to extract. The two images input by the moving object extraction apparatus may be any images. For example, two consecutive image frames constituting moving image data, two images taken using the continuous shooting function of a digital still camera, etc. It is good also as what inputs. Since these images are closely related in time series, it is possible to extract a moving object effectively.

  The statistical processing unit may obtain two distributions using only one of the two input images, or may obtain two distributions using both. If both are used, the number of samples for which statistics are obtained increases, so that two distributions can be obtained satisfactorily.

  The moving body constituent color estimation unit obtains the ratio of the second distribution to the first distribution, and obtains the ratio of the first distribution to the second distribution for each class, A pixel value belonging to a class in which the ratio is equal to or less than the second threshold value may be estimated as a moving object constituent color constituting the moving object.

In the moving object extraction apparatus having the above configuration,
The first distribution and the second distribution obtained by the statistical processing unit may be a relative frequency distribution.

  In this way, if the relative frequency obtained by dividing the frequency by the number of samples is used, the tendency of the distribution of pixel values in the difference area and the entire image area can be compared on the same scale. Can be suitably estimated.

In the moving object extraction apparatus having the above configuration,
The statistical processing unit obtains the first and second distributions for each of Y, U, and V components of pixel values,
The moving object composition color estimation unit may estimate the moving object composition color for each of the Y, U, and V components.

  The Y component refers to the luminance component of the pixel value, the U component refers to the difference between the luminance component and the red component, and the V component refers to the difference between the luminance component and the blue component. In this way, if the moving object constituent color is estimated for each component of the pixel value, the moving object color can be estimated in more detail. In addition, it is good also as what estimates a moving body structural color about R (red), G (green), and B (blue) each component instead of YUV.

In the moving object extraction device having such a configuration,
The statistical processing unit may obtain the first distribution and the second distribution by making the class width of the Y component relatively larger than the class width of the U component and the V component.

  In this way, the sensitivity of the change in the Y component can be made insensitive, so that the influence of shadows existing in the image, changes in shooting conditions such as changes in the amount of solar radiation, and changes in the aperture value can be suppressed. It is possible to estimate the moving object constituent color.

  Moreover, the said moving body structure color estimation part is good also as a thing with which the said 2nd threshold value is set for every said Y, U, and V component.

  By doing so, it is possible to estimate the moving object constituent color with high accuracy. Further, as a method of making the sensitivity of the Y component insensitive, as described above, there is also a method of making the class width of the Y component larger than the class width of the other component. It is also possible to adopt a method of lowering the threshold value.

In the moving object extraction apparatus having the above configuration,
Furthermore, it is good also as a thing provided with the moving body extraction part which extracts the pixel applicable to the said moving body structural color from the target image which extracts a moving body. With such a configuration, it is possible to extract a moving object from an arbitrary image using the moving object composition color estimated by the moving object composition color estimation unit.

In the moving object extraction apparatus having the above configuration,
Furthermore, a correction unit that corrects a shift between the two images may be provided before the difference area is specified. By doing so, it is possible to suppress the influence of camera shake and panning at the time of image capturing, so that the difference area can be suitably specified.

  In the present invention, the various aspects described above can be applied by appropriately combining or omitting some of them. The present invention can also be configured as a moving object extraction method for extracting a moving object imaged in an image by a computer, a computer program for extracting a moving object imaged in an image, and the like. In any configuration, the various aspects described above can be applied.

  The computer program may be recorded on a computer-readable recording medium. As the recording medium, for example, various media such as a flexible disk, a CD-ROM, a DVD-ROM, a magneto-optical disk, a memory card, and a hard disk can be used.

Hereinafter, embodiments of the present invention will be described in the following order based on examples.
A. Outline of operation of moving object extraction device:
B. Schematic configuration of moving object extraction device:
C. Moving object extraction processing:
D. Variation:

A. Outline of operation of moving object extraction device:
FIG. 1 is an explanatory diagram showing an outline of the operation of the moving object extraction apparatus 100 in the present embodiment. The moving object extraction apparatus 100 is an apparatus for taking moving image data from a digital video camera 200 connected by a DV cable 300 or the like and extracting moving objects captured in the moving image. A moving body refers to a subject that moves relative to a stationary background.

  In the figure, the moving image data taken from the digital video camera 200 is divided into four frames (1 to 4). In this moving image data, a mountain is imaged as a background, and a ball is imaged as a moving object. As shown in the figure, the ball is gradually moving from the upper left to the lower right in the frame. The moving object extraction apparatus 100 can remove only a ball as a moving object by removing a mountain as a background from an arbitrary frame of such moving image data. Below, the structure and process of this moving body extraction apparatus 100 are demonstrated in detail. In this specification, “frame” and “image” are synonymous terms.

B. Schematic configuration of moving object extraction device:
FIG. 2 is an explanatory diagram showing a schematic configuration of the moving object extraction device 100. The moving object extraction device 100 includes a CPU 110, a RAM 120, a ROM 130 in which a BIOS (Basic Input / Output System) is recorded, an IEEE 1394 interface 140 for connecting the digital video camera 200, an HDD (Hard Disk Drive) 150 in which various software are installed, and the like. These are connected to each other via a predetermined bus. The moving body extraction device 100 can be configured as, for example, a personal computer, a DVD recorder, a hard disk recorder, or the like.

  The HDD 150 is installed with an OS (Operating System) 160 and a moving object extraction program 170 operating on the OS 160. The moving object extraction program 170 is a program for extracting moving objects from moving image data. The CPU 110 develops these programs in the RAM 120 and executes them. Note that the OS 160 and the moving object extraction program 170 may be recorded in the ROM 130 as firmware.

  A block diagram of functions realized by the CPU 110 executing the moving object extraction program 170 is shown in the lower part of the figure. As illustrated, the moving object extraction apparatus 100 includes a deviation correction unit 171, a difference area specifying unit 172, a statistical processing unit 173, a moving object configuration color estimation unit 174, and a moving object extraction unit 175 as functional units.

  The shift correction unit 171 shifts a frame from which a moving object is extracted (hereinafter referred to as an “extraction target frame”) and a frame (hereinafter referred to as a “comparison target frame”) immediately before the extraction target frame in time series. Correct. For example, in FIG. 1, when the frame 3 is the extraction target frame, the immediately preceding frame 2 is the comparison target frame. The deviation between the two frames can be corrected based on, for example, a gradient method that is a well-known technique. By using the gradient method, it is possible to correct the deviation between the two images by comparing the gradients of the luminance changes of the two images on the XY plane and superimposing both images at the position where the gradient has the strongest correlation.

  FIG. 3 is an explanatory diagram illustrating a method for correcting a shift between frames. FIG. 3A shows an example in which mountains and balls are imaged in the extraction target frame F1 and the comparison target frame F2. In these frames, the mountains are green, the balls are white, and the other parts are black. Comparing these two frames, it can be seen that the mountain captured in the comparison target frame F2 is slightly rotated with respect to the mountain captured in the extraction target frame F1. It can also be seen that the position of the ball relative to the mountain is different between the two frames.

  FIG. 3B shows a result of superimposing these two frames by correcting the deviation of the two frames by the gradient method. In these frames, since the area occupied by the mountain is larger than the area occupied by the ball, the correlation of the gradient of the luminance change of the mountain is stronger than the ball. Therefore, if the gradient method is used, two images are superimposed according to the position where the mountain exists.

  The difference area specifying unit 172 (FIG. 2) obtains a luminance difference for each corresponding pixel in the extraction target frame and the comparison target frame after the deviation is corrected. And the area | region where this difference becomes more than a predetermined threshold value is specified as a difference area. As the threshold value, for example, when the possible range of luminance is 0 to 255, it can be set to 64 or the like. In addition to the luminance, the difference area may be specified using other pixel values such as saturation, hue, and RGB value.

  When two consecutive frames constituting a moving image are compared, a portion where the pixel value hardly changes between the two images is likely to be the background. On the other hand, there is a high possibility that the part where the pixel value has changed greatly is the part where the moving object is imaged. Therefore, it can be considered that the difference area specified by the difference area specifying unit 172 includes a large number of pixel values constituting the moving object.

  FIG. 4 is an explanatory diagram showing a method for specifying a difference area. FIG. 4A shows two frames that have been overlapped with the shift corrected by the shift correction unit 171. In the two frames shown in FIG. 4A, the peaks are the same green color, so there is no difference in luminance. Therefore, the difference is zero and is not specified as the difference region. On the other hand, since the position of the ball portion is different between the two frames, the difference to be obtained is the difference between the black and white luminances, and the value is maximum. Therefore, the imaged part of the ball in both frames is specified as the difference area. In FIG. 4B, the difference area specified in this way is shown with hatching.

  FIG. 5 is an explanatory diagram showing the relative frequency distributions Nz and Nd obtained by the statistical processing unit 173. First, as shown in FIG. 5A, the statistical processing unit 173 (see FIG. 2) obtains a relative frequency distribution Nz of pixel values with a predetermined class width for pixels constituting the entire extraction target frame. Further, as shown in FIG. 5B, the relative frequency distribution Nd of the pixel values is obtained for the pixels in the difference area in the extraction target frame. The class width can be set to 32 when the range of pixel values is 0 to 255, for example. These two distributions are obtained for the Y (luminance), U (luminance and red color difference) and V (luminance and blue color difference) components of the pixel, respectively. FIG. 5 shows only the distribution of the Y component.

  FIG. 6 is a semilogarithmic graph showing the result of the ratio R calculated based on the two distributions shown in FIG. As shown in the figure, the moving object constituent color estimation unit 174 (see FIG. 2) obtains a ratio R (= Nd / Nz) of the relative frequency distribution Nd to the relative frequency distribution Nz for each class. Then, a pixel value belonging to a class in which the ratio R is equal to or higher than a predetermined threshold T2 (for example, 1.2 or higher) is estimated to be a pixel value constituting a moving object (hereinafter referred to as “moving object constituent color”). A large value of the ratio R means that pixel values belonging to the class have a small proportion in the entire extraction target frame, but a large proportion in the difference area. This is because there is a high possibility that the pixel value is configured.

  According to the illustrated example, the luminances belonging to the class in which the value of the ratio R is equal to or greater than the threshold T2 are 159 to 223. Therefore, a color having luminance in this range is estimated as a moving object constituent color. For such moving body constituent colors, not only the luminance but also the U component and the V component are obtained in the same manner.

  The moving object extraction unit 175 (FIG. 2) determines whether or not each of the YUV components of all the pixels in the extraction target frame corresponds to the moving object constituent color, and extracts pixels that satisfy the range of all the component values. . Thereby, a moving body will be extracted. FIG. 7 shows an example in which a moving object is extracted from the extraction target frame F1 shown in FIG.

C. Moving object extraction processing:
FIG. 8 is a flowchart of the moving object extraction process executed by the CPU 110 of the moving object extraction apparatus 100 according to the moving object extraction program. This process is executed when the moving object extraction program is started.

  First, the CPU 110 inputs moving image data from the digital video camera 200 via the IEEE 1394 interface 140 (step S10). The moving image data may be input not only from the digital video camera 200 but also from the HDD 150, CD-ROM, DVD, memory card, or the like.

  Next, the CPU 110 selects an extraction target frame from the input moving image data (step S11). Such selection may be specified by the user, or may be automatically specified by the CPU 110 at a predetermined frame interval. When the extraction target frame is selected, the immediately preceding frame is automatically selected as the comparison target frame. Note that the comparison target frame may be arbitrarily designated by the user or the like instead of the frame immediately before the extraction target frame.

  Thereafter, the CPU 110 corrects the shift between the extraction target frame and the comparison target frame by the shift correction unit 171 (step S12). Then, the difference area specifying unit 172 calculates a difference in pixel value for each corresponding pixel of the two frames (step S13), and an area where the difference is equal to or greater than a threshold value T1 (for example, T1 = 30) is set as a difference area. Specify (step S14).

  Next, the CPU 110 obtains the relative frequency distribution Nz of the pixel values for the pixels of the entire extraction target frame by the statistical processing unit 173 (step S15), and further compares the relative pixel values of the pixels in the difference area in the extraction target frame. The frequency distribution Nd is obtained (step S16).

  Next, the CPU 110 calculates a ratio R of the two relative frequency distributions by the moving object configuration color estimation unit 174 (step S17), and sets a pixel value at which the ratio R is equal to or greater than a threshold T2 (for example, T2 = 1.2) as a moving object. Estimated as a constituent color (step S18).

  Finally, the CPU 110 extracts a moving object by extracting pixels corresponding to the moving object constituent color from the extraction target frame by the moving object extraction unit 175 (step S19).

  According to the embodiment described above, it is easy to prepare moving image data without specifying the color or texture of a moving object or preparing an image in which only the background is captured in advance. A moving object can be extracted from an arbitrary frame.

D. Variation:
(1) When extracting the moving object, the moving object extraction apparatus 100 may divide the extraction target frame and the comparison target frame into a plurality of blocks, and perform the above-described moving object extraction processing for each corresponding block. FIG. 9 shows the extraction target frame F1 and the comparison target frame F2 divided into a plurality of blocks. In this way, moving objects can be extracted for each block.

  For example, when a white ball is imaged as a moving object in a frame and a white signboard is imaged in the background, when the moving object composition color is estimated by the method described in the embodiment, the white color of the ball is the moving object composition color. Is estimated as Therefore, when a moving body is extracted based on this moving body constituent color, not only the ball but also a part of the background signboard is extracted as a moving body. However, if the frame is divided into blocks, it is possible to extract only the ball as a moving object if the ball and the signboard are in different blocks.

(2) In the above embodiment, the moving object is extracted from the extraction target frame. However, once the moving object composition color is estimated, the moving object can be extracted from frames other than the extraction target frame by applying this moving object composition color to any other frame. By doing so, even when the moving object is stationary over two or more frames, it is possible to appropriately extract the moving object. In addition, it is possible to prevent a part of the background from being erroneously extracted as a moving object from a frame in which no moving object originally exists. In addition, since it is not necessary to estimate the moving object composition color for each frame, it is possible to extract moving objects at high speed from a plurality of frames constituting moving image data.

(3) In the above embodiment, the relative frequency distribution Nz is obtained using only the pixels in the extraction target frame. However, the relative frequency distribution Nz may be obtained by using the pixels in the comparison target frame in addition to the pixels of the extraction target frame. By doing so, the number of samples when calculating the relative frequency can be increased, so that it is possible to estimate the moving body constituent color with high accuracy. Similarly, when obtaining the relative frequency distribution Nd, not only the difference area in the extraction target frame but also the difference area in the comparison target frame may be used.

(4) When obtaining the relative frequency distributions Nz and Nd, the class width of the Y component is determined from the class width of the other components, for example, the class width of the Y component is 64 and the class width of the other components is 32. It may be enlarged.

  FIG. 10 is an explanatory diagram for explaining the effect when the class width of the Y component is increased. In the figure, a ball reflecting light as a moving body is shown. For example, when the class width of the Y component is narrow, there is a possibility that a portion reflecting light is not extracted as a moving object even with a ball having a uniform color. However, if the class width of the Y component is wide, there is an increased possibility that even a portion reflecting light with a large luminance change is appropriately extracted as a moving object.

  That is, by making the class width of the Y component larger than the class width of the other components, it is possible to make the luminance change sensitivity insensitive. Therefore, it is possible to estimate the moving body composition color while suppressing influences such as shadows existing in the frames, changes in the amount of solar radiation between frames, changes in image capturing conditions such as changes in the aperture value, and the like. .

(5) The threshold value T2 applied to the ratio R may be set for each of the Y, U, and V components. By doing so, it is possible to estimate the moving object constituent color with high accuracy. In such a case, the threshold value of the Y component may be lower than the threshold values of the other components. By doing so, the sensitivity of the luminance change can be made insensitive, so that the same effect as the effect described in the modification (4) can be obtained.

(6) In the embodiment, the case where a moving object is extracted from a moving image has been described. However, the object from which a moving object is extracted is not limited to a moving image, and the above embodiment can be applied to a case where a moving object is extracted from a still image. In such a case, step S10 of the moving object extraction process may be omitted, and in step S11, the user may individually specify a still image for the extraction target frame and the comparison target frame.

  As mentioned above, although the Example and modification of this invention were demonstrated, this invention is not limited to these, It cannot be overemphasized that a various structure can be taken in the range which does not deviate from the meaning. For example, a function realized by software may be realized by hardware.

It is explanatory drawing which shows the outline | summary of operation | movement of a moving body extraction apparatus. It is explanatory drawing which shows schematic structure of a moving body extraction apparatus. It is explanatory drawing which shows the correction method of the gap between frames. It is explanatory drawing which shows the identification method of a difference area | region. It is explanatory drawing which shows the example which calculated | required two relative frequency distribution about the luminance component. It is a semilogarithmic graph which shows the result of having calculated ratio R. It is explanatory drawing which shows the example from which the moving body was extracted from the extraction object flame | frame. It is a flowchart of a moving body extraction process. It is explanatory drawing which shows the extraction object frame divided | segmented into the block, and a comparison object frame. It is explanatory drawing explaining the effect at the time of enlarging the class width of Y component.

Explanation of symbols

100 ... Moving object extraction device 110 ... CPU
120 ... RAM
130 ... ROM
140 ... IEEE 1394 interface 150 ... hard disk drive 160 ... operating system 170 ... moving object extraction program 171 ... deviation correction unit 172 ... difference area specifying unit 173 ... statistic processing unit 174 .. .Moving object composition color estimation unit 175 ... Moving object extraction unit 200 ... Digital video camera 300 ... DV cable

Claims (10)

A moving object extraction device for extracting a moving object from an image,
An input unit for inputting two images in which pixel values are recorded for each pixel;
A difference area specifying unit that calculates a difference between pixel values for each corresponding pixel of the two images, and specifies an area in which the difference is equal to or greater than a first threshold as a difference area;
A first distribution is obtained with a predetermined class width for pixel values of the entire image of at least one image of the two images, and further, a pixel value within an area corresponding to the difference area of the image is calculated with the class width. A statistical processing unit for obtaining a second distribution;
A ratio of the second distribution to the first distribution is obtained for each class, and a pixel value belonging to a class in which the ratio is equal to or greater than a second threshold is estimated to be a moving object constituent color constituting the moving object. A moving object color estimation unit;
A moving body extraction apparatus comprising: The moving body extraction device according to claim 1,
The moving object extraction device, wherein the first distribution and the second distribution obtained by the statistical processing unit are relative frequency distributions. The moving body extraction device according to claim 1,
The statistical processing unit obtains the first and second distributions for each of Y, U, and V components of pixel values,
The moving object configuration color estimation unit estimates the moving object composition color for each of the Y, U, and V components. The moving object extraction device according to claim 3,
The moving object extraction device, wherein the statistical processing unit obtains the first and second distributions by making a class width of the Y component relatively larger than a class width of the U component and the V component. The moving object extraction device according to claim 3,
The moving object extraction device, wherein the moving object constituent color estimation unit sets the second threshold value for each of the Y, U, and V components. The moving body extraction device according to claim 1,
Furthermore, a moving body extraction apparatus provided with the moving body extraction part which extracts the pixel applicable to the said moving body structural color from the target image which extracts a moving body. The moving body extraction device according to claim 1,
Furthermore, a moving body extraction apparatus provided with the shift | offset | difference correction | amendment part which correct | amends the shift | offset | difference of these two images before specifying the said difference area. A moving object extraction method for extracting a moving object from an image by a computer,
Inputting two images in which pixel values are recorded for each pixel;
Obtaining a difference between pixel values for each corresponding pixel of the two images, and specifying an area where the difference is equal to or greater than a first threshold as a difference area;
A first distribution is obtained with a predetermined class width for pixel values of the entire image of at least one image of the two images, and further, a pixel value within an area corresponding to the difference area of the image is calculated with the class width. Obtaining a second distribution;
A ratio of the second distribution to the first distribution is obtained for each class, and a pixel value belonging to a class in which the ratio is equal to or greater than a second threshold is estimated to be a moving object constituent color constituting the moving object. Process,
A moving body extraction method comprising: A computer program for extracting a moving object from an image,
A function of inputting two images in which pixel values are recorded for each pixel;
A function of obtaining a difference between pixel values for each corresponding pixel of the two images, and specifying an area where the difference is equal to or greater than a first threshold as a difference area;
A first distribution is obtained with a predetermined class width for the pixel values of the entire image of at least one image of the two images, and further, the pixel values in the area corresponding to the difference area of the image are calculated with the class width. A function for obtaining a second distribution;
A ratio of the second distribution to the first distribution is obtained for each class, and a pixel value belonging to a class in which the ratio is equal to or greater than a second threshold is estimated to be a moving object constituent color constituting the moving object. Function and
A computer program for realizing a computer.   The computer-readable recording medium which recorded the computer program of Claim 9.

Images