JP2011053951A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve the accuracy of recognition of a subject in a moving image. <P>SOLUTION: An image processing apparatus extracts an area in which image information is changed between a previous frame (first frame) concerned with a moving image and a subsequent frame (second frame) subsequent to the previous frame, sets a search area of the subsequent frame based on a subject area concerned with the detection of the subject in the previous frame and the extracted area, and discriminates the subject in the set search area of the subsequent frame. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method for moving images.

  2. Description of the Related Art Conventionally, an image processing method for automatically detecting a specific subject pattern from a still image is very useful, and is used for, for example, determination of a human face. Such an image processing method can be used in many fields such as a teleconference, a man-machine interface, security, a monitor system for tracking a human face, and image compression.

  In recent years, detection of a subject from a moving image has been performed. In order to detect a face in a moving image in real time, an area that has not changed in time is determined, and the area is detected. A method of excluding from processing is disclosed (for example, see Patent Document 1).

JP 2005-174352 A

  However, the above-described method is effective in shortening the image processing time related to the recognition of the subject in the moving image, but it is difficult to improve the accuracy of the recognition of the subject in the moving image.

  The present invention has been made in view of such problems, and an object thereof is to improve the accuracy of recognition of a subject in a moving image.

  Therefore, the present invention provides an extraction unit that extracts an area in which image information changes between a first frame relating to a moving image and a second frame following the first frame, and the first unit Setting means for setting the search area of the second frame based on the subject area relating to the detection of the subject in the frame and the area extracted by the extracting means, and the second set by the setting means Discriminating means for discriminating a subject within a frame search area.

  According to the present invention, it is possible to improve the accuracy of recognition of a subject in a moving image.

It is a figure which shows the structure of an image processing apparatus. It is a figure which shows the flowchart which concerns on the process regarding an initial stage frame. It is a figure which shows an example of a collation pattern. It is a figure which shows the flowchart which concerns on the process regarding a subsequent frame.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1A is a diagram illustrating a hardware configuration of the image processing apparatus according to the present embodiment. The image processing apparatus includes a CPU (Central Processing Unit) 1, a storage device 2, an input device 3, an output device 4, and an imaging device 5. Each device is configured to be able to communicate with each other, and is connected by a bus or the like.

The CPU 1 controls the operation of the image processing apparatus and executes a program stored in the storage device 2.
The storage device 2 is a storage device such as a magnetic storage device or a semiconductor memory, and stores a program read based on the operation of the CPU 1, data that must be stored for a long time, and the like.
In the present embodiment, the CPU 1 performs processing according to the procedure of the program stored in the storage device 2, thereby realizing functions in the image processing apparatus and processing according to flowcharts described later.

The input device 3 is a mouse, a keyboard, a touch panel device, a button, or the like, and inputs various instructions.
The output device 4 is a liquid crystal panel, an external monitor, a speaker, or the like, and outputs various types of information.
The imaging device 5 is a camcorder or the like, and includes an imaging device such as a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor). Note that moving image data captured by the imaging device 5 is stored in the storage device 2 or the like. The moving image includes a series of a plurality of frames, and has a still image corresponding to each frame.

  Note that the hardware configuration of the image processing apparatus is not limited to this. For example, the image processing device may include an I / O device for performing communication between various devices. The I / O device is an input / output unit such as a memory card or a USB cable, a wired or wireless transmission / reception unit, and the like.

  FIG. 1B is a diagram illustrating a functional configuration of the image processing apparatus according to the present embodiment. The processing and functions of the image processing apparatus are as follows: image input unit 10, image memory unit 20, image reduction unit 30, matching pattern extraction unit 40, brightness correction unit 50, face discrimination unit 60, face probability distribution integration unit 70, face area output. Unit 80, change area extraction unit 90, and search area setting unit 100.

The image input unit 10 reads the moving image data captured by the imaging device 5, extracts image data for each frame from the moving image data, and inputs the extracted image data to the image memory unit 20. The image input unit 10 may be configured to read moving image data from a storage medium that stores moving image data. The image input unit 10 may be configured to read moving image data stored in a server or the like via the Internet or the like.
The image memory unit 20 is a storage area provided in the storage device 2. The image memory unit 20 temporarily stores the image data output from the image input unit 10. Note that the image memory unit 20 may temporarily store moving image data.

The image reduction unit 30 reduces the image data corresponding to each frame of the moving image data stored in the image memory unit 20 according to a predetermined magnification, and outputs a plurality of reduced images having different sizes.
The collation pattern extraction unit 40 extracts a predetermined partial area from the image data reduced by the image reduction unit 30 as a collation target pattern (so-called collation pattern).
The brightness correction unit 50 corrects the brightness distribution of the matching pattern extracted by the matching pattern extraction unit 40.
The face discriminating unit 60 outputs a face probability for discriminating whether the collation pattern extracted by the collation pattern extraction unit 40 and corrected by the luminance correction unit 50 is a face pattern or a non-face pattern.

The face probability distribution integration unit 70 holds a distribution of face probabilities corresponding to each partial area extracted from a plurality of reduced images, and integrates the distribution of face probabilities between a plurality of frames.
The face area output unit 80 outputs a partial area corresponding to a matching pattern that is identified as a face based on the result of integration by the face probability distribution integration unit 70 to the output device 4 or the like.

The change area extraction unit 90 outputs an area (change area) in which image data (image information) changes between frames due to the movement of the subject.
The search region setting unit 100 sets a search region in the subsequent frame from the result of the face probability distribution integration unit 70 in the previous frame and the change region between frames extracted by the change region extraction unit 90.

  FIG. 2 is a flowchart illustrating processing performed by the image processing apparatus. The operation of the image processing apparatus in the initial frame will be described with reference to FIG.

First, the image input unit 10 inputs image data to be processed into the image memory unit 20 (step S101).
Here, the input image data is, for example, two-dimensional array data composed of 8-bit pixels, and is composed of R, G, B, and three surfaces. At this time, when the image data is compressed by a method such as JPEG (Joint Photographic Experts Group), the image data is decompressed in accordance with a decompression method corresponding to the compression, and the image data is configured by RGB pixels.
Further, in the present embodiment, the image input unit 10 generates luminance image data including luminance components (for example, image data excluding color difference components) from luminance data included in image data configured by RGB pixels. This applies to subsequent processing. The luminance image data is stored in the image memory unit 20. When YCrCb data is input as image data, the Y component may be directly used as luminance data, and luminance image data may be generated.

Next, the image reducing unit 30 reads the luminance image data from the image memory unit 20, and generates luminance image data (so-called reduced luminance image data) reduced to a predetermined magnification (step S102). In the present embodiment, a configuration is adopted in which luminance image data reduced to a predetermined magnification is generated and the luminance image data of a plurality of sizes is sequentially detected (see, for example, Reference 1). Therefore, faces of various sizes can be detected. For example, the image reduction unit 30 performs a reduction process to images different in magnification by about 1.2 times a plurality of times.
Reference 1: Rowley et al, "Neural network-based face detection", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.20, NO.1, JANUARY 1998

Here, Reference 1 proposes a method of detecting a face pattern in an image using a neural network. Hereinafter, a face detection method according to Reference 1 will be briefly described.
First, image data targeted for face detection is read into the memory, and a predetermined area to be matched with the face is cut out from the read image. Then, one output can be obtained by calculation using a neural network with the distribution of pixel values in the cut out region as an input. At this time, the weights and threshold values of the neural network are learned in advance using a huge number of face image patterns and non-face image patterns. For example, if the output of the neural network is 0 or more, it is determined that the face is non-face. Then, the face is detected from the image by, for example, scanning the cutout position of the image pattern to be collated with the face which is an input of the neural network in the vertical and horizontal directions from the entire image. Further, in order to cope with detection of faces of various sizes, the read images are sequentially reduced at a predetermined rate, and the above-described face detection scanning is performed on the images.

Next, the collation pattern extraction unit 40 extracts and sets a partial area having a predetermined size from the reduced luminance image data as a collation pattern (step S103).
Here, the collation pattern will be described with reference to FIG.
The column A shown in FIG. 3 shows each reduced luminance image reduced by the image reduction unit 30. In step S103, a partial area (for example, a rectangular area) having a predetermined size with respect to each reduced luminance image. Is cut out. That is, a rectangular area having the same size is set in each reduced luminance image, and the rectangular areas are sequentially extracted as a collation pattern.
In addition, the column B shown in FIG. 3 shows a state of clipping in the middle of repeating scanning in the vertical and horizontal directions from each reduced luminance image. As shown in the drawing, when a face is discriminated by extracting a collation pattern from an image with a large reduction ratio, the face is detected in a large region with respect to the image.

  Next, the brightness correction unit 50 normalizes the brightness of the partial area cut out by the matching pattern extraction unit 40 based on the distribution (step S104). For example, the luminance correction unit 50 performs luminance correction such as histogram smoothing. Even if the luminance distribution of the subject pattern to be captured changes depending on the illumination condition, it is possible to reduce the accuracy of collation in the subject as much as possible by correcting the luminance.

Next, the face discriminating unit 60 discriminates whether the collation pattern extracted by the collation pattern extraction unit 40 and corrected by the luminance correction unit 50 is a face pattern or a non-face pattern, and whether it is a face pattern. A face probability is calculated as an index indicating whether or not (step S105).
Here, a known method (for example, refer to References 1, 2, and 3) may be used as the face discrimination method.
Reference 2: Schneiderman and Kanade, "A statistical method for 3D object detection applied to faces and cars", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR2000)
Reference 3: Viola and Jones, "Rapid Object Detection using Boosted Cascade of Simple Features", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'01)

Here, in Reference Document 2, the face probability of the matching pattern is regarded as an integrated model of statistical distributions regarding a plurality of appearances, and processing related to discrimination is performed.
Reference 3 focuses on speeding up the processing and uses AdaBoost to effectively combine many weak classifiers to improve the accuracy of face discrimination, while each weak classifier is a Haar-type rectangular feature. The rectangular feature value is calculated at high speed using the integral image. Also, the discriminators obtained by AdaBoost learning are connected in series to form a cascade type face detector. This cascade type face detector first removes a pattern candidate that is clearly not a face on the spot by using a simple classifier (that is, a classifier having a smaller calculation amount) in the preceding stage. Then, only for other candidates, it is determined whether or not it is a face by using a later complex discriminator having higher discrimination performance (that is, a discriminator having a larger calculation amount). In this way, since it is not necessary to make a complicated determination for all candidates, the processing becomes faster.

For example, in this embodiment, the output value of the neural network is output as the face probability value. However, in order to increase the accuracy of the value output as the face probability, the relationship between the output value of the neural network and the value of the face probability is stored in advance in a table instead of the output value itself of the neural network, and the table is referred to. Then, the face probability value may be output. The table can be prepared based on the statistical distribution of the output values of the prepared neural network with a sufficient number of face image patterns prepared in advance.
When a face is discriminated by a plurality of discriminators as the face discriminating unit, a weighted average or the like of output values of the plurality of discriminators may be calculated to output a face probability value.

Then, the face probability distribution integration unit 70 integrates the face probability of the partial area obtained by the face determination unit 60 with the value of the corresponding partial area in the face probability distribution stored in advance, and updates the face probability distribution. (Step S106).
For example, let P (s, x, y) be the face probability obtained by the face discrimination unit 60 at the reduction ratio s and the cutout positions x and y. Here, if the value corresponding to the reduction ratio s and the cutout position x, y (in this case, a predetermined value is set as an initial value) in the face probability distribution is P _OLD (s, x, y), A value P _NEW (s, x, y) in the integrated face probability distribution is calculated by the following equation (1). However, α is a predetermined integration parameter and satisfies 0 <α <1.

P _NEW (s, x, y) = α • P (s, x, y) + (1−α) • P _OLD (s, x, y) (1)

  Here, in the processing from step S103 to step S106, scanning is repeated in the vertical and horizontal order as shown in FIG. 3 for each reduced luminance image output from the image reduction unit 30. In addition, reduction processes with different magnifications are sequentially applied, and the processes from step S102 to step S106 are repeated.

  Then, the face area output unit 80 determines that the value in the face probability distribution updated by the face probability distribution integration unit 70 is a predetermined value after the search is completed in all reduced luminance images at a plurality of predetermined magnifications. The region having the maximum value (may be a local maximum value) in the face probability distribution is output to the output device 4 or the like as a face region (subject region) (step S107). In other words, since the face probability distribution is provided for each reduced luminance image, the face area output unit 80 can output the face area for each reduced luminance image to the output device 4 or the like.

  Next, with reference to FIG. 4, the operation of the image processing apparatus in the subsequent frame performed using the detection result of the initial frame will be described.

  First, the image input unit 10 inputs luminance image data corresponding to a subsequent frame (for example, a second frame) subsequent to an initial frame (for example, a first frame) to the image memory unit 20 (step S201).

Next, the change area extraction unit 90, which is an example of an extraction unit, outputs an area (change area) in which the luminance data changes between frames due to the movement of the subject (step S202).
For example, the change area extraction unit 90 performs a difference calculation of the luminance value of the luminance image of each frame, and sets an area including pixels whose luminance value difference exceeds a predetermined threshold as the change area.
More specifically, the change area extraction unit 90 performs binarization processing in order to identify pixels in the change area and pixels in areas other than the change area. Further, in order to reduce the influence of the image noise, the change area extraction unit 90 repeats the smoothing process on the luminance image data after the binarization process based on the luminance image data of a predetermined neighboring area, so that the predetermined area or more Will be integrated into the changing areas. Note that the change area extraction unit 90 may calculate the change area by performing a morphological opening process on the luminance image data after the binarization process, for example.
Further, the change area extraction unit 90 extracts the skin color probability distribution from the color image of each frame, in addition to or instead of the configuration of calculating the change area based on the difference in the luminance value of the luminance image of each frame, You may make it calculate a change area | region based on the difference of the skin color probability in each flame | frame. Note that the skin color probability is calculated by applying a method (for example, see Reference 4) performed by a mixed Gaussian model representing the skin color probability distribution.
Reference 4: Jones and Rehg, "Statistical color models with application to skin detection", International Journal of Computer Vision, VOL.46, NO.1, JANUARY 2002

In the case of a moving image shot by a fixed camera, the change area can be extracted by the above simple process. However, if the camera is not fixed, the change area does not necessarily correspond to the moving subject area. Therefore, you may make it extract an area | region by the following processes. In this case, first, the change area extraction unit 90 extracts a motion vector over the entire image from the luminance image data of a plurality of frames. Then, the change area extraction unit 90 calculates camera motion parameters based on the motion vector distribution. Furthermore, the change area extraction unit 90 corrects the movement amount corresponding to the camera motion parameter calculated from the motion vector distribution, and extracts a motion component due to the subject motion separated from the camera motion. Then, the change area extraction unit 90 sets an area in which the motion component due to the movement of the subject is a predetermined value or more as the change area. Further, the change area extraction unit 90 performs a smoothing process on the extracted change area in consideration of the influence of the image noise as described above.
As a method for extracting motion vectors from luminance image data of a plurality of frames and calculating camera motion parameters, a known method (for example, see Reference 5) can be used.
Reference 5: Takekawa, Miyajima, “Three-dimensional motion and shape analysis from time-series images”, Computer vision technology review and future prospects, New Technology Communications (1998)

Next, the search area setting unit 100, which is an example of a setting unit, in the subsequent frame from the result of the face area output unit 80 in the previous frame (for example, the face area) and the change area between the frames extracted by the change area extraction unit 90. A search area is set (step S203).
More specifically, the search area setting unit 100 first extracts the area output by the face area output unit 80 as a first search area. Then, the search area setting unit 100 extracts the change area between frames extracted by the change area extraction unit 90 as the second search area. Here, when the reduction process is performed, the search area setting unit 100 sets the area obtained by reducing the change area by the same reduction process as the area in the luminance image obtained by the reduction process, and the second Extract as a search area. Then, the search area setting unit 100 sets the logical sum of the first search area and the second search area as the search area.
It should be noted that as the region used as the first search region (for example, the face region), all regions in which the value in the face probability distribution that is the output of the face probability distribution integration unit 70 is a predetermined value or more may be used. Further, the first search area may output an area including a predetermined neighborhood area with respect to the face area. For example, the neighborhood area is a partial area in contact with the face area.

  Next, the image reducing unit 30 reads the luminance image data from the image memory unit 20, and generates luminance image data reduced to a predetermined magnification (step S204).

  Next, the collation pattern extraction unit 40 sets a partial area of a predetermined size to be extracted from the reduced luminance image data (step S205), and is an area belonging to the search area set by the search area setting unit 100. It is determined whether or not (step S206). Here, the area belonging to the search area refers to a partial area including a part or all of the search area.

  If the partial area set in step S205 belongs to the search area, the matching pattern extraction unit 40 extracts and sets the partial area belonging to the search area as a matching pattern (step S207). If the partial area set in step S205 does not belong to the search area, the process returns to step S205, and scanning is repeated in the vertical and horizontal order. Note that if scanning is completed for one luminance image or one reduced luminance image, the process returns to step S204.

  Next, the luminance correction unit 50 normalizes the luminance of the partial region cut out by the matching pattern extraction unit 40 based on the distribution, similarly to the processing of step S104 (step S208).

  Next, the face determination unit 60, which is an example of a determination unit, performs a process similar to the process of step S105 (step S209). That is, the face discriminating unit 60 discriminates whether the collation pattern (subject to be discriminated in the search area) extracted by the collation pattern extraction unit 40 and corrected by the luminance correction unit 50 is a face pattern or a non-face pattern. In addition, the face probability is calculated.

Then, the face probability distribution integration unit 70, which is an example of an integration unit, performs the same process as the process of step S106 (step S210). That is, the face probability distribution integration unit 70 corresponds to a value (corresponding to the search area of the subsequent frame) corresponding to the partial area of the face probability distribution calculated up to the previous frame and the face probability (discrimination result of the subject in the search area of the subsequent frame). The face probability distribution is updated in combination with the subject discrimination result in the area to be processed. The face probability obtained by the face discriminating unit 60 is P (s, x, y), and the value in the face probability distribution (in this case, the integrated result of the face probability distribution up to the previous frame) is P _OLD (s, x, y). Then, the value P _NEW (s, x, y) in the face probability distribution after the integration is calculated by the above equation (1).

  As described above, the processing from step S205 to step S210 is repeated in the vertical and horizontal order for each reduced luminance image output from the image reduction unit 30. In addition, reduction processes with different magnifications are sequentially applied, and the processes from step S204 to step S210 are repeated. That is, the search for the face pattern is performed in the area set as the search area for each reduced luminance image.

Then, the face area output unit 80, which is an example of an output unit, performs a process similar to the process of step S107 (step S211). That is, the face area output unit 80 outputs an area related to the subject based on the integrated result. More specifically, the face region output unit 80 determines a region in which the value in the face probability distribution updated by the face probability distribution integration unit 70 is equal to or greater than a predetermined value and has a maximum value in the face probability distribution. Is output to the output device 4 or the like.
Note that the processing from step S201 to step S211 is sequentially repeated until the processing is completed for all frames in the moving image data.

Note that if the movement of the subject is slower than the frame interval, the processing does not necessarily have to be performed for all frames, and detection processing (for example, see Reference 6) is performed at a predetermined frame interval. Also good.
Reference 6: Mikolajczyk et al, "Face detection in a video sequence-a temporal approarch", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'01)

  Here, in Reference Document 6, based on the method described in Reference Document 2, the face state of the next frame is predicted from the face detection result of a predetermined frame, and the face detection result is applied to the face determination process. Proposes a way to update. Also, a method for performing a full search every 5 frames is proposed.

  In the present embodiment, the face of a person is detected as the subject pattern, but other subject patterns different from the person may be adopted.

<Other embodiments>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

As described above, according to each of the embodiments described above, it is possible to improve the accuracy of subject recognition in a moving image.
That is, according to the configuration of each embodiment, it is possible to provide an apparatus that detects a predetermined subject from a moving image with high speed and high accuracy. That is, speeding up can be realized by narrowing the search area to the subject area and the inter-frame change area. Further, since the subject area is included as the search area, it is possible to cope with a case where a temporal change between frames is small. Further, since the search area includes a change area between frames, it is possible to cope with the appearance of a new subject. Furthermore, since the detection results in a plurality of frames are integrated, the detection can be performed stably and with high accuracy compared to the case of subject detection for each frame.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims.・ Change is possible.

10 image input unit, 20 image memory unit, 30 image reduction unit, 40 collation pattern extraction unit, 50 brightness correction unit, 60 face discrimination unit, 70 face probability distribution integration unit, 80 face region output unit, 90 change region extraction unit, 100 Search area setting section

Claims (4)

Extraction means for extracting a region in which image information changes between a first frame relating to a moving image and a second frame subsequent to the first frame;
Setting means for setting a search area for the second frame based on a subject area relating to detection of a subject in the first frame and an area extracted by the extracting means;
Discriminating means for discriminating a subject within the search area of the second frame set by the setting means;
An image processing apparatus. A result of the determination of the subject in the search region of the second frame by the determination means, and a result of determination of the subject in the region of the first frame corresponding to the search region of the second frame. Integration means to integrate;
Output means for outputting a subject area related to detection of a subject in the second frame based on the result of integration by the integration means;
The image processing apparatus according to claim 1, further comprising: An extraction step of extracting an area in which image information changes between a first frame relating to a moving image and a second frame subsequent to the first frame;
A setting step for setting a search region for the second frame based on a subject region related to detection of a subject in the first frame and the region extracted in the extraction step;
A discriminating step for discriminating a subject within the search area of the second frame set in the setting step;
An image processing method. An extraction step of extracting an area in which image information changes between a first frame relating to a moving image and a second frame subsequent to the first frame;
A setting step for setting a search region for the second frame based on a subject region related to detection of a subject in the first frame and the region extracted in the extraction step;
A discriminating step for discriminating a subject within the search area of the second frame set in the setting step;
A program that causes a computer to execute.

Images