JP5778522B2 - Image processing apparatus and image processing program - Google Patents

Description

  The present invention relates to an image processing device and an image processing program for generating a higher-definition image than a frame image from a plurality of frame images captured by an imaging device such as a video camera.

  2. Description of the Related Art Conventionally, a technique (hereinafter referred to as a super-resolution technique) that generates a higher-definition image (hereinafter referred to as a high-definition image) than a frame image captured by an imaging device such as a video camera. Various proposals have been made (for example, see Patent Documents 1 to 3). In general, an imaging apparatus such as a video camera is more expensive as the resolution is higher. This super-resolution technique is useful for reducing the cost of building various systems that use an imaging apparatus.

  A system that uses an imaging apparatus is required to have a recognition rate of an imaged object depending on the use and purpose of the system. For example, in a place where an unspecified number of people gather, such as a station, a downtown area, a shopping center, etc., a plurality of security cameras are installed, and a monitoring system for monitoring suspicious persons and the like is constructed. In this monitoring system, a certain recognition rate is required for a person's face captured in a frame image captured by a security camera. The monitoring system uses a plurality of security cameras to monitor suspicious persons over a wide area. If a relatively high-resolution security camera is used to satisfy the required face recognition rate, the cost for constructing the system increases. Therefore, by using a security camera with a relatively low resolution and utilizing the above-described super-resolution technology, the cost for system construction is reduced and the required face recognition rate is secured. ing.

  In addition to the above-described monitoring system, the system using this super-resolution technology recognizes the license plate number written on the license plate of the vehicle traveling on the road and recognizes the vehicle between the two points. There is a traffic control system that gets travel time.

JP 2007-157001 A JP-A-2005-130443 JP-A-2005-72935

  However, the super-resolution technique proposed in Patent Documents 1 and 2 and the like is a high-definition object image (for example, a face image) from a plurality of frame images for an object such as a person captured in a frame image. When generating the image, the difference in the orientation of the object between the imaging frames was not taken into consideration.

  An object of the present invention is to take into account the difference in the orientation of a target object such as the face of a person being imaged between a plurality of frame images, so that a higher-definition image can be generated for the target object than before. An object is to provide an image processing apparatus and an image processing program.

  In order to achieve the above object, the image processing apparatus of the present invention is configured as follows.

  A frame image captured by the imaging device is input to the image input unit. The imaging device may be, for example, a video camera that outputs a frame image (moving image) of about 30 frames per second, a camera that captures images in response to an operation of a release button, or a camera that has a continuous shooting function. May be.

The extraction unit extracts a plurality of frame images from which the target object (for example, the face of the target person or a vehicle license plate) is captured from the frame image input to the image input unit. The reference frame image selection unit selects a reference frame image from a plurality of frame images extracted by the extraction unit. For example, a frame image in which the orientation of the target person's face (target object) is closest to the front is selected as a reference frame image from among a plurality of frame images extracted by the extraction unit.
Further, the dividing unit divides the imaging region of the target object into a grid according to a predetermined number of pixels for each frame image extracted by the extracting unit. The feature point setting unit sets a plurality of feature points for the target object that is the processing target.

The non-rigid deformation unit is a target in which the position of each feature point set by the feature point setting unit is captured in the reference frame image for each non-reference frame image that the reference frame image selection unit has not selected as the reference frame image. The orientation of the target object imaged in the non-reference frame image is determined by Free Form Deformations (FFD) that moves each lattice point divided by the division unit in a lattice shape so as to match the position of the corresponding feature point of the object. Non-rigid deformation is performed in accordance with the direction of the target object captured in the reference frame image .

The high-definition image generation unit uses the reference frame image selected by the reference frame image selection unit, the pixel value of each grid point divided by the division unit in a lattice shape, and the non-rigid deformation unit performs non-rigid deformation. A high-definition image of the target object determined using the pixel values of the corresponding lattice points of the reference frame image is generated .

  Moreover, it is good also as a structure which performs the image quality improvement process which reduces noise and an optical blur with respect to the high definition image which the high definition image generation part produced | generated.

  Therefore, considering the difference in the orientation of the target object such as the face of the person being imaged between the plurality of frame images, a higher definition image than the frame images is obtained for the target object from the plurality of frame images. Can be generated.

  According to the present invention, it is possible to generate a high-definition image for a target object in consideration of a difference in direction of the target object such as the face of a person being captured between a plurality of frame images.

It is a block diagram which shows the structure of the principal part of an image processing apparatus. It is a flowchart which shows the high-definition image generation process in an image processing apparatus. It is a figure explaining a non-rigid body deformation process. It is a figure explaining a non-rigid body deformation process. It is a figure explaining a non-rigid body deformation process.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described.

  FIG. 1 is a block diagram showing a configuration of a main part of the image processing apparatus. The image processing apparatus 1 includes a control unit 2, an image input unit 3, an image processing unit 4, and an output unit 5.

  The control unit 2 controls the operation of each part of the main body of the image processing apparatus 1.

  A video camera 10 is connected to the image input unit 3. The video camera 10 is a security camera, for example. The video camera 10 is installed so that a monitoring target area for monitoring the intrusion of a suspicious person, the leaving of a suspicious object, or the like is within the imaging field of view. The video camera 10 outputs a moving image obtained by imaging the monitoring target area. For example, the video camera 10 outputs about 30 frames per second of a frame image obtained by imaging the monitoring target area. The video camera 10 inputs a frame image obtained by imaging the monitoring target area to the image input unit 3.

  Here, the configuration in which the video camera 10 is connected to the image input unit 3 is taken as an example, but a recording medium such as a hard disk on which a moving image captured by the video camera 10 is recorded is connected to the image input unit 3 and the image is input. The input unit 3 may be configured to sequentially read frame images constituting a moving image captured by the video camera 10 from the recording medium.

  The image processing unit 4 processes the frame image input to the image input unit 3 and detects a target object (here, a human face being imaged) captured in the frame image. The image processing unit 4 performs high-definition image generation processing for generating a high-definition image for the face of the person using a plurality of frame images in which the face of the person as the target object is captured. The image processing unit 4 includes a microcomputer that executes the high-definition image generation processing, an image memory that is used as a working area, and the like. The image processing program according to the present invention is a program for operating this microcomputer and is installed in the image processing unit 4 in advance.

  The output unit 5 outputs a frame image input to the image input unit 3, a high-definition image generated by the image processing unit 4 through a high-definition image generation process, and the like. A display device, a face authentication device, and the like are connected to the output unit 5. A display device (not shown) connected to the output unit 5 displays a frame image input to the image input unit 3 and a high-definition image (face image of the subject) generated by the image processing unit 4. In addition, a face authentication apparatus (not shown) connected to the output unit 5 performs face authentication processing for the target person using the high-definition image (the target person's face image) generated by the image processing unit 4.

  Hereinafter, the operation of the image processing apparatus 1 will be described.

  The image input unit 3 is input with about 30 frames per second of a frame image of the imaging area captured by the connected video camera 10. The image processing apparatus 1 waits for an operator to perform an input operation for designating a target person in an operation unit (not shown). The operator performs an input operation for designating the target person while confirming the captured image of the video camera 10 displayed on the display device connected to the output unit 5. When an input operation for designating a target person is performed, the image processing apparatus 1 executes the following high-definition image generation processing.

  Here, the case where the operator designates the target person is taken as an example. However, for example, when a moving image captured by the video camera 10 is processed and a person who takes a peculiar action is detected, this person is selected. It is good also as a structure designated automatically as a subject person.

  FIG. 2 is a flowchart showing this high-definition image generation processing. This high-definition image generation process is executed in the image processing unit 4.

  The image processing unit 4 selects a frame image in which the face (target object) of the target person designated this time is captured from the frame images input to the image input unit 3 by a predetermined number of frames (for example, 30). Frame) (s1).

  The image processing unit 4 selects a reference frame image as a reference from a plurality of frame images captured in s1 (s2).

  In s2, the frame image with the subject's face closest to the front is selected as the reference frame image from the frame images captured in s1. For example, the subject's face is divided in the vertical direction by a straight line (partition line) passing through the center of the subject's nose and the center of the mouth. A frame image in which the absolute value of the difference between the length from this dividing line to the end of the right cheek and the length from this dividing line to the end of the left cheek is selected as the reference frame image.

  In order to reduce the processing load in s2, a frame image located in the middle in time among the plurality of frame images captured in s1 may be used as the reference frame image, or the oldest frame image in time is selected. The reference frame image may be used, the latest frame image in time may be used as the reference frame image, or the reference frame image may be selected by other methods.

  For each non-reference frame image that has not been selected as the reference frame image in s2, the image processing unit 4 matches the orientation of the face of the person who is the target object with the orientation of the face of the person who is the target object in the reference image. Deformation is performed (s3). This non-rigid deformation is performed by Free Form Deformations (FFD).

  Non-rigid deformation by FFD is well known and will be described briefly. FIG. 3A is a face image of the subject captured in the reference frame image extracted in s2. 3B is a face image in which the subject's face is slightly tilted to the left, and FIG. 3C is a face image in which the subject's face is slightly tilted to the right. FIG. 3B and FIG. 3C are non-reference frame images.

  With respect to the reference frame image shown in FIG. 3A, the face of the person who is the target object is divided into a lattice shape so as to have a predetermined number of pixels (see FIG. 4A). A lattice point (node) at which the line divided in the vertical direction and the line divided in the horizontal direction intersect corresponds to one pixel. The pixel has an appropriate size centered on the lattice point. Similarly, in the non-reference frame images (FIG. 3B and FIG. 3C) that perform non-rigid deformation, the face of the person who is the target object has a predetermined number of pixels as in the reference frame image. (See FIGS. 4B and 4C).

  The image processing unit 4 sets a plurality of feature points for the subject's face. The image processing unit 4 performs the following processing for each non-reference frame image.

  Each grid point set on the non-reference frame image is given a coefficient in the horizontal direction and the vertical direction. The image processing unit 4 calculates the positions (coefficients) of lattice points that match the positions of a plurality of feature points (feature points of the subject's face) in the non-reference frame image with the positions on the reference frame image. The image processing unit 4 moves each grid point of the non-reference frame image based on the coefficient calculated for the grid point. At this time, the image of the region surrounded by the four adjacent grid points is deformed according to the movement of these grid points. FIG. 5A is a reference frame image, and FIG. 5B and FIG. 5C are non-reference frame images obtained by performing non-rigid deformation on s3. FIG. 5B is an example in which the non-reference frame image shown in FIG. 4B is non-rigidly deformed, and FIG. 5C is a non-rigid body image of the non-reference frame image shown in FIG. This is a modified example.

  As shown in FIG. 5, the non-rigid deformation of s <b> 3 matches the orientation of the subject's face in each non-reference frame image with the orientation of the subject's face in the reference frame image.

  The image processing unit 4 generates a high-definition image for the target object using the non-reference frame image subjected to non-rigid deformation (s4). In the process of generating the high-definition image, the reference frame image may or may not be used. Here, description will be made assuming that a reference frame image is used.

  s4 is a process for determining a pixel value for each pixel in the reference frame image. Here, for each pixel, the average pixel value of the corresponding pixel in the reference frame image and the non-reference frame image subjected to non-rigid body deformation in s3 is determined as the pixel value of that pixel.

  In the case where the reference frame image is not used, the average of the pixel values of the corresponding pixels in the non-reference frame image subjected to non-rigid deformation may be set to the pixel value of the corresponding pixel. In addition, here, for each pixel, the pixel value is an average of the pixel values of the corresponding pixels in each frame image, but may be calculated by other methods such as a mean square.

  A high-definition image is an image having a pixel value determined by the above process for each pixel.

  The image processing unit 4 performs image quality improvement processing for reducing noise and optical blur on the high-definition image generated in s4 (s5). In s5, known unsharp masking and edge enhancement are executed as image quality improvement processing.

  The image processing apparatus 1 outputs from the output unit 5 a high-definition image that has undergone image quality improvement processing in s5.

  As described above, the image processing apparatus 1 generates a high-definition image for the face image of the person in consideration of the difference in the orientation of the face of the person among the plurality of frame images in the above-described high-definition image generation process. can do.

  In the above example, the target object is a person's face, but the target object is not limited to a person's face, and may be another type of object such as a vehicle license plate.

  Further, the number of pixels of the high-definition image to be generated can be arbitrarily set by adjusting the size of the grid that divides the reference frame image or the non-reference frame image.

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus 2 ... Control part 3 ... Image input part 4 ... Image processing part 5 ... Output part 10 ... Video camera

Claims (4)

An image input unit for inputting a frame image captured by the imaging device;
An extraction unit that extracts a plurality of frame images of a target object that is a processing target from the frame image input to the image input unit;
A reference frame image selection unit that selects a reference frame image from a plurality of frame images extracted by the extraction unit;
A dividing unit that divides the imaging region of the target object into a grid according to a predetermined number of pixels for each frame image of the plurality of frames extracted by the extraction unit;
A feature point setting unit for setting a plurality of feature points for a target object to be processed;
For each non-reference frame image extracted by the extraction unit and not selected as the reference frame image by the reference frame image selection unit , the position of each feature point set by the feature point setting unit is captured in the reference frame image. The target imaged in the non-reference frame image by Free Form Deformations (FFD) that moves each grid point divided by the division unit in a grid shape so as to match the position of the corresponding feature point of the target object being processed A non-rigid deformation part that performs non-rigid deformation to match the direction of the object with the direction of the target object imaged in the reference frame image;
With respect to the reference frame image selected by the reference frame image selection unit, the pixel value of each lattice point divided by the dividing unit in a lattice shape is used for the non-reference frame image obtained by performing non-rigid deformation by the non-rigid deformation unit . An image processing apparatus comprising: a high-definition image generation unit that generates a high-definition image of a target object determined using pixel values of corresponding grid points . The target object is the face image of the target person,
The image processing apparatus according to claim 1 , wherein the feature point setting unit sets a plurality of feature points to a target object using a feature amount of a face part of the target person. The image processing apparatus according to claim 1 , further comprising an image quality improvement processing unit that reduces noise and optical blur for a high-definition image of the target object generated by the high-definition image generation unit. An extraction process for extracting a plurality of frame images in which a target object to be processed is imaged from the frame image input to the image input unit;
A reference frame image selection process for selecting a reference frame image from a plurality of frame images extracted by the extraction process;
A division process for dividing the imaging region of the target object into a grid according to a predetermined number of pixels for each frame image of a plurality of frames extracted by the extraction process;
A feature point setting process for setting a plurality of feature points for a target object to be processed;
For each non-reference frame image extracted in the extraction process and not selected as a reference frame image in the reference frame image selection process, the position of each feature point set in the feature point setting process is captured in the reference frame image. The target imaged in the non-reference frame image by Free Form Deformations (FFD) that moves each grid point divided in the grid pattern by the division processing so as to match the position of the corresponding feature point of the target object being processed A non-rigid deformation process for performing non-rigid deformation to match the direction of the object with the direction of the target object imaged in the reference frame image;
With respect to the reference frame image selected by the reference frame image selection process, the pixel values of the respective lattice points divided in a lattice shape by the division process are converted into the non-reference frame image obtained by performing non-rigid deformation by the non-rigid deformation process . An image processing program for causing a computer to execute high-definition image generation processing for generating a high-definition image of a target object determined using pixel values of corresponding grid points .

Images