JP5460499B2 - Image processing apparatus and computer program - Google Patents

Description

  The present invention relates to an image processing apparatus and a computer program thereof. In particular, the present invention relates to an image processing apparatus that associates positions of corresponding points among a plurality of images and a computer program thereof.

It is required to calculate to which position on a plurality of images a certain feature point on the subject corresponds between a plurality of images obtained by imaging a subject having a three-dimensional shape.
For example, Non-Patent Document 1 estimates a subject position shift between two images, a subject rotation center and angle, and a subject enlargement or reduction center and enlargement rate (or reduction rate). The method is described. In the technique described in this document, the Fourier transform of the patches respectively included in the two images is calculated, and the affine transformation between both patches is estimated using the two largest Fourier components.

  Non-Patent Document 2 describes a technique for estimating a positional deviation between feature points by comparing Gabor wavelet coefficients measured at feature points included in two image patches. . The system described in Non-Patent Document 2 collates one input face image with a face template, and recognizes an input face based on the result.

Stefan Kruger, Andrew Calway, “Image Registration using Multiresolution Frequency Domain Correlation”, Proceedings of the British Machine Vision Conference BMVC98, 1998, pp. 316-325 Laurenz Wiskott, Jean-Marc Fellous, Norbert Kruger, Christoph von der Malsburg, “Face Recognition by Elastic Bunch Graph Matching”, TR96-08, Institut fur Neuroinformatik, Ruhr-Universitat Bochum, 1996

  However, the technique described in Non-Patent Document 1 only estimates a global affine transformation between the entire patches (that is, all pixels of both patches). In reality, not only such global affine transformation but also non-linear warping between patches exists between a plurality of images obtained by imaging a subject. The technique described in Non-Patent Document 1 Then, points corresponding to such nonlinear distortion cannot be associated between images.

  In the technique described in Non-Patent Document 1 or 2, two images obtained by photographing the same subject are given, and one of the two images is used as a reference and the other is used as a test. Warping to match the reference image. However, in general, a large number (three or more) of images (image patches) are given, and it is unclear which image should be used as a reference when associating the positions of points included in these images. is there.

  The present invention has been made in consideration of the above-described circumstances. When the pixel position on the image is made to correspond to the position of the subject surface based on the image obtained by photographing the subject having a three-dimensional shape, the present invention is nonlinear. Provided is an image processing apparatus that can perform alignment in response to mechanical distortion.

  [1] In order to solve the above-described problem, an image processing apparatus according to an aspect of the present invention obtains the position of pixels included in a plurality of frame images obtained by imaging a three-dimensional object on the surface of the three-dimensional object. A reference image of the resolution is generated by applying a low-pass filter with a predetermined resolution to the sum of the pasted images acquired by the pasted image acquisition unit and the pasted image acquisition unit that acquires the pasted image to be obtained And calculating a positional deviation amount of pixels included in the pasted image based on the pasted image and the reference image, and calculating the positional deviation amount of the pixels of the pasted image based on the obtained positional deviation amount. And a warping processing unit for aligning positions.

  The pasted image acquired by the image acquisition unit is an image of a three-dimensional object, and the position on the three-dimensional object corresponding to each pixel is estimated. However, although this estimation has a predetermined accuracy, it may include a positional deviation. The reference image generation unit generates a reference image with a predetermined resolution based on the plurality of pasted images. That is, a reference image can be constructed based on a pasted image without giving a reference image separately or designating what should be a reference. In the above, the sum of images may be a simple sum, or a sum of products of pixel values obtained by weighting each image and a weight value. As an example, a weight value determined by an angle formed by the imaging direction and the surface normal of the three-dimensional object is used as the weight value, but is not limited thereto. The warping processing unit calculates a positional deviation amount for each pixel of the pasted image based on the obtained reference image and each pasted image. In other words, the positional deviation amount of each pixel for one pasted image is not limited to that corresponding to affine transformation, and may correspond to nonlinear distortion.

  [2] Further, according to one aspect of the present invention, in the image processing apparatus, the pasted image obtained by the warping processing unit aligning the pixel positions at the predetermined resolution may be used as the pasted image. The acquisition unit acquires the reference image, and the reference image generation unit generates the reference image at the next resolution, and controls to sequentially repeat the alignment of the pixel positions of the pasted image from the low resolution side to the high resolution side. And a control unit.

  With the above configuration, first the misalignment alignment is performed on the low resolution side, then the misalignment alignment is performed at the upper resolution, and the misalignment alignment is gradually performed while increasing the resolution sequentially, and this is the highest resolution. Will be repeated.

  [3] Further, according to one aspect of the present invention, in the image processing apparatus described above, a texture image storage unit that stores a texture image representing a texture of a surface of a three-dimensional object, and a plurality of the alignments that are aligned by the warping processing unit A texture image writing unit that writes the texture image obtained by combining the pasted images into the texture image storage unit;

  With the above configuration, a texture image can be obtained by combining a plurality of pasted images. At this time, a plurality of pasted images may be combined (added) with weighting by weight values.

  [4] Further, according to one aspect of the present invention, in the above-described image processing apparatus, two-dimensional coordinate value data corresponding to a feature point of a face is identified as a person in the image processing apparatus as the three-dimensional object. Included in the read image frame based on the face feature database unit stored in association with information and angle data representing the head posture, and the two-dimensional coordinate value data corresponding to the feature points read from the face feature database unit A face area detecting / collating unit for estimating two-dimensional coordinate value data of a facial feature point; a face model storing unit for storing three-dimensional coordinate value data corresponding to mesh vertices in a predetermined generic model; and the face area detecting unit Two-dimensional coordinate value data of the feature points estimated by the matching unit, and three-dimensional coordinate value data corresponding to the mesh vertices read from the face model storage unit A position / posture estimation unit that estimates three-dimensional coordinate value data of the feature points included in the image frame and angle data representing a head posture of the image frame, and the position / posture estimation unit Based on the estimated three-dimensional coordinate value data of the feature points and the angle data, a corrected face model is generated by warping the mesh vertices in the generic model, and the corrected face model corresponding to the mesh vertices A mesh warping unit for calculating three-dimensional coordinate value data of mesh vertices in the image, and a plurality of rendering processes when the angle data representing the head posture is changed based on the corrected face model generated by the mesh warping unit. Generate a composite face image model and calculate the two-dimensional coordinate value data of the feature points And a database registration unit that registers the two-dimensional coordinate value data of the feature points calculated by the rendering unit in the face feature database unit in association with the corresponding angle data, the face feature database unit Stores at least image feature information in the vicinity of the feature point in association with the person identification information and the angle data of the head posture, and the rendering unit writes the texture written by the texture image writing unit. The image is read from the texture image storage unit, and rendering processing based on the texture image is performed. The database registration unit is configured to correspond to the image feature information based on the result of rendering processing performed by the rendering unit. In association with angle data in the facial feature database section It is characterized by registering.

[4A] Further, according to one aspect of the present invention, in the image processing apparatus according to [4], the texture mapping unit may determine an angle between an optical axis of a line of sight with respect to the corrected face model and a surface normal of the corrected face model. Based on this, the weight of the direction when mapping the shading / color information included in the image frame to the two-dimensional face texture image is adjusted.
[4B] Further, according to one aspect of the present invention, in the image processing device according to [4], the texture mapping unit is a distance from a position corresponding to the feature point in the corrected face model in the two-dimensional face texture image. And adjusting the weight of the distance when mapping the shading / color information included in the image frame to the two-dimensional face texture image, and the degree of change of the weight of the distance with respect to the distance is set to the first The variable is variable according to the parameter, and the degree of change in the weight of the direction with respect to the angle between the optical axis of the line of sight and the surface normal is variable according to the second parameter.

  [5] In addition, according to one aspect of the present invention, a pasted image for obtaining a pasted image obtained by estimating positions of pixels included in a plurality of frame images obtained by capturing a three-dimensional object on the surface of the three-dimensional object. An acquisition unit; a reference image generation unit that generates a reference image of the resolution by applying a low-pass filter with a predetermined resolution to a sum of the plurality of the pasted images acquired by the pasted image acquisition unit; and the pasting A warping processing unit that calculates a positional deviation amount of pixels included in the pasted image based on the image and the reference image, and aligns the pixel positions of the pasted image based on the obtained positional deviation amount. A computer program that causes a computer to function as an image processing apparatus.

According to the present invention, when the position of a pixel on the image is made to correspond to the position of the subject surface based on an image of a subject having a three-dimensional shape, the amount of positional deviation is calculated corresponding to nonlinear distortion. And alignment can be performed.
Further, according to the present invention, even when an image to be used as a reference cannot be specified, a reference image can be constructed based on the captured image, and the amount of pixel displacement can be calculated based on the relationship with the reference image.

It is a block diagram which shows the function structure of the image processing apparatus by the 1st Embodiment of this invention. It is a block diagram which shows the more detailed functional structure of the texture mapping part by the embodiment. It is the schematic which showed the relationship between the texture image which the texture image memory | storage part by the same embodiment memorize | stores, and a three-dimensional model of a to-be-photographed object. It is the flowchart which showed the process sequence by the embodiment. It is a block diagram which shows the function structure of the face image processing apparatus by the 2nd Embodiment of this invention. In the embodiment, it is the figure which showed typically the feature point position which the face area detection collation part estimated from the face image area | region. In the embodiment, it is the figure which showed the data structure of the information regarding the correspondence memorize | stored in a mesh vertex allocation information storage part. In the same embodiment, it is the figure which showed the data structure of the information regarding the correspondence memorize | stored in a pixel allocation information storage part. In the same embodiment, it is a figure which shows the data structure of the variable template structure which a database registration part produces | generates. 4 is a flowchart illustrating a procedure in which a face area detection / collation unit detects a face image area from each image frame of video data, collates it with face feature data, and outputs a tracking result or a recognition result in the embodiment. 4 is a flowchart showing a procedure of processing for estimating a three-dimensional feature point position and a head posture for each image frame from two-dimensional feature point positions in each image frame of video data in the embodiment. In the same embodiment, it is the flowchart which showed the procedure of the mesh warping process which a mesh warping part performs. In the same embodiment, it is a figure which shows the mesh of a three-dimensional CG face model. In the embodiment, it is a figure for demonstrating the warping process of the mesh vertex of a three-dimensional CG face model. 5 is a flowchart showing a procedure of UV texture image correction processing in the embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a block diagram illustrating a functional configuration of the image processing apparatus according to the first embodiment. As illustrated, the image processing apparatus 1 includes an image data storage unit 11, a reverse pose conversion processing unit 12, a texture mapping processing unit 14, and a texture image storage unit 15.
The image data storage unit 11 stores image data obtained by photographing a certain subject using a camera or the like. The image data includes a plurality of image frames for a certain subject. The data of each image frame includes the color value (for example, RGB value) of each pixel. Here, each image frame may be a still image taken at a time interval, or may be one image frame included in a video (moving image). Here, the plurality of image frames may have different positions, directions, distances, angles of view, and the like when the subject is imaged.
The texture image storage unit 15 includes a texture image representing the texture of the entire surface of the subject or at least a part of the surface. The texture image represents the color, shading, pattern, or texture of the subject surface. The texture image stored in the texture image storage unit 15 is an image of a two-dimensional orthogonal coordinate system plane represented by u coordinates and v coordinates, and the plane corresponds to the surface of the subject. In other words, one pixel (specified by coordinates (u, v)) on the planar image corresponds to a certain point on the surface of the subject. The texture image data is represented by the color value (for example, RGB value) of each pixel.

The inverse pose conversion processing unit 12 estimates the pose of the subject photographed on the frame image based on the frame image read from the image data storage unit 11. Here, the pose is the position and orientation of the subject and the degree of enlargement / reduction in relation to the camera that captured the frame image. A pose is an amount represented by a combination of a parallel movement of a subject in space, a rotation amount (for example, a rotation amount about each axis about three axes orthogonal to each other), and an enlargement / reduction amount. is there. The factors that determine enlargement / reduction are the distance between the camera and the subject and the angle of view of the lens used for shooting. The inverse pose conversion processing unit 12 recognizes a plurality of feature points captured in the frame image, and estimates the pose based on the relationship between the feature points. Then, the inverse pose conversion processing unit 12 can normalize the position, orientation, and enlargement / reduction of the subject by applying inverse conversion of the pose estimated above to each frame image. Further, the reverse pose conversion processing unit 12 reads out information on the three-dimensional shape of the subject from a three-dimensional shape model storage unit (not shown). The information on the three-dimensional shape of the subject is, for example, the three-dimensional coordinate value of each mesh vertex when the subject is represented by a mesh. The inverse pose conversion processing unit 12 obtains a correspondence relationship (mapping relationship) between the pixel on the frame image and the pixel on the texture image from the frame image obtained by inversely converting the pose and the information on the three-dimensional shape. presume. The reverse pose conversion processing unit 12 develops a frame image using this mapping relationship to obtain a pasted image. That is, the reverse pose conversion processing unit 12 obtains the kth pasted image s _k (u, v) from the kth (0 ≦ k <N _frames ) frame image. N _frames is the number of image frames.

The inverse pose conversion processing unit 12 also calculates a direction weight α ^k (u, v) in each pixel of the pasted image. Here, the direction weight is a weight value given by, for example, α ^k (u, v) = cos ^m (θ ^k (u, v)). Here, m is an integer of 1 or more determined as appropriate. Θ ^k (u, v) is the surface normal of the subject and the optical axis of the line of sight (camera light) corresponding to the pixel at coordinates (u, v) on the k-th pasted image. Angle) and − (π / 2) ≦ θ ^k (u, v) ≦ (π / 2). The pose of the subject in the k-th frame image has already been calculated, and the direction of the surface normal on the subject surface corresponding to the coordinates (u, v) on the pasted image is the information on the three-dimensional shape. Therefore, the inverse pose conversion processing unit 12 can calculate θ ^k (u, v). The inverse pose conversion processing unit 12 writes the direction weight α ^k (u, v) in the corner pixel calculated here in a memory (not shown) so that it can be referred to later.

  The texture mapping unit 14 performs mapping from the frame image stored in the image data storage unit 11 to the texture image stored in the texture image storage unit 15. In other words, the texture mapping unit 14 specifies one pixel on the texture image corresponding to the pixel corresponding to the subject surface imaged in each frame image. Furthermore, the texture mapping unit 14 can create a texture image from one or a plurality of frame images using this mapping result, and write the texture image in the texture image storage unit 15.

  That is, the texture mapping unit 14 pastes the sum of contributions from each frame image on the texture image by texture mapping. Excluding the normalization term regarding the number of frame images, the sum at the coordinates (u, v) is as f (u, v) in the following equation (1).

Here, α ^k (u, v) is the weight of the direction at the coordinates (u, v), and r (hat) _k (u, v) is the k-th image frame of the texture image at the coordinates (u, v). S _k (u, v) = p _k (r (hat) _k (u, v)) is an RGB value at the estimated position. The value of the direction weight has already been obtained by the reverse pose conversion processing unit 12.

The texture mapping unit 14 _aligns N _frames pasted images of s _k (u, v), 0 ≦ k <N _frames . That is, s _k (u, v) in the k-th pasted image is replaced with s _k (u + m ^k (u, v), v + n ^k (u, v)). Here, (m ^k (u, v), n ^k (u, v)) is the optimum amount of positional deviation for aligning the k-th pasted image to another pasted image.
A method of calculating the amount of misalignment by the texture mapping unit 14 and correcting it will be described later.

FIG. 2 is a block diagram showing a more detailed functional configuration inside the texture mapping unit 14. As illustrated, the texture mapping unit 14 includes a pasted image acquisition unit 51-0, 51-1, 51-2,..., A warping processing unit 52-0, 52-1,. An image generation unit 53, a control unit 54, and a texture image writing unit 55 are included. The reference image generation unit 53 includes image addition units 61-0, 61-1,... And Gaussian window processing units 62-0, 62-1,. Yes.
In the figure, only the configuration relating to the (k−1) -th, k-th, and k + 1-th frame images is shown, and the others are omitted. However, in practice, the texture mapping unit 14 uses the 0th to ( It has a configuration corresponding to the N _frames −1) th frame images. Further, in the figure, the pasted image acquisition unit 51-2 for storing the pasted image of the second resolution is shown and the configuration at the subsequent stage is omitted, but in reality, the texture mapping unit 14 is omitted. Includes a warping processing section and a pasted image acquisition section in the subsequent stage (corresponding to a higher resolution).

  Although not shown in the figure, the texture mapping unit 14 performs the processing described below using the data read from the image data storage unit 11 and the data acquired from the reverse pose conversion processing unit 12. .

The pasted image acquisition units 51-0, 51-1, 51-2,... Acquire and store image data in the process for generating an image to be pasted on the texture image. Specifically, the pasted image acquisition units 51-0, 51-1, 51-2,... Position the positions of the pixels included in the plurality of frame images obtained by capturing the three-dimensional object on the surface of the three-dimensional object. A pasted image obtained by estimation is acquired. The pasted image acquisition unit 51-0 acquires a pasted image having a 0th resolution (resolution will be described later). The pasted image acquisition unit 51-1 acquires a pasted image having the first resolution. Similarly, up to the highest resolution (N _res −1), the pasted image acquisition unit corresponding to each resolution acquires and holds the pasted image.

The reference image generation unit 53, based on the pasted image of each resolution read from the pasted image acquisition unit 51-0, 51-1, 51-2, ..., the reference image q ₀ (u, v of each resolution). ), Q ₁ (u, v),. Specifically, the reference image generation unit 53 generates a reference image of the resolution by applying a low-pass filter with a predetermined resolution to the sum of the plurality of pasted images. A detailed procedure for generating the reference image will be described later.
The image adding units 61-0, 61-1,... Add weighted pasted images corresponding to the 0th to (N _frames −1) th frame images in the respective resolutions. The weight value α (u, v) used here is the weight in the above-described direction.
The Gaussian window processing units 62-0, 62-1,... Are output to the output of the image addition units 61-0, 61-1,. Performs low-pass filter processing using (window function).

The warping processing units 52-0, 52-1,... Are the pasted images at the resolution read from the pasted image acquisition units 51-0, 51-1,. Based on q ₀ (u, v), q ₁ (u, v),..., the amount of positional deviation at each pixel of the pasted image is estimated, and the following ( A pasted image with a higher resolution is generated. In other words, the warping processing units 52-0, 52-1,... Calculate the positional deviation amount of the pixels included in the pasted image based on the pasted image and the reference image, and the obtained positional deviation amount. The pixel positions of the pasted image are aligned. As a result of the alignment, a pasted image in which the misalignment is aligned is obtained. Then, the generated pasted image is transferred to the resolution pasted image acquisition unit 51.

  The control unit 54 causes the pasted image acquisition unit to acquire a pasted image obtained by the warping processing unit aligning the pixel positions at a predetermined resolution, and generates a reference image at the next resolution at the reference image generation unit. At the same time, the control is performed so that the alignment of the pixel positions of the pasted image is sequentially repeated from the low resolution side to the high resolution side. A specific processing procedure including repetition by the control of the control unit 54 will be described later.

  The texture image writing unit 55 sequentially outputs a texture image obtained by synthesizing a plurality of pasted images whose misalignments are aligned by the warping processing units 52-0, 52-1,. Write to.

Here, the resolution will be described. The resolution has N _res stages from the 0th resolution (low resolution) to the (N _res −1) th resolution (highest resolution). The value of N _res is appropriately set according to the resolution of the target image.
Then, as an example, the resolution of the i-th (0 ≦ _{i <N res)} is made to be ^{b i.} Here, b is an integer determined as appropriate. For example, when b = 2, the i-th resolution is 2 ⁱ . That is, in order from the 0th, the resolution is a series of 1, 2, 4, 8, 16,. Then, the value of N _res is determined so that the resolution of the target image can be fully used. As an example, when the resolution of the target image is 2048 (vertical) × 2048 (horizontal), if b = 2 and N _res = 12, the first, second, fourth, A series of resolutions of 8, 16, 32, 64, 128, 256, 512, 1024, and 2048 will be used.

FIG. 3 is a diagram illustrating the relationship between the subject to be imaged and the texture image held by the texture image storage unit 15. FIG. 4A shows a state in which texture mapping is performed on a 3D object CG (computer graphics) model representing a subject, and FIG. 4B shows a mesh of a 3D object CG model on a texture image. Shows the state of overlapping. As shown in the figure, one point (n-th mesh vertex x ⁿ (vector corresponding to a three-dimensional coordinate value) on the surface of the subject and coordinate values (u ⁿ , v ⁿ ) that are pixel positions on the texture image. (0 ≦ u ⁿ ≦ 1, 0 ≦ v ⁿ ≦ 1, 0 ≦ n <N _VT ), where N _VT is the total number of mesh vertices used here. The processing apparatus 1 stores, for each ⁿ , data (not shown) in a table that holds the three-dimensional coordinate values of ^xn and the coordinate values (u ⁿ , v ⁿ ) of pixels on the UV texture image in association with each other. Yes.
Here, an example of a human face (head) has been shown as a three-dimensional object that is a subject, but in practice, not only this but all three-dimensional objects that can be subjects can be handled.

FIG. 4 is a flowchart showing a processing procedure of the image processing apparatus 1. Hereinafter, a processing procedure will be described along this flowchart.
First, in step S1, the pasted image acquisition unit 51-0 sets initial pasted images s ₀ (u, v), s ₁ (u, v),..., S _k (u, v),. _.. , S _Nframes (u, v) is acquired and stored. The initial pasted image of the kth frame (0 ≦ k <N _frames ) is based on the kth frame image, and is converted into the uv orthogonal coordinate system based on the processing of the inverse pose conversion processing unit 12. It is an image that has been.
In step S2, the texture mapping unit 14 initializes the value of the variable i to 0. This variable i represents the resolution stage.

(A0) Construction of Reference Image Next, in step S3, the reference image generation unit 53 constructs a reference image having the i-th resolution based on the pasted image already obtained. Since i = 0, the reference image generation unit 53 reads the pasted image from the pasted image acquisition unit 51-0. Then, the reference image generation unit 53 constructs the reference image q ₀ (u, v) with a low resolution (0th resolution). This image is obtained by applying a low-pass filter to the weighted sum of pasted images from each video frame by a two-dimensional Gaussian window possessed by a low-resolution Gabor wavelet. The reference image generation unit 53 generates a reference image q ₀ (u, v) by calculation represented by the following equation (2).

  At this time, the image adding unit 61-0 calculates the weighted sum of the pasted image, and the Gaussian window processing unit 62-0 performs the convolution calculation of the two-dimensional Gaussian window of the resolution.

(B0) Calculation of Warping with Reference Image Next, in step S4, the warping processing unit 52-0 uses the low resolution Gabor wavelet to calculate the reference image q at each pixel of each pasted image s _k (u, v). A positional deviation from ₀ (u, v) is estimated. Note that i = 0.
A specific calculation method for estimating the displacement amount will be described later.
By this calculation, a misregistration amount estimated from only a low-resolution (0th resolution) image component (m ^k ₀ (u, v), n ^k ₀ (u, v)) is obtained. .
In step S5, the warping processing unit 52-0 warps the pasted image using the positional deviation amount obtained in the above step. That is, for each k, a pasted image of s _k (u + m ^k ₀ (u, v), v + n ^k ₀ (u, v)) is obtained. Then, the warping processing unit 52-0 supplies the pasted image obtained by the warping to the pasted image acquisition unit 51-1.

Next, in step S6, the texture mapping unit 14 determines whether the highest resolution processing has been completed. When the process up to the maximum resolution is completed (step S6: YES), the process of the entire flowchart is terminated, and when the process is not completed (step S6: NO), the process proceeds to step S7.
When the process proceeds to step S7, in this step, the texture mapping unit 14 adds 1 to the variable i. Thus, the value of the variable i represents the next resolution stage. Here, i = 1. When the process of this step is finished, the process proceeds to step S3.

(A1) Construction of Reference Image at Next Resolution In step S3, the reference image generation unit 53 constructs a reference image at the i-th resolution based on the pasted image already obtained. Since i = 1, the reference image q ₁ (u, v) is constructed with the next highest resolution (first resolution). This reference image q ₁ (u, v) reflects the amount of displacement obtained at a low resolution (0th resolution), and then adds the weighted sum of the pasted image from each video frame. A low-pass filter is applied by a two-dimensional Gaussian window possessed by the first resolution Gabor wavelet. The reference image generation unit 53 generates a reference image q ₁ (u, v) by calculation represented by the following equation (3).

  At this time, the image adding unit 61-1 calculates the weighted sum of the pasted image, and the Gaussian window processing unit 62-1 calculates the convolution of the two-dimensional Gaussian window of the resolution.

(B1) Calculation of Warp with Reference Image Next, in step S4, the warping processing unit 52-1 uses the first Gabor wavelet to generate a reference image for each pixel of each pasted image s _k (u, v). The positional deviation from q ₀ (u, v) is estimated. Note that i = 0.
In other words, the warping processing unit 52-1 executes each of the pasted images s _k (u + m ^k ₀ (u, v), v + n ^k ₀ (u, v)) warped at a low resolution (the 0th resolution). In a pixel, the amount of positional deviation from the reference image q ₁ (u, v) is estimated. The method for calculating the amount of displacement is the same as in the case of the 0th resolution. By this calculation, a misregistration amount estimated from only the image component of the first resolution, (m ^k ₁ (u, v), n ^k ₁ (u, v)), is obtained.

In step S5, the warping processing unit 52-1 warps the pasted image using the positional deviation amount obtained in the above step. That is, for each k, a paste of s _k (u + m ^k ₀ (u, v) + m ^k ₁ (u, v), v + n ^k ₀ (u, v) + n ^k ₁ (u, v))) To get an image. Then, the warping processing unit 52-1 supplies the pasted image obtained by the warping to the pasted image acquisition unit 51-2.

In steps S6 and S7, the texture mapping unit 14 again adds 1 to the variable i, i.e., sets i = 2, and proceeds to the next resolution process.
As described above, the amount of displacement is calculated for each of (a0), (b0), (a1), (b1) and the 0th and first resolutions, and warping is performed based on the amount of displacement. Thereafter, the same processing is performed for the second and subsequent resolutions.
General processing at the i-th resolution (i = 0, 1, 2,..., N _res −1) is as follows.

(A) Construction of Reference Image with i-th Resolution In step S3, the reference image generation unit 53 uses the already obtained pasted image as a reference image q _i (u, v) with the i-th resolution. ) Build. This image is obtained by applying a low-pass filter to the weighted sum of pasted images from each video frame by a two-dimensional Gaussian window possessed by a low-resolution Gabor wavelet. q _i (u, v) is expressed by the following equation (4).

Here, u ^k _i (u, v) and v ^k _i (u, v) are coordinate values obtained by aligning displacement amounts from the 0th resolution to the i-th resolution, respectively. (5) and (6).

(B) Calculation of warp with reference image In step S4, the warping processing unit (52-i) uses the Gabor wavelet at the i-th resolution to calculate the pasted image warped at the resolution before the i-th resolution. The amount of positional deviation between each pixel and the reference image is estimated. When 0 <i, the pasted image warped at the (i−1) -th resolution is s _k (u ^k _i−1 (u, v), v ^k _i−1 (u, v)). is there.
The method for calculating the amount of displacement is the same as the method described for the 0th and 1st resolutions. By this calculation, a displacement amount estimated from only the image component of the i-th resolution, (m ^k _i (u, v), n ^k _i (u, v)), is obtained.
By aligning the misalignment amounts, a pasted image s _k (u ^k _i (u, v), v ^k _i (u, v)) warped at the i-th resolution is obtained.

  In step S5, the warping processing unit (52-i) warps the pasted image using the positional deviation amount obtained in step S4. Then, the warping processing unit (52-i) supplies the pasted image obtained by the warping to the pasted image acquisition unit corresponding to the next resolution.

(A) Construction of a reference image and (b) calculation of a warp with a reference image at the i-th resolution are as described above. By repeating this process from the 0th resolution to the highest resolution N _res −1, the nonlinear alignment can be accurately calculated.

The Gabor wavelet gr ^{, φ} (x (arrow)) at the coordinates represented by the vector x (arrow) is as shown in the equations (7) and (8).

Further, Gaussian g _r (x (arrow)) having the same is as shown in Expression (9) (r = 1, 2,...).

  Then, the texture mapping unit 14 pastes the misaligned aligned pasted image on the texture image. The following formula (10) represents an example of the texture image after the pasted image is pasted.

In Expression (10), t (u, v) is a color value at the coordinates (u, v) of the texture image after pasting the pasted image. α ^k ′ (u, v) is a weight in the direction corresponding to the frame image k (0 ≦ k <N _frames ) in the coordinates (u, v) after aligning the positional deviation. s _k ′ (u, v) is a pasted image corresponding to the frame image k after alignment of the positional deviation. t ₀ (u, v) is an initial value of the texture image. When the initial value of the texture image is not used, equation (10) may be used with t ₀ (u, v) = 0 uniformly. Γ is a parameter value for adjusting the weight of the initial value of the texture image.
In this way, the texture mapping unit 14 performs a calculation of adding and pasting (synthesizing) the pasted image after aligning the positional deviation appropriately weighted and added to the texture image. Then, the texture image writing unit 55 in the texture mapping unit 14 writes the obtained texture image into the texture image storage unit 15.

[Method for estimating misalignment]
Next, the details of the calculation method for estimating the displacement described in step S4 of FIG. 4 will be described.
Here, “jet” (jet) will be described first. A jet is an amount corresponding to one pixel on an image in uv coordinates. The jet J (u, v) at the coordinates (u, v) is an aggregate of 40 values J _j (u, v) (where 0 ≦ j <40, j is an integer).
Here, j is an index value, j is another index value μ (where 0 ≦ μ <8, μ is an integer) and ν (where 0 ≦ ν <5, ν is an integer), and j = Μ + 8ν.
Note that ν is an index value related to the frequency of the Gabor wavelet. Μ is an index value related to the direction of the Gabor wavelet.

The above-mentioned J _j (u, v) constituting the jet is represented by the following equation (11).

The vector x (arrow) indicates coordinates (u, v). Ψ _j (x (arrow)) is a Gabor kernel represented by the following equation (12).

  The Gabor function expressed by Equation (12) above is constrained by a Gaussian envelope function, and the vector k (arrow) is a wave vector. This wave vector k (arrow) is expressed by the following equation (13).

In addition, kv represented using the said index value (nu) is as the following formula _| equation (14).

Further, θ _μ expressed using the index value μ is as shown in the following formula (15).

J _j (u, v) defined as described above can be expressed as the following Expression (16).

Then, the positional deviation amount is estimated using the jet defined as described above. The similarity S _[Phi between two jets (J and J'), is expressed by the following approximate expression (17).

Here, the deviation vector d (arrow) = (d _x , d _y ) is obtained so that the similarity S _Φ between the jets is maximized. For this purpose, the condition of equation (18) is set.

Deviation so as to satisfy the condition amount vector d (arrow) (J, J') Solving, solutions according to the following equation (19) is obtained (in the case of _{Γ xx Γ yy -Γ xy Γ yx} ≠ 0 ).

  The displacement vector obtained here is the positional displacement amount at the resolution. In the above formula, the following formulas (20) to (24) are provided.

[Second Embodiment]
Next, a second embodiment will be described. The second embodiment is an application of the image processing apparatus according to the first embodiment, and is a face image processing apparatus that models a face image and registers it in a database. In the following, items specific to the present embodiment will be described, and description of items common to the first embodiment will be omitted.

FIG. 5 is a block diagram showing a functional configuration of the face image processing apparatus according to the embodiment. As shown in the figure, the face image processing apparatus 100 (image processing apparatus) includes an image data storage unit 110, a face area detection collation unit 120, a three-dimensional estimation unit 130, a rendering unit 140, and a database registration unit 150. And a face feature database unit 160.
The face image processing apparatus 100 in the figure operates by being set to either the registration mode or the recognition mode. The face image processing apparatus 100 uses an image frame of video data or a plurality of still image data (still image data is also referred to as an image frame hereinafter) including a human face image as an input source. An image region is detected, and processing for generating and registering face feature data from the face image region (registration mode) or verification processing (recognition mode) is executed. In addition, the face image processing apparatus 100 executes a process of approximating and registering or collating face feature data corresponding to a face image of a head posture that is not included in an image frame.

  Note that there is no significant difference in the processing of this embodiment depending on whether the input source is video data or a plurality of still image data. Therefore, in the following description of the present embodiment, a process of generating facial feature data based on a human face image included in each image frame of video data, performing approximate synthesis, and registering or collating will be described. Processing when the input source is still image data is essentially the same as when the input source is video data.

  The image data storage unit 110 stores video data including a face image of a person to be registered. The face area detection / collation unit 120 reads video data from the image data storage unit 110, detects a face image area for each image frame, and estimates the positions of a plurality of feature points. This estimation process is performed for temporally continuous image frames. Then, the face area detection / collation unit 120 collates the estimated feature point with a variable template structure (details will be described later) stored in the face feature database unit 160, and the tracking result of the feature point Alternatively, the recognition result is output. That is, the face area detection / collation unit 120 performs tracking or recognition of feature points using a feature point-based face image matching algorithm, and the tracking of the feature points and the detection of the face area itself can be performed using existing techniques. For example, reference [Clippingdale, S., Takayuki Ito, “Unified approach to face detection / tracking / recognition of moving images,” IEICE technical report, PRMU98-200, Osaka University, 1999.] FAVRET system described in references [Clippingdale, S., Ito, T., “A Unified Approach to Video Face Detection, Tracking and Recognition,” Proc. ICIP'99, Kobe, Japan, 1999.] can do. The FAVRET system can deal with video (moving images), and can also deal with head postures that change over time. The data representation in the FAVRET system is based on the data representation in the EBGM algorithm (that is, the position of each feature point and the Gabor wavelet feature measured at that position). In the FAVRET system, the recognition target face images for a plurality of head postures are registered in the face image database in order to realize the correspondence to the plurality of head postures and the operation in the real time video, and the features in each image frame The point search is started from the position estimated in the previous image frame. That is, the FAVRET system reduces the search range by executing the feature point tracking process.

  The face area detection / collation unit 120 estimates the position of a plurality of feature points from the face image area for each image frame, regardless of whether the registration mode or the recognition mode is set. 2D coordinate value) is output as a result of matching. When the recognition mode is set, the face area detection / collation unit 120 calculates the similarity with the face feature data for each registered person stored in the face feature database unit 160 and has the highest similarity. Outputs match score for facial feature data. This match score is, for example, information that associates the name of a registered person or a name for identifying a registered person (such as a nickname), identification information (such as an identification number) of the person, and the calculated similarity. It is.

  FIG. 6 is a diagram schematically showing the feature point positions estimated from the face image area by the face area detection collating unit 120. In the figure, the feature point positions estimated by the face area detection / collation unit 120 correspond to positions corresponding to the eyes, the corners of the eyes, and the mouth of the face image area (two each), and the nose, nose, and tip of the lip. Position (one for each). Note that the face area detection / collation unit 120 estimates only the feature point position of the visible part according to the head posture of the person among the above-described feature point positions.

  Returning to FIG. 5, the three-dimensional estimation unit 130 estimates a three-dimensional position of each feature point, and a generic (person unspecified) three-dimensional computer graphics face mesh model (three-dimensional CG face model, Warping mesh shape of generic model. In this warping, the three-dimensional estimation unit 130 matches the three-dimensional positions of mesh vertices corresponding to these feature points in the three-dimensional CG face model to the estimated three-dimensional positions of the respective feature points, and matches the mesh shape. This refers to processing for generating a corrected three-dimensional CG face model (corrected face model). The three-dimensional CG face model and the modified three-dimensional CG face model are models that represent a solid made up of polygons composed of meshes. These polygons constitute the head-shaped surface in this model. The solid represented by this model according to the human head shape is roughly monotonously convex over the entire head. Then, the 3D estimation unit 130 pastes a texture from the video data to the modified 3D CG face model, and outputs a CG face model to which the face included in the video data is applied.

  As shown in the figure, the three-dimensional estimation unit 130 includes a position / posture estimation unit 131, a mesh warping unit 132, a texture mapping unit 133, a face model storage unit 134, a mesh vertex assignment information storage unit 135, A texture image storage unit 136 and a pixel allocation information storage unit 137 are included.

  The position / orientation estimation unit 131 uses the feature point two-dimensional coordinate value, which is the feature point position in each image frame of the video data, estimated by the face area detection collation unit 120, and the feature point that is the three-dimensional position of each feature point Three-dimensional coordinate values and head posture angle data for each image frame (rotational position (angle) around the vertical axis, or rotational position (angle) around the horizontal axis, or a combination thereof) Are estimated in association with each other.

  The mesh warping unit 132 warps the mesh vertex of the 3D CG face model stored in the face model storage unit 134 to the feature point 3D coordinate value position of each feature point estimated by the position / posture estimation unit 131. Then, a modified three-dimensional CG face model is generated.

The mesh vertex assignment information storage unit 135, when the mesh warping unit 132 warps the mesh vertices of the 3D CG face model, mesh vertices corresponding to the assignment relationship between each mesh vertex and a triangle having the mesh vertex as a vertex. The allocation information is stored.
FIG. 7 is a schematic diagram illustrating a data structure of mesh vertex assignment information stored in the mesh vertex assignment information storage unit 135. As shown in the figure, the mesh vertex assignment information storage unit 135 is tabular data, including a three-dimensional coordinate value of a mesh vertex and a triangle to which the mesh vertex is assigned (including a polygon other than a triangle). Even in the case of being configured, the information is stored in association with the identification information of dividing the polygon into triangles as appropriate, as will be described later. The primary key in this table is the three-dimensional coordinate value of the mesh vertex.

The texture mapping unit 133 converts the texture including the image information such as the shade and color of the video data stored in the image data storage unit 110 into the UV texture image (stored in the texture image storage unit 136 at a weight ratio described later). A CG face model is generated by synthesizing a UV texture image at a position corresponding to the facial feature point in the two-dimensional face texture image) and at least its vicinity and pasting it on the modified three-dimensional CG face model. The UV texture image is face image data in a two-dimensional plane with orthogonal U and V axes. The UV texture image is mapped to the 3D CG face model by associating the pixel position (u, v) (where 0 ≦ u ≦ 1, 0 ≦ v ≦ 1) and the mesh vertex of the 3D CG face model. Is done. As a result of this mapping process, colors and shades of RGB colors are expressed on the texture surface of the modified three-dimensional CG face model.
The texture mapping unit 133 has the same configuration and function as the texture mapping unit 14 described in the first embodiment. Then, as will be described later, the texture mapping unit 133 calculates the pixel displacement amount when the pasted image is pasted on the UV texture image, and aligns the pixel position using the calculated displacement amount.

The pixel allocation information storage unit 137 stores pixel allocation information indicating the correspondence between the pixel position of the UV texture image near the feature point and the assigned triangle when the texture mapping unit 133 pastes the UV texture image.
FIG. 8 is a schematic diagram illustrating a data structure of pixel allocation information stored in the pixel allocation information storage unit 137. As shown in the figure, the pixel allocation information is tabular data, and the pixel position (u, v) of the pixel on the UV texture image and the triangle to which the pixel is allocated (the mesh is a polygon other than a triangle). Are also stored in association with the identification information of dividing the polygon into triangles as appropriate, as will be described later. The primary key in this table is the pixel position of the UV image.

  The rendering unit 140 rotates the CG face model output from the three-dimensional estimation unit 130 to a predetermined head posture, performs a rendering process, and generates a composite face image model. The rendering unit 140 generates a combined face image model of all head postures necessary for registration. The number of head postures necessary for this registration is, for example, for each of 19 patterns in increments of 10 degrees from 0 degrees to 180 degrees in the left-right direction of the person to be registered (the face front is at a position of 90 degrees). The horizontal direction is 0 degree, and there are 5 patterns of 0 degree, ± 15 degrees, and ± 30 degrees in the vertical direction, for a total of 95 patterns.

The database registration unit 150 uses a predetermined number of resolutions and a predetermined number of azimuths for Gabor wavelet features (image feature information) centered on feature points that are visible in each synthetic face image model rendered by the rendering unit 140. Perform the convolution process. Then, the database registration unit 150 generates a variable template structure, which is face feature data necessary for recognizing a registered person, and stores the variable template structure in the face feature database unit 160.
FIG. 9 is a schematic diagram showing the data structure of the variable template structure generated by the database registration unit 150. The variable template structure shown in the figure has, for each registered person, the name of the registered person or a name for identifying the registered person, and person identification information including identification information. Each person has a head posture index (angle data) corresponding to the number of head postures. Furthermore, for each head posture index, feature point information corresponding to the number of feature points is included. The feature point information includes a feature flag (integer starting from 0) as a key, a visibility flag indicating whether the feature point corresponding to the feature point number is visible or not, and the feature of the feature point Point two-dimensional coordinate value (however, the two-dimensional coordinate value is indicated only when the visibility flag is visible), and a predetermined number of resolutions (wavelet size) × Gabor wavelet features (image information in a predetermined number of directions) ) In association with the value.
The head posture index is, for example, a combination of a rotational position (angle) centered on the vertical axis (y direction) of the head and a rotational position (angle) centered on the horizontal (x direction) axis. Is index data representing. However, the head posture index may be data of these angles themselves.

  For example, the database registration unit 150 creates one piece of person identification information and a head posture index corresponding to the 95 types of head postures described above as members of the variable template structure for each registered person. Further, for each head posture, feature point information for each of the nine feature points (feature point numbers from 0 to 8) shown in FIG. 6 is created as a member. Furthermore, for each of these feature points, members of a visibility flag and a feature point two-dimensional coordinate value are created, and 40 Gabor wavelet features in 5 resolutions × 8 directions (every 22.5 degrees). Create the value data as a member.

Next, the operation of each part of the face image processing apparatus 100 will be described.
[Face area detection / collation processing]
FIG. 10 is a flowchart illustrating a procedure in which the face area detection / collation unit 120 detects a face image area from each image frame of video data, collates it with face feature data, and outputs a tracking result or a recognition result. In step S601, the face area detection collation unit 120 sets the mode to either the registration mode or the recognition mode. This mode setting is performed by, for example, designation by operating an operation unit (not shown) of the face image processing apparatus 100.

Next, in step S602, the face area detection collation unit 120 reads the image frames of the video data stored in the image data storage unit 110 in order of oldest time information.
Next, in step S603, the face area detection collation unit 120 moves to the process of step S604 when the image frame is read (S603: YES), and when the image frame is not read (S603: NO), The process ends.

  In step S604, the face area detection collation unit 120 detects a face image area from the read image frame. At this time, for example, the face area detection collation unit 120 scans an image frame to acquire color information, and extracts a pixel of a specific color corresponding to the color of the human body. A method of detecting a face image region using a known face recognition algorithm for recognizing a facial part such as an eye, nose, or mouth as a shape, or a method using a combination thereof is used.

Next, in step S605, the face area detection collation unit 120 estimates a feature point position from the face image area. The feature points are those shown in FIG. In addition, when the site | part of a feature point position is invisible, the face image area | region detection collation part 120 does not estimate the feature point position. The feature point position estimation processing by the face area detection / collation unit 120 is performed as follows, for example. Based on the assumption that the feature point in the current image frame is present at or near the same position as the feature point in the previous image frame, the face area detection / collation unit 120 uses the same position as the initial position and sets the initial position as the center. Search for neighborhoods in range. Since the feature point positions estimated in the past image frames are registered as feature point time-dimensional coordinate values in the variable template structure of the face feature database unit 160, the face image region detection collation unit 120 uses the variable template structure. The initial position can be set by reading the feature point two-dimensional coordinate value of the previous frame from the body. The face image area detection collation unit 120 performs a search by measuring Gabor wavelet features from a low resolution area to a high area. The Gabor wavelet feature is a complex number, and the cosine waveform of the real part and the sine waveform of the imaginary part are 90 degrees out of phase. Therefore, the face area detection collation unit 120 estimates a positional shift using coefficients for each of the real part and the imaginary part, and repeatedly performs a search process while shifting the position. Then, the search process is terminated when the positional deviation converges. The position at this time is estimated to be the feature point position having the maximum similarity, that is, the estimated feature point two-dimensional coordinate value. The number of feature points estimated in the face image area per image frame is _NFP .

  Next, in step S606, the face area detection collation unit 120 divides the process according to the mode set in the process of step S601. If the setting mode is the registration mode (S606: registration mode), the process proceeds to step S607. If the setting mode is the recognition mode (S606: recognition mode), the process proceeds to step S608.

In step S607, the face area detection collation unit 120 in the registration mode outputs the feature point two-dimensional coordinate values that are the _NFP feature point positions estimated in the process of step S605, and returns to the process of step S602. On the other hand, in step S608, the face area detecting matching unit 120 of the recognition mode, outputs the estimated N _FP pieces of two-dimensional coordinate values the feature point in the processing of step S605, the match about the highest face characteristic data of the similarity A score is generated and output, and the process returns to step S602.

[Three-dimensional position / head posture estimation processing of feature points]
FIG. 11 shows a process of estimating a three-dimensional feature point position and a head posture for each image frame from two-dimensional feature point positions (feature point two-dimensional coordinate values) in each image frame of video data. It is a flowchart which shows a procedure. In step S <b> 701, the position / posture estimation unit 131 captures N _FP feature point two-dimensional coordinate values for each of the N _frames image _frames from the face area detection collation unit 120. At this time, the j-th (0 ≦ j <N _FP ) feature point two-dimensional coordinate value in the k-th (0 ≦ k <N _frames ) image frame is represented by an element (x ^k _j , Y ^k _j ) and is represented by a vector y (bold field) ^k _j (hat). Here, the element (x ^k _j , y ^k _j ) is an x-coordinate estimated value of the j-th feature point when the k-th image frame is represented by a two-dimensional plane with orthogonal x-axis and y-axis, y It corresponds to the coordinate estimated value. That is, y (bold field) ^k _j in equation (25) is a two-dimensional vector of real numbers (R). Here, the description “(bold)” indicates that the character immediately before the description is a bold typeface, and that the expression is a matrix or a vector. Similarly, the description “(hat)” means that the value of the expression is an estimated value.

Next, in step S < _b > 702, the position / posture estimation unit 131 determines that the j-th (0 ≦ j <N _FP ) feature point in the k-th (0 ≦ k <N _frames ) image frame is visible or not. Visibility v ^k _j (hat) ε {0, 1} that is estimated to be visible is acquired. Values are 0 for invisible and 1 for visible. In the registration mode, the visibility v ^k _j (hat) for each image frame is input from the outside depending on whether or not the feature point position is visible.

Next, in step S < _b > 703, the position / posture estimation unit 131 stores the mesh vertex position corresponding to the j-th (0 ≦ j <N _FP ) feature point in the three-dimensional CG face model from the face model storage unit 134. The vertex three-dimensional coordinate value m (bold body) _j ∈ R ³ (where 0 ≦ j <N _FP ) is read.

  Next, in step S704, the position / orientation estimation unit 131 calculates a feature point three-dimensional coordinate value, which is a three-dimensional position of each feature point in the video data, based on the data obtained by the processing from steps S701 to S703. The head posture for each image frame is estimated in association with it. This estimation process is a well-known document such as “Simon Clipping Dell, Masato Fujii, Nobuyuki Yagi,“ Study of 3D facial feature point estimation from 2D observation data with shielding and noise ”, IEICE Technical Report, PRMU2008. -42, 2008/06, pp. 133-138. ”, The estimation process of the three-dimensional feature data is applied from the two-dimensional feature data. Alternatively, “face three-dimensional model estimation processing” described later is applied.

Next, in step S705, the position and orientation estimating section 131, _{N FP} pieces of three-dimensional feature point coordinates m _(bold) in the image data _j (hat) ∈R ³ (where, 0 ≦ j _{<N FP)} And head posture Q (bold body) ^k (hat) ∈ R ^{2 × 3} (where 0 ≦ k <N _frames ) in the k-th image frame, and two-dimensional feature points in the k-th image frame Centroid y (bold body) ^k (bar) ∈ R ² (where 0 ≦ k <N _frames ).

  The estimation algorithm in the process of step S704 belongs to the nonlinear conditional least square method estimation class, and minimizes the square error represented by equation (26) under the orthogonality condition represented by equation (27).

In Expression (27), I ₂ is a two-dimensional unit matrix, and λ is a constant.

[Mesh warping process]
Next, the mesh warping unit 132 has a feature point three-dimensional coordinate value m (bold body) _j (hat) ∈ R ³ (where 0 ≦ j <N _FP ) of each estimated feature point, By warping the mesh vertex indicated by the mesh vertex three-dimensional coordinate value x (bold body) ⁿ ∈ R ³ (where 0 ≦ n <N _VT ) of the three-dimensional CG face model stored in the face model storage unit 134 A modified 3D CG face model is generated.
FIG. 12 is a flowchart showing a procedure of mesh warping processing executed by the mesh warping unit 132. FIG. 13 is a diagram showing a mesh of a three-dimensional CG face model. Hereinafter, a description will be given along the flowchart of FIG.

In step S801, the mesh warping unit 132 has a predetermined number of mesh vertices (referred to as feature point vertices) of the 3D CG face model corresponding to the feature points and a boundary portion between the 3D CG face model and the outside world. Designate mesh vertices (called fixed vertices). Then, the mesh warping unit 132 sets N _tri triangles i (where 0 ≦ i <N _tri ) having these mesh vertices as vertices. The coordinates of the vertices of these triangles are x (bold) ^{i, l} , where 0 ≦ i <N _tri , 0 ≦ l (el) <3.
FIG. 13 schematically shows feature point vertices (colored circles), fixed vertices (hatched circles), and triangles having these as vertices.

Next, in step S802, the mesh warping unit 132 obtains the n-th mesh vertex three-dimensional coordinate value x (bold body) ⁿ ∈ R ³ , 0 ≦ n <of the three-dimensional CG face model according to the following equation (28). N _VT is assigned to the i _min (n) -th triangle (where 0 ≦ n <N _vt ) among the triangles i (0 ≦ i <N _tri ) set in step S801. Then, the mesh warping unit 132 writes the mesh vertex three-dimensional coordinate value and the identification information of the triangle to which the mesh vertex is assigned to the mesh vertex assignment information stored in the mesh vertex assignment information storage unit 135. If the mesh vertex allocation information storage unit 135 is referred to when the mesh vertices already calculated in the second and subsequent processes are selected again, the mesh warping unit 132 does not need to perform the calculation process again and is efficient. Can be processed.

  Note that the function f (x) in Expression (28) is a penalty function, and the value of this penalty function is 0 when the mesh vertex is included in the triangle.

Next, in step S803, the mesh warping unit 132 sets the n-th mesh vertex three-dimensional coordinate value x (bold body) ⁿ , 0 ≦ n <N _VT to the n-th modified mesh represented by the following equation (29). Replace with the vertex three-dimensional coordinate value x (bold) ⁿ (hat) (0 ≦ n <N _VT ).

In Expression (29), p = i _min (n), x (bold body) ^{p, l} (hat), 0 ≦ l (el) <3 is the three-dimensional position / head posture of the feature point described above The feature point three-dimensional coordinate value m (bold) _j (hat), 0 ≦ j <N _FP corresponding to the mesh point and the fixed vertex estimated in the estimation process.

FIG. 14 is a diagram for explaining a warping process for mesh vertices of a three-dimensional CG face model. The figure shows three mesh vertex three-dimensional coordinate values x (bold body) ^{p, l} (l (el)) among the feature points and fixed vertices estimated in the above-described three-dimensional position / head posture estimation processing of feature points. ∈ {0, 1, 2}) is replaced with the modified mesh vertex three-dimensional coordinate value x (bold) ^{p, l} (hat), l (el) ∈ {0, 1, 2} estimated from the video data Further, in accordance with the replaced feature point vertex and fixed vertex, the mesh vertex three-dimensional coordinate value x (bold body) ^{n of} the general mesh vertex is corrected mesh vertex three-dimensional coordinate value x (bold body) ⁿ (hat) ) Shows how it was replaced. In the figure, the mesh vertex three-dimensional coordinate value x (bold body) ^{p, 1} is a fixed vertex, and the mesh vertex three-dimensional coordinate value x (bold body) ^{p, l} , l (el) ε {0,2} is Mesh vertices corresponding to feature points.

[Texture mapping process]
When a texture is pasted (mapped) on the modified 3D CG face model generated by warping the face image of the video data in accordance with the 3D CG face model, the modified 3D CG face model after the mapping is added to an arbitrary head. When rendering with the head posture, the facial texture of the person to be registered is reflected in the head posture. Since the texture of the texture reflected on the surface of the rendered modified three-dimensional CG face model is determined by the UV texture image in addition to the head posture and lighting conditions, in order to match the model to the face of a specific person, In this manner, the UV texture image is corrected.

The relationship between the texture-mapped three-dimensional CG face model and the UV texture image has already been shown in FIG. FIG. 6A shows a state in which texture mapping is performed on a three-dimensional CG face model, and FIG. 6B is a diagram in which a mesh is superimposed on a UV texture image. As shown in the figure, when the mesh is superimposed on the UV texture image, the n-th mesh vertex three-dimensional coordinate value x (bold body) ⁿ (0 ≦ n <N _VT ) and the pixel position on the UV texture image. Coordinate values (u ⁿ , v ⁿ ) (0 ≦ u ⁿ ≦ 1, 0 ≦ v ⁿ ≦ 1, 0 ≦ n <N _VT ) are related. Specifically, the data of the n-th mesh vertex three-dimensional coordinate value x (bold body) ⁿ uses the pixel coordinate values (u ⁿ , v ⁿ ) on the UV texture image as a pointer to the UV texture image. By having them in association, both of these are related.

FIG. 15 is a flowchart showing a procedure of UV texture image correction processing. In step S <b> 1201, the texture mapping unit 133 divides a polygon that is a quadrilateral or more forming a mesh of the three-dimensional CG face model into triangles. That is, the texture mapping unit 133 converts an N-gon formed by vertices corresponding to the mesh vertex three-dimensional coordinate value x (bold body) ^j (0 ≦ j <N) of the three-dimensional CG face model to {x ⁰ , x ¹ , x ² }, {x ⁰ , x ² , x ³ },..., {X ⁰ , x ^N−2 , x ^N−1 } (x is a bold body) (N−2) ) Divide into triangles. Thus, when the number of triangles provided on the 3D CG face model by the texture mapping unit 133 is N _uvtri , the i-th triangle {x ^{i, 0} , x ^{i, 1} , x ^{i, 2} } (x is (Bold body) (0 ≦ i <N _uvtri ), i-th triangle {(u ^{i, 0} , v ^{i, 0} ), (u ^{i, 1} , v ^{i, 1} ), (u ⁱ ) in the UV texture image ^{, 2} , v ^{i, 2} )} are associated with each other.

Next, in step S1202, the texture mapping unit 133 sets the pixel corresponding to one mesh vertex among the mesh vertices corresponding to the feature points and the vicinity thereof (within a radius = R pixel) in the UV texture image. Select certain pixels (the coordinates of those pixels are represented by (u, v)).
Next, when the pixel (u, v) is selected in step S1203 (step S1203: YES), the process proceeds to step S1204. On the other hand, when the pixel (u, v) is not selected (step S1203: NO), the process proceeds to step S1210.

In step S1204, the texture mapping unit 133 assigns the selected pixel (u, v) to the i _min (u, v) -th triangle on the UV texture image using Expression (30).

The texture mapping unit 133 uses each pixel (u, v) and the triangle number i _min (u, v) assigned to the pixel as pixel allocation information stored in the pixel allocation information storage unit 137. Write. The texture mapping unit 133 needs to perform the calculation process again if the pixel allocation information storage unit 137 is referred to when the pixel (u, v) already calculated in the second and subsequent processes is selected again. Efficient processing.

Here, i _min (u, v) is expressed as “imin” (the same applies hereinafter). The vertex of the imin-th triangle to which the pixel (u, v) is assigned in the process of step S1204 is defined as (u ^{imin, l} , ^{vimin, l} ) (l (el) = 0, 1, 2), and these correspond The attached three-dimensional coordinate value of the corrected mesh vertex is x (bold body) ^{imin, l} (hat) (l (el) = 0, 1, 2), and the surface method at the vertex of the corresponding three-dimensional CG face model Let the line be n (bold) ^{imin, l} (l (0), 1, 2).

Next, in step S1205, the texture mapping unit 133 estimates a projection position in each image frame of the video data. Specifically, the texture mapping unit 133 calculates the corrected mesh vertex three-dimensional coordinate value x (bold body) ^{imin, l} (hat) (l (el) = 0, 1, 2) by the equation (31) as described above. _Is converted based on the head posture Q (bold body) ^k (hat), 0 ≦ k <N _frames estimated in the process of step S705 of the three-dimensional position / head posture estimation processing of the feature points. The centroid y (bold body) ^k (bar), 0 ≦ k <N _frames of the two-dimensional feature points in the k-th image frame is added to obtain the projection position r (bold body) ^{imin, l} _k (hat). Ask.

This projected position r (bold body) ^{imin, l} _k (hat) is an estimated value of the position at which the vertex is projected onto the kth image frame of the video data when it is not occluded (visible).

Next, in step S1206, the texture mapping unit 133 estimates the projection position of each pixel on the UV texture image in each image frame of the video data. Specifically, the texture mapping unit 133 uses the projection position r (bold body) ^{imin, l} _k (hat) obtained in the process of step ^{S1205 and} b (bold body) ^imin = b () obtained in the process of step S1204. Bold body) A weighted combination r (bold body) _k (u, v) (hat) by ^imin (u, v) is calculated as in the following equation (32), and a pixel (u, v) on the UV texture image is calculated. ), RGB values p (bold body) _k (r (bold body) _k (u, v) (hat)) (0 ≦ k <N _frames ) in the k-th image frame of the video data are obtained. The RGB value p (bold body) _k (r (bold body) _k (u, v) (hat)) is the light / dark information contained in the original image frame.

In step S1207, the texture mapping unit 133 estimates a surface normal at a position corresponding to the pixel (u, v) of the UV texture image in the three-dimensional CG face model. Specifically, the texture mapping unit 133 ^{calculates the} surface normal n (bold body) ^{imin, l} (l (el) = 0, 1, 2), b (bold body) ^imin = determined in the process of step S1204. The weighted combination n (bold body) (u, v) (hat) by b (bold body) ^imin (u, v) is calculated as in the following equation (33).

Next, in step S1208, the texture mapping unit 133 determines the Q (bold body) which is the surface normal rotated based on the head posture Q (bold body) ^k (hat) in the ^k-th image frame of the video data. ) ^k (hat) n (bold) (u, v) (hat) and the line of sight of the optical axis (the optical axis of the camera) ^[001] is the angle between the ^{T θ k (u, v)} ( hat) Is used to calculate the direction weight α ^k (u, v) by the following equation (34). Note that the optical axis of the line of sight is a virtual observer's line of sight that looks at the head posture Q (bold body) ^k (hat) reflected in the image frame, or the head posture Q (bold body) ^k (hat). It is the optical axis of the imaging lens of a virtual camera.

The direction weight α ^k (u, v) enables the value of m in the equation (34) to be adjusted by setting a parameter (second parameter). By adjusting this parameter, the surface normal Q (bold body) ^k (hat) n (bold body) (u, b) at the position where the pixel (u, v) of the UV texture image appears in the k-th image frame of the video data. v) The degree of change in the weight α ^k (u, v) in the direction with respect to the angle between the (hat) and the optical axis of the line of sight (the optical axis of the camera) can be made variable.

Generally, in the rendering process of 3D computer graphics, the burden of Z buffer calculation for determining whether or not a portion of the surface of the 3D model is blocked is large. However, in the texture mapping process according to the present embodiment, since the shape of the human face has a substantially convex shape, the weight α ^k (u, v) in the above direction is used, so that The calculation of whether or not the position r (bold body) _k (u, v) (hat) (0 ≦ k <N _frames ) _appearing in each image frame of the video data of the pixel (u, v) is occluded is omitted. can do. That is, the larger the direction weight α ^k (u, v), the more suitable the position r (bold body) _k (u, v) (hat) in the head posture is facing the camera. Therefore, there is a high probability that the position is not shielded. In addition, the value of the direction weight α ^k (u, v) decreases monotonously as it deviates from that direction.
Due to this property of the direction weight, the texture mapping unit 133 applies the texture included in the image frame to the UV based on the angle between the optical axis of the line of sight with respect to the 3D CG face model and the surface normal of the 3D CG face model. It is preferable to adjust the direction weight α ^k (u, v) when mapping to the texture image. By doing so, the texture resolution can be kept high.

Next, in step S1209, the texture mapping unit 133 determines the feature point three-dimensional coordinate value m (bold body) _j (hat) estimated in step S704 of the feature point three-dimensional position / head posture estimation process described above. , 0 ≦ j <N _FP of pixel distance (u _j , v _j ) on the UV texture image associated with the mesh vertex corresponding to _FP , 0 ≦ j <N _{FP according} to the Euclidean distance in the UV texture image The weight δ (u, v) is calculated by the following equation (35).

The distance weight δ (u, v) allows the value of n in the equation (35) to be adjusted by setting a parameter (first parameter). By adjusting this parameter, when the distance weight δ (u, v) decreases as the distance from the feature point on the UV texture image increases, the degree of change in the distance weight δ (u, v) with respect to the distance is variable. Can be.
Note that the three-dimensional estimation unit 130 stores, in a memory, data obtained as a result of the calculation in steps S1202 to S1209 for each pixel. Then, when the process of step S1209 ends, the process returns to step S1202.

When the process proceeds from step S1203 to step S1210, in step S1210, the texture mapping unit 133 uses the distance weight δ (u, v) and the direction weight α ^k (u, v) to perform the processing on the UV texture image. The RGB vector value at the pixel (u, v) is updated. Specifically, the texture mapping unit 133 calculates (synthesizes) the RGB vector t (bold body) (u, v) at the pixel (u, v) according to the equation (36), and performs an initial operation at the pixel (u, v). The texture value t (bold body) ₀ (u, v) is updated with t (bold body) (u, v).

At this time, the texture mapping unit 133 (texture image writing unit) is a method similar to the method described in the first embodiment, and the pasted image r (bold image) already obtained sequentially from the low resolution to the high resolution. Body) _k (u, v) (hat) is used to generate a reference image with a predetermined resolution, and based on the reference image and each pasted image, a pixel displacement amount of the pasted image is calculated, and Then, the position of the pixel is aligned using the obtained positional deviation amount. At this time, the weight value that has already been calculated, that is, the direction weight α ^k (u, v) in each pixel for each frame image is appropriately used. The sequential alignment processing from the low resolution to the maximum resolution means that each pixel of the pixel (u, v) of the pasted image r (bold body) _k (u, v) (hat) is represented by r ( Bold) _k (u + m ^k (u, v), v + n ^k (u, v)) (hat). In the following processing for pasting to a texture image, the position after the alignment will be described again as (u, v).

In Expression (36), t (bold body) ₀ (u, v) is an initial RGB vector value (texture value) at pixel (u, v) of the UV texture image on the three-dimensional CG face model. The constant γ adjusts the balance between this t (bold body) ₀ (u, v) and the texture to be pasted from the video data to the three-dimensional CG face model. w (bold body) (u, v) is an adjustment amount for adjusting the average luminance of the video data to the average luminance of t (bold body) ₀ (u, v). In this adjustment, for example, the luminance of the video data is adjusted to the luminance of the texture of the three-dimensional CG face model by preprocessing such as histogram equalization.

Further, as shown in Expression (36), the texture mapping unit 133 determines the RGB value p (bold body) _k (r (bold body) _k (u, v) (hat)) in the k-th image frame in the direction. Is calculated by multiplying by the weight α ^k (u, v). That is, the texture mapping unit 133 performs adjustment based on the weight in this direction when mapping the shading / color information included in the original image frame to the two-dimensional face texture image.
Similarly, the texture mapping unit 133 performs a calculation by multiplying the sum of RGB values (adjusted by the weight in the above direction) for N _frames image frames by a weight δ (u, v). Yes. That is, the texture mapping unit 133 performs adjustment based on the weight of this distance when mapping the shading / color information included in the original image frame to the two-dimensional face texture image.
As described above, the value of m in the equation (34) can be adjusted by setting a parameter (second parameter), whereby the surface normal of the three-dimensional CG face model and the optical axis of the line of sight (camera optical axis). The degree of change in the direction weight α ^k (u, v) with respect to the angle between the two is variable. In addition, the value of n in the equation (35) can be adjusted by setting a parameter (first parameter), thereby changing the distance weight δ (u, v) with respect to the distance from the feature point on the UV texture image. The degree is variable. In this way, when both m and n are variable, the balance between the weights of both can be adjusted. For example, when the degree of change in the weight of the direction relative to the angle is adjusted to be relatively gentle, even if the angle formed by the surface normal and the optical axis of the line of sight is large to some extent, It is possible to register in the database a feature amount in which the RGB value in the image frame that is the result of imaging is reflected to some extent.

[Rendering]
The rendering unit 140 rotates the modified 3D CG face model corrected by the 3D estimation unit 130 to a predetermined head posture (θ _y , θ _x1 ), performs rendering processing, and generates a composite face image model. Θ _y is an angle around the vertical axis of the face, and θ _x1 is an angle around the horizontal axis of the face. The rendering unit 140 performs rotation about the vertical axis of the three-dimensional CG face model, and then rotation about the horizontal axis. Therefore, the vertical rotation axis is the y-axis of the three-dimensional CG face model, and the horizontal rotation axis is the horizontal direction and an axis parallel to the face front. That is, the horizontal rotation axis itself rotates by rotation about the vertical rotation axis.

Specifically, the rendering unit 140 determines that each feature point estimated from the feature point three-dimensional coordinate value m (bold body) _j (hat) ∈ R ³ , 0 ≦ j <N _FP is the head posture (θ _y , Visibility v (bold body) ^{θy, θx1} _j ∈ {0, indicating whether or not the image R ^{θy, θx1} (z (bold body)), z (bold body) ∈ R ² rendered with θ _x1 ) 1} (0 ≦ j <N _FP ) and v (bold body) ^{θy, θx1} _j = 1 (visible), the position at which the feature point appears in the image R ^{θy, θx1} (z (bold body)) Rendering processing is performed by calculating z (bold body) ^{θy, θx1} _j . That is, the rendering unit 140 calculates two-dimensional coordinate value data of feature points.

[Database registration process]
The database registration unit 150 is visible (v (bold body) ^{θy, θx1} _j = 1) on the image R ^{θy, θx1} (z (bold body)) rendered by the rendering unit 140. body) ^{[theta] y,} in ^θx1 _j, Gabor wavelet features _{^{f θy, θx1 j (r,}} θ _{a), N res} pieces of resolution _{r∈ {r 0, r 1,} ···, r Nres-1} _{and, N orn} One-point convolution measurement is performed according to the equation (37) with the individual orientations φ∈ {φ ₀ ,..., Φ _Norn−1 }.

Next, the database registration unit 150 generates the variable template structure shown in FIG. 9, stores the data in each member, and registers it in the face feature database unit 160. Specifically, the database registration unit 150 stores, for each registered person, the name of the person to be registered or the name for identifying the registered person, and identification information in the person identification information, and heads corresponding to the number of head postures. The head posture index and the feature point information in each head posture are stored. The head posture index is a head posture (θ _y , θ _x1 ). In the feature point information, the visibility flag v ^{θy, θx1} _j shown corresponding to the feature point number jth (0 ≦ j <N _FP ), and the feature point two-dimensional coordinate value z (bold body) of the feature point ^{[theta] y,} and stores the ^θx1 _{j, N res} number of resolutions (wavelets size) × _{N orn} number of Gabor wavelet features f ^{[theta] y} at an ^{azimuth, θx1} _j ^{(r, θ)} and the.

  As described above, according to the present embodiment, while performing alignment corresponding to the non-linear positional deviation, the labor for registering information related to feature points for a plurality of head postures is reduced, and the registration video is recorded. Alternatively, head postures that are not included in a plurality of still image frames can be easily registered.

  Note that all or part of the functions of the image processing apparatus or the face image processing apparatus in the above-described embodiment may be realized by a computer. In that case, the program for realizing the control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” dynamically holds a program for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It is also possible to include those that hold a program for a certain time, such as a volatile memory inside a computer system serving as a server or client in that case. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

Although a plurality of embodiments have been described above, the present invention can also be implemented in the following modifications.
For example, in the second embodiment, a face image processing apparatus that processes an image of a human face (head) has been described. However, the subject is not limited to this, and an arbitrary subject having a three-dimensional shape is targeted. An image processing apparatus may be used. However, when the subject has a certain degree of rigidity, the estimation accuracy of the positional relationship between the feature points is improved. In addition, when the subject surface undergoes a certain amount of nonlinear deformation (however, this is not the case), an effect specific to the present invention can be obtained.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  The present invention can be widely used for modeling a three-dimensional object based on a captured image. It can also be used in the field of human machine interfaces.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 11 Image data storage part 12 Reverse pose conversion process part 14 Texture mapping part 15 Texture image storage part 51-0, 51-1, 51-2, ... Pasted image acquisition part 52-0, 52- DESCRIPTION OF SYMBOLS 1, ... Warping processing part 53 Reference image generation part 54 Control part 55 Texture image writing part 61-0, 61-1, ... Image addition part 62-0, 62-1, ... Gaussian window processing part 100 face image processing apparatus (image processing apparatus)
DESCRIPTION OF SYMBOLS 110 Image data memory | storage part 120 Face area | region detection collation part 130 Three-dimensional estimation part 131 Position and attitude | position estimation part 132 Mesh warping part 133 Texture mapping part 134 Face model memory | storage part 135 Mesh vertex allocation information storage part 136 Texture image memory | storage part 137 Pixel allocation Information storage unit 140 Rendering unit 150 Database registration unit 160 Face feature database unit

Claims (5)

A pasted image acquisition unit that acquires a pasted image obtained by estimating the positions of the pixels included in the plurality of frame images obtained by capturing the three-dimensional object on the surface of the three-dimensional object;
A reference image generation unit that generates a reference image of the resolution by applying a low-pass filter with a predetermined resolution to the sum of the pasted images acquired by the pasted image acquisition unit;
Based on the pasted image and the reference image, the amount of pixel misregistration included in the pasted image is calculated, and the pixel value of the pixel of the pasted image is separated from the pixel value by the amount of misalignment. The warping processing part to be replaced with,
An image processing apparatus comprising: In the predetermined resolution, the warping processing unit causes the pasted image acquisition unit to acquire the pasted image obtained by replacing the pixel value of the pixel with the pixel value of the pixel separated by the positional deviation amount , A control unit that controls the reference image generation unit to generate the reference image at the next resolution and to sequentially repeat replacement of pixel values of the pixels of the pasted image from the low resolution side to the high resolution side;
The image processing apparatus according to claim 1, further comprising: A texture image storage unit for storing a texture image representing the texture of the surface of the three-dimensional object;
A texture image writing unit for writing the texture image obtained by synthesizing a plurality of the pasting images obtained by the replacement of the warping process the pixel values that by the unit to the texture image storage unit,
The image processing apparatus according to claim 2, further comprising: The three-dimensional object is a face;
A face feature database unit for storing two-dimensional coordinate value data corresponding to facial feature points in association with person identification information and angle data representing a head posture;
A face area detection collation unit that estimates the two-dimensional coordinate value data of the facial feature points included in the read image frame based on the two-dimensional coordinate value data corresponding to the feature points read from the facial feature database unit;
A face model storage unit for storing three-dimensional coordinate value data corresponding to mesh vertices in a predetermined generic model;
Included in the image frame based on the two-dimensional coordinate value data of the feature points estimated by the face area detection collation unit and the three-dimensional coordinate value data corresponding to the mesh vertices read from the face model storage unit A position / posture estimator for estimating three-dimensional coordinate value data of the feature points and angle data representing a head posture of the image frame;
Based on the three-dimensional coordinate value data and the angle data of the feature point estimated by the position / posture estimation unit, a corrected face model is generated by warping the mesh vertex in the generic model, and the mesh vertex is generated. Correspondingly, a mesh warping unit for calculating three-dimensional coordinate value data of mesh vertices in the modified face model,
Based on the corrected face model generated by the mesh warping unit, a plurality of synthesized face image models are generated by performing rendering processing when the angle data representing the head posture is changed, and two-dimensional coordinate values of the feature points A rendering unit for calculating data;
A database registration unit that registers the two-dimensional coordinate value data of the feature points calculated by the rendering unit in the face feature database unit in association with the corresponding angle data;
With
The face feature database unit stores at least image feature information in the vicinity of the feature points in association with the person identification information and the angle data of the head posture;
The rendering unit reads the texture image written by the texture image writing unit from the texture image storage unit, and performs a rendering process based on the texture image.
The database registration unit registers the image feature information based on the result of the rendering process performed by the rendering unit in the face feature database unit in association with the corresponding angle data.
The image processing apparatus according to claim 3. A pasted image acquisition unit that acquires a pasted image obtained by estimating the positions of the pixels included in the plurality of frame images obtained by capturing the three-dimensional object on the surface of the three-dimensional object;
A reference image generation unit that generates a reference image of the resolution by applying a low-pass filter with a predetermined resolution to the sum of the pasted images acquired by the pasted image acquisition unit;
Based on the pasted image and the reference image, a pixel displacement amount included in the pasted image is calculated, and a pixel value of the pixel of the pasted image is calculated based on the obtained positional displacement amount. A warping processor that replaces the pixel values of pixels separated by an amount ;
A computer program that causes a computer to function as an image processing apparatus.

Images