JP6244885B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for processing an image including a face.

  In an image obtained by photographing a spectacle wearer, the wearer's facial contour may be distorted by the spectacles, so that the wearer may be photographed as an unwilling image. In order to recognize such a phenomenon as a problem and improve the appearance of the spectacle wearer on the image, a distortion region where the image looks unnatural is estimated by the spectacles, a virtual contour is obtained, and the surrounding area is obtained by complementing with the skin color data. (Patent Document 1).

  Moreover, as a method for determining the outline of the glasses, there is a method disclosed in Patent Document 2. In this method, the deviation of the face contour is detected to determine the contour of the glasses.

JP 2008-250407 A JP 2010-003232 A

  However, in the conventional image processing technique described above, only the contour line is corrected, and the distance between eyes and the size of the eyes that characterize an individual remain unchanged when wearing glasses, which may be unnatural. Furthermore, assuming that face recognition processing using the pupil distance as a parameter is performed, there is a possibility that the recognition result may be flawed when wearing glasses or not, but the above conventional technique solves this problem. Can not.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an image processing apparatus, an image processing method, and a program for eliminating the unnaturalness of the face image of a spectacle wearer.

  The present invention for solving the problems of the above conventional example is an image processing apparatus, an acquisition means for acquiring an image of a face imaged, and if there is a lens portion of the glasses in the acquired image, the glasses An estimation means for estimating which pixel in the lens portion of the lens and the estimated pixel in the lens portion of the spectacles are mapped to a pixel at another position in the lens portion of the spectacles Conversion means.

  Here, the image processing apparatus further includes an extraction unit that extracts a contour line of a captured face in the acquired image, and the estimation unit uses the extracted face contour line in the acquired image. When it is determined that there is a lens portion, it may be estimated which pixel is in the lens portion of the spectacles.

  Further, the image processing apparatus includes an extraction unit that extracts a contour line of a captured face in the acquired image, and the conversion unit enlarges or reduces an image part in the lens part of the spectacles. Conversion that maps to pixels at other positions may be performed, and the conversion means may set an enlargement or reduction ratio so that the extracted contour lines are continuous.

  The estimation means further estimates a position corresponding to the lens center of the spectacles from the estimated lens portion of the spectacles, and the conversion means performs the enlargement or reduction of the estimated lens center of the spectacles. It is good also as enlarging or shrinking radially centering on the position corresponding to.

  Further, the image processing apparatus further includes means for detecting an image part of a pupil from the face part, and the conversion means determines a position corresponding to the center of the detected image part of the pupil when performing enlargement or reduction. The enlargement or reduction may be performed radially as the center.

  Each of the spectacle lenses further includes means for obtaining conversion table data, which is a parameter relating to mapping of the image in the lens portion, and the conversion means calculates the estimated pixels in the spectacle lens portion. The conversion table data corresponding to the lens type of the spectacles may be used to perform conversion that maps to pixels at other positions in the lens portion of the spectacles.

  In addition, the image processing apparatus further includes an extraction unit that extracts a contour line of a captured face in the acquired image, and the conversion unit includes, as conversion table data corresponding to a lens type of the glasses, It is also possible to select and use conversion table data for which the extracted contour line is determined to be continuous. Further, the conversion means may perform conversion after correcting the face included in the image to face forward.

  The image processing method according to one aspect of the present invention includes a step in which the conversion unit acquires an image in which a face is captured, and the estimation unit has a lens part of the glasses if the acquired image has a lens portion. Estimating which pixel in the portion is, and converting means for converting the estimated pixel in the lens portion of the spectacles to a pixel in another position in the lens portion of the spectacles. Steps to perform.

  According to still another aspect of the present invention, there is provided a program for acquiring an image obtained by capturing an image of a face, a pixel in a lens portion of the spectacles if the acquired image includes a lens portion of the spectacles. And a conversion unit that performs conversion for mapping a pixel in the lens part of the spectacles to a pixel at another position in the lens part of the spectacles. .

  According to the present invention, the unnaturalness of the face image of the spectacle wearer can be eliminated.

It is a block diagram showing an example of an image processing apparatus according to an embodiment of the present invention. It is a functional block diagram showing the example of the image processing apparatus which concerns on embodiment of this invention. It is explanatory drawing showing the example of the process regarding the discontinuous point in the image processing apparatus which concerns on embodiment of this invention. It is explanatory drawing showing the example of a coordinate utilized by the process which concerns on the image processing apparatus which concerns on embodiment of this invention. It is explanatory drawing showing the example of another coordinate utilized by the process which concerns on the image processing apparatus which concerns on embodiment of this invention. It is explanatory drawing showing the example of the storage state of the conversion data table which the image processing apparatus which concerns on embodiment of this invention utilizes. It is explanatory drawing showing a part of example of the content of the conversion data table which the image processing apparatus which concerns on embodiment of this invention utilizes. It is explanatory drawing which represents typically the conversion using the conversion data table by the image processing apparatus which concerns on embodiment of this invention. It is a flowchart figure showing the process example of the image processing apparatus which concerns on embodiment of this invention.

  Embodiments of the present invention will be described with reference to the drawings. As illustrated in FIG. 1, the image processing apparatus 1 according to the present embodiment includes a control unit 11, a storage unit 12, an operation unit 13, a display unit 14, and an input / output unit 15. The

  The control unit 11 according to the present embodiment is a program control device such as a CPU, and operates according to a program stored in the storage unit 12. In the present embodiment, the control unit 11 acquires information of an image in which a face is imaged, and if there is a lens part of the spectacles in the acquired image information, which is the pixel in the lens part of the spectacles. Estimate if there is. The control unit 11 performs conversion for mapping the estimated pixel in the lens portion of the spectacles to the pixel at another position in the spectacle lens portion. Details of the processing of the control unit 11 will be described later.

  The storage unit 12 includes a memory device and a disk device. The storage unit 12 holds a program executed by the control unit 11. The program may be provided by being stored in a computer-readable recording medium such as a DVD-ROM and stored in the storage unit 12. The program may be distributed via a network and stored in the storage unit 12.

  In an example of the present embodiment, the storage unit 12 stores image information to be processed by the control unit 11. The storage unit 12 also operates as a work memory for the control unit 11.

  The operation unit 13 is a keypad or the like, and accepts a user instruction operation and outputs it to the control unit 11. The display unit 14 is a signal generation unit that generates and outputs a television signal representing an image to be displayed in accordance with, for example, an instruction input from the control unit 11. The display unit 14 may include a liquid crystal display or the like and display an image input from the control unit 11.

  The input / output unit 15 is a network interface or the like, for example, and receives information transmitted via the network according to an instruction input from the control unit 11 and outputs the information to the control unit 11. The input / output unit 15 may include a memory card reader. In this case, the input / output unit 15 reads image information and the like from an inserted memory card, for example, and outputs the image information to the control unit 11.

  In an example of the present embodiment, the control unit 11 functionally includes an image acquisition unit 21, an estimation unit 22, and a conversion unit 23 as illustrated in FIG. The image acquisition unit 21 acquires information on the image of the face captured from the input / output unit 15 or the storage unit 12. For example, the image of the face imaged may be selected and read out from the image information stored in the storage unit 12 by selecting the image information of the face imaged by the user's instruction operation on the operation unit 13.

  The estimation unit 22 uses the image information acquired by the image acquisition unit 21 as a processing target, and estimates which of the pixels in the lens portion of the glasses is the image information of the processing target. In addition, this estimation part 22 will complete | finish a process, if there is no part estimated as the pixel in the lens part of spectacles (it is judged that there is no lens part). The conversion unit 23 performs conversion for mapping the estimated pixel in the lens portion of the spectacles to the pixel at another position in the spectacle lens portion. Next, specific processing contents of the estimation unit 22 and the conversion unit 23 will be described together with some examples.

[Example in which the lens part is inferred from the discontinuity of the contour line]
As an example, the estimation unit 22 estimates which pixel is in the lens portion of the glasses as follows. That is, the estimation unit 22 according to an example of the present embodiment extracts a portion that is recognized as a human face from the image information to be processed. A widely known face recognition process can be applied to this process.

  In the following description, the estimation unit 22 and the conversion unit 23 perform processing for each extracted portion (for each face image). Here, for the sake of simplicity, a case where there is only one face image will be described as an example. However, if there are a plurality of faces, the following processing may be repeated for each face.

  The estimation unit 22 includes information representing the position of the pupil, information representing the vertical direction of the face, and information representing a midline passing through the face (a line intersecting the head and the sagittal plane) from the extracted face image portions. For example, information that characterizes the face is generated. Hereinafter, an axis parallel to the median line is referred to as a Y axis. For the Y axis, the upward direction of the face is the positive direction. An axis orthogonal to the Y axis and having the right side of the face (right side facing the face) as the positive direction is defined as the X axis. The origin of each axis is appropriately determined as, for example, the intersection of a line segment connecting the positions of the left and right pupils and the median line. Information such as the position of the pupil is expressed as coordinates on the XY orthogonal axes.

  The estimation unit 22 according to an example of the present embodiment further extracts a face outline from the extracted face image portion. That is, the estimation unit 22 in this example also operates as the extraction unit of the present invention. Since there are widely known methods such as Canny's edge detection method for extracting the face contour, detailed description thereof is omitted here.

  The estimation unit 22 checks whether there is a portion where the extracted contour line is discontinuous. Specifically, the estimation unit 22 sets a plurality of points on the extracted contour line. In general, since the contour line is a closed curve, a point is taken inside the closed curve (may be the center of the face), and a plurality of virtual line segments are drawn radially from this point at a predetermined angle. These intersection points may be obtained and these intersection points may be set as the plurality of points.

  The estimator 22 generates a curve representing a contour with at least one (generally a plurality) B-spline curve (or a group of B-spline curves if there are a plurality) for a plurality of points set on the contour line. Perform the process. At this time, a plurality of sets of B-spline curve group candidates are obtained while different points are used for the curve fitting process, and the fitting is evaluated according to a predetermined information criterion for each set. The pair of B spline curves having the highest evaluation is referred to. If there is a pair of B spline curves having the same point as an end point (node) in the group of B spline curves included in the group, the pair of B spline curves is included. Spline curve whose length in the Y-axis direction is longer than the length in the X-axis direction (the absolute value of the difference between the Y-coordinate components at both ends of the B-spline curve is greater than the absolute value of the difference in the X-coordinate components) Select. Then, it is estimated that the outline of the selected B-spline curve is discontinuous at the position of the node at the same position as the other B-spline curve, and the coordinates of the node are determined as discontinuous points (discontinuous points). ) As coordinates.

Here, since the contour line is originally a closed curve, when a discontinuous point is found in the selected B-spline curve β1, the node of the same discontinuous point as the selected B-spline curve β1 (one side end) The other side end of the other B-spline curve β2 having the above is also a discontinuous point. That is, both ends of the B-spline curve β2 are discontinuous points. Further, another B spline curve β3 having the same discontinuous node as the other end of the B spline curve β2 is referred to as a “B spline curve to be adjacent”. Then, the estimation unit 22 is a discontinuous point in a selected B-spline curve and a discontinuous point in a B-spline curve that should be adjacent to the discontinuous point. Discontinuous points connected by a B-spline curve are set as a pair, and each coordinate is stored. Further, at this time, the B spline curve including the discontinuous points and selected as described above, as illustrated in FIG.
(1) A B-spline curve including discontinuous points, where the Y coordinate of the discontinuous point is larger than the Y coordinate of the pupil position E, and the Y coordinate of the other end point is larger than the Y coordinate of the discontinuous point ( B1)
(2) A B-spline curve including discontinuous points, where the Y coordinate of the discontinuous point is larger than the Y coordinate of the pupil position E, and the Y coordinates of the other end points are smaller than the Y coordinate of the discontinuous point ( B2)
(3) A B-spline curve including a discontinuous point, where the Y coordinate of the discontinuous point is smaller than the Y coordinate of the pupil position E, and the Y coordinate of the other end point is larger than the Y coordinate of the discontinuous point ( B3)
(4) A B-spline curve including discontinuous points, where the Y coordinate of the discontinuous point is smaller than the Y coordinate of the pupil position E, and the Y coordinates of the other end points are smaller than the Y coordinate of the discontinuous point ( B4)
There are four cases. The discontinuous points included in the B1 B spline curve are the outer discontinuous points, the discontinuous points included in the B2 B spline curve are the inner discontinuous points, and the discontinuous points included in the B3 B spline curve. The inner discontinuous point and the discontinuous points included in the B4 B-spline curve are associated with attribute information indicating that they are outer discontinuous points. One of the coordinates of the discontinuous points included in the stored set is the coordinate of the outer discontinuous point, and the other is the coordinate of the inner discontinuous point.

  Discontinuity detection methods using B-spline curves are described in Imoto, Konishi, “Discontinuous curve estimation and change point search using B-splines”, Hiroshima University lecture (http://bonsai.hgc.jp/ ~ imoto / imoto_hiroshima2000.pdf), which is a well-known method, so a more detailed explanation is omitted here.

  In addition, here, the edge detection is used for the detection of the contour line, and the method using the B spline for the detection of the discontinuous point is exemplified. However, the detection of the contour line of the face and the detection method of the discontinuous point in the present embodiment are Other methods may be used.

  For example, when there is a discontinuous point detected by the above processing, the estimation unit 22 assumes that the face is worn with glasses with a lens frame at the position of the discontinuous point, and represents the position of the discontinuous point. Is output. In consideration of the fact that glasses are usually in the vicinity of the eyes, the estimation unit 22 detects eyes from the face (there is a widely known method for eye detection, so a detailed description here) Is a face in which eyeglasses with a lens frame are worn at the position of the discontinuous point when a discontinuous point is detected within a predetermined radius range centered on the detected eye position) Information indicating the position of the discontinuous point may be output.

  Specifically, as illustrated in FIG. 4A, a plurality of sets of points (discontinuous points) 101 on the contour line that are discontinuous by the spectacle lens are generally detected by the processing so far. In FIG. 4A, the B-spline curve β1 including the end point 101b, the B-spline curve β2 connecting the end point 101b and the end point 101a, and the end point 101a and the end point 101c, which correspond to the outer contour portion of the lens frame of the glasses. And a B-spline curve β3 (representing the contour portion inside the lens frame of the glasses) are detected. Of these, B-spline curves β1, β3 having a length in the Y-axis direction longer than the length in the X-axis direction are selected, and discontinuous points 101b, 101a, 101c,. Further, from the relationship between the discontinuous point and the other end points in the selected B-spline curve, the discontinuous point 101b is set as an outer discontinuous point, and the discontinuous points 101a and 101c are set as inner discontinuous points. Furthermore, since the discontinuous points 101a, b in the B spline curves β1, β3 to be adjacent to each other are connected by the B spline curve β2, both ends of which are discontinuous points, the estimator 22 uses these discontinuous points 101a, 101b, Each coordinate of b is stored as a set. In the same manner, the estimating unit 22 then sets the discontinuous points 101a, b,..., P, q (the coordinates of the discontinuous points 101a, 101b are stored as a set in the example of FIG. 4A). Using the left and right pupil positions 102L and 102R specified above, it is estimated which pixel is in the lens portion by the following process, for example.

  That is, the estimation unit 22 according to an example includes a pupil position 102L on the left side (X-axis negative direction) of the detected face midline, and further to the left side (X-axis negative direction) of the pupil position 102L. The rectangle 103L circumscribing the discontinuous points 101a, b,. The estimation unit 22 also includes a pupil position 102R on the right side (X-axis positive direction) of the detected face midline, and a right side (X-axis positive direction) further than the pupil position 102R. A rectangle 103R circumscribing the continuous points 101p, q... Is obtained.

In addition, the estimation unit 22 according to another example of the present embodiment detects an eye range including the pupil position 102L on the left side of the detected face midline (this range is detected in the process during face recognition). And a point A located at a position separated by a predetermined distance on the midline side from the right end side (end on the midline side of the eye) of the eye range. This predetermined method includes a method in which the distance is set as a constant in advance and a value obtained by multiplying the distance from the right end of the eye range to the median line by a predetermined constant (a constant less than 1). There can be.
The estimation unit 22 may obtain a rectangle 103L circumscribing the point A and the discontinuous points 101a, b,... On the left side of the pupil position 102L.
In this example, the estimation unit 22 specifies the eye range including the pupil position 102R on the right side of the detected face midline, and is more than the left end side (end of the eye midline side) of the eye range. A point A 'located at a distance determined by a predetermined method on the midline side is obtained by the same method as described above.
The estimation unit 22 may obtain a rectangle 103R circumscribing the point A ′ and the discontinuous points 101p, q... On the right side of the pupil position 102R.
In the estimation unit 22 according to still another example of the embodiment of the present invention, as illustrated in FIG. 4B, the midline passes through the pupil position 102L on the left side of the detected midline of the face. .., And points 101a ′, 101b ′,... Located at positions symmetrical to the discontinuous points 101a, b,. The estimation unit 22 in this example may obtain the discontinuous points 101a, b... And a rectangle 103L circumscribing the obtained points 101a ′, 101b ′.
The estimation unit 22 in this example also uses a line segment l ′ that passes through the pupil position 102R on the right side of the detected midline of the face and is parallel to the midline, and the discontinuous point 101p, Points 101p ′, 101q ′,... at positions symmetrical to q. The estimation unit 22 in this example may obtain discontinuous points 101p, q... And a rectangle 103R circumscribing the obtained points 101p ′, 101q ′.
Then, the estimation unit 22 according to the present embodiment uses information (for example, a set of vertex coordinates of each of the rectangles 103L and R) that specifies the rectangles 103L and R obtained by any of the methods illustrated here as a lens portion. Information to be expressed (information defining pixels in the lens portion) is output. At this time, the estimation unit 22 may also output information representing the position coordinates of the pupil positions 102L and R and the discontinuous points 101a, b,... P, q,.

  The conversion unit 23 performs conversion for mapping the pixel in the lens part of the spectacles estimated by the estimation unit 22 to the pixel at another position in the lens part of the spectacles. Specifically, the conversion unit 23 according to an example includes, for each of the rectangles 103L and R specified by the information output from the estimation unit 22, a pupil position 102 in the rectangle and a discontinuous point in the rectangle. The following processing is executed using a set of 101 coordinates.

  For each set of coordinates of the discontinuous point 101, the conversion unit 23 associates the discontinuous point 101i associated with the attribute information of the inner discontinuous point and the attribute information of the outer discontinuous point included in the set. The distances (may be Euclidean distances) between the discontinuous point 101e and the pupil position 102 are calculated as di and de, respectively. Further, the ratio m = de / di (corresponding to the enlargement / reduction ratio) is calculated.

  Then, the conversion unit 23 performs an enlargement / reduction process for scaling the pixel estimated to be in the lens portion of the glasses to a position m times the distance from the pupil position. Specifically, the conversion unit 23 virtually stores a pixel block (converted pixel block) having the same shape and size as the pixel block (original pixel block) in the lens portion of the glasses, such as the rectangle 103, in the storage unit 12. Then, each pixel of the converted pixel block is sequentially selected as a target pixel, and a distance d and direction (an angle from the X axis, etc.) from the pupil position to the target pixel are calculated. Then, the conversion unit 23 sets a value of a pixel at a distance of d / m from the pupil position in the same direction as the value of the target pixel in the original pixel block. Alternatively, the conversion unit 23 weights the value of the target pixel by averaging the plurality of pixels in the vicinity of the point having a distance of d / m in the same direction from the pupil position (the distance from the point to the center of each pixel). Set to the value of When the setting of all the pixel values in the post-conversion pixel block is completed, the conversion unit 23 copies the value of each pixel in the post-conversion pixel block to the value of each corresponding pixel in the original pixel block. The process ends.

  As a result, a part of the face imaged in a distorted manner by the lens is enlarged / reduced radially around the pupil position, and an image approximate to an image estimated to have been imaged without the lens is obtained.

[Example of processing by estimating the lens center instead of the pupil center]
In the above example, the enlargement / reduction process is executed with the pupil position as the center. However, the present embodiment is not limited to this, and the center of the lens may be the center of the enlargement / reduction process. In this case, the estimation unit 22 detects a frame of the glasses (each unit such as a rim, a bridge, and an armor) from the face image portion. The detection process is described by Yasuyuki Saito, Yukiko Kenmochi, Kazuya Kotani, “Extraction and removal of eyeglass frame regions from facial images using parametric eyeglass frame models”, IEICE Transactions D, J82-D2 (5): Since a widely known method such as the method described in 880-890, 1999 (particularly the eyeglass frame extraction process of FIG. 4 in the same paper) can be adopted, detailed description thereof is omitted here.

  The estimation unit 22 estimates which pixel is in the lens portion from the detected spectacle frame, and calculates the coordinates of the center of the lens. Specifically, there are types of spectacle frames such as a full rim in which the rim surrounds the entire lens, a half rim in which the rim is arranged only at a part of the upper (or lower) portion of the lens, a two-point without rim, and the like. However, since the glasses without the bridge and the armor are not generally used, the estimation unit 22 uses the eyeglass frame for the ends of the bridge and the armor (corresponding to both ends of the lens in the X-axis direction). The coordinate information is always obtained.

  Below, the process which estimates the pixel in the lens part in the case of various frames of a full rim, a half rim, and a two-point by the estimation part 22 is demonstrated.

(A) In the case of a full rim In the case of a full rim, the estimation unit 22 can obtain the coordinates of each pixel on the contour of the rim by the method exemplified above. The contour of the rim can be estimated as it is as the outer peripheral contour of the lens portion, and the pixels within the range surrounded by the pixels on the contour of the rim are estimated as the pixels in the lens portion. Therefore, the estimation unit 22 may calculate the average (or arithmetic average) of the coordinates of each pixel on the contour of the rim, and use the result of the calculation as the coordinates of the center of the lens.

(B) In the case of a half rim In the case of a half rim, the estimation unit 22 can obtain the coordinates of some pixels on the contour of the rim by the method exemplified above. In this example, the estimation unit 22 also detects discontinuous points of the face contour as described above. Then, the coordinates of some pixels on the contour of the rim, the coordinates of the ends of the bridge and the armature, and the coordinates of the discontinuous points are obtained.

  Since each of these coordinates is considered to be in the outer peripheral portion of the lens, the estimation unit 22 forms a contour line including these coordinates by using a curve fitting technique such as a Snakes curve, and this formed contour Processing is performed assuming that the line is the outer peripheral contour of the lens portion. In other words, in this example, the estimation unit 22 calculates the average of the coordinates of each pixel on the formed contour line (which may be an arithmetic average) and sets the result of the calculation as the coordinate of the center of the lens.

(C) In the case of two points In the case of glasses without a rim such as two points, the estimation unit 22 also detects discontinuous points of the face contour as described above. And the estimation part 22 will acquire the coordinate of a bridge | bridging, the edge part of an armor, and a discontinuous point. Since each of these coordinates is considered to be in the outer peripheral portion of the lens, the estimation unit 22 forms a contour line including these coordinates by using a curve fitting technique such as a Snakes curve, and this formed contour Processing is performed assuming that the line is the outer peripheral contour of the lens portion. In other words, in this example, the estimation unit 22 calculates the average of the coordinates of each pixel on the formed contour line (which may be an arithmetic average) and sets the result of the calculation as the coordinate of the center of the lens. Alternatively, the estimation unit 22 obtains a rectangle circumscribing the obtained bridge and end of the arm and the coordinates of the discontinuous points, and performs processing on the assumption that this rectangle is the outer peripheral contour of the lens portion. That is, in this example, the estimation unit 22 may calculate the obtained rectangular center coordinates (intersection of diagonal lines) and set the result of the calculation as the coordinates of the center of the lens.

  The estimation unit 22 outputs information for specifying the outer contour of the lens portion obtained by the above processing and the coordinates of the center of the lens portion.

  The conversion unit 23 performs conversion for mapping pixels in the outer periphery contour (in the lens portion) of the lens portion specified by the information output from the estimation unit 22 to pixels at other positions in the lens portion of the spectacles. Specifically, as illustrated in FIG. 5, the conversion unit 23 associates attribute information of inner discontinuous points included in each set of coordinates of the discontinuous points 101 obtained by the estimation unit 22. The distances (which may be Euclidean distances) between the discontinuous point 101i, the discontinuous point 101e associated with the attribute information of the outer discontinuous point, and the coordinates of the center 104 of the lens are calculated as di and de, respectively. Further, the ratio m = de / di (corresponding to the enlargement / reduction ratio) is calculated.

  Then, the conversion unit 23 changes each pixel in the outer peripheral contour of the lens part (in the lens part) specified by the information output from the estimation unit 22 to a position m times the distance from the center of the lens center 104. Execute scaling processing to double. Specifically, the conversion unit 23 virtually generates a pixel block having the same shape as the pixel block in the outer peripheral contour of the lens portion specified by the information output from the estimation unit 22 in the storage unit 12. Then, the conversion unit 23 sequentially selects each pixel in the virtually generated pixel block as a target pixel, and the distance d and direction (angle from the X axis) from the pixel corresponding to the center 104 of the lens to the target pixel. Etc.). Then, the conversion unit 23 sets a value of a pixel at a distance of d / m from the lens center 104 in the above direction in the pixel block in the original face image as the value of the target pixel. Alternatively, the conversion unit 23 weights the value of the target pixel by averaging a plurality of pixels in the vicinity of a point having a distance of d / m from the lens center 104 in the above direction (the distance from the point to the center of each pixel). To the value of). When the setting of all the pixel values in the virtually generated pixel block is completed, the conversion unit 23 copies the value of each pixel in the pixel block to the value of each corresponding pixel in the original face image. The process ends.

  As a result, a part of the face imaged in a distorted manner by the lens is enlarged and reduced radially around the center 104 of the lens, and an image approximating an image estimated to have been imaged without the lens is obtained.

[Another example of lens part detection]
In another example of the present embodiment, as processing for estimating which pixel in the lens portion is the estimation unit 22, information on the brightness of each pixel in the face portion is generated, and the brightness of adjacent pixels is calculated. By detecting the part where the difference exceeds the predetermined threshold as a tint edge, curve fitting is performed between the pixels where the edge is detected around the pupil position, and which pixel is in the lens part May be estimated. In this case, frame detection is not always necessary.

  In the description so far, as a process in which the estimation unit 22 estimates which pixel is in the lens portion, a method of finding and using discontinuous points on the face contour, and a method of detecting a frame of glasses Although the method using the brightness difference of the pixels has been described, these may be used not only independently but also in combination. That is, it is possible to estimate all of the pixels that are determined to be in the lens portion by any method as the pixels in the lens portion, or the lens portion by any one of a plurality of methods (or all methods). It is also possible to estimate all of the pixels in the lens portion as pixels in the lens portion.

[Another example of conversion processing]
Furthermore, in an example of the present embodiment, the conversion unit 23 may perform the following processing instead of performing the enlargement / reduction processing inside the image portion that is the lens portion as described above.

  That is, in another example of the present embodiment, the storage unit 12 stores in advance the lens portion for each type of eyeglass lens (the type differs depending on the lens “degree”, whether or not astigmatism correction is performed). The conversion table data, which is a parameter relating to the mapping of the image, is associated and stored (FIG. 6). Here, as illustrated in FIG. 7, the conversion table data includes N × M (N and M are positive integers) pieces of two-dimensional coordinate information (dest). Specifically, in the conversion table data illustrated in FIG. 7, in the two-dimensional coordinate information (dest), i is moved to the right from the upper left corner and j is moved downward (where i ≦ N, j ≦ M). The coordinate information (coordinate information of the position of (i, j)) of a certain pixel is set as (i ′, j ′), for example. This means that the pixel located at the position (i ′, j ′) in the original image is moved to the position (i, j).

  Converter 23, when performing the conversion, the center (pre what pixels previously determined whether the pixel of the center, or the (N / 2, M / 2) of the two-dimensional coordinate information (dest) And the coordinates of the center of the lens portion estimated by the estimation unit 22 are made to coincide with each other. Then, the conversion unit 23 refers to the value on the two-dimensional coordinate information (dest) corresponding to each pixel included in the lens portion estimated by the estimation unit 22, and generates an image in the lens portion in which the pixel value is mapped. Then, the value of each pixel in the lens portion in the original image is replaced with the value of the corresponding pixel in the generated image.

  This process is schematically described with reference to FIG. In FIG. 8, it is assumed that there are 3 × 3 pixels corresponding to the image in the original lens portion, and the pixel values are A, B,..., I from the upper left to the lower right. . The center is the pixel at the position of the second row and the second column. Also, here, copying the pixel value of the η-th row ξ-column in the original pixel block is expressed as (ξ, η), and as 2D coordinate information (dest) for 5 × 3 pixels, from the upper left to the right Go to (1,1), (3,1), (3,1), (3,1), (5,1) down one line and again from the left end to (1,2), (3,2), (3,2), (3,2), (5,2) and (1,3), (3,3), (3,3), (3,3), (5) , 3). The center of the two-dimensional coordinate information (dest) for 5 × 3 pixels is the pixel at the position of the second row and the third column.

  In this case, the conversion unit 23 newly generates a 3 × 3 image having the same shape as the image of the original lens portion (the center is the same as the original image in the second row and the second column). The pixel at the position of the first row and the first column in the generated image is set as the target pixel. When the conversion unit 23 matches the second row and third column, which is the center of the two-dimensional coordinate information (dest), with the second row and second column, which is the center of the generated image, the conversion unit 23 To find the pixel corresponding to the pixel of interest. In this case, since the first row and the second column, which are pixels moved by one pixel from the center second row and the third column to the left column and the first row, are the corresponding pixels, the value (3, 1) is read out. This position (3, 1), that is, the first row and the third column corresponds to a pixel in the first row and the second column (pixel having a pixel value of B) in the original image of the lens portion.

  Then, the conversion unit 23 sets the pixel value at the position of the first row and the first column of the generated image to the pixel value “B” at the position of the corresponding first row and the second column in the original lens portion image.

  Thereafter, the conversion unit 23 sets the pixel value at the position of the first row and the second column to the pixel value “B” of the first row and the second column in the original image of the lens by the same procedure, and sets the first row and the third column. Is set to the pixel value “B” in the first row and the second column in the original lens portion image, and so on. When the setting of each pixel of 3 × 3 pixels having the same shape as the original lens partial image is completed in this way, the conversion unit 23 replaces the image for which this pixel has been set with the original lens partial image. In this example, the image in the lens portion is replaced with the state where the pixels in each row are “B”, “E”, and “H”, respectively.

  In addition, although a method of directly describing the corresponding pixel is adopted here, the present embodiment is not limited to this. If a so-called transformation matrix (affine transformation matrix, perspective transformation matrix, etc.) can be generated, a transformation matrix is generated, and the transformation matrix is stored in the storage unit 12 for each lens type as transformation table data. Also good. When performing conversion in this case, the conversion unit 23 maps the image in the lens portion using the conversion matrix. Since this process is widely known as a process using a remap function in OpenCV, for example, detailed description thereof is omitted.

When the conversion unit 23 according to such an example is used, the conversion unit 23 sequentially reads conversion table data corresponding to each lens type stored in the storage unit 12, and converts the image in the lens portion into the respective conversion table. An image after conversion using data is generated. Then, among the generated images, for each set estimated by the estimation unit 22, an image in which the difference between the positions of the pixels corresponding to the two discontinuous points is minimized is selected, and the image in the original lens portion is selected. May be replaced with the selected image. That is, conversion table data determined to be continuous with the extracted contour line may be selected and used.
Further, the conversion table data may be prepared for each type of eyeglass lens and for each face direction. In this example, the conversion unit 23 acquires information on the angle of the face image portion when the face image portion is detected. For obtaining this angle information, a widely known method such as a method of detecting by the position of the nose may be employed.
The conversion unit 23 may select conversion table data for which the extracted contour line is determined to be continuous from the conversion table data for each lens type corresponding to the acquired angle.

According to this example, conversion that is not limited to simple enlargement / reduction is possible, and therefore, it is possible to deal with glasses that may cause complex image distortion, such as bifocal glasses. Further, by selecting the conversion table data, it is possible to estimate the lens type of the corresponding eyeglass, and the estimation result of this lens type can also be used for another process.
Further, in the selection of the conversion table data, for each person, the conversion table data corresponding to the lens of the glasses worn by the person may be stored in association with the information specifying the person in advance. Good. According to this example, when conversion table data is selected by the above method, information associated with the selected table data is output, so that processing such as specifying a person based on the information can be performed.
Alternatively, when the image processing apparatus 1 according to the present embodiment is not used for personal authentication but is used for the purpose of reducing a sense of incongruity when wearing glasses, the converting unit 23 indicates that the extracted contour line is continuous. Instead of selecting conversion table data determined to be, or in addition to this, the person is recognized while wearing glasses, or the user is input information specifying the person, and the result of the recognition or input It is also possible to select conversion table data associated with information for specifying a person specified by the specified information.
Also in this case, conversion table data may be prepared for each piece of information for identifying a person and for each face direction. Also in this case, the conversion unit 23 acquires information on the angle of the face image portion when the face image portion is detected. Then, the conversion unit 23 converts the conversion table data for each piece of information specifying the person corresponding to the acquired angle, the conversion table associated with the result of recognition or the information specifying the person specified by the input information. The data should be selected.

  In the processing described so far, after the face image portion is detected, processing for converting the face image portion to the front direction may be performed, and then the processing of the estimation unit 22 and the conversion unit 23 may be performed. In this case, a process of returning to the original angle after conversion may be performed. For example, Yasuhiro Mukakawa, Yuichi Nakamura, Yuichi Ota “Generating face images with arbitrary direction and expression by combining multiple face images”, electronic information There are technologies such as IEICE Transactions D-II, J80-D2 (6): 1555-1562, 1997, and these can be adopted. Various methods are known, such as a method for estimating the orientation of the face based on the relative position between the median line and the pupil, and thus detailed description thereof is omitted here.

  The image processing apparatus 1 according to an example of the embodiment of the present invention has the above configuration and operates as follows. The image processing apparatus 1 that has received the image to be processed starts the process illustrated in FIG. 9 and detects a face portion from the image to be processed (S1). Then, one of the detected face portions that has not yet been selected is selected as the attention portion (S2), and information on the facial feature amounts (such as pupil position information and contour line information) included in the attention portion is selected. Is generated (S3), and a process of detecting the pixel in the lens portion of the glasses from the target portion and attempting to estimate the lens center is executed (S4).

  In this process S4, as described above, the process of detecting the frame of the glasses, the difference in brightness, the discontinuity of the contour line, and the like are used to detect which pixel in the lens portion is. If the center of the lens part (position corresponding to the lens center of the glasses, hereinafter referred to as the lens center) can be estimated by the process of detecting the frame of the glasses or the process of using the brightness difference, the coordinates of the lens center Information will also be obtained. In general, a spectacle lens is formed by processing a circular lens in accordance with the shape of the frame, but the center of the lens before processing is the center of the lens. The position corresponding to the center of the lens is usually processed and adjusted so as to overlap the position of the pupil when the spectacle wearer faces the front. If only the discontinuous point can be detected, it is impossible to estimate the center of the lens. In this case, the discontinuous point on the contour line is a detection result indicating which pixel is in the lens portion.

  The image processing apparatus 1 checks whether or not the pixel in the lens portion has been detected in the process S4 (whether a discontinuous point has been found or a frame has been detected) (S5). If yes (Yes), it is further determined whether or not the lens center has been found (S6).

  If the lens center is found here (if Yes), a conversion process for converting the image in the lens is executed around the lens center (S7). In the conversion processing of this processing S7, the lens is enlarged or reduced radially around the center of the lens, the coordinates of the discontinuous point on the contour line (the position of the pixel at the position of the discontinuous point) are converted, and the contour line is continuous. You may perform the process to convert. In addition, as processing of this processing S7, the coordinates of the discontinuous points on the contour line (pixel positions at the discontinuous points) are converted by the information directly representing the mapping destination of the pixel value, and the contour lines are continuous. The conversion process may be performed as described above. Further, in this processing S7, the coordinates of the discontinuous points on the contour line (the positions of the pixels at the discontinuous points) are converted by the mapping process using the transformation matrix so that the contour lines are continuous. You may perform the process converted into. Since the lens portions of the glasses are generally on the left and right sides of the face, the image processing apparatus 1 performs the processes from S4 to S7 for each.

  The image processing apparatus 1 checks whether or not processing for all face images included in the image to be processed has been completed (S8). When it is determined that the processing for all the face images has been completed (in the case of Yes), the processed image is output via the display unit 14 or the input / output unit 15 (or stored in the storage unit 12). The process ends. If it is determined in process S8 that there is an unselected face image (in the case of No), the process returns to process S2 and continues.

  If it is determined in process S5 that it is not possible to detect which pixel in the lens portion is (that is, No), the process proceeds to process S8 and the process is continued. Further, if it is determined in step S6 that the lens center has not been found (if No), a conversion process for converting the image in the lens around the pupil position is executed (S9). In this process S9, for example, the enlargement / reduction is performed radially around the pupil position, the coordinates of the discontinuous points on the contour line (the position of the pixel at the position of the discontinuous point) are converted, and the contour line is converted to be continuous. Perform the process. Since the lens portions of the glasses are generally on the left and right sides of the face, the image processing apparatus 1 executes the process of process S9 for each. Then, the image processing apparatus 1 proceeds to the process S8 and continues the process.

  According to the present embodiment, when the center of the lens is found, distortion caused by the lens of the glasses, including the position of the pupil, can be eliminated, so that it is used when performing human face recognition such as the pupil interval. Parameters are also corrected, and the accuracy of face recognition can be improved. When the type of spectacle lens can be specified, processing such as estimating the type of spectacle lens currently used can be executed based on the face image.

  Furthermore, in the above-described processing according to the present embodiment, processing is performed to convert the discontinuous points of the contour line for each of the pair of spectacle lenses. That is, the enlargement / reduction ratio (ratio m) is calculated for each lens, but the present embodiment is not limited to this. For example, when the enlargement / reduction ratio can be calculated only for one lens depending on the orientation of the face, conversion using this enlargement / reduction ratio may be performed for the other lens.

  Furthermore, in the present embodiment, the conversion process by the conversion unit 23 is performed only when the ratio m corresponding to the enlargement / reduction ratio exceeds 1, that is, in the case of a near-eye spectacle lens (the contour line is distorted toward the face center). Also good.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 11 Control part, 12 Storage part, 13 Operation part, 14 Display part, 15 Input / output part, 21 Image acquisition part, 22 Estimation part, 23 Conversion part

Claims (12)

Obtaining means for obtaining an image of a face imaged;
If there is a lens part of the spectacles in the acquired image, an estimation means for estimating which pixel is in the lens part of the spectacles;
Conversion means for performing conversion to map pixels in the estimated lens portion of the spectacles to pixels at other positions in the lens portion of the spectacles; and
Extracting means for extracting the contour of the face imaged in the acquired image;
The estimating means further estimates a position corresponding to the lens center of the glasses from the estimated lens portion of the glasses,
The converting means performs conversion for mapping to pixels at other positions in the lens portion of the spectacles by enlarging or reducing an image portion in the lens portion of the spectacles, and the converting means converts the extracted contour line Conversion means for setting an enlargement or reduction ratio so as to be continuous, and when enlarging or reducing, image processing is performed to enlarge or reduce radially about a position corresponding to the estimated lens center of the eyeglasses apparatus. Obtaining means for obtaining an image of a face imaged;
If there is a lens part of the spectacles in the acquired image, an estimation means for estimating which pixel is in the lens part of the spectacles;
Conversion means for performing conversion for mapping pixels in the estimated lens portion of the spectacles to pixels at other positions in the lens portion of the spectacles;
Extraction means for extracting the contour of the face imaged in the acquired image; and
Means for detecting an image portion of the pupil from the face portion;
The converting means performs conversion for mapping to pixels at other positions in the lens portion of the spectacles by enlarging or reducing an image portion in the lens portion of the spectacles, and the converting means converts the extracted contour line Is a conversion means for setting an enlargement or reduction ratio so as to be continuous, and when enlargement or reduction is performed, enlargement or reduction is performed radially about a position corresponding to the center of the image portion of the detected pupil. Image processing device. Obtaining means for obtaining an image of a face imaged;
If there is a lens part of the spectacles in the acquired image, an estimation means for estimating which pixel is in the lens part of the spectacles;
Conversion means for performing conversion to map pixels in the estimated lens portion of the spectacles to pixels at other positions in the lens portion of the spectacles; and
For each type of eyeglass lens, including means for obtaining conversion table data that is a parameter relating to mapping of an image in the lens part,
The conversion unit maps the pixel in the estimated spectacle lens portion to a pixel at another position in the spectacle lens portion using the conversion table data corresponding to the type of the spectacle lens. An image processing apparatus that performs conversion. The image processing apparatus according to claim 3 ,
Extracting means for extracting the contour of the face imaged in the acquired image;
The conversion means selects and uses, as the conversion table data corresponding to the lens type of the eyeglasses, conversion table data that is determined to be continuous with the extracted contour line from the acquired conversion table data. apparatus. Obtaining means for obtaining an image of a face imaged;
If there is a lens part of the spectacles in the acquired image, an estimation means for estimating which pixel is in the lens part of the spectacles, and
The pixels in the lens portion of the estimated glasses, seen including converting means, the performing conversion to map the pixels of the other positions in the lens portion of the spectacles,
The image processing apparatus that performs conversion after correcting a face included in the image in a front direction. The image processing apparatus according to claim 5,
Extracting means for extracting the contour of the face imaged in the acquired image;
The converting means performs conversion for mapping to pixels at other positions in the lens portion of the spectacles by enlarging or reducing an image portion in the lens portion of the spectacles, and the converting means converts the extracted contour line An image processing apparatus that sets an enlargement or reduction ratio so that the images are continuous. The image processing apparatus according to any one of claims 1 to 4 , wherein:
The image processing apparatus that performs conversion after correcting a face included in the image in a front direction. The image processing apparatus according to any one of claims 1 to 7 ,
Furthermore, it includes an extracting means for extracting the contour line of the face imaged in the acquired image,
The estimation means uses the extracted facial contour line to determine which pixel is in the lens portion of the glasses when it is determined that there is a lens portion of the glasses in the acquired image. An image processing apparatus to be estimated. An obtaining means for obtaining an image of the face imaged;
An estimating means, if there is a lens part of the spectacles in the acquired image, estimating which pixel is in the lens part of the spectacles;
Transforming means for mapping the pixels in the estimated lens portion of the spectacles to pixels at other positions in the spectacle lens portion; and
The extracting means includes a step of extracting a contour of the face imaged in the acquired image;
The estimating means further estimates a position corresponding to the lens center of the glasses from the estimated lens portion of the glasses;
When the conversion means sets an enlargement or reduction ratio so that the extracted contour line is continued, and performs enlargement or reduction, the eyeglasses radially center around a position corresponding to the estimated lens center of the eyeglasses. An image processing method for performing conversion for mapping to pixels at other positions in the lens portion of the spectacles by enlarging or reducing the image portion in the lens portion. An obtaining means for obtaining an image of the face imaged;
The estimating means, if there is a lens part of the spectacles in the acquired image, estimating which pixel is in the lens part of the spectacles; and
Converting means for mapping a pixel in the estimated lens portion of the spectacles to a pixel at another position in the spectacle lens portion;
An image processing method for performing conversion after correcting a face included in the image in a front direction in the step of performing the conversion. Computer
Obtaining means for obtaining an image of a face imaged;
If there is a lens part of the spectacles in the acquired image, an estimation means for estimating which pixel is in the lens part of the spectacles;
Conversion means for performing conversion to map pixels in the estimated lens portion of the spectacles to pixels at other positions in the lens portion of the spectacles; and
Function as an extracting means for extracting the contour line of the face imaged in the acquired image;
When functioning as the estimation means, further, from the estimated lens portion of the spectacles, the position corresponding to the lens center of the spectacles is estimated,
When functioning as the conversion means, by enlarging or reducing the image portion in the lens portion of the glasses, conversion is performed to map to pixels at other positions in the lens portion of the glasses, as the conversion means When functioning, the enlargement or reduction rate is set so that the extracted contour line is continuous, and when performing the enlargement or reduction, enlargement is performed radially around the position corresponding to the estimated lens center of the glasses. Or a program that causes reduction. Computer
Obtaining means for obtaining an image of a face imaged;
If there is a lens part of the spectacles in the acquired image, an estimation means for estimating which pixel is in the lens part of the spectacles, and
Functioning as a conversion means for performing conversion to map the pixels in the estimated lens part of the spectacles to pixels at other positions in the lens part of the spectacles;
A program for performing conversion after correcting a face included in the image to face forward when functioning as the conversion means.

Images