JP7439899B2 - Verification auxiliary device, verification auxiliary method and program - Google Patents

Description

The present invention relates to a verification assisting device, a verification assisting method, and a program storage medium.

In some cases, a person photographed by a security camera or the like is visually compared with a person registered in a database.
Patent Document 1 discloses that when a user visually compares a three-dimensional image of a person registered in a database with a photographed person, the user changes the orientation and size of the three-dimensional image to create a two-dimensional image that has been photographed. A technology has been disclosed that makes it easier to compare the information.

Utility model registration No. 3204175 Japanese Patent Application Publication No. 2015-25859

In the technique described in Patent Document 1, collation by a user is facilitated by changing the orientation and size of three-dimensional images registered in a database. However, when a person appearing in a captured image is wearing glasses, the position and size of the eyes reflected in the lenses appear different due to the refraction of the lenses compared to when the person is seen with the naked eye. This becomes noticeable depending on the person's face direction, etc., so that when a user visually compares a two-dimensional image and a three-dimensional image, the accuracy of matching whether they are the same person or not decreases. Patent Document 1 does not take the above situation into consideration.

SUMMARY OF THE INVENTION In view of the above-mentioned problems, it is an object of the present invention to improve matching accuracy by reducing changes in appearance caused by wearing glasses.

According to one aspect of the present invention, a detection unit detects an input image including a person wearing a lens in at least one input image of a person; A matching auxiliary device is provided that includes a corrected image that is an image obtained by correcting the position and size of the eye reflected in a lens, and an output unit that outputs a three-dimensional registered image that is a target for matching with the corrected image. .

According to one aspect of the present invention, an input image including a person wearing a lens is detected in at least one input image of a person, and the image is reflected in the lens using the power of the lens estimated in the input image. A matching assistance method is provided that outputs a corrected image, which is an image in which the position and size of the eyes have been corrected, and a three-dimensional registered image, which is a target for matching with the corrected image.

According to one aspect of the present invention, a computer is provided with a process of detecting an input image including a person wearing a lens in at least one input image of a person, using the power of the lens estimated in the input image. A program that stores a program for executing a process of outputting a corrected image, which is an image obtained by correcting the position and size of the eye reflected in the lens, and a three-dimensional registered image, which is a target for comparison with the corrected image. A storage medium is provided.

According to one aspect of the present invention, there is provided a detection unit that detects an input image including a person wearing glasses having lenses in at least one input image of a person, and a power of the lens estimated in the input image. an output unit that outputs the input image and a three-dimensional corrected image in which the eye position and size of a three-dimensional registered image to be matched with the input image are corrected using is provided.

According to one aspect of the present invention, an input image including a person wearing glasses having lenses is detected from at least one input image of a person, and the power of the lens estimated in the input image is used. A matching auxiliary method is provided that outputs the input image and a three-dimensional corrected image in which the position and size of eyes in a three-dimensional registered image to be matched with the input image are corrected.

According to one aspect of the present invention, a process of detecting an input image including a person wearing glasses having lenses in at least one input image of a person is performed in a computer; A program for executing a process of outputting the input image and a three-dimensional corrected image in which the eye position and size of the three-dimensional registered image to be matched with the input image are corrected using a frequency. A program storage medium is provided that stores a program.

According to the present invention, there are provided a verification assisting device, a verification assisting method, and a program that can improve verification accuracy by reducing changes in appearance caused by wearing glasses.

1 is a diagram illustrating an example of the overall configuration of a verification assistance system in a first embodiment; FIG. FIG. 2 is a block diagram showing the functions of the verification assistance system in the first embodiment. FIG. 3 is a conceptual diagram illustrating the state of an imaging target in the first to fourth embodiments. FIG. 3 is a conceptual diagram illustrating the state of an imaging target in the first to fourth embodiments. FIG. 3 is a block diagram showing the functions of a frequency estimator in the first embodiment. FIG. 3 is a conceptual diagram for explaining calculation processing executed by the collation assisting system in the first to fourth embodiments. FIG. 3 is a conceptual diagram for explaining calculation processing executed by the collation assisting system in the first to fourth embodiments. 3 is a flowchart illustrating an example of image processing in the first embodiment. It is a flowchart which shows an example of machine verification in a 1st embodiment. 3 is a flowchart illustrating an example of user input reception in the first embodiment. FIG. 3 is a block diagram showing the functions of a verification assistance system in a second embodiment. 7 is a flowchart illustrating an example of image processing in the second embodiment. 12 is a flowchart illustrating an example of user input reception in the second embodiment. FIG. 3 is a block diagram showing the functions of the verification assist system in the third and fourth embodiments. 12 is a flowchart illustrating an example of processing in a third embodiment. 12 is a flowchart illustrating an example of processing in a fourth embodiment. FIG. 2 is a block diagram showing an example of the hardware configuration of an image matching system in first to fourth embodiments. This is an example of a calculation formula used for processing in the present invention. FIG. 3 is a block diagram illustrating a modification applicable to the embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings. In the drawings, similar or corresponding elements are denoted by the same reference numerals, and the description thereof may be omitted or simplified.

<Background>
A person who wears glasses for vision correction looks different in facial features compared to a person who does not wear glasses. This is because the position and size of the eyes appear to change due to the refraction phenomenon of the lens. For administrators who check images captured by cameras, it is difficult to visually determine whether a person whose facial features appear to have changed due to wearing glasses corresponds to one of the person images registered in the database. do not have. In the embodiment described later, a system for assisting an administrator (user) will be described.

<First embodiment>
The configuration of the verification assistance system 1 of this embodiment will be explained using FIGS. 1 and 2. FIG. 1 shows an example of the overall configuration of a verification assistance system 1 in this embodiment. The verification assistance system is an information processing system that includes a processing device 100, an imaging device 200, and a user terminal 300. Each device and terminal are connected via a network. A specific example of the situation shown in FIG. 1 is a situation where a person is imaged by a camera installed on the street.

The imaging device 200 is a terminal that images a person and obtains a captured image, and is, for example, a security camera installed on the street. The imaging device 200 images a person passing through the imaging section, and outputs the captured image to the processing device 100.

FIG. 2 is a functional block diagram showing the configuration of the verification assistance system 1. As shown in FIG. The verification assistance system 1 includes a processing device 100, an imaging device 200, and a user terminal 300. The processing device 100 includes an input section 110, a detection section 120, an extraction section 130, a frequency estimation section 140, an image processing section 150, a storage section 160, a collation section 170, and an output section 180. The imaging device 200 includes an imaging section 210. User terminal 300 includes a display section 310 and an operation reception section 320.

The imaging unit 210 included in the imaging device 200 images a person passing through the imaging section, and outputs the captured image to the processing device 100. Note that the imaging unit 210 images the imaging section in time series according to a set frame rate, for example. The imaging unit 210 is not limited to the above-described specific example, and may take an image at a timing when receiving an instruction from the outside, or may take a still image at a predetermined timing. In this embodiment, the captured image only needs to include at least the head of the person, and does not need to include the whole body of the person.

The input unit 110 receives input of a captured image output by the imaging unit 210. Here, the target (input image) for which the input unit 110 receives input may be all of the captured images captured by the imaging unit 210, or may be a part of the captured images. For example, the image capture unit 210 may extract captured images at predetermined time intervals from the captured images captured in chronological order, and may sequentially receive input. As an example of extracting captured images at predetermined time intervals, captured images may be extracted by thinning out some of the captured images captured by the imaging unit 210 according to a predetermined frame rate. The objects that the input unit 110 accepts are not limited to the above.

The detection unit 120 detects a human head and glasses from the captured image received by the input unit 110. Here, to detect a person's head and glasses, for example, a trained model that is trained by machine learning on an image of a head wearing glasses is used. As the machine learning method, for example, deep learning via a multilayer neural network may be used. The method used to detect the head of a person and the wearing of glasses is not limited to the above. The detection unit 120 may detect the detection of a human head and the wearing of glasses in a stepwise manner, or may detect each detection independently. In this case, wearing of glasses may be detected after detecting the human head. Further, the detection unit 120 does not necessarily need to detect the head, and may detect glasses.

The detection unit 120 detects an image in which the outline of the face is reflected inside the lens of the glasses from among the captured images in which the wearing of glasses is detected.

Next, the state of the face reflected in the captured image will be explained using FIG. 3. FIG. 3 is an example of a face in which the outline of the face is reflected inside the lens of the glasses. In the figure, FL indicates the outline of a person's face, and GL1 and GL2 indicate lenses. Further, VFL indicates the outline of the face reflected inside the lens. 3A shows a state in which the person's face faces forward to the imaging unit 210 (face inclination angle is 0 degrees), and FIG. 3B shows a state in which the person's face does not face forward to the imaging unit 210 (face inclination angle is 0 degrees). angle is not 0 degrees). FIG. 3B shows, as an example, a state in which the person faces to the right. The state in which the face faces forward is a state in which the inclination angle of the face with respect to the imaging direction is close to 0 degrees and smaller than a predetermined threshold. The state where the face is not facing forward is a state where the inclination angle of the face with respect to the imaging direction is larger than a predetermined threshold value. The predetermined threshold value is arbitrarily set. As shown in FIG. 3A, when the face is facing forward to the imaging unit 210 (the inclination angle of the face is 0 degrees), the outline of the face is difficult to be reflected inside the lenses GL1 and GL2. On the other hand, as shown in FIG. 3B, when the face direction is not in front of the imaging unit 210 (the inclination angle of the face is not 0 degrees), the lens It is easy to see the outline of the face inside GL1.

The detection unit 120 estimates the inclination angle of the face in the captured image with respect to the imaging direction. The inclination angle of the face is the angle at which the person's face is oriented with respect to a straight line passing through the imaging unit 210 and the person's head in three-dimensional space. Specifically, when a person's face is facing the imaging unit 210, the inclination angle is close to 0 degrees and smaller than a predetermined threshold. The target image in which the detection unit 120 detects a face may be a two-dimensional image extracted by the extraction unit 130 described later.

The extraction unit 130 extracts a captured image in which the contour of the face is reflected inside the lens as a two-dimensional image. The extraction unit 130 extracts an image in which the outline of the face is reflected inside the lens, as illustrated in FIG. 3B. In other words, the extraction unit 130 extracts an image in which a person's face is turned diagonally to the extent that the outline of the face is reflected inside the lens when viewed from the imaging unit 210. The extraction unit 130 may extract images in which the inclination angle of the face is greater than or equal to an arbitrarily set first threshold, or images in which the inclination angle of the face is less than or equal to an arbitrarily set second threshold. Here, the second threshold may be set to a value larger than the first threshold.

“Extraction” executed by the extraction unit 130 refers to a process of extracting some captured images from among a plurality of captured images. Specifically, from a plurality of captured images captured in time series according to a predetermined frame rate, only captured images in which wearing of glasses is detected may be extracted. Note that "extraction" may be a concept that includes a process of cutting out a part of the region within the captured image. For example, in a captured image that includes a plurality of people, in addition to cutting out a region where a head wearing glasses is shown, processing may also be performed to cut out a region where a person is shown. The process referred to by "extraction" is not limited to the above specific example.

The power estimation unit 140 estimates the power of glasses worn by a person in the two-dimensional image extracted by the extraction unit 130. Details of the frequency estimation unit 140 will be explained using FIG. 4. FIG. 4 is a functional block diagram of the frequency estimation section 140. The frequency estimation section 140 includes a calculation section 141 , a learning model storage section 142 , and an estimation section 143 .

The calculation unit 141 calculates the distance in three-dimensional space between the person's face imaged by the imaging unit 210 and the imaging unit 210. To calculate the distance, for example, the size of the area showing the face on the captured image or the length of the area showing the person may be used, but the distance calculation method is not limited to this, and those skilled in the art can use well-known techniques as appropriate. can be applied.

The calculation unit 141 calculates the difference between the positions of the contours of the face reflected on the inside and outside of the lens (herein referred to as "contour difference"). The concept of contour difference will be explained in detail using FIG. FIG. 5 is a diagram schematically showing a part of a face wearing a lens. The calculation unit 141 calculates the horizontal inter-pixel distance (first scanning distance E1) between the horizontal position of the center point of the eye and the position E4 of the facial contour in the area inside the lens. The calculation unit 141 calculates the horizontal inter-pixel distance (second scanning distance E2) between the horizontal position of the center point of the eye and the position E3 of the facial contour in the area outside the lens.

The calculation unit 141 calculates a contour difference (normalized contour difference) by normalizing the difference between the first scanning distance E1 and the second scanning distance E2 using the distance between the imaging unit 210 and the face. The specific method of normalization is not limited, and for example, the difference between the first scanning distance E1 and the second scanning distance E2 may be normalized by dividing it by the distance between the imaging unit 210 and the face. Any calculation formula that takes into account the distance between the imaging unit 210 and the face may be used when normalizing the contour difference.

The calculation unit 141 calculates the angle of incidence of light that enters the imaging unit 210 from an object appearing at an arbitrary pixel on the captured image. An example of a method for calculating the incident angle Ψ from a real object to the imaging unit 210 corresponding to an arbitrary pixel position on a captured image will be described in detail using FIGS. 6 and 17. FIG. 6 is a diagram schematically showing the optical system of the imaging unit 210 that images a person. A CCD (Charge-Coupled Device) sensor CS detects light incident from a person via a camera lens P. Calculation formula (1) in FIG. 17 is a formula showing the process of deriving the incident angle Ψ. The variables forming calculation formula (1) will be explained. The incident angle Ψ is calculated from the center position of the captured image showing the entire imaging range of the imaging unit 210, with the first pixel distance xs from the center position in the captured image showing the entire imaging range of the imaging unit 210 as a variable. The calculation formula shown in FIG. 17 ( It can be expressed as 1). The calculation unit 141 calculates the angle of incidence at the position of the contour of the face reflected in the area inside the lens. For example, calculation formula (1) shown in FIG. 17 can be used for calculation.

The learning model storage unit 142 stores a regression model that outputs the power of the lens when the incident angle and the normalized contour difference are input. The regression model is a trained model that has previously learned the combination of the incident angle, normalized contour difference, and lens power.

The estimation unit 143 inputs the incident angle and the contour difference calculated by the calculation unit 141 into the regression model stored in the learning model storage unit 142, and estimates the power of the lens.

The regression model described above may be one that is trained to input the incident angle, the contour difference before normalization, and the distance between the imaging unit 210 and the face, and output the power of the lens. In this case, there is no need to normalize the contour difference, so the processing load can be reduced.

An example of the function of the power estimating unit 140 described above is the positional relationship between the outline reflected on the inside of the lens and the outline reflected on the outside of the lens in the captured image, and the relationship between the photographing unit 210 that photographed the person wearing the lens and the person. The power of the lens is estimated using the positional relationship. In addition, in order to estimate the power of the lens, the power estimating unit 140 also uses a method such as the contour of the face reflected inside the lens of the glasses, as described in Patent Document 2 (Japanese Patent Application Laid-open No. 2015-25859). A method of estimating the power of the lens may be used based on the position of the face, the position of the outline of the face visible outside the lens, and the inclination angle of the face. The method of estimating the power of the lens is not limited to the above.

The image processing unit 150 performs image correction on the two-dimensional image using the lens power information estimated by the power estimation unit 140, and generates a corrected image (corrected image) corresponding to the two-dimensional image. do. The area to be corrected is the area inside and around the lens in the image, and the outline of the face and the position and size of the person's eyes reflected in the area inside the lens of which power has been estimated are corrected. Specifically, the area reflected inside the lens is corrected so that the area reflected inside the lens is enlarged and displayed so that the contour reflected inside the lens and the contour reflected outside the lens are located on the same curve.

The storage unit 160 stores three-dimensional images (three-dimensional registered images) of a plurality of registrants in association with registrant information. Examples of registrant information items include registrant identification information, name, registration date and time, and the like.

The matching unit 170 matches the person appearing in the two-dimensional image extracted by the extraction unit 130 with the plurality of three-dimensional registered images stored in the storage unit 160, and matches the person imaged by the imaging unit 210 with the plurality of three-dimensional registered images. Calculate the degree of similarity with the image. Here, the matching unit 170 sets three-dimensional registered images whose degree of similarity is equal to or higher than a threshold value as a target for visual matching by the user. A plurality of three-dimensional registered images may be set as targets for visual verification.

The output unit 180 outputs the three-dimensional registered image set as a target for visual verification and the corrected image processed by the image processing unit 150 (corrected image).

User terminal 300 includes a display section 310 and an operation reception section 320. The user terminal 300 is a terminal that provides information to the user or receives user operations. Specifically, the user terminal 300 prompts the user to determine whether a three-dimensional registered image of a person registered in a database (not shown) and a person captured by a security camera or the like are the same person. It is a device. Alternatively, the device may be a device for checking images captured by a user using a security camera or the like, but is not limited thereto.

The display unit 310 displays the three-dimensional registered image output by the output unit 180 and the corrected image processed by the image processing unit 150. The display unit 310 may display the three-dimensional registered image and the corrected image at the same time. At this time, the display unit 310 may display the plurality of three-dimensional registered images simultaneously, or may display the plurality of three-dimensional registered images sequentially according to the degree of similarity. The display unit 310 may display a plurality of corrected images simultaneously. Further, the display unit 310 may display the three-dimensional registered image and the corrected image on the same screen, for example.

The user determines whether the at least one three-dimensional registered image displayed by the display unit 310 and the corrected image show the same person, and inputs the determination result to the operation reception unit 320. The operation reception unit 320 receives the input of the judgment result determined by the user through the user's operation. The operation accepted by the operation reception unit 320 is, for example, an operation of selecting an image of the same person as the corrected image to be compared among the three-dimensional registered images displayed on the display unit 310.

Next, the operation of the verification assistance system of this embodiment will be explained using FIGS. 7 to 9. 7 to 9 are flowcharts showing an example of processing in this embodiment.

FIG. 7 shows an example of a flow related to image processing. First, the input unit 110 receives input of a captured image output by the imaging device 200 (S101). The input unit 110 may, for example, sequentially receive input of images captured in time series according to a predetermined frame rate.

The detection unit 120 detects a human head and glasses from the captured image received by the input unit 110 (S102). The detection unit 120 may detect the detection of a human head and the wearing of glasses in a stepwise manner, or may detect each detection independently. If the detection unit 120 does not detect glasses in the captured image (S103, NO), the process returns to step S101 and receives the input of the next captured image (S101). When the detection unit 120 detects glasses from the captured image (S103, YES), the detection unit 120 detects the outline of the face reflected inside the lens of the glasses from the captured image in which the wearing of glasses is detected. It is determined whether there is one (S104).

If it is determined that the outline is reflected inside the lens (S105, YES), the extraction unit 130 extracts a captured image in which the outline is reflected inside the lens from among the captured images (S106). If it is determined that the outline of the face is not reflected inside the lens (S105, NO), the process returns to step S101.

The power estimation unit 140 estimates the power of the lens worn by the person in the image extracted by the extraction unit 130 (S107).

The image processing unit 150 corrects the image extracted by the extraction unit 130 using the lens power information estimated by the power estimation unit 140, and generates a corrected image (corrected image) (S108). .

FIG. 8 shows an example of a processing flow related to verification by the processing device 100. The matching unit 170 matches the person appearing in the two-dimensional image extracted by the extraction unit 130 with the plurality of three-dimensional registered images stored in the storage unit 160, and matches the person imaged by the imaging unit 210 with the plurality of registrants. The degree of similarity is calculated (S109). Here, if there is a three-dimensional registered image with a degree of similarity equal to or higher than the threshold value (S110, YES), the three-dimensional registered image is set as a target for visual verification by the user (S111). If there is no three-dimensional registered image with a degree of similarity equal to or higher than the threshold (S110, NO), the process ends.

Note that in this embodiment, after the processing flow (S109 to S111) related to verification by the apparatus shown in FIG. 8, a series of flows related to image processing (S101 to S108) shown in FIG. 7 may be executed. In this case, the power of the lens is estimated only for images in which the lens is recognized, so the processing load on the device can be reduced.

The flow related to verification (S109 to S111) and the flow related to image processing (S101 to S108) may be processed in parallel. A flow related to image processing (S101 to S108) may be executed before a flow related to verification (S109 to S111). The sequential relationship between the series of flows related to image processing shown in FIG. 7 (S101 to S108) and the processing flow related to verification by the apparatus shown in FIG. 8 (S109 to S111) is not limited.

FIG. 9 shows an example of a processing flow related to image display. The output unit 180 outputs the three-dimensional registered image set as a target for visual verification and the corrected image (corrected image) processed by the image processing unit 150 (S112).

The display unit 310 displays the three-dimensional registered image output by the output unit 180 and the corrected image processed by the image processing unit 150 (S113).

The operation receiving unit 320 receives the input of the judgment result determined by the user through the user's operation (S114). The operation accepted by the operation reception unit 320 is, for example, an operation of selecting an image of the same person as the corrected image to be compared among the three-dimensional registered images displayed on the display unit 310.

Note that the processing flow (S112 to S113) related to image display shown in FIG. 9 is a series of flows related to image processing (S101 to S108) shown in FIG. It is executed after S109 to S111).

As a result, even if the verification target is a person wearing lenses, changes in appearance or facial impression caused by wearing lenses can be reduced, allowing users to perform visual verification efficiently and accurately. be able to.

<Second embodiment>
The verification assist system 1 of this embodiment will be described below. The second embodiment is different from the first embodiment in that the processing device 100 includes a three-dimensional image processing unit 190 and performs correction on the three-dimensional registered image based on estimated lens power information. do. Note that explanations of parts common to the first embodiment will be omitted.

FIG. 10 is a functional block diagram showing the configuration of the verification assistance system 1 in this embodiment. The processing device 100 includes an input section 110, a detection section 120, an extraction section 130, a frequency estimation section 140, an image processing section 150, a storage section 160, a matching section 170, an output section 180, and a three-dimensional image processing section. and a processing section 190.

The three-dimensional image processing unit 190 corrects the image around the eyes of the person in the three-dimensional registered image that the matching unit 170 has set as a target for visual matching, and creates a corrected three-dimensional image ( 3D corrected image). Specifically, a predetermined three-dimensional registered image is corrected according to lens power information estimated by the power estimation unit 140. The predetermined three-dimensional registered image is a three-dimensional registered image that the matching unit 170 sets as a target for visual matching. Known three-dimensional computer graphics are used for this correction. An example of correction according to lens power information will be described. For example, since concave lenses are used in glasses for correcting myopia, the eyes of a person wearing the glasses appear smaller than before wearing them due to the refraction of the lenses. In this case, the three-dimensional image processing unit 190 uses the estimated power information of the concave lens to reproduce changes in facial features that appear to be refracted by the lens when the person shown in the three-dimensional registered image wears a concave lens. , Correct the image around the eyes.

The storage unit 160 stores a plurality of three-dimensional registered images in association with registrant information. Examples of registrant information items include registrant identification information, name, registration date and time, and the like. Note that the storage unit 160 may store the three-dimensional corrected image generated by the three-dimensional image processing unit 190.

The matching unit 170 matches the person appearing in the two-dimensional image extracted by the extraction unit 130 with the three-dimensional registered image showing a plurality of registrants stored in the storage unit 160, and matches the person imaged by the imaging unit 210 and the plurality of registrants. Calculate the degree of similarity with the registrant. The matching unit 170 sets the three-dimensional registered images for which the obtained degree of similarity is greater than or equal to the threshold value as a target for visual verification by the user. A plurality of three-dimensional registered images may be set as targets for visual verification.

The output unit 180 outputs a three-dimensional corrected image showing the person set as a target for visual verification and the image extracted by the extraction unit 130. The three-dimensional corrected image is a three-dimensional image generated by the image processing unit 150.

The display unit 310 displays a three-dimensional corrected image showing a person set as a target for visual verification and a two-dimensional image extracted by the extraction unit 130. The display unit 310 may display the three-dimensional corrected image and the two-dimensional image simultaneously. At this time, the display unit 310 may display the plurality of three-dimensional images simultaneously, or may display the plurality of three-dimensional images sequentially according to the obtained similarity. The display unit 310 may display multiple two-dimensional images simultaneously. Further, the display unit 310 may display the three-dimensional registered image and the corrected image on the same screen, for example.

The user determines whether at least one three-dimensional image displayed by the display section 310 and the two-dimensional image show the same person, and inputs the determination result to the operation reception section 320. The operation reception unit 320 receives the input of the judgment result determined by the user through the user's operation. The operation accepted by the operation reception unit 320 is, for example, an operation of selecting an image of the same person as the two-dimensional image to be compared among the three-dimensional images displayed on the display unit 310.

Next, the operation of the verification assistance system of this embodiment will be explained using FIGS. 11 and 12. FIG. 11 shows an example of a processing flow related to image processing in this embodiment. Note that a description of processes that overlap with those of the first embodiment will be omitted. In this embodiment, the processing flow related to verification by the device is the same as in the first embodiment (S109 to S111).

The power estimation unit 140 estimates the power of glasses worn by the person in the image extracted by the extraction unit 130 (S107). Next, the three-dimensional image processing unit 190 corrects the image around the person's eyes using the estimated lens power information in the three-dimensional registered image that the matching unit 170 has set as a target for visual matching, and A corrected image is generated (S208).

The storage unit 160 stores the three-dimensionally corrected image in association with the registrant information (S209).

The output unit 180 outputs the three-dimensional corrected image showing the person set as the target for visual verification and the image extracted by the extraction unit 130 (S212).

The display unit 310 displays the three-dimensional corrected image showing the person set as the target for visual verification and the two-dimensional image extracted by the extraction unit 130 (S213). The display unit 310 may display the three-dimensional corrected image and the two-dimensional image simultaneously. At this time, the display unit 310 may display the plurality of three-dimensional images simultaneously, or may display the plurality of three-dimensional images sequentially depending on the degree of similarity. The display unit 310 may display multiple two-dimensional images simultaneously.

The user determines whether at least one three-dimensional image displayed by the display section 310 and the two-dimensional image show the same person, and inputs the determination result to the operation reception section 320. The operation receiving unit 320 receives the input of the judgment result determined by the user through the user's operation (S114).

In this embodiment, after the processing flow (S109 to S111) related to verification by the apparatus shown in FIG. 8, a series of flows related to image processing (S101 to S107, S208 to S209) shown in FIGS. May be executed. In this case, the power of the lens is estimated only for images in which the lens is recognized, so the processing load on the device can be reduced.

The flow related to verification (S109 to S111) and the flow related to image processing (S101 to S107, S208 to S209) may be processed in parallel. Before the flow related to verification (S109 to S111), the flow related to image processing (S101 to S107, S208 to S209) may be executed. The sequential relationship between the series of flows related to image processing shown in FIGS. 7 and 11 (S101 to S107, S208 to S209) and the processing flow related to verification by the apparatus shown in FIG. 8 (S109 to S111) is not limited.

As described above, in the matching assistance system of the present embodiment, when the imaged person is wearing glasses, the three-dimensional registered image held in the database is corrected according to the change in appearance due to wearing the glasses. As a result, even if the verification target is a person wearing glasses, changes in appearance or facial impression due to wearing glasses can be reduced, allowing the user to perform visual verification efficiently and accurately. be able to.

<Third embodiment>
The configuration of the verification assistance system 1 of this embodiment will be explained using FIG. 13. FIG. 13 is a functional block diagram of the verification assistance system 1 in this embodiment. The verification assistance system 1 includes a detection section 120 and an output section 180.

The detection unit 120 detects an input image including a person wearing a lens, from at least one input image of a person.

The output unit 180 outputs a corrected image, which is an image obtained by correcting the input image using lens power information estimated from the input image detected by the detection unit 120, and a three-dimensional registered image, which is a target for matching with the corrected image. Outputs . The corrected image is, for example, an image obtained by correcting the position and size of the eyes reflected in the lens in the input image.

Next, the operation of the verification assistance system of this embodiment will be explained using FIG. 14. FIG. 14 is a flowchart showing an example of processing in this embodiment.

The detection unit 120 detects an input image including a person wearing a lens in the captured image (S307).

The output unit 180 outputs a corrected image, which is an image obtained by correcting the input image using lens power information estimated from the input image detected by the detection unit 120, and a three-dimensional registered image, which is a target for matching with the corrected image. is output (S308).

As a result, even if the verification target is a person wearing glasses, changes in appearance or facial impression due to wearing glasses can be reduced, allowing the user to perform visual verification efficiently and accurately. be able to.

<Fourth embodiment>
The configuration of the verification assistance system 1 of this embodiment will be explained using FIG. 13 similarly to the third embodiment. This embodiment differs from the third embodiment in that the function of the output unit 180 is different. FIG. 13 is a functional block diagram of the verification assistance system 1 in this embodiment. The verification assistance system 1 includes a detection section 120 and an output section 180.

The detection unit 120 detects an input image including a person wearing glasses having lenses from at least one input image of a person.

The output unit 180 outputs the input image and a three-dimensional corrected image obtained by correcting the three-dimensional registered image using the power of the lens estimated from the input image. The three-dimensional corrected image outputted by the output unit 180 is, for example, an image in which the eye position and size of the three-dimensional registered image that is to be compared with the input image are corrected.

Next, the operation of the verification assistance system of this embodiment will be explained using FIG. 15. FIG. 15 is a flowchart showing an example of processing in this embodiment.

The detection unit 120 detects an input image including a person wearing glasses having lenses from at least one input image of a person (S407).

The output unit 180 outputs the input image and a three-dimensional corrected image obtained by correcting the three-dimensional registered image using the power of the lens estimated from the input image (S408).

As a result, even if the verification target is a person wearing glasses, changes in appearance or facial impression due to wearing glasses can be reduced. It is possible to generate an image that allows the user to perform visual verification efficiently and accurately.

(Hardware configuration)
Next, an example of a hardware configuration in which the processing device 100, the imaging device 200, and the user terminal 300 in each of the above-described embodiments are implemented using one or more computers will be described. Each of the functional units included in the processing device 100, the imaging device 200, and the user terminal 300 includes at least one CPU (Central Processing Unit) of any computer, at least one memory, a program loaded into the memory, and at least one computer that stores the program. It is realized by any combination of hardware and software, centering on a storage unit such as a hard disk, a network connection interface, etc. Those skilled in the art will understand that there are various modifications to this implementation method and device. Note that the storage unit can store not only programs stored before the device is shipped, but also programs downloaded from a storage medium such as an optical disk, a magneto-optical disk, or a semiconductor flash memory, or from a server on the Internet.

FIG. 16 is a block diagram illustrating the hardware configurations of the processing device 100, the imaging device 200, and the user terminal 300. As shown in FIG. 16, the processing device 100, the imaging device 200, and the user terminal 300 have a processor 1A, a memory 2A, an input/output interface 3A, a peripheral circuit 4A, a communication interface 5A, and a bus 6A. The peripheral circuit 4A includes various modules. The processing device 100, the imaging device 200, and the user terminal 300 do not need to have the peripheral circuit 4A. Note that the processing device 100, the imaging device 200, and the user terminal 300 may be composed of a plurality of physically and/or logically separated devices. In this case, each of the plurality of devices can include the above hardware configuration.

The bus 6A is a data transmission path through which the processor 1A, memory 2A, input/output interface 3A, peripheral circuit 4A, and communication interface 5A exchange data with each other. The processor 1A is an arithmetic processing device such as a CPU, a GPU (Graphics Processing Unit), or a microprocessor. The processor 1A can execute processing according to various programs stored in the memory 2A, for example.

The memory 2A is, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory), and stores programs and various data.

The input/output interface 3A includes an interface for acquiring information from an input device, an external device, an external storage unit, an external sensor, a camera, etc., an interface for outputting information to an output device, an external device, an external storage unit, etc. including. Input devices include, for example, a touch panel, keyboard, mouse, microphone, camera, and the like. Examples of the output device include a display, a speaker, a printer, and a lamp.

The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

The communication interface 5A allows the processing device 100, the imaging device 200, and the user terminal 300 to communicate with each other, and also allows the processing device 100, the imaging device 200, and the user terminal 300 to communicate with each other. Note that some functions of the processing device 100, the imaging device 200, and the user terminal 300 may be configured by a computer.

<Modified example>
Modifications applicable to the above embodiment will be described. If the power estimating unit 140 fails to estimate the power of the lens in the captured image of the target, the process may be performed again on captured images taken before and after the captured image. Further, the image processing unit 150 may perform image processing on the captured image so as to remove the rim of the glasses.

Another modification will be explained. The imaging unit 210 may include a device that acquires three-dimensional depth information (depth information). In that case, the calculation unit 141 does not need to calculate the distance between the imaging unit 210 and the person (or the person's face) from the two-dimensional image, and the estimation unit 143 uses the depth information acquired by the imaging unit 210 to The power of the lens may also be estimated. The matching unit 170 also matches the person appearing in the two-dimensional image extracted by the extraction unit 130 with a plurality of three-dimensional registered images stored in the storage unit 160, as well as the corrected image processed by the image processing unit 150. It may be designed to match the three-dimensional registered image with the three-dimensional registered image.

Other modifications applicable to the above embodiment will be described. The imaging device 200 may execute some or all of the processing executed by the processing device 100. For example, as shown in FIG. 18, the imaging device 200 may include a detection unit 120, an extraction unit 130, and a frequency estimation unit 140 in addition to the imaging unit 210. In this case, in the imaging device 200, the power of the lens may be estimated, and the image processing unit 150 included in the processing device 100 may acquire information regarding the power of the lens from the imaging device 200. In this case, the imaging device 200 may execute the processes of S101 to S107. Further, the imaging device 200 may extract only the captured images in which the contour difference of the person appearing in the captured images is equal to or greater than a threshold value, and may perform a process of estimating the power of the lens or a process of transmitting the extracted images to the processing device 100. According to the above-mentioned modification, it is possible to reduce the processing load on the processing device 100 and the load related to the transmission processing of the collation assisting system in the present invention.

Note that the configurations of the embodiments described above may be combined or some components may be replaced. Further, the configuration of the present invention is not limited to the above-described embodiments, and various changes may be made without departing from the gist of the present invention. Part or all of the above embodiments may be described as in the following additional notes, but are not limited to the following.

(Additional note 1)
a detection unit that detects an input image including a person wearing a lens in at least one input image of a person;
Outputting a corrected image, which is an image in which the position and size of the eye reflected in the lens is corrected using the power of the lens estimated in the input image, and a three-dimensional registered image, which is a target for matching with the corrected image. an output section to
A verification auxiliary device comprising:

(Additional note 2)
a display unit that displays the corrected image and the three-dimensional registered image on the same screen;
The verification auxiliary device according to Supplementary Note 1, further comprising:

(Additional note 3)
The display unit displays three-dimensional registered images whose degree of similarity obtained by matching the input image corresponding to the corrected image with the three-dimensional registered images of a plurality of registrants is equal to or higher than a threshold.
Verification auxiliary device described in Appendix 2.

(Additional note 4)
a detection unit that detects an input image including a person wearing glasses having lenses in at least one input image of a person;
Outputting the input image and a three-dimensional corrected image in which the eye position and size of a three-dimensional registered image that is a target for comparison with the input image is corrected using the power of the lens estimated in the input image. an output section to
A verification auxiliary device comprising:

(Appendix 5)
The matching auxiliary device according to supplementary note 4, further comprising a display unit that displays the three-dimensionally corrected image and the input image on the same screen.

(Appendix 6)
The display unit displays the three-dimensional corrected image corresponding to the three-dimensional registered image for which the degree of similarity obtained by matching with the input image is equal to or higher than a threshold.
Verification auxiliary device described in Appendix 5.

(Appendix 7)
Using the positional relationship between the outline reflected inside the lens and the outline reflected outside the lens in the input image, and the positional relationship between the person and the imaging unit that photographed the person wearing the lens, The verification auxiliary device according to any one of Supplementary Notes 1 to 6, further comprising an estimation unit that estimates the power of the lens.

(Appendix 8)
Supplementary note: The estimation unit estimates the power of the lens by using an image in which a distance between an outline reflected on the inside and outside of the lens worn by the person is a threshold value or more in a plurality of input images of a person. 7. Verification auxiliary device according to 7.

(Appendix 9)
The estimation unit estimates the power of the lens reflected in the input image for which the degree of similarity obtained by matching with any of the three-dimensional registered images of a plurality of registrants is equal to or higher than a threshold value. , the verification auxiliary device according to appendix 7 or 8.

(Appendix 10)
detecting an input image including a person wearing a lens in at least one input image of a person;
Outputting a corrected image, which is an image in which the position and size of the eye reflected in the lens is corrected using the power of the lens estimated in the input image, and a three-dimensional registered image, which is a target for matching with the corrected image. do,
Matching aid method.

(Appendix 11)
to the computer,
A process of detecting an input image including a person wearing a lens in at least one input image of a person;
Outputting a corrected image, which is an image in which the position and size of the eye reflected in the lens is corrected using the power of the lens estimated in the input image, and a three-dimensional registered image, which is a target for matching with the corrected image. processing,
A program storage medium that stores a program for executing.

(Appendix 12)
detecting an input image including a person wearing glasses having lenses from at least one input image of a person;
Outputting the input image and a three-dimensional corrected image in which the eye position and size of a three-dimensional registered image that is a target for comparison with the input image is corrected using the power of the lens estimated in the input image. do,
Matching aid method.

(Appendix 13)
to the computer,
A process of detecting an input image including a person wearing glasses having lenses in at least one input image of a person;
Outputting the input image and a three-dimensional corrected image in which the eye position and size of a three-dimensional registered image that is a target for comparison with the input image is corrected using the power of the lens estimated in the input image. processing,
A program storage medium that stores a program for executing.

1 Matching assistance system 100 Processing device 110 Input section 120 Detection section 130 Extraction section 140 Frequency estimation section 141 Calculation section 142 Learning model storage section 143 Estimation section 150 Image processing section 160 Storage section 170 Matching section 180 Output section 190 Three-dimensional image processing section 200 Imaging device 210 Imaging section 300 User terminal 310 Display section 320 Operation reception section 1A Processor 2A Memory 3A Input/output interface 4A Peripheral circuit 5A Communication interface 6A Bus E1 First scanning distance E2 Second scanning distance E3, E4, FL Face contour VFL Contour of the face inside the lens GL1, GL2 Lens Ψ Incident angle Θ Person's face direction Φ Angle of view xs 1st pixel distance c 2nd pixel distance XL Number of pixels of captured image P Camera lens CS CCD sensor

Claims (8)

a detection unit that detects an input image including a person wearing glasses having lenses in at least one input image of a person;
3D correction in which the eye position and size of a 3D registered image including the person not wearing glasses, which is a target for comparison with the input image, is corrected using the power of the lens estimated in the input image; an output unit that outputs a subsequent image and the input image;
A verification auxiliary device comprising: The verification auxiliary device according to claim 1, further comprising a display unit that displays the three-dimensional corrected image and the input image on the same screen. The display unit displays the three-dimensional corrected image corresponding to the three-dimensional registered image for which the degree of similarity obtained by matching with the input image is equal to or higher than a threshold.
The verification auxiliary device according to claim 2. Using the positional relationship between the outline reflected inside the lens and the outline reflected outside the lens in the input image, and the positional relationship between the person and the imaging unit that photographed the person wearing the lens, The verification auxiliary device according to any one of claims 1 to 3, further comprising an estimation unit that estimates the power of the lens. The estimating unit estimates the power of the lens by using an image in which a distance between an outline reflected on the inside and outside of the lens worn by the person is equal to or greater than a threshold among a plurality of input images of a person. Verification auxiliary device according to item 4. The estimation unit estimates the power of the lens reflected in the input image for which the degree of similarity obtained by matching with any of the three-dimensional registered images of a plurality of registrants is equal to or higher than a threshold value. , A verification auxiliary device according to claim 4 or 5. The computer is
detecting an input image including a person wearing a lens in at least one input image of a person;
3D correction in which the eye position and size of a 3D registered image including the person not wearing glasses, which is a target for comparison with the input image, is corrected using the power of the lens estimated in the input image; outputting a subsequent image and the input image;
Matching aid method. to the computer,
A process of detecting an input image including a person wearing a lens in at least one input image of a person;
3D correction in which the eye position and size of a 3D registered image including the person not wearing glasses, which is a target for comparison with the input image, is corrected using the power of the lens estimated in the input image; a process of outputting a subsequent image and the input image;
A program to run.

Images