WO2021181642A1 - Collation assistance device, collation assistance method, and program storage medium - Google Patents

Collation assistance device, collation assistance method, and program storage medium Download PDF

Info

Publication number
WO2021181642A1
WO2021181642A1 PCT/JP2020/010980 JP2020010980W WO2021181642A1 WO 2021181642 A1 WO2021181642 A1 WO 2021181642A1 JP 2020010980 W JP2020010980 W JP 2020010980W WO 2021181642 A1 WO2021181642 A1 WO 2021181642A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
lens
input image
dimensional
collation
Prior art date
Application number
PCT/JP2020/010980
Other languages
French (fr)
Japanese (ja)
Inventor
秀治 古明地
那孟 鈴木
Original Assignee
日本電気株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日本電気株式会社 filed Critical 日本電気株式会社
Priority to PCT/JP2020/010980 priority Critical patent/WO2021181642A1/en
Priority to JP2022505677A priority patent/JP7439899B2/en
Publication of WO2021181642A1 publication Critical patent/WO2021181642A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis

Definitions

  • the present invention relates to a collation assisting device, a collation assisting method, and a program storage medium.
  • Patent Document 1 describes a two-dimensional image captured by changing the orientation and size of the three-dimensional image when the user visually collates the three-dimensional image of the person registered in the database with the captured person. A technique for facilitating collation with is disclosed.
  • Patent Document 1 In the technique described in Patent Document 1, the collation by the user is facilitated by changing the orientation and size of the three-dimensional image registered in the database. However, when the person shown in the captured image wears spectacles, the position and size of the eyes reflected in the lens look different from those of the naked eye due to the refraction of the lens. Since this becomes remarkable depending on the face orientation of the person and the like, when the user visually compares the two-dimensional image and the three-dimensional image, the matching accuracy of whether or not they are the same person is lowered. Patent Document 1 does not consider the above situation.
  • the detection unit that detects the input image including the person wearing the lens and the power of the lens estimated in the input image are used.
  • a collation assisting device including a corrected image which is an image in which the position and size of the eyes reflected on the lens are corrected, and an output unit which outputs a three-dimensional registered image which is a collation target with the corrected image. ..
  • an input image including a person wearing a lens is detected and reflected on the lens using the power of the lens estimated in the input image.
  • a collation assisting method for outputting a corrected image which is an image in which the position and size of an eye are corrected, and a three-dimensional registered image, which is a collation target between the corrected image.
  • a process of detecting an input image including a person wearing a lens in at least one input image obtained by capturing a person on a computer, and using the power of the lens estimated in the input image are used.
  • a storage medium is provided.
  • a detection unit that detects an input image including a person wearing glasses having a lens, and a power of the lens estimated from the input image.
  • a collation assisting device including a three-dimensional corrected image in which the eye position and size of the three-dimensional registered image to be collated with the input image are corrected using the above, and an output unit for outputting the input image.
  • an input image including a person wearing glasses having a lens is detected, and the power of the lens estimated in the input image is used.
  • a collation assisting method for outputting a three-dimensional corrected image obtained by correcting the eye position and size of a three-dimensional registered image to be collated with the input image and the input image.
  • a program storage medium for storing images is provided.
  • a collation assisting device a collation assisting method, and a program capable of improving collation accuracy by reducing changes in appearance due to wearing eyeglasses are provided.
  • FIG. 1 shows an overall configuration example of the collation assisting system 1 according to the present embodiment.
  • the collation assistance system is an information processing system including a processing device 100, an image pickup device 200, and a user terminal 300. Each device and terminal is connected via a network.
  • a specific example of the situation shown in FIG. 1 is a situation in which a person is photographed by a camera installed on the street.
  • the image pickup device 200 is a terminal that captures a person and obtains a captured image, for example, a security camera installed on a street.
  • the image pickup device 200 takes an image of a person passing through the image pickup section, and outputs the captured image to the processing device 100.
  • FIG. 2 is a functional block diagram showing the configuration of the collation assistance system 1.
  • the collation assistance system 1 includes a processing device 100, an image pickup device 200, and a user terminal 300.
  • the processing device 100 includes an input unit 110, a detection unit 120, an extraction unit 130, a frequency estimation unit 140, an image processing unit 150, a storage unit 160, a collation unit 170, and an output unit 180.
  • the imaging device 200 includes an imaging unit 210.
  • the user terminal 300 includes a display unit 310 and an operation reception unit 320.
  • the imaging unit 210 included in the imaging device 200 images a person passing through the imaging section and outputs the captured image to the processing device 100.
  • the imaging unit 210 images the imaging section in time series according to, for example, a set frame rate.
  • the image pickup unit 210 is not limited to the above specific example, and may be one that captures an image at a timing that receives an instruction from the outside, or may be one that captures a still image at a predetermined timing.
  • the captured image may include at least the head of the person and may not include the whole body of the person.
  • the input unit 110 receives the input of the captured image output by the imaging unit 210.
  • the target (input image) for which the input unit 110 accepts the input may be all the captured images captured by the imaging unit 210, or may be a part of the captured images.
  • the captured images may be extracted at predetermined time intervals and sequentially received for input.
  • a captured image obtained by thinning out a part of the captured images captured by the imaging unit 210 according to a predetermined frame rate may be extracted.
  • the target accepted by the input unit 110 is not limited to the above.
  • the detection unit 120 detects the human head and eyeglasses from the captured image received by the input unit 110.
  • a trained model in which the image of the head wearing the spectacles is learned by machine learning is used.
  • the machine learning method for example, deep learning via a multi-layer neural network may be used.
  • the method used to detect the wearing of a person's head and eyeglasses is not limited to the above.
  • the detection unit 120 may detect the detection of the human head and the wearing of the spectacles in a stepwise manner, or may independently detect each of them. In this case, the wearing of eyeglasses may be detected after detecting the human head. Further, the detection unit 120 does not necessarily have to detect the head, and may detect eyeglasses.
  • the detection unit 120 detects an image in which the contour of the face is reflected inside the lens of the spectacles from the captured images that have detected the wearing of the spectacles.
  • FIG. 3 is an example showing the state of the face in which the outline of the face is reflected inside the lens of the spectacles.
  • FL in the figure shows the outline of a person's face
  • GL1 and GL2 show a lens.
  • VFL shows the contour of the face reflected inside the lens.
  • FIG. 3A shows a state in which the person's face faces the front of the imaging unit 210 (face tilt angle is 0 degrees)
  • FIG. 3B shows a state in which the face does not face the front of the imaging unit 210 (face tilt). The angle is not 0 degrees).
  • FIG. 3B shows, as an example, a state in which a person faces to the right.
  • the state in which the face faces the front is a state in which the tilt angle of the face is close to 0 degrees with respect to the imaging direction and is smaller than a predetermined threshold value.
  • the state in which the face is not facing the front is a state in which the inclination angle of the face with respect to the imaging direction is larger than a predetermined threshold value.
  • a predetermined threshold value is arbitrarily set. As shown in FIG. 3A, when the face orientation is in front of the imaging unit 210 (the tilt angle of the face is 0 degrees), the contour of the face is difficult to be reflected inside the lenses GL1 and GL2. On the other hand, as shown in FIG.
  • the lens when the face orientation is not in front of the imaging unit 210 (the tilt angle of the face is not 0 degrees), the lens is affected by the refraction phenomenon as compared with the case where the face orientation is in front.
  • the outline of the face is easily reflected inside the GL1.
  • the detection unit 120 estimates the tilt angle of the face with respect to the imaging direction for the face reflected in the captured image.
  • the tilt angle of the face is an angle at which the face of the person is facing with reference to a straight line passing through the imaging unit 210 and the head of the person in the three-dimensional space. Specifically, when the face of the person faces the imaging unit 210, the inclination angle is close to 0 degrees and smaller than a predetermined threshold value.
  • the image of the target for which the detection unit 120 detects the face may be a two-dimensional image extracted by the extraction unit 130, which will be described later.
  • the extraction unit 130 extracts an captured image in which the outline of the face is reflected inside the lens as a two-dimensional image. As illustrated in FIG. 3B, the extraction unit 130 extracts an image in which the contour of the face is reflected inside the lens. In other words, the extraction unit 130 extracts an image in which the face of the person is oriented obliquely so that the outline of the face is reflected inside the lens when viewed from the image pickup unit 210.
  • the extraction unit 130 may extract an image in which the inclination angle of the face is equal to or more than the first threshold value arbitrarily set, or an image in which the inclination angle of the face is equal to or less than the second threshold value arbitrarily set.
  • the second threshold value may be set to a value larger than the first threshold value.
  • the “extraction” executed by the extraction unit 130 refers to a process of extracting a part of the captured images from a plurality of captured images. Specifically, from a plurality of captured images captured in time series according to a predetermined frame rate, only the captured images in which the wearing of eyeglasses is detected may be extracted.
  • the “extraction” may be a concept including a process of cutting out a part of the captured image. For example, in the captured image including a plurality of people, one area in which the head wearing glasses is reflected may be cut out, or one area in which the person is reflected may be cut out.
  • the process pointed to by "extraction” is not limited to the above specific example.
  • the power estimation unit 140 estimates the power of the glasses worn by a person in the two-dimensional image extracted by the extraction unit 130.
  • the details of the frequency estimation unit 140 will be described with reference to FIG. FIG. 4 is a functional block diagram of the frequency estimation unit 140.
  • the frequency estimation unit 140 includes a calculation unit 141, a learning model storage unit 142, and an estimation unit 143.
  • the calculation unit 141 calculates the distance between the face of the person imaged by the imaging unit 210 and the imaging unit 210 in the three-dimensional space.
  • the distance for example, the size of the area showing the face on the captured image or the length of the area showing the person may be used, but the method for calculating the distance is not limited to this, and those skilled in the art will appropriately know the technique. Can be applied.
  • the calculation unit 141 calculates the difference in the position of the contour of the face reflected on the inside and the outside of the lens (referred to as "contour difference" in the present specification).
  • contour difference the difference in the position of the contour of the face reflected on the inside and the outside of the lens.
  • the concept of contour difference will be described in detail with reference to FIG.
  • FIG. 5 is a diagram schematically showing a part of a face to which a lens is attached.
  • the calculation unit 141 calculates the horizontal inter-pixel distance (first scanning distance E1) between the horizontal position of the center point of the eye and the position E4 of the contour of the face in the region inside the lens.
  • the calculation unit 141 calculates the horizontal inter-pixel distance (second scanning distance E2) between the horizontal position of the center point of the eye and the position E3 of the contour of the face in the region outside the lens.
  • the calculation unit 141 calculates the contour difference (normalized contour difference) obtained by normalizing the difference between the first scanning distance E1 and the second scanning distance E2 using the distance between the imaging unit 210 and the face.
  • the specific method of normalization is not limited, and for example, the difference between the first scanning distance E1 and the second scanning distance E2 may be normalized by dividing by the distance between the imaging unit 210 and the face. In normalizing the contour difference, any calculation formula may be used as long as the distance between the imaging unit 210 and the face is taken into consideration.
  • the calculation unit 141 calculates the incident angle of the light incident on the imaging unit 210 from an object reflected in an arbitrary pixel on the captured image.
  • An example of a method of calculating the incident angle ⁇ from a real object corresponding to an arbitrary pixel position on the captured image to the imaging unit 210 will be described in detail with reference to FIGS. 6 and 17.
  • FIG. 6 is a diagram schematically showing an optical system of an imaging unit 210 that images a person.
  • the CCD (Charge-Coupled Device) sensor CS detects light incident from a person via the camera lens P.
  • the calculation formula (1) in FIG. 17 is a formula showing the derivation process of the incident angle ⁇ . The variables constituting the calculation formula (1) will be described.
  • the incident angle ⁇ is reflected in the captured image from the center position of the captured image showing the entire imaging range of the imaging unit 210, with the first pixel distance xs from the center position in the captured image indicating the entire imaging range of the imaging unit 210 as a variable.
  • the calculation formula shown in FIG. 17 using the second pixel distance c to the center position of the face, the number of pixels XL of the captured image, the face orientation ⁇ detected by the detection unit 120, and the angle of view ⁇ of the imaging unit 210 ( It can be expressed by 1).
  • the calculation unit 141 calculates the incident angle at the position of the contour of the face reflected in the region inside the lens. For the calculation, for example, the calculation formula (1) shown in FIG. 17 can be used.
  • the learning model storage unit 142 stores a regression model that outputs the power of the lens when the incident angle and the normalized contour difference are input.
  • the regression model is a trained model in which the combination of the incident angle and the normalized contour difference and the dioptric power of the lens is learned in advance.
  • the estimation unit 143 inputs the incident angle and contour difference calculated by the calculation unit 141 into the regression model stored in the learning model storage unit 142, and estimates the power of the lens.
  • the regression model described above may be learned to input the incident angle, the contour difference before normalization, and the distance between the imaging unit 210 and the face, and output the power of the lens. In this case, since it is not necessary to normalize the contour difference, the processing load can be reduced.
  • An example of the function of the power estimation unit 140 described above is the positional relationship between the contour reflected inside the lens and the contour reflected outside the lens in the captured image, and the photographing unit 210 and the person who photographed the person wearing the lens.
  • the power of the lens is estimated using the positional relationship.
  • the power estimation unit 140 for example, as described in Patent Document 2 (Japanese Unexamined Patent Publication No. 2015-25859), outlines the face reflected inside the lens of the spectacles.
  • a method of estimating the power of the lens based on the position of the lens, the position of the contour of the face visible on the outside of the lens, and the inclination angle of the face may be used.
  • the method for estimating the power of the lens is not limited to the above.
  • the image processing unit 150 corrects the image of the two-dimensional image using the power information of the lens estimated by the power estimation unit 140, and generates a corrected image (corrected image) corresponding to the two-dimensional image. do.
  • the area to be corrected is the area inside and around the lens in the image, and the contour of the face and the position and size of the eyes of the person reflected in the area in the lens of which the power is estimated are corrected. Specifically, the area reflected inside the lens is corrected so as to be enlarged so that the contour reflected inside the lens and the contour reflected outside the lens are located on the same curve.
  • the storage unit 160 stores three-dimensional images (three-dimensional registered images) of a plurality of registrants in association with registrant information.
  • items of registrant information include registrant identification information, name, registration date and time, and the like.
  • the collation unit 170 collates a person reflected in the two-dimensional image extracted by the extraction unit 130 with a plurality of three-dimensional registered images stored in the storage unit 160, and collates the person imaged by the imaging unit 210 with a plurality of three-dimensional registrations. Calculate the similarity with the image.
  • the collation unit 170 sets a three-dimensional registered image having a similarity equal to or higher than a threshold value as a collation target by the user's visual inspection. There may be a plurality of three-dimensional registered images set as targets for visual collation.
  • the output unit 180 outputs a three-dimensional registered image set as a target for visual collation and a corrected image (corrected image) processed by the image processing unit 150.
  • the user terminal 300 includes a display unit 310 and an operation reception unit 320.
  • the user terminal 300 is a terminal that provides information to the user or accepts the user's operation. Specifically, the user terminal 300 prompts the user to determine whether or not the three-dimensional registered image of the person registered in the database (not shown) and the person captured by the security camera or the like are the same person. It is a device. In addition, the device may be a device for confirming an image captured by a user with a security camera or the like, but the present invention is not limited to this.
  • the display unit 310 displays the three-dimensional registered image output by the output unit 180 and the corrected image processed by the image processing unit 150.
  • the display unit 310 may display the three-dimensional registered image and the corrected image at the same time.
  • the display unit 310 may display a plurality of three-dimensional registered images at the same time, or may sequentially display a plurality of three-dimensional registered images according to the degree of similarity.
  • the display unit 310 may display a plurality of corrected images at the same time. Further, the display unit 310 may display, for example, the three-dimensional registered image and the corrected image on the same screen.
  • the user determines whether or not at least one three-dimensional registered image displayed by the display unit 310 and the corrected image indicate the same person, and inputs the determination result to the operation reception unit 320.
  • the operation reception unit 320 accepts the input of the judgment result judged by the user by the operation of the user.
  • the operation accepted by the operation receiving unit 320 is, for example, an operation of selecting an image of the same person as the corrected image to be collated from the three-dimensional registered images displayed on the display unit 310.
  • FIGS. 7 to 9 are flowcharts showing an example of the processing in the present embodiment.
  • FIG. 7 shows an example of a flow related to image processing.
  • the input unit 110 receives the input of the captured image output by the imaging device 200 (S101).
  • the input unit 110 may sequentially accept inputs of images captured in time series according to a predetermined frame rate, for example.
  • the detection unit 120 detects the human head and eyeglasses from the captured image received by the input unit 110 (S102).
  • the detection unit 120 may detect the detection of the human head and the wearing of the spectacles in a stepwise manner, or may independently detect each of them. If the detection unit 120 does not detect the spectacles from the captured image (S103, NO), the process returns to step S101 and the input of the next captured image is accepted (S101).
  • the detection unit 120 detects the spectacles from the captured image (S103, YES)
  • the detection unit 120 shows the contour of the face inside the lens of the spectacles from the captured image detected that the spectacles are worn. Whether or not it is determined (S104). Twice
  • the extraction unit 130 extracts the captured image in which the contour is reflected on the inside of the lens from the captured image (S106).
  • the process returns to the process of step S101.
  • the image processing unit 150 corrects the image extracted by the extraction unit 130 using the power information of the lens estimated by the power estimation unit 140, and generates a corrected image (corrected image) (S108). ..
  • FIG. 8 shows an example of a processing flow related to collation by the processing device 100.
  • the collation unit 170 collates the person reflected in the two-dimensional image extracted by the extraction unit 130 with the plurality of three-dimensional registered images stored in the storage unit 160, and the person imaged by the imaging unit 210 and the plurality of registrants.
  • the similarity of is calculated (S109).
  • S110, YES the three-dimensional registered image is set as a target for collation by the user's visual inspection
  • S110, NO the process ends.
  • a series of flows (S101 to S108) related to the image processing shown in FIG. 7 may be executed. In that case, since the power of the lens is estimated only for the image in which the lens is recognized, the processing load of the device can be reduced.
  • the flow related to collation (S109 to S111) and the flow related to image processing (S101 to S108) may be processed in parallel.
  • the flow related to image processing (S101 to S108) may be executed before the flow related to collation (S109 to S111).
  • the context of the series of flows related to image processing (S101 to S108) shown in FIG. 7 and the processing flows related to collation by the apparatus shown in FIG. 8 (S109 to S111) is not limited.
  • FIG. 9 shows an example of a processing flow related to image display.
  • the output unit 180 outputs a three-dimensional registered image set as a target for visual collation and a corrected image (corrected image) processed by the image processing unit 150 (S112).
  • the display unit 310 displays the three-dimensional registered image output by the output unit 180 and the corrected image processed by the image processing unit 150 (S113).
  • the operation reception unit 320 accepts the input of the judgment result judged by the user by the operation of the user (S114).
  • the operation accepted by the operation receiving unit 320 is, for example, an operation of selecting an image of the same person as the corrected image to be collated from the three-dimensional registered images displayed on the display unit 310.
  • the processing flow (S112 to S113) related to the display of the image shown in FIG. 9 is a series of flows (S101 to S108) related to the image processing shown in FIG. 7 and a processing flow related to collation by the apparatus shown in FIG. 8 (S101 to S108). It is executed after S109 to S111).
  • the collation target is a person wearing a lens
  • the second embodiment is different from the first embodiment in that the processing device 100 includes a three-dimensional image processing unit 190 and corrects the three-dimensional registered image based on the estimated power information of the lens. do.
  • the description of the parts common to the first embodiment will be omitted.
  • FIG. 10 is a functional block diagram showing the configuration of the collation assistance system 1 in the present embodiment.
  • the processing device 100 includes an input unit 110, a detection unit 120, an extraction unit 130, a frequency estimation unit 140, an image processing unit 150, a storage unit 160, a collation unit 170, an output unit 180, and a three-dimensional image. It is provided with a processing unit 190.
  • the three-dimensional image processing unit 190 corrects the image around the eyes of a person in the three-dimensional registered image set by the collating unit 170 as the target of visual collation, and the corrected three-dimensional image corresponding to the three-dimensional registered image ( Generate an image after 3D correction).
  • the predetermined three-dimensional registered image is corrected according to the power information of the lens estimated by the power estimation unit 140.
  • the predetermined three-dimensional registered image is a three-dimensional registered image set by the collation unit 170 as a target for visual collation.
  • Known three-dimensional computer graphics are used for this correction. An example of correction according to the power information of the lens will be described.
  • the three-dimensional image processing unit 190 uses the estimated power information of the concave lens to reproduce the change in facial features that appears to be refracted by the lens when the person indicated by the three-dimensional registered image wears the concave lens. , Correct the image around the eyes.
  • the storage unit 160 stores a plurality of three-dimensional registered images in association with the registrant information. Examples of items of registrant information include registrant identification information, name, registration date and time, and the like.
  • the storage unit 160 may store the three-dimensional corrected image generated by the three-dimensional image processing unit 190.
  • the collation unit 170 collates a person reflected in the two-dimensional image extracted by the extraction unit 130 with a three-dimensional registered image indicating a plurality of registrants stored in the storage unit 160, and collates a plurality of persons imaged by the imaging unit 210. Calculate the degree of similarity with the registrant of.
  • the collation unit 170 sets the obtained three-dimensional registered image having a similarity equal to or higher than the threshold value as a collation target by the user's visual inspection. There may be a plurality of three-dimensional registered images set as targets for visual collation.
  • the output unit 180 outputs a three-dimensional corrected image showing a person set as a target for visual collation and an image extracted by the extraction unit 130.
  • the three-dimensional corrected image is a three-dimensional image generated by the image processing unit 150.
  • the display unit 310 displays a three-dimensional corrected image showing a person set as a target for visual collation and a two-dimensional image extracted by the extraction unit 130.
  • the display unit 310 may display the three-dimensional corrected image and the two-dimensional image at the same time.
  • the display unit 310 may display a plurality of three-dimensional images at the same time, or may sequentially display a plurality of three-dimensional images according to the obtained similarity.
  • the display unit 310 may display a plurality of two-dimensional images at the same time. Further, the display unit 310 may display, for example, the three-dimensional registered image and the corrected image on the same screen.
  • the user determines whether or not at least one three-dimensional image displayed by the display unit 310 and the two-dimensional image indicate the same person, and inputs the determination result to the operation reception unit 320.
  • the operation reception unit 320 accepts the input of the judgment result judged by the user by the operation of the user.
  • the operation received by the operation receiving unit 320 is, for example, an operation of selecting an image of the same person as the two-dimensional image to be collated from the three-dimensional images displayed on the display unit 310.
  • FIG. 11 shows an example of a processing flow related to image processing in the present embodiment. The description of the process that overlaps with the first embodiment will be omitted.
  • the processing flow related to the collation by the apparatus is the same as that in the first embodiment (S109 to S111).
  • the power estimation unit 140 estimates the power of the glasses worn by a person in the image extracted by the extraction unit 130 (S107).
  • the three-dimensional image processing unit 190 corrects the image around the eyes of a person using the estimated lens power information in the three-dimensional registered image set by the collation unit 170 as the target of visual collation, and three-dimensionally. A corrected image is generated (S208).
  • the storage unit 160 stores the three-dimensional corrected image in association with the registrant information (S209).
  • the output unit 180 outputs a three-dimensional corrected image showing a person set as a target for visual collation and an image extracted by the extraction unit 130 (S212).
  • the display unit 310 displays a three-dimensional corrected image showing a person set as a target for visual collation and a two-dimensional image extracted by the extraction unit 130 (S213).
  • the display unit 310 may display the three-dimensional corrected image and the two-dimensional image at the same time.
  • the display unit 310 may display a plurality of three-dimensional images at the same time, or may sequentially display a plurality of three-dimensional images according to the degree of similarity.
  • the display unit 310 may display a plurality of two-dimensional images at the same time.
  • the user determines whether or not at least one three-dimensional image displayed by the display unit 310 and the two-dimensional image indicate the same person, and inputs the determination result to the operation reception unit 320.
  • the operation receiving unit 320 receives the input of the judgment result judged by the user by the operation of the user (S114).
  • the flow related to collation (S109 to S111) and the flow related to image processing (S101 to S107, S208 to S209) may be processed in parallel.
  • the flow related to image processing (S101 to S107, S208 to S209) may be executed before the flow related to collation (S109 to S111).
  • the context of the series of flows related to image processing shown in FIGS. 7 and 11 (S101 to S107, S208 to S209) and the processing flow related to collation by the apparatus shown in FIG. 8 (S109 to S111) is not limited.
  • the collation assistance system of the present embodiment when the imaged person is wearing glasses, the three-dimensional registered image held by the database is corrected according to the change in appearance due to wearing the glasses. As a result, even if the collation target is a person wearing spectacles, it is possible to reduce changes in appearance and facial impression due to wearing spectacles, and the user can perform visual collation efficiently and accurately. be able to.
  • FIG. 13 is a functional block diagram of the collation assistance system 1 according to the present embodiment.
  • the collation assistance system 1 includes a detection unit 120 and an output unit 180.
  • the detection unit 120 detects an input image including a person wearing a lens in at least one input image obtained by capturing a person.
  • the output unit 180 is a three-dimensional registered image that is a collation target between the corrected image, which is an image obtained by correcting the input image using the power information of the lens estimated from the input image detected by the detection unit 120, and the corrected image. And output.
  • the corrected image is, for example, an image in which the position and size of the eyes reflected on the lens in the input image are corrected.
  • FIG. 14 is a flowchart showing an example of the processing in the present embodiment.
  • the detection unit 120 detects an input image including a person wearing a lens in the captured image (S307).
  • the output unit 180 is a three-dimensional registered image that is a collation target between the corrected image, which is an image obtained by correcting the input image using the power information of the lens estimated from the input image detected by the detection unit 120, and the corrected image. And is output (S308).
  • the collation target is a person wearing spectacles
  • FIG. 13 is a functional block diagram of the collation assistance system 1 according to the present embodiment.
  • the collation assistance system 1 includes a detection unit 120 and an output unit 180.
  • the detection unit 120 detects an input image including a person wearing eyeglasses having a lens in at least one input image obtained by capturing a person.
  • the output unit 180 outputs a three-dimensional corrected image obtained by correcting the three-dimensional registered image and the input image by using the power of the lens estimated from the input image.
  • the three-dimensional corrected image output by the output unit 180 is, for example, an image in which the position and size of the eyes of the three-dimensional registered image to be collated with the input image are corrected.
  • FIG. 15 is a flowchart showing an example of the processing in the present embodiment.
  • the detection unit 120 detects an input image including a person wearing eyeglasses having a lens in at least one input image obtained by capturing a person (S407).
  • the output unit 180 outputs the three-dimensional corrected image corrected by the three-dimensional registered image and the input image by using the power of the lens estimated from the input image (S408).
  • the collation target is a person wearing glasses, it is possible to reduce changes in appearance and changes in facial impression due to wearing glasses. It is possible to generate an image for the user to perform visual collation efficiently and accurately.
  • Each functional unit included in the processing device 100, the image pickup device 200, and the user terminal 300 includes at least one CPU (Central Processing Unit) of an arbitrary computer, at least one memory, a program loaded into the memory, and at least a program for storing the program. It is realized by any combination of hardware and software centering on a storage unit such as one hard disk, an interface for network connection, and the like. It will be understood by those skilled in the art that there are various modifications of this realization method and apparatus.
  • the storage unit can store not only programs stored before the device is shipped, but also storage media such as optical disks, magneto-optical disks, and semiconductor flash memories, and programs downloaded from servers on the Internet.
  • FIG. 16 is a block diagram illustrating the hardware configurations of the processing device 100, the imaging device 200, and the user terminal 300.
  • the processing device 100, the image pickup device 200, and the user terminal 300 include a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, a communication interface 5A, and a bus 6A.
  • the peripheral circuit 4A includes various modules.
  • the processing device 100, the image pickup device 200, and the user terminal 300 do not have to have the peripheral circuit 4A.
  • the processing device 100, the image pickup device 200, and the user terminal 300 may be composed of a plurality of physically and / or logically separated devices. In this case, each of the plurality of devices can be provided with the above hardware configuration.
  • the bus 6A is a data transmission path for the processor 1A, the memory 2A, the input / output interface 3A, the peripheral circuit 4A, and the communication interface 5A to send and receive data to and from each other.
  • the processor 1A is, for example, an arithmetic processing unit such as a CPU, a GPU (Graphics Processing Unit), or a microprocessor.
  • the processor 1A can execute the process according to various programs stored in the memory 2A, for example.
  • the memory 2A is, for example, a memory such as a RAM (RandomAccessMemory) or a ROM (ReadOnlyMemory), and stores a program and various data.
  • a RAM RandomAccessMemory
  • ROM ReadOnlyMemory
  • the input / output interface 3A includes an interface for acquiring information from an input device, an external device, an external storage unit, an external sensor, a camera, etc., an interface for outputting information to an output device, an external device, an external storage unit, etc. including.
  • the input device is, for example, a touch panel, a keyboard, a mouse, a microphone, a camera, or the like.
  • the output device is, for example, a display, a speaker, a printer, a lamp, or the like.
  • Processor 1A can issue commands to each module and perform calculations based on the calculation results.
  • the communication interface 5A realizes that the processing device 100, the image pickup device 200, and the user terminal 300 communicate with each other with the external device, and also realizes that the processing device 100, the image pickup device 200, and the user terminal 300 communicate with each other.
  • a computer may configure some functions of the processing device 100, the imaging device 200, and the user terminal 300.
  • ⁇ Modification example> An example of modification applicable to the above-described embodiment will be described. If the power estimation unit 140 fails to estimate the power of the lens in the captured image of the target, the captured images taken before and after the captured image may be processed again. Further, the image processing unit 150 may perform image processing on the captured image so as to remove the edges of the spectacles.
  • the imaging unit 210 may include a device that acquires three-dimensional depth information (depth information).
  • the calculation unit 141 does not need to calculate the distance between the image pickup unit 210 and the person (or the face of the person) from the two-dimensional image
  • the estimation unit 143 uses the depth information acquired by the image pickup unit 210 to use the depth information.
  • the power of the lens may be estimated.
  • the collation unit 170 collates the person reflected in the two-dimensional image extracted by the extraction unit 130 with the plurality of three-dimensional registered images stored in the storage unit 160, and the corrected image processed by the image processing unit 150. It may be designed to collate with the three-dimensional registered image.
  • the image pickup apparatus 200 may execute a part or all of the processes executed by the processing apparatus 100.
  • the image pickup apparatus 200 may include a detection unit 120, an extraction unit 130, and a power estimation unit 140 in addition to the image pickup unit 210.
  • the image processing device 200 may estimate the power of the lens, and the image processing unit 150 included in the processing device 100 may acquire information regarding the power of the lens from the image pickup device 200.
  • the image pickup apparatus 200 may execute the processes of S101 to S107.
  • the imaging device 200 may be a target of a process of extracting only an captured image in which the contour difference of a person reflected in the captured image is equal to or greater than a threshold value and estimating the power of the lens, or a process of transmitting the image to the processing device 100. According to the above modification, it is possible to reduce the processing load of the processing device 100 and the load related to the transmission processing of the collation assisting system in the present invention.
  • a detection unit that detects an input image including a person wearing a lens in at least one input image of a person. Outputs a three-dimensional registered image that is a collation target between the corrected image, which is an image in which the position and size of the eyes reflected on the lens are corrected using the power of the lens estimated in the input image, and the corrected image. Output section and A collation assisting device comprising.
  • Appendix 2 A display unit that displays the corrected image and the three-dimensional registered image on the same screen.
  • the display unit displays a three-dimensional registered image having a similarity equal to or higher than a threshold value obtained by collating the input image corresponding to the corrected image with the three-dimensional registered image of a plurality of registrants.
  • the collation assisting device according to Appendix 2.
  • a detection unit that detects an input image including a person wearing eyeglasses having a lens in at least one input image of a person.
  • the three-dimensional corrected image obtained by correcting the eye position and size of the three-dimensional registered image to be collated with the input image using the power of the lens estimated in the input image, and the input image are output.
  • Output section and A collation assisting device comprising.
  • Appendix 5 The collation assisting device according to Appendix 4, further comprising a display unit that displays the three-dimensional corrected image and the input image on the same screen.
  • the display unit displays the three-dimensional corrected image corresponding to the three-dimensional registered image whose similarity obtained by collation with the input image is equal to or higher than the threshold value.
  • the collation assisting device according to Appendix 5.
  • Appendix 7 The positional relationship between the contour reflected inside the lens and the contour reflected outside the lens in the input image, and the positional relationship between the imaging unit and the person who photographed the person wearing the lens are used.
  • the collation assisting device according to any one of Appendix 1 to 6, further comprising an estimation unit for estimating the power of the lens.
  • the estimation unit estimates the power of the lens by using an image in which the distance between the contours reflected on the inside and the outside of the lens worn by the person is equal to or more than a threshold value in a plurality of input images obtained by capturing the person. 7.
  • the estimation unit estimates the power of the lens reflected in the input image with respect to the input image whose similarity obtained by collation with any of the three-dimensional registered images of a plurality of registrants is equal to or more than a threshold value.
  • the collation assisting device according to Appendix 7 or 8.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Collating Specific Patterns (AREA)
  • Image Analysis (AREA)

Abstract

A collation assistance device according to the present invention is provided with: a detection unit that, in order to assist in collating a two-dimensional input image against a previously registered three-dimensional image, detects, from at least one image in which a person is captured, an input image that includes a person wearing a lens; and an output unit for outputting a post-correction image, in which the position and size of an eye reflected by the lens are corrected using the power of the lens estimated from the input image, and a three-dimensional registered image, which is to be collated with the post-correction image.　

Description

照合補助装置、照合補助方法及びプログラム記憶媒体Verification assistance device, verification assistance method and program storage medium
　本発明は、照合補助装置、照合補助方法及びプログラム記憶媒体に関する。 The present invention relates to a collation assisting device, a collation assisting method, and a program storage medium.
　防犯カメラ等で撮像された人物と、データベースに登録された人物とを目視で照合することがあった。
　特許文献１には、データベースに登録された人物の三次元像と、撮像した人物とをユーザが目視で照合するにあたり、三次元像の向きや大きさを変更することで、撮像した二次元画像とを照合しやすくする技術が開示されている。 In some cases, a person photographed by a security camera or the like is visually collated with a person registered in a database.
Patent Document 1 describes a two-dimensional image captured by changing the orientation and size of the three-dimensional image when the user visually collates the three-dimensional image of the person registered in the database with the captured person. A technique for facilitating collation with is disclosed.
実用新案登録第３２０４１７５号公報Utility Model Registration No. 3204175 特開２０１５－２５８５９号公報Japanese Unexamined Patent Publication No. 2015-25859
　特許文献１に記載された技術においては、データベースに登録された三次元像の向きや大きさを変えることで、ユーザによる照合を容易にする。しかし、撮像画像に映る人物が眼鏡を装着していると、レンズの屈折によりレンズ内に映る目の位置や大きさが裸眼のときと比較して異なって見える。これは人物の顔向き等に応じて顕著になるため、ユーザが目視で二次元画像と三次元画像とを比較するときに同一人物か否かの照合精度が低下する。特許文献１では上記の状況は考慮されていない。 In the technique described in Patent Document 1, the collation by the user is facilitated by changing the orientation and size of the three-dimensional image registered in the database. However, when the person shown in the captured image wears spectacles, the position and size of the eyes reflected in the lens look different from those of the naked eye due to the refraction of the lens. Since this becomes remarkable depending on the face orientation of the person and the like, when the user visually compares the two-dimensional image and the three-dimensional image, the matching accuracy of whether or not they are the same person is lowered. Patent Document 1 does not consider the above situation.
　そこで、本発明は上記の問題を鑑み、眼鏡の装着により生じる外観の変化を軽減させることで、照合精度を向上させることを課題とする。 Therefore, in view of the above problems, it is an object of the present invention to improve the collation accuracy by reducing the change in appearance caused by wearing the spectacles.
　本発明の一観点によれば、人物を撮像した少なくとも一つの入力画像において、レンズを装着した人物を含む入力画像を検出する検出部と、前記入力画像において推定した前記レンズの度数を用いて当該レンズに映る目の位置及び大きさを補正した画像である補正後画像と、前記補正後画像との照合対象である三次元登録画像を出力する出力部と、を備える照合補助装置が提供される。 According to one aspect of the present invention, in at least one input image of a person, the detection unit that detects the input image including the person wearing the lens and the power of the lens estimated in the input image are used. Provided is a collation assisting device including a corrected image which is an image in which the position and size of the eyes reflected on the lens are corrected, and an output unit which outputs a three-dimensional registered image which is a collation target with the corrected image. ..
　本発明の一観点によれば、人物を撮像した少なくとも一つの入力画像において、レンズを装着した人物を含む入力画像を検出し、前記入力画像において推定した前記レンズの度数を用いて当該レンズに映る目の位置及び大きさを補正した画像である補正後画像と、前記補正後画像との照合対象である三次元登録画像を出力する、照合補助方法が提供される。 According to one aspect of the present invention, in at least one input image obtained by capturing a person, an input image including a person wearing a lens is detected and reflected on the lens using the power of the lens estimated in the input image. Provided is a collation assisting method for outputting a corrected image, which is an image in which the position and size of an eye are corrected, and a three-dimensional registered image, which is a collation target between the corrected image.
　本発明の一観点によれば、コンピュータに、人物を撮像した少なくとも一つの入力画像において、レンズを装着した人物を含む入力画像を検出する処理、前記入力画像において推定した前記レンズの度数を用いて当該レンズに映る目の位置及び大きさを補正した画像である補正後画像と、前記補正後画像との照合対象である三次元登録画像を出力する処理、を実行させるためのプログラムを記憶するプログラム記憶媒体が提供される。 According to one aspect of the present invention, a process of detecting an input image including a person wearing a lens in at least one input image obtained by capturing a person on a computer, and using the power of the lens estimated in the input image are used. A program that stores a program for executing a process of outputting a corrected image, which is an image in which the position and size of the eyes reflected on the lens are corrected, and a three-dimensional registered image, which is a collation target of the corrected image. A storage medium is provided.
　本発明の一観点によれば、人物を撮像した少なくとも一以上の入力画像において、レンズを有する眼鏡を装着した人物を含む入力画像を検出する検出部と、前記入力画像において推定した前記レンズの度数を用いて前記入力画像との照合対象である三次元登録画像の目の位置及び大きさを補正した三次元補正後画像と、前記入力画像と、を出力する出力部と、を備える照合補助装置が提供される。 According to one aspect of the present invention, in at least one or more input images of a person, a detection unit that detects an input image including a person wearing glasses having a lens, and a power of the lens estimated from the input image. A collation assisting device including a three-dimensional corrected image in which the eye position and size of the three-dimensional registered image to be collated with the input image are corrected using the above, and an output unit for outputting the input image. Is provided.
　本発明の一観点によれば、人物を撮像した少なくとも一以上の入力画像において、レンズを有する眼鏡を装着した人物を含む入力画像を検出し、前記入力画像において推定した前記レンズの度数を用いて前記入力画像との照合対象である三次元登録画像の目の位置及び大きさを補正した三次元補正後画像と、前記入力画像と、を出力する、照合補助方法が提供される。 According to one aspect of the present invention, in at least one input image of a person, an input image including a person wearing glasses having a lens is detected, and the power of the lens estimated in the input image is used. Provided is a collation assisting method for outputting a three-dimensional corrected image obtained by correcting the eye position and size of a three-dimensional registered image to be collated with the input image and the input image.
　本発明の一観点によれば、コンピュータに、人物を撮像した少なくとも一以上の入力画像において、レンズを有する眼鏡を装着した人物を含む入力画像を検出する処理、前記入力画像において推定した前記レンズの度数を用いて前記入力画像との照合対象である三次元登録画像の目の位置及び大きさを補正した三次元補正後画像と、前記入力画像と、を出力する処理、を実行させるためのプログラムを記憶するプログラム記憶媒体が提供される。 According to one aspect of the present invention, a process of detecting an input image including a person wearing glasses having a lens in at least one input image of a person captured by a computer, the lens estimated in the input image A program for executing a process of outputting a three-dimensional corrected image obtained by correcting the eye position and size of a three-dimensional registered image to be collated with the input image using a frequency, and the input image. A program storage medium for storing images is provided.
　本発明によれば、眼鏡の装着による外観の変化を軽減させることで、照合精度を向上することができる照合補助装置、照合補助方法及びプログラムが提供される。 According to the present invention, a collation assisting device, a collation assisting method, and a program capable of improving collation accuracy by reducing changes in appearance due to wearing eyeglasses are provided.
第１の実施形態における照合補助システムの全体構成例を示す図である。It is a figure which shows the whole configuration example of the collation assistance system in 1st Embodiment. 第１の実施形態における照合補助システムの機能を示すブロック図である。It is a block diagram which shows the function of the collation assistance system in 1st Embodiment. 第１から第４の実施形態における撮像対象の状態を説明する概念図である。It is a conceptual diagram explaining the state of the image pickup object in 1st to 4th Embodiment. 第１から第４の実施形態における撮像対象の状態を説明する概念図である。It is a conceptual diagram explaining the state of the image pickup object in 1st to 4th Embodiment. 第１の実施形態における度数推定部の機能を示すブロック図である。It is a block diagram which shows the function of the frequency estimation part in 1st Embodiment. 第１から第４の実施形態における照合補助システムが実行する計算処理を説明するための概念図である。It is a conceptual diagram for demonstrating the calculation process executed by the collation assistance system in 1st to 4th Embodiment. 第１から第４の実施形態における照合補助システムが実行する計算処理を説明するための概念図である。It is a conceptual diagram for demonstrating the calculation process executed by the collation assistance system in 1st to 4th Embodiment. 第１の実施形態における画像処理の一例を示すフローチャートである。It is a flowchart which shows an example of image processing in 1st Embodiment. 第１の実施形態における機械照合の一例を示すフローチャートである。It is a flowchart which shows an example of the machine collation in 1st Embodiment. 第１の実施形態におけるユーザ入力受付の一例を示すフローチャートである。It is a flowchart which shows an example of the user input reception in 1st Embodiment. 第２の実施形態における照合補助システムの機能を示すブロック図である。It is a block diagram which shows the function of the collation assistance system in 2nd Embodiment. 第２の実施形態における画像処理の一例を示すフローチャートである。It is a flowchart which shows an example of image processing in 2nd Embodiment. 第２の実施形態におけるユーザ入力受付の一例を示すフローチャートである。It is a flowchart which shows an example of the user input reception in 2nd Embodiment. 第３及び第４の実施形態における照合補助システムの機能を示すブロック図である。It is a block diagram which shows the function of the collation assistance system in 3rd and 4th Embodiment. 第３の実施形態における処理の一例を示すフローチャートである。It is a flowchart which shows an example of the process in 3rd Embodiment. 第４の実施形態における処理の一例を示すフローチャートである。It is a flowchart which shows an example of the process in 4th Embodiment. 第１から第４の実施形態における画像照合システムのハードウェア構成例を示すブロック図である。It is a block diagram which shows the hardware configuration example of the image collation system in 1st to 4th Embodiment. 本発明における処理に用いる計算式の一例である。This is an example of a calculation formula used for the processing in the present invention. 本発明の実施形態に適用可能な変形例を説明するブロック図である。It is a block diagram explaining the modification which can apply to embodiment of this invention.
　以下、図面を参照して、本発明の例示的な実施形態を説明する。図面において同様の要素又は対応する要素には同一の符号を付し、その説明を省略又は簡略化することがある。 Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings. Similar elements or corresponding elements may be designated by the same reference numerals in the drawings, and the description thereof may be omitted or simplified.
＜背景＞
　視力矯正用の眼鏡を装着する人物は、眼鏡を装着していないときと比較して顔立ちが異なって見える。これはレンズの屈折現象により目の位置と大きさが変化して見えるためである。カメラが撮像した映像を確認する管理者にとって、眼鏡の装着により顔立ちが変化して見える人物が、データベースに登録された人物画像の何れかに該当するか否かを目視で判断することは容易ではない。後述する実施形態では、管理者（ユーザ）を補助するシステムについて説明する。 <Background>
People who wear eyeglasses for vision correction look different in facial features than when they do not wear eyeglasses. This is because the position and size of the eyes appear to change due to the refraction phenomenon of the lens. It is not easy for an administrator who checks the image captured by the camera to visually determine whether or not a person whose facial features change due to wearing glasses corresponds to any of the person images registered in the database. No. In the embodiment described later, a system that assists the administrator (user) will be described.
＜第１の実施形態＞
　本実施形態の照合補助システム１の構成を図１及び図２を用いて説明する。図１は、本実施形態における照合補助システム１の全体構成例を示す。照合補助システムは、処理装置１００と、撮像装置２００と、ユーザ端末３００とを備える情報処理システムである。各装置及び端末は、ネットワークを介して接続される。図１が示す状況の具体例は、街頭に設置されたカメラで人物を撮像する状況である。 <First Embodiment>
The configuration of the collation assisting system 1 of the present embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 shows an overall configuration example of the collation assisting system 1 according to the present embodiment. The collation assistance system is an information processing system including a processing device 100, an image pickup device 200, and a user terminal 300. Each device and terminal is connected via a network. A specific example of the situation shown in FIG. 1 is a situation in which a person is photographed by a camera installed on the street.
　撮像装置２００は、人物を撮像して撮像画像を得る端末であり、例えば街頭に設置される防犯カメラである。撮像装置２００は撮像区画を通行する人物を撮像し、撮像画像を処理装置１００に出力する。 The image pickup device 200 is a terminal that captures a person and obtains a captured image, for example, a security camera installed on a street. The image pickup device 200 takes an image of a person passing through the image pickup section, and outputs the captured image to the processing device 100.
　図２は、照合補助システム１の構成を示す機能ブロック図である。照合補助システム１は、処理装置１００と、撮像装置２００と、ユーザ端末３００とを備える。処理装置１００は、入力部１１０と、検出部１２０と、抽出部１３０と、度数推定部１４０と、画像処理部１５０と、記憶部１６０と、照合部１７０と、出力部１８０を備える。撮像装置２００は、撮像部２１０を備える。ユーザ端末３００は、表示部３１０と操作受付部３２０を備える。 FIG. 2 is a functional block diagram showing the configuration of the collation assistance system 1. The collation assistance system 1 includes a processing device 100, an image pickup device 200, and a user terminal 300. The processing device 100 includes an input unit 110, a detection unit 120, an extraction unit 130, a frequency estimation unit 140, an image processing unit 150, a storage unit 160, a collation unit 170, and an output unit 180. The imaging device 200 includes an imaging unit 210. The user terminal 300 includes a display unit 310 and an operation reception unit 320.
　撮像装置２００が備える撮像部２１０は、撮像区画を通行する人物を撮像し、撮像画像を処理装置１００に出力する。なお、撮像部２１０は、例えば、設定されたフレームレートに応じて撮像区画を時系列的に撮像する。撮像部２１０は、上記の具体例に限定されず、外部からの指示を受けたタイミングで撮像するものであってもよいし、所定のタイミングで静止画を撮像するものであってもよい。本実施形態において撮像画像は少なくとも人物の頭部が含まれていればよく、人物の全身を含まなくともよい。 The imaging unit 210 included in the imaging device 200 images a person passing through the imaging section and outputs the captured image to the processing device 100. The imaging unit 210 images the imaging section in time series according to, for example, a set frame rate. The image pickup unit 210 is not limited to the above specific example, and may be one that captures an image at a timing that receives an instruction from the outside, or may be one that captures a still image at a predetermined timing. In the present embodiment, the captured image may include at least the head of the person and may not include the whole body of the person.
　入力部１１０は、撮像部２１０が出力した撮像画像の入力を受け付ける。ここで、入力部１１０が入力を受け付ける対象（入力画像）は、撮像部２１０が撮像した全ての撮像画像であってもよいし、一部であってもよい。例えば、撮像部２１０が時系列的に撮像した撮像画像のうち、所定の時間間隔で撮像画像を抽出し、順次入力を受け付けるものであってもよい。所定の時間間隔で撮像画像を抽出する一例として、所定のフレームレートに応じて撮像部２１０が撮像した撮像画像の中から一部を間引いた撮像画像を抽出するものであってもよい。入力部１１０が受け付ける対象は上記に限定されない。 The input unit 110 receives the input of the captured image output by the imaging unit 210. Here, the target (input image) for which the input unit 110 accepts the input may be all the captured images captured by the imaging unit 210, or may be a part of the captured images. For example, among the captured images captured by the imaging unit 210 in time series, the captured images may be extracted at predetermined time intervals and sequentially received for input. As an example of extracting captured images at predetermined time intervals, a captured image obtained by thinning out a part of the captured images captured by the imaging unit 210 according to a predetermined frame rate may be extracted. The target accepted by the input unit 110 is not limited to the above.
　検出部１２０は、入力部１１０が受け付けた撮像画像の中から、人間の頭部及び眼鏡を検出する。ここで人物の頭部及び眼鏡の検出には、例えば、機械学習により眼鏡を装着した頭部の画像を学習した学習済みモデルが用いられる。機械学習の手法は、例えば多層のニューラルネットワークを介したディープラーニングを用いてもよい。人物の頭部及び眼鏡の装着検出に用いられる手法は上記に限定されない。検出部１２０は、人間の頭部の検出と、眼鏡の装着とを、段階的に検出するものであってもよいし、それぞれ独立して検出するものであってもよい。この場合、人間の頭部を検出した後に、眼鏡の装着を検出してもよい。また、検出部１２０は、必ずしも頭部を検出する必要はなく、眼鏡を検出するものであってもよい。 The detection unit 120 detects the human head and eyeglasses from the captured image received by the input unit 110. Here, for the detection of the head of a person and the spectacles, for example, a trained model in which the image of the head wearing the spectacles is learned by machine learning is used. As the machine learning method, for example, deep learning via a multi-layer neural network may be used. The method used to detect the wearing of a person's head and eyeglasses is not limited to the above. The detection unit 120 may detect the detection of the human head and the wearing of the spectacles in a stepwise manner, or may independently detect each of them. In this case, the wearing of eyeglasses may be detected after detecting the human head. Further, the detection unit 120 does not necessarily have to detect the head, and may detect eyeglasses.
　検出部１２０は、眼鏡の装着を検出した撮像画像の中から、眼鏡が有するレンズの内側に顔の輪郭が映る画像を検出する。 The detection unit 120 detects an image in which the contour of the face is reflected inside the lens of the spectacles from the captured images that have detected the wearing of the spectacles.
　次に、図３を用いて撮像画像に映る顔の状態を説明する。図３は眼鏡のレンズの内側に顔の輪郭が映っている顔の状態を示す一例である。図中のFLは人物の顔の輪郭を示し、ＧＬ１及びＧＬ２はレンズを示す。また、ＶＦＬはレンズの内側に映る顔の輪郭を示している。図３Ａは撮像部２１０に対し人物の顔が正面を向いた状態を（顔の傾斜角度が０度）示し、図３Ｂは撮像部２１０に対し顔向きが正面を向いていない状態（顔の傾斜角度が０度でない）を示す。図３Ｂでは一例として、人物が右を向いた状態を示す。顔が正面を向いた状態とは、撮像方向に対して顔の傾斜角度が０度に近く、所定の閾値よりも小さい状態である。顔が正面を向いていない状態とは、撮像方向に対する顔の傾斜角度が所定の閾値よりも大きい状態である。所定の閾値は任意に設定される。図３Ａに示すように、顔向きが撮像部２１０に対して正面である場合（顔の傾斜角度が０度）、レンズＧＬ１及びＧＬ２の内側に顔の輪郭が映りにくい。一方、図３Ｂに示すように、顔向きが撮像部２１０に対し正面でない場合は（顔の傾斜角度が０度でない）、顔向きが正面である場合と比較して、屈折現象の影響によりレンズＧＬ１の内側に顔の輪郭が映りやすい。 Next, the state of the face reflected in the captured image will be described with reference to FIG. FIG. 3 is an example showing the state of the face in which the outline of the face is reflected inside the lens of the spectacles. FL in the figure shows the outline of a person's face, and GL1 and GL2 show a lens. In addition, VFL shows the contour of the face reflected inside the lens. FIG. 3A shows a state in which the person's face faces the front of the imaging unit 210 (face tilt angle is 0 degrees), and FIG. 3B shows a state in which the face does not face the front of the imaging unit 210 (face tilt). The angle is not 0 degrees). FIG. 3B shows, as an example, a state in which a person faces to the right. The state in which the face faces the front is a state in which the tilt angle of the face is close to 0 degrees with respect to the imaging direction and is smaller than a predetermined threshold value. The state in which the face is not facing the front is a state in which the inclination angle of the face with respect to the imaging direction is larger than a predetermined threshold value. A predetermined threshold value is arbitrarily set. As shown in FIG. 3A, when the face orientation is in front of the imaging unit 210 (the tilt angle of the face is 0 degrees), the contour of the face is difficult to be reflected inside the lenses GL1 and GL2. On the other hand, as shown in FIG. 3B, when the face orientation is not in front of the imaging unit 210 (the tilt angle of the face is not 0 degrees), the lens is affected by the refraction phenomenon as compared with the case where the face orientation is in front. The outline of the face is easily reflected inside the GL1.
　検出部１２０は、撮像画像に映る顔について、撮像方向に対する顔の傾斜角度を推定する。顔の傾斜角度とは、三次元空間において撮像部２１０と人物の頭部とを通る直線を基準に、当該人物の顔が向いている角度である。具体的には、人物の顔が撮像部２１０を向いている場合、傾斜角度は０度に近く所定の閾値よりも小さい。検出部１２０が顔を検出する対象の画像は、後述する抽出部１３０が抽出した二次元画像であってもよい。 The detection unit 120 estimates the tilt angle of the face with respect to the imaging direction for the face reflected in the captured image. The tilt angle of the face is an angle at which the face of the person is facing with reference to a straight line passing through the imaging unit 210 and the head of the person in the three-dimensional space. Specifically, when the face of the person faces the imaging unit 210, the inclination angle is close to 0 degrees and smaller than a predetermined threshold value. The image of the target for which the detection unit 120 detects the face may be a two-dimensional image extracted by the extraction unit 130, which will be described later.
　抽出部１３０は、レンズの内側に顔の輪郭が映る撮像画像を二次元画像として抽出する。抽出部１３０は、図３Ｂに例示するように、レンズの内側に顔の輪郭が映っている画像を抽出する。換言すると、抽出部１３０は、撮像部２１０から見てレンズの内側に顔の輪郭が映る程度に人物の顔が斜めを向いている画像を抽出する。抽出部１３０は、顔の傾斜角度が任意に設定した第１の閾値以上である画像や、任意に設定した第２の閾値以下である画像を抽出してもよい。ここで第２の閾値は第１の閾値より大きい値を設定してもよい。 The extraction unit 130 extracts an captured image in which the outline of the face is reflected inside the lens as a two-dimensional image. As illustrated in FIG. 3B, the extraction unit 130 extracts an image in which the contour of the face is reflected inside the lens. In other words, the extraction unit 130 extracts an image in which the face of the person is oriented obliquely so that the outline of the face is reflected inside the lens when viewed from the image pickup unit 210. The extraction unit 130 may extract an image in which the inclination angle of the face is equal to or more than the first threshold value arbitrarily set, or an image in which the inclination angle of the face is equal to or less than the second threshold value arbitrarily set. Here, the second threshold value may be set to a value larger than the first threshold value.
　抽出部１３０が実行する「抽出」とは、複数ある撮像画像の中から、一部の撮像画像を抜き出す処理を指す。具体的には、所定のフレームレートに応じて時系列的に撮像された複数の撮像画像において、眼鏡の装着が検出された撮像画像のみを抽出するものであってもよい。なお、「抽出」とは、撮像画像内において、一部の領域を切り出す処理を含む概念であってもよい。例えば、複数の人物を含んだ撮像画像内において、眼鏡を装着した頭部が映る一領域を切り出す他、人物が映る一領域を切り出す処理であってもよい。「抽出」が指す処理は上記の具体例に限定されない。 The "extraction" executed by the extraction unit 130 refers to a process of extracting a part of the captured images from a plurality of captured images. Specifically, from a plurality of captured images captured in time series according to a predetermined frame rate, only the captured images in which the wearing of eyeglasses is detected may be extracted. The "extraction" may be a concept including a process of cutting out a part of the captured image. For example, in the captured image including a plurality of people, one area in which the head wearing glasses is reflected may be cut out, or one area in which the person is reflected may be cut out. The process pointed to by "extraction" is not limited to the above specific example.
　度数推定部１４０は、抽出部１３０が抽出した二次元画像において、人物が装着する眼鏡の度数を推定する。図４を用いて度数推定部１４０の詳細を説明する。図４は、度数推定部１４０の機能ブロック図である。度数推定部１４０は、計算部１４１、学習モデル記憶部１４２、推定部１４３を有する。 The power estimation unit 140 estimates the power of the glasses worn by a person in the two-dimensional image extracted by the extraction unit 130. The details of the frequency estimation unit 140 will be described with reference to FIG. FIG. 4 is a functional block diagram of the frequency estimation unit 140. The frequency estimation unit 140 includes a calculation unit 141, a learning model storage unit 142, and an estimation unit 143.
　計算部１４１は、撮像部２１０が撮像した人物の顔と撮像部２１０との三次元空間における距離を算出する。距離の算出には、例えば撮像画像上における顔を示す領域の大きさや、人物を示す領域の長さを用いてもよいが、距離の算出手法はこれに限定されず、当業者は適宜周知技術を適用することができる。 The calculation unit 141 calculates the distance between the face of the person imaged by the imaging unit 210 and the imaging unit 210 in the three-dimensional space. For the calculation of the distance, for example, the size of the area showing the face on the captured image or the length of the area showing the person may be used, but the method for calculating the distance is not limited to this, and those skilled in the art will appropriately know the technique. Can be applied.
　計算部１４１は、レンズの内側と外側のそれぞれに映る顔の輪郭の位置の差（本明細書において「輪郭差」と称する）を計算する。図５を用いて輪郭差の概念を詳細に説明する。図５は、レンズを装着した顔の一部を模式的に示した図である。計算部１４１は、レンズの内側の領域における、目の中心点の水平方向位置と顔の輪郭の位置Ｅ４との水平方向のピクセル間距離（第１走査距離Ｅ１）を計算する。計算部１４１は、レンズの外側の領域において、目の中心点の水平方向位置と顔の輪郭の位置Ｅ３との水平方向のピクセル間距離（第２走査距離Ｅ２）を計算する。 The calculation unit 141 calculates the difference in the position of the contour of the face reflected on the inside and the outside of the lens (referred to as "contour difference" in the present specification). The concept of contour difference will be described in detail with reference to FIG. FIG. 5 is a diagram schematically showing a part of a face to which a lens is attached. The calculation unit 141 calculates the horizontal inter-pixel distance (first scanning distance E1) between the horizontal position of the center point of the eye and the position E4 of the contour of the face in the region inside the lens. The calculation unit 141 calculates the horizontal inter-pixel distance (second scanning distance E2) between the horizontal position of the center point of the eye and the position E3 of the contour of the face in the region outside the lens.
　計算部１４１は、第１走査距離Ｅ１と第２走査距離Ｅ２の差分を、撮像部２１０と顔との距離を用いて正規化した輪郭差（正規化輪郭差）を計算する。正規化の具体的な手法は限定されず、例えば、第１走査距離Ｅ１と第２走査距離Ｅ２との差分は、撮像部２１０と顔との距離で除算することで正規化してもよい。輪郭差の正規化にあたり、撮像部２１０と顔との距離が考慮される計算式であればよい。 The calculation unit 141 calculates the contour difference (normalized contour difference) obtained by normalizing the difference between the first scanning distance E1 and the second scanning distance E2 using the distance between the imaging unit 210 and the face. The specific method of normalization is not limited, and for example, the difference between the first scanning distance E1 and the second scanning distance E2 may be normalized by dividing by the distance between the imaging unit 210 and the face. In normalizing the contour difference, any calculation formula may be used as long as the distance between the imaging unit 210 and the face is taken into consideration.
　計算部１４１は、撮像画像上の任意のピクセルに映る物体から撮像部２１０へ入射する光の入射角を計算する。図６及び図１７を用いて撮像画像上の任意のピクセル位置に対応する実物体から撮像部２１０への入射角Ψの計算方法の一例を詳細に説明する。図６は人物を撮像する撮像部２１０の光学系を模式的に示す図である。ＣＣＤ（Ｃｈａｒｇｅ－Ｃｏｕｐｌｅｄ　Ｄｅｖｉｃｅ）センサＣＳは、カメラレンズＰを介し、人物から入射した光を検出する。図１７の計算式（１）は入射角Ψの導出過程を示す式である。計算式（１）を構成する変数について説明する。入射角Ψは、撮像部２１０の撮像範囲全体を示す撮像画像における中心位置からの第１ピクセル距離ｘｓを変数として、撮像部２１０の撮像範囲全体を示す撮像画像の中心位置から当該撮像画像に映る顔の中心位置までの第２ピクセル距離ｃと、当該撮像画像の画素数ＸＬと、検出部１２０が検出した顔向きΘと、撮像部２１０の画角Φを用いた図１７に示す計算式（１）で表現できる。計算部１４１は、レンズの内側の領域に映る顔の輪郭の位置において、入射角を計算する。計算には例えば図１７に示す計算式（１）を用いることができる。 The calculation unit 141 calculates the incident angle of the light incident on the imaging unit 210 from an object reflected in an arbitrary pixel on the captured image. An example of a method of calculating the incident angle Ψ from a real object corresponding to an arbitrary pixel position on the captured image to the imaging unit 210 will be described in detail with reference to FIGS. 6 and 17. FIG. 6 is a diagram schematically showing an optical system of an imaging unit 210 that images a person. The CCD (Charge-Coupled Device) sensor CS detects light incident from a person via the camera lens P. The calculation formula (1) in FIG. 17 is a formula showing the derivation process of the incident angle Ψ. The variables constituting the calculation formula (1) will be described. The incident angle Ψ is reflected in the captured image from the center position of the captured image showing the entire imaging range of the imaging unit 210, with the first pixel distance xs from the center position in the captured image indicating the entire imaging range of the imaging unit 210 as a variable. The calculation formula shown in FIG. 17 using the second pixel distance c to the center position of the face, the number of pixels XL of the captured image, the face orientation Θ detected by the detection unit 120, and the angle of view Φ of the imaging unit 210 ( It can be expressed by 1). The calculation unit 141 calculates the incident angle at the position of the contour of the face reflected in the region inside the lens. For the calculation, for example, the calculation formula (1) shown in FIG. 17 can be used.
　学習モデル記憶部１４２は、入射角と正規化輪郭差を入力するとレンズの度数を出力する回帰モデルを記憶する。回帰モデルは、予め入射角及び正規化輪郭差と、レンズの度数との組み合わせを学習した学習済みモデルである。 The learning model storage unit 142 stores a regression model that outputs the power of the lens when the incident angle and the normalized contour difference are input. The regression model is a trained model in which the combination of the incident angle and the normalized contour difference and the dioptric power of the lens is learned in advance.
　推定部１４３は、学習モデル記憶部１４２が記憶する回帰モデルに、計算部１４１が計算した入射角及び輪郭差を入力し、レンズの度数を推定する。 The estimation unit 143 inputs the incident angle and contour difference calculated by the calculation unit 141 into the regression model stored in the learning model storage unit 142, and estimates the power of the lens.
　上述した回帰モデルは、入射角と、正規化する前の輪郭差と、撮像部２１０と顔との距離とを入力し、レンズの度数を出力するように学習したものであってもよい。この場合、輪郭差を正規化する必要が無いため処理負荷を軽減することができる。 The regression model described above may be learned to input the incident angle, the contour difference before normalization, and the distance between the imaging unit 210 and the face, and output the power of the lens. In this case, since it is not necessary to normalize the contour difference, the processing load can be reduced.
　上述した度数推定部１４０の機能の一例は、撮像画像におけるレンズの内側に映る輪郭と、レンズの外側に映る輪郭との位置関係と、レンズを装着する人物を撮影した撮影部２１０と人物との位置関係を用いてレンズの度数を推定するものである。他にも、度数推定部１４０はレンズの度数を推定するために、例えば、特許文献２（特開２０１５－２５８５９号公報）に記載されているように、眼鏡のレンズの内側に映る顔の輪郭の位置と、レンズの外側に見える顔の輪郭の位置及び顔の傾斜角度とに基づき、レンズの度数を推定する手法を用いてもよい。レンズの度数を推定する手法は上記に限定されない。 An example of the function of the power estimation unit 140 described above is the positional relationship between the contour reflected inside the lens and the contour reflected outside the lens in the captured image, and the photographing unit 210 and the person who photographed the person wearing the lens. The power of the lens is estimated using the positional relationship. In addition, in order to estimate the power of the lens, the power estimation unit 140, for example, as described in Patent Document 2 (Japanese Unexamined Patent Publication No. 2015-25859), outlines the face reflected inside the lens of the spectacles. A method of estimating the power of the lens based on the position of the lens, the position of the contour of the face visible on the outside of the lens, and the inclination angle of the face may be used. The method for estimating the power of the lens is not limited to the above.
　画像処理部１５０は、度数推定部１４０が推定したレンズの度数情報を用いて、二次元画像に対し画像の補正を行い、当該二次元画像に対応する補正後の画像（補正後画像）を生成する。補正の対象となる領域は画像内の当該レンズの内側及び周辺の領域であり、度数を推定した対象のレンズ内の領域に映る顔の輪郭、人物の目の位置及び大きさを補正する。具体的には、レンズの内側に映る輪郭と、レンズの外側に映る輪郭とが同一曲線上に位置するように、レンズの内側に映る領域を拡大表示するように補正する。 The image processing unit 150 corrects the image of the two-dimensional image using the power information of the lens estimated by the power estimation unit 140, and generates a corrected image (corrected image) corresponding to the two-dimensional image. do. The area to be corrected is the area inside and around the lens in the image, and the contour of the face and the position and size of the eyes of the person reflected in the area in the lens of which the power is estimated are corrected. Specifically, the area reflected inside the lens is corrected so as to be enlarged so that the contour reflected inside the lens and the contour reflected outside the lens are located on the same curve.
　記憶部１６０は、複数の登録者の三次元画像（三次元登録画像）を登録者情報と関連付けて記憶する。登録者情報の項目の例として、登録者の識別情報、氏名、登録日時等が挙げられる。 The storage unit 160 stores three-dimensional images (three-dimensional registered images) of a plurality of registrants in association with registrant information. Examples of items of registrant information include registrant identification information, name, registration date and time, and the like.
　照合部１７０は、抽出部１３０が抽出した二次元画像に映る人物と、記憶部１６０が格納する複数の三次元登録画像との照合を行い、撮像部２１０が撮像した人物と複数の三次元登録画像との類似度を算出する。ここで、照合部１７０は、類似度が閾値以上の三次元登録画像をユーザの目視による照合の対象として設定する。目視照合の対象として設定する三次元登録画像は複数であってもよい。 The collation unit 170 collates a person reflected in the two-dimensional image extracted by the extraction unit 130 with a plurality of three-dimensional registered images stored in the storage unit 160, and collates the person imaged by the imaging unit 210 with a plurality of three-dimensional registrations. Calculate the similarity with the image. Here, the collation unit 170 sets a three-dimensional registered image having a similarity equal to or higher than a threshold value as a collation target by the user's visual inspection. There may be a plurality of three-dimensional registered images set as targets for visual collation.
　出力部１８０は、目視による照合の対象として設定された三次元登録画像と、画像処理部１５０により処理された補正後の画像（補正後画像）とを出力する。 The output unit 180 outputs a three-dimensional registered image set as a target for visual collation and a corrected image (corrected image) processed by the image processing unit 150.
　ユーザ端末３００は表示部３１０と操作受付部３２０を備える。ユーザ端末３００は、ユーザに対して情報を提供し、あるいはユーザの操作を受け付ける端末である。具体的には、ユーザ端末３００は、ユーザに対して不図示のデータベースに登録された人物の三次元登録画像と、防犯カメラ等で撮像した人物とが同一人物であるか否かの判断を促す装置である。他にも、ユーザが防犯カメラ等で撮像した映像を確認する装置であってもよいが、これに限定されない。 The user terminal 300 includes a display unit 310 and an operation reception unit 320. The user terminal 300 is a terminal that provides information to the user or accepts the user's operation. Specifically, the user terminal 300 prompts the user to determine whether or not the three-dimensional registered image of the person registered in the database (not shown) and the person captured by the security camera or the like are the same person. It is a device. In addition, the device may be a device for confirming an image captured by a user with a security camera or the like, but the present invention is not limited to this.
　表示部３１０は、出力部１８０により出力された三次元登録画像と画像処理部１５０により処理された補正後画像を表示する。表示部３１０は、三次元登録画像と補正後画像を同時に表示してもよい。このとき表示部３１０は、複数の三次元登録画像を同時に表示してもよいし、類似度に応じて複数の三次元登録画像を順次表示してもよい。表示部３１０は、複数の補正後画像を同時に表示してもよい。また、表示部３１０は、例えば、三次元登録画像と補正後画像を同一画面上に表示してもよい。 The display unit 310 displays the three-dimensional registered image output by the output unit 180 and the corrected image processed by the image processing unit 150. The display unit 310 may display the three-dimensional registered image and the corrected image at the same time. At this time, the display unit 310 may display a plurality of three-dimensional registered images at the same time, or may sequentially display a plurality of three-dimensional registered images according to the degree of similarity. The display unit 310 may display a plurality of corrected images at the same time. Further, the display unit 310 may display, for example, the three-dimensional registered image and the corrected image on the same screen.
　ユーザは、表示部３１０が表示した少なくとも一つの三次元登録画像と、補正後画像とが同一人物を示すか否かを判断し、操作受付部３２０に判断結果を入力する。操作受付部３２０は、ユーザの操作により、ユーザが判断した判断結果の入力を受け付ける。操作受付部３２０が受け付ける操作とは、例えば、表示部３１０に表示された三次元登録画像の中で、照合対象である補正後画像と同一人物である画像を選択する操作である。 The user determines whether or not at least one three-dimensional registered image displayed by the display unit 310 and the corrected image indicate the same person, and inputs the determination result to the operation reception unit 320. The operation reception unit 320 accepts the input of the judgment result judged by the user by the operation of the user. The operation accepted by the operation receiving unit 320 is, for example, an operation of selecting an image of the same person as the corrected image to be collated from the three-dimensional registered images displayed on the display unit 310.
　次に、本実施形態の照合補助システムの動作について、図７～図９を用いて説明する。図７～図９は本実施形態における処理の一例を示すフローチャートである。 Next, the operation of the collation assisting system of the present embodiment will be described with reference to FIGS. 7 to 9. 7 to 9 are flowcharts showing an example of the processing in the present embodiment.
　図７は画像処理に係るフローの一例を示す。まず、入力部１１０は、撮像装置２００が出力した撮像画像の入力を受け付ける（Ｓ１０１）。入力部１１０は、例えば、所定のフレームレートに応じて時系列的に撮像された画像の入力を順次受け付けるものであってもよい。 FIG. 7 shows an example of a flow related to image processing. First, the input unit 110 receives the input of the captured image output by the imaging device 200 (S101). The input unit 110 may sequentially accept inputs of images captured in time series according to a predetermined frame rate, for example.
　検出部１２０は、入力部１１０が受け付けた撮像画像の中から、人間の頭部及び眼鏡を検出する（Ｓ１０２）。検出部１２０は、人間の頭部の検出と、眼鏡の装着とを、段階的に検出するものであってもよいし、それぞれ独立して検出するものであってもよい。検出部１２０が撮像画像の中から眼鏡を検出しない場合は（Ｓ１０３、ＮＯ）、ステップＳ１０１の処理に戻り、次の撮像画像の入力を受け付ける（Ｓ１０１）。検出部１２０が撮像画像の中から眼鏡を検出した場合（Ｓ１０３、ＹＥＳ）、検出部１２０は、眼鏡の装着を検出した撮像画像の中から、眼鏡が有するレンズの内側に顔の輪郭が映っているか否かを判定する（Ｓ１０４）。　 The detection unit 120 detects the human head and eyeglasses from the captured image received by the input unit 110 (S102). The detection unit 120 may detect the detection of the human head and the wearing of the spectacles in a stepwise manner, or may independently detect each of them. If the detection unit 120 does not detect the spectacles from the captured image (S103, NO), the process returns to step S101 and the input of the next captured image is accepted (S101). When the detection unit 120 detects the spectacles from the captured image (S103, YES), the detection unit 120 shows the contour of the face inside the lens of the spectacles from the captured image detected that the spectacles are worn. Whether or not it is determined (S104). Twice
　レンズの内側に輪郭が映ると判定した場合（Ｓ１０５、ＹＥＳ）、抽出部１３０は、撮像画像の中から、レンズの内側に輪郭が映る撮像画像を抽出する（Ｓ１０６）。レンズの内側に顔の輪郭が映っていないと判定した場合（Ｓ１０５、ＮＯ）、ステップＳ１０１の処理に戻る。 When it is determined that the contour is reflected on the inside of the lens (S105, YES), the extraction unit 130 extracts the captured image in which the contour is reflected on the inside of the lens from the captured image (S106). When it is determined that the contour of the face is not reflected inside the lens (S105, NO), the process returns to the process of step S101.
　度数推定部１４０は、抽出部１３０が抽出した画像において、人物が装着するレンズの度数を推定する（Ｓ１０７）。 The power estimation unit 140 estimates the power of the lens worn by a person in the image extracted by the extraction unit 130 (S107).
　画像処理部１５０は、度数推定部１４０が推定したレンズの度数情報を用いて、抽出部１３０が抽出した画像において画像の補正を行い、補正後の画像（補正後画像）を生成する（Ｓ１０８）。 The image processing unit 150 corrects the image extracted by the extraction unit 130 using the power information of the lens estimated by the power estimation unit 140, and generates a corrected image (corrected image) (S108). ..
　図８は、処理装置１００による照合に係る処理フローの一例を示す。照合部１７０は、抽出部１３０が抽出した二次元画像に映る人物と、記憶部１６０が格納する複数の三次元登録画像との照合を行い、撮像部２１０が撮像した人物と複数の登録者との類似度を算出する（Ｓ１０９）。ここで、類似度が閾値以上の三次元登録画像がある場合には（Ｓ１１０、ＹＥＳ）、当該三次元登録画像をユーザの目視による照合の対象として設定する（Ｓ１１１）。類似度が閾値以上の三次元登録画像が無い場合（Ｓ１１０、ＮＯ）、処理を終了する。 FIG. 8 shows an example of a processing flow related to collation by the processing device 100. The collation unit 170 collates the person reflected in the two-dimensional image extracted by the extraction unit 130 with the plurality of three-dimensional registered images stored in the storage unit 160, and the person imaged by the imaging unit 210 and the plurality of registrants. The similarity of is calculated (S109). Here, when there is a three-dimensional registered image whose similarity is equal to or higher than the threshold value (S110, YES), the three-dimensional registered image is set as a target for collation by the user's visual inspection (S111). When there is no three-dimensional registered image whose similarity is equal to or higher than the threshold value (S110, NO), the process ends.
　なお、本実施形態において、図８で示す装置による照合に係る処理フロー（Ｓ１０９～Ｓ１１１）の後に、図７で示す画像処理に係る一連のフロー（Ｓ１０１～Ｓ１０８）が実行されてもよい。その場合、レンズを認識した画像のみに対してレンズの度数推定を行うため、装置の処理負荷を軽減することができる。 In the present embodiment, after the processing flow (S109 to S111) related to the collation by the apparatus shown in FIG. 8, a series of flows (S101 to S108) related to the image processing shown in FIG. 7 may be executed. In that case, since the power of the lens is estimated only for the image in which the lens is recognized, the processing load of the device can be reduced.
　照合に係るフロー（Ｓ１０９～Ｓ１１１）と、画像処理に係るフロー（Ｓ１０１～Ｓ１０８）とが並列で処理されてもよい。照合に係るフロー（Ｓ１０９～Ｓ１１１）の前に、画像処理に係るフロー（Ｓ１０１～Ｓ１０８）が実行されてもよい。図７で示す画像処理に係る一連のフロー（Ｓ１０１～Ｓ１０８）と、図８で示す装置による照合に係る処理フロー（Ｓ１０９～Ｓ１１１）の前後関係は限定されない。 The flow related to collation (S109 to S111) and the flow related to image processing (S101 to S108) may be processed in parallel. The flow related to image processing (S101 to S108) may be executed before the flow related to collation (S109 to S111). The context of the series of flows related to image processing (S101 to S108) shown in FIG. 7 and the processing flows related to collation by the apparatus shown in FIG. 8 (S109 to S111) is not limited.
　図９は、画像の表示に係る処理フローの一例を示す。出力部１８０は、目視による照合の対象として設定された三次元登録画像と、画像処理部１５０により処理された補正後の画像（補正後画像）とを出力する（Ｓ１１２）。 FIG. 9 shows an example of a processing flow related to image display. The output unit 180 outputs a three-dimensional registered image set as a target for visual collation and a corrected image (corrected image) processed by the image processing unit 150 (S112).
　表示部３１０は、出力部１８０により出力された三次元登録画像と画像処理部１５０により処理された補正後画像を表示する（Ｓ１１３）。 The display unit 310 displays the three-dimensional registered image output by the output unit 180 and the corrected image processed by the image processing unit 150 (S113).
　操作受付部３２０は、ユーザの操作により、ユーザが判断した判断結果の入力を受け付ける（Ｓ１１４）。操作受付部３２０が受け付ける操作とは、例えば、表示部３１０に表示された三次元登録画像の中で、照合対象である補正後画像と同一人物である画像を選択する操作である。 The operation reception unit 320 accepts the input of the judgment result judged by the user by the operation of the user (S114). The operation accepted by the operation receiving unit 320 is, for example, an operation of selecting an image of the same person as the corrected image to be collated from the three-dimensional registered images displayed on the display unit 310.
　なお、図９で示す画像の表示に係る処理フロー（Ｓ１１２～Ｓ１１３）は、図７で示す画像処理に係る一連のフロー（Ｓ１０１～Ｓ１０８）と、図８で示す装置による照合に係る処理フロー（Ｓ１０９～Ｓ１１１）の後に実行される。 The processing flow (S112 to S113) related to the display of the image shown in FIG. 9 is a series of flows (S101 to S108) related to the image processing shown in FIG. 7 and a processing flow related to collation by the apparatus shown in FIG. 8 (S101 to S108). It is executed after S109 to S111).
　これにより、照合対象がレンズを装着した人物であっても、レンズの装着による外見の変化や、顔の印象の変化を軽減することができ、ユーザは目視での照合を効率的かつ精度良く行うことができる。 As a result, even if the collation target is a person wearing a lens, it is possible to reduce changes in appearance and facial impression due to wearing the lens, and the user can perform visual collation efficiently and accurately. be able to.
＜第２の実施形態＞
　以下、本実施形態の照合補助システム１について説明する。第２の実施形態では、処理装置１００は、三次元画像処理部１９０を備え、推定したレンズの度数情報を基に三次元登録画像に対して補正を行う点で第１の実施形態とは相違する。なお、第１の実施形態と共通する箇所の説明は省略する。 <Second embodiment>
Hereinafter, the collation assistance system 1 of the present embodiment will be described. The second embodiment is different from the first embodiment in that the processing device 100 includes a three-dimensional image processing unit 190 and corrects the three-dimensional registered image based on the estimated power information of the lens. do. The description of the parts common to the first embodiment will be omitted.
　図１０は、本実施形態における照合補助システム１の構成を示す機能ブロック図である。処理装置１００は、入力部１１０と、検出部１２０と、抽出部１３０と、度数推定部１４０と、画像処理部１５０と、記憶部１６０と、照合部１７０と、出力部１８０と、三次元画像処理部１９０とを備える。 FIG. 10 is a functional block diagram showing the configuration of the collation assistance system 1 in the present embodiment. The processing device 100 includes an input unit 110, a detection unit 120, an extraction unit 130, a frequency estimation unit 140, an image processing unit 150, a storage unit 160, a collation unit 170, an output unit 180, and a three-dimensional image. It is provided with a processing unit 190.
　三次元画像処理部１９０は、照合部１７０が目視照合の対象として設定した三次元登録画像において、人物の目の周辺の画像を補正し、三次元登録画像に対応する補正後の三次元画像（三次元補正後画像）を生成する。具体的には、度数推定部１４０が推定したレンズの度数情報に応じて、所定の三次元登録画像に対して補正を行う。所定の三次元登録画像とは、照合部１７０が目視照合の対象として設定した三次元登録画像である。この補正には、公知の三次元コンピュータグラフィックスが用いられる。レンズの度数情報に応じた補正の一例を説明する。例えば、近視矯正用の眼鏡には凹レンズが用いられるため、眼鏡を装着した人物の目は、レンズの屈折の影響により装着前と比較して小さく見える。この場合、三次元画像処理部１９０は、推定した凹レンズの度数情報を用いて、三次元登録画像が示す人物が凹レンズを装着した場合に、レンズにより屈折して見える顔立ちの変化を再現するように、目の周辺の画像を補正する。 The three-dimensional image processing unit 190 corrects the image around the eyes of a person in the three-dimensional registered image set by the collating unit 170 as the target of visual collation, and the corrected three-dimensional image corresponding to the three-dimensional registered image ( Generate an image after 3D correction). Specifically, the predetermined three-dimensional registered image is corrected according to the power information of the lens estimated by the power estimation unit 140. The predetermined three-dimensional registered image is a three-dimensional registered image set by the collation unit 170 as a target for visual collation. Known three-dimensional computer graphics are used for this correction. An example of correction according to the power information of the lens will be described. For example, since a concave lens is used for myopia correction spectacles, the eyes of a person wearing the spectacles look smaller than before the spectacles due to the influence of the refraction of the lens. In this case, the three-dimensional image processing unit 190 uses the estimated power information of the concave lens to reproduce the change in facial features that appears to be refracted by the lens when the person indicated by the three-dimensional registered image wears the concave lens. , Correct the image around the eyes.
　記憶部１６０は、複数の三次元登録画像を登録者情報と関連付けて記憶する。登録者情報の項目の例として、登録者の識別情報、氏名、登録日時等が挙げられる。なお、記憶部１６０は、三次元画像処理部１９０が生成した、三次元補正後画像を記憶してもよい。 The storage unit 160 stores a plurality of three-dimensional registered images in association with the registrant information. Examples of items of registrant information include registrant identification information, name, registration date and time, and the like. The storage unit 160 may store the three-dimensional corrected image generated by the three-dimensional image processing unit 190.
　照合部１７０は、抽出部１３０が抽出した二次元画像に映る人物と、記憶部１６０が格納する複数の登録者を示す三次元登録画像との照合を行い、撮像部２１０が撮像した人物と複数の登録者との類似度を算出する。照合部１７０は、得られた類似度が閾値以上の三次元登録画像をユーザの目視による照合の対象として設定する。目視照合の対象として設定する三次元登録画像は複数であってもよい。 The collation unit 170 collates a person reflected in the two-dimensional image extracted by the extraction unit 130 with a three-dimensional registered image indicating a plurality of registrants stored in the storage unit 160, and collates a plurality of persons imaged by the imaging unit 210. Calculate the degree of similarity with the registrant of. The collation unit 170 sets the obtained three-dimensional registered image having a similarity equal to or higher than the threshold value as a collation target by the user's visual inspection. There may be a plurality of three-dimensional registered images set as targets for visual collation.
　出力部１８０は、目視による照合の対象として設定された人物を示す三次元補正後画像と、抽出部１３０が抽出した画像を出力する。三次元補正後画像とは、画像処理部１５０により生成された三次元画像である。 The output unit 180 outputs a three-dimensional corrected image showing a person set as a target for visual collation and an image extracted by the extraction unit 130. The three-dimensional corrected image is a three-dimensional image generated by the image processing unit 150.
　表示部３１０は、目視による照合の対象として設定された人物を示す三次元補正後画像と、抽出部１３０が抽出した二次元画像を表示する。表示部３１０は、三次元補正後画像と二次元画像を同時に表示してもよい。このとき表示部３１０は、複数の三次元画像を同時に表示してもよいし、得られた類似度に応じて複数の三次元画像を順次表示してもよい。表示部３１０は、複数の二次元画像を同時に表示してもよい。また、表示部３１０は、例えば、三次元登録画像と補正後画像を同一画面上に表示してもよい。 The display unit 310 displays a three-dimensional corrected image showing a person set as a target for visual collation and a two-dimensional image extracted by the extraction unit 130. The display unit 310 may display the three-dimensional corrected image and the two-dimensional image at the same time. At this time, the display unit 310 may display a plurality of three-dimensional images at the same time, or may sequentially display a plurality of three-dimensional images according to the obtained similarity. The display unit 310 may display a plurality of two-dimensional images at the same time. Further, the display unit 310 may display, for example, the three-dimensional registered image and the corrected image on the same screen.
　ユーザは、表示部３１０が表示した少なくとも一つの三次元画像と、二次元画像とが同一人物を示すか否かを判断し、操作受付部３２０に判断結果を入力する。操作受付部３２０は、ユーザの操作により、ユーザが判断した判断結果の入力を受け付ける。操作受付部３２０が受け付ける操作とは、例えば、表示部３１０に表示された三次元画像の中で、照合対象である二次元画像と同一人物である画像を選択する操作である。 The user determines whether or not at least one three-dimensional image displayed by the display unit 310 and the two-dimensional image indicate the same person, and inputs the determination result to the operation reception unit 320. The operation reception unit 320 accepts the input of the judgment result judged by the user by the operation of the user. The operation received by the operation receiving unit 320 is, for example, an operation of selecting an image of the same person as the two-dimensional image to be collated from the three-dimensional images displayed on the display unit 310.
　次に、本実施形態の照合補助システムの動作について、図１１及び図１２を用いて説明する。図１１は本実施形態における画像処理に係る処理フローの一例を示す。なお、第１の実施形態と重複する処理の説明は省略する。本実施形態において、装置による照合に係る処理フローは第１の実施形態と同様である（Ｓ１０９～Ｓ１１１）。 Next, the operation of the collation assisting system of the present embodiment will be described with reference to FIGS. 11 and 12. FIG. 11 shows an example of a processing flow related to image processing in the present embodiment. The description of the process that overlaps with the first embodiment will be omitted. In the present embodiment, the processing flow related to the collation by the apparatus is the same as that in the first embodiment (S109 to S111).
　度数推定部１４０は、抽出部１３０が抽出した画像において、人物が装着する眼鏡の度数を推定する（Ｓ１０７）。次に、三次元画像処理部１９０は、照合部１７０が目視照合の対象として設定した三次元登録画像において、推定したレンズの度数情報を用いて人物の目の周辺の画像を補正し、三次元補正後画像を生成する（Ｓ２０８）。 The power estimation unit 140 estimates the power of the glasses worn by a person in the image extracted by the extraction unit 130 (S107). Next, the three-dimensional image processing unit 190 corrects the image around the eyes of a person using the estimated lens power information in the three-dimensional registered image set by the collation unit 170 as the target of visual collation, and three-dimensionally. A corrected image is generated (S208).
　記憶部１６０は、三次元補正後画像を登録者情報に関連付けて記憶する（Ｓ２０９）。 The storage unit 160 stores the three-dimensional corrected image in association with the registrant information (S209).
　出力部１８０は、目視による照合の対象として設定された人物を示す三次元補正後画像と、抽出部１３０が抽出した画像を出力する（Ｓ２１２）。 The output unit 180 outputs a three-dimensional corrected image showing a person set as a target for visual collation and an image extracted by the extraction unit 130 (S212).
　表示部３１０は、目視による照合の対象として設定された人物を示す三次元補正後画像と、抽出部１３０が抽出した二次元画像を表示する（Ｓ２１３）。表示部３１０は、三次元補正後画像と二次元画像を同時に表示してもよい。このとき表示部３１０は、複数の三次元画像を同時に表示してもよいし、類似度に応じて複数の三次元画像を順次表示してもよい。表示部３１０は、複数の二次元画像を同時に表示してもよい。 The display unit 310 displays a three-dimensional corrected image showing a person set as a target for visual collation and a two-dimensional image extracted by the extraction unit 130 (S213). The display unit 310 may display the three-dimensional corrected image and the two-dimensional image at the same time. At this time, the display unit 310 may display a plurality of three-dimensional images at the same time, or may sequentially display a plurality of three-dimensional images according to the degree of similarity. The display unit 310 may display a plurality of two-dimensional images at the same time.
　ユーザは、表示部３１０が表示した少なくとも一つの三次元画像と、二次元画像とが同一人物を示すか否かを判断し、操作受付部３２０に判断結果を入力する。操作受付部３２０は、ユーザの操作により、ユーザが判断した判断結果の入力を受け付ける（Ｓ１１４）。 The user determines whether or not at least one three-dimensional image displayed by the display unit 310 and the two-dimensional image indicate the same person, and inputs the determination result to the operation reception unit 320. The operation receiving unit 320 receives the input of the judgment result judged by the user by the operation of the user (S114).
　なお、本実施形態において、図８で示す装置による照合に係る処理フロー（Ｓ１０９～Ｓ１１１）の後に、図７及び図１１で示す画像処理に係る一連のフロー（Ｓ１０１～Ｓ１０７、Ｓ２０８～Ｓ２０９）が実行されてもよい。その場合、レンズを認識した画像のみに対してレンズの度数推定を行うため、装置の処理負荷を軽減することができる。 In the present embodiment, after the processing flow (S109 to S111) related to the collation by the apparatus shown in FIG. 8, a series of flows (S101 to S107, S208 to S209) related to the image processing shown in FIGS. 7 and 11 follow. It may be executed. In that case, since the power of the lens is estimated only for the image in which the lens is recognized, the processing load of the device can be reduced.
　照合に係るフロー（Ｓ１０９～Ｓ１１１）と、画像処理に係るフロー（Ｓ１０１～Ｓ１０７、Ｓ２０８～Ｓ２０９）とが並列で処理されてもよい。照合に係るフロー（Ｓ１０９～Ｓ１１１）の前に、画像処理に係るフロー（Ｓ１０１～Ｓ１０７、Ｓ２０８～Ｓ２０９）が実行されてもよい。図７及び図１１で示す画像処理に係る一連のフロー（Ｓ１０１～Ｓ１０７、Ｓ２０８～Ｓ２０９）と、図８で示す装置による照合に係る処理フロー（Ｓ１０９～Ｓ１１１）の前後関係は限定されない。 The flow related to collation (S109 to S111) and the flow related to image processing (S101 to S107, S208 to S209) may be processed in parallel. The flow related to image processing (S101 to S107, S208 to S209) may be executed before the flow related to collation (S109 to S111). The context of the series of flows related to image processing shown in FIGS. 7 and 11 (S101 to S107, S208 to S209) and the processing flow related to collation by the apparatus shown in FIG. 8 (S109 to S111) is not limited.
　以上のように、本実施形態の照合補助システムでは、撮像した人物が眼鏡を装着した状態である場合に、眼鏡の装着による外見の変化に応じて、データベースが持つ三次元登録画像を補正する。これにより、照合対象が眼鏡を装着した人物であっても、眼鏡の装着による外見の変化や、顔の印象の変化を軽減することができ、ユーザは目視での照合を効率的かつ精度良く行うことができる。 As described above, in the collation assistance system of the present embodiment, when the imaged person is wearing glasses, the three-dimensional registered image held by the database is corrected according to the change in appearance due to wearing the glasses. As a result, even if the collation target is a person wearing spectacles, it is possible to reduce changes in appearance and facial impression due to wearing spectacles, and the user can perform visual collation efficiently and accurately. be able to.
＜第３の実施形態＞
　本実施形態の照合補助システム１の構成について図１３を用いて説明する。図１３は、本実施形態における照合補助システム１の機能ブロック図である。照合補助システム１は、検出部１２０と、出力部１８０とを備える。 <Third embodiment>
The configuration of the collation assistance system 1 of the present embodiment will be described with reference to FIG. FIG. 13 is a functional block diagram of the collation assistance system 1 according to the present embodiment. The collation assistance system 1 includes a detection unit 120 and an output unit 180.
　検出部１２０は、人物を撮像した少なくとも一つの入力画像において、レンズを装着した人物を含む入力画像を検出する。 The detection unit 120 detects an input image including a person wearing a lens in at least one input image obtained by capturing a person.
　出力部１８０は、検出部１２０が検出した入力画像から推定したレンズの度数情報を用いて入力画像を補正した画像である補正後画像と、当該補正後画像との照合対象である三次元登録画像とを出力する。補正後画像とは、例えば、入力画像において当該レンズに映る目の位置及び大きさを補正した画像である。 The output unit 180 is a three-dimensional registered image that is a collation target between the corrected image, which is an image obtained by correcting the input image using the power information of the lens estimated from the input image detected by the detection unit 120, and the corrected image. And output. The corrected image is, for example, an image in which the position and size of the eyes reflected on the lens in the input image are corrected.
　次に、本実施形態の照合補助システムの動作について、図１４を用いて説明する。図１４は本実施形態における処理の一例を示すフローチャートである。 Next, the operation of the collation assistance system of the present embodiment will be described with reference to FIG. FIG. 14 is a flowchart showing an example of the processing in the present embodiment.
　検出部１２０は、撮像画像においてレンズを装着した人物を含む入力画像を検出する（Ｓ３０７）。 The detection unit 120 detects an input image including a person wearing a lens in the captured image (S307).
　出力部１８０は、検出部１２０が検出した入力画像から推定したレンズの度数情報を用いて入力画像を補正した画像である補正後画像と、当該補正後画像との照合対象である三次元登録画像とを出力する（Ｓ３０８）。 The output unit 180 is a three-dimensional registered image that is a collation target between the corrected image, which is an image obtained by correcting the input image using the power information of the lens estimated from the input image detected by the detection unit 120, and the corrected image. And is output (S308).
　これにより、照合対象が眼鏡を装着した人物であっても、眼鏡の装着による外見の変化や、顔の印象の変化を軽減することができ、ユーザは目視での照合を効率的かつ精度良く行うことができる。 As a result, even if the collation target is a person wearing spectacles, it is possible to reduce changes in appearance and facial impression due to wearing spectacles, and the user can perform visual collation efficiently and accurately. be able to.
＜第４の実施形態＞
　本実施形態の照合補助システム１の構成について、第３の実施形態と同様に図１３を用いて説明する。本実施形態と第３の実施形態は、出力部１８０の機能が異なる点で相違する。図１３は、本実施形態における照合補助システム１の機能ブロック図である。照合補助システム１は、検出部１２０と、出力部１８０とを備える。 <Fourth Embodiment>
The configuration of the collation assistance system 1 of the present embodiment will be described with reference to FIG. 13 as in the third embodiment. The present embodiment and the third embodiment are different in that the functions of the output unit 180 are different. FIG. 13 is a functional block diagram of the collation assistance system 1 according to the present embodiment. The collation assistance system 1 includes a detection unit 120 and an output unit 180.
　検出部１２０は、人物を撮像した少なくとも一以上の入力画像において、レンズを有する眼鏡を装着した人物を含む入力画像を検出する。 The detection unit 120 detects an input image including a person wearing eyeglasses having a lens in at least one input image obtained by capturing a person.
　出力部１８０は、入力画像から推定した前記レンズの度数を用いて、三次元登録画像を補正した三次元補正後画像と、当該入力画像と、を出力する。出力部１８０が出力する三次元補正後画像は、例えば、入力画像との照合対象である三次元登録画像の目の位置及び大きさを補正した画像である。 The output unit 180 outputs a three-dimensional corrected image obtained by correcting the three-dimensional registered image and the input image by using the power of the lens estimated from the input image. The three-dimensional corrected image output by the output unit 180 is, for example, an image in which the position and size of the eyes of the three-dimensional registered image to be collated with the input image are corrected.
　次に、本実施形態の照合補助システムの動作について、図１５を用いて説明する。図１５は本実施形態における処理の一例を示すフローチャートである。 Next, the operation of the collation assistance system of the present embodiment will be described with reference to FIG. FIG. 15 is a flowchart showing an example of the processing in the present embodiment.
　検出部１２０は、人物を撮像した少なくとも一以上の入力画像において、レンズを有する眼鏡を装着した人物を含む入力画像を検出する（Ｓ４０７）。 The detection unit 120 detects an input image including a person wearing eyeglasses having a lens in at least one input image obtained by capturing a person (S407).
　出力部１８０は、入力画像から推定した前記レンズの度数を用いて、三次元登録画像を補正した三次元補正後画像と、当該入力画像と、を出力する（Ｓ４０８）。 The output unit 180 outputs the three-dimensional corrected image corrected by the three-dimensional registered image and the input image by using the power of the lens estimated from the input image (S408).
　これにより、照合対象が眼鏡を装着した人物であっても、眼鏡の装着による外見の変化や、顔の印象の変化を軽減することができる。ユーザが目視での照合を効率的かつ精度良く行うための画像を生成することができる。 As a result, even if the collation target is a person wearing glasses, it is possible to reduce changes in appearance and changes in facial impression due to wearing glasses. It is possible to generate an image for the user to perform visual collation efficiently and accurately.
（ハードウェア構成）
　次に、上述した各実施形態における、処理装置１００、撮像装置２００、ユーザ端末３００を、一つ以上のコンピュータを用いて実現するハードウェア構成の一例について説明する。処理装置１００、撮像装置２００、ユーザ端末３００が備える各機能部は、任意のコンピュータの少なくとも1つのＣＰＵ（Central Processing Unit）、少なくとも１つのメモリ、メモリにロードされるプログラム、そのプログラムを格納する少なくとも１つのハードディスク等の記憶ユニット、ネットワーク接続用インターフェース等を中心にハードウェアとソフトウエアの任意の組合せによって実現される。この実現方法、装置には種々の変形例があることは、当業者には理解されるところである。なお記憶ユニットは、装置の出荷以前から格納されているプログラムのほか、光ディスク、光磁気ディスク、半導体フラッシュメモリ等の記憶媒体やインターネット上のサーバ等からダウンロードされたプログラムをも格納可能である。 (Hardware configuration)
Next, an example of a hardware configuration in which the processing device 100, the image pickup device 200, and the user terminal 300 in each of the above-described embodiments is realized by using one or more computers will be described. Each functional unit included in the processing device 100, the image pickup device 200, and the user terminal 300 includes at least one CPU (Central Processing Unit) of an arbitrary computer, at least one memory, a program loaded into the memory, and at least a program for storing the program. It is realized by any combination of hardware and software centering on a storage unit such as one hard disk, an interface for network connection, and the like. It will be understood by those skilled in the art that there are various modifications of this realization method and apparatus. The storage unit can store not only programs stored before the device is shipped, but also storage media such as optical disks, magneto-optical disks, and semiconductor flash memories, and programs downloaded from servers on the Internet.
　図１６は、処理装置１００、撮像装置２００、ユーザ端末３００のハードウェア構成を例示するブロック図である。図１６に示すように、処理装置１００、撮像装置２００、ユーザ端末３００は、プロセッサ１Ａ、メモリ２Ａ、入出力インターフェース３Ａ、周辺回路４Ａ、通信インターフェース５Ａ、バス６Ａを有する。周辺回路４Ａには、様々なモジュールが含まれる。処理装置１００、撮像装置２００、ユーザ端末３００は周辺回路４Ａを有さなくてもよい。なお、処理装置１００、撮像装置２００、ユーザ端末３００は物理的及び／又は論理的に分かれた複数の装置で構成されてもよい。この場合、複数の装置各々が上記のハードウェア構成を備えることができる。 FIG. 16 is a block diagram illustrating the hardware configurations of the processing device 100, the imaging device 200, and the user terminal 300. As shown in FIG. 16, the processing device 100, the image pickup device 200, and the user terminal 300 include a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, a communication interface 5A, and a bus 6A. The peripheral circuit 4A includes various modules. The processing device 100, the image pickup device 200, and the user terminal 300 do not have to have the peripheral circuit 4A. The processing device 100, the image pickup device 200, and the user terminal 300 may be composed of a plurality of physically and / or logically separated devices. In this case, each of the plurality of devices can be provided with the above hardware configuration.
　バス６Ａは、プロセッサ１Ａ、メモリ２Ａ、入出力インターフェース３Ａ、周辺回路４Ａ、通信インターフェース５Ａが相互にデータを送受信するためのデータ伝送路である。プロセッサ１Ａは、例えばＣＰＵ、ＧＰＵ（Graphics Processing Unit）やマイクロプロセッサ等の演算処理装置である。プロセッサ１Ａは、例えば、メモリ２Ａに記憶された各種プログラムに従って処理を実行することが可能である。 The bus 6A is a data transmission path for the processor 1A, the memory 2A, the input / output interface 3A, the peripheral circuit 4A, and the communication interface 5A to send and receive data to and from each other. The processor 1A is, for example, an arithmetic processing unit such as a CPU, a GPU (Graphics Processing Unit), or a microprocessor. The processor 1A can execute the process according to various programs stored in the memory 2A, for example.
　メモリ２Ａは、例えばＲＡＭ（Random Access Memory）やＲＯＭ（Read Only Memory）などのメモリであり、プログラムや各種データを記憶する。 The memory 2A is, for example, a memory such as a RAM (RandomAccessMemory) or a ROM (ReadOnlyMemory), and stores a program and various data.
　入出力インターフェース３Ａは、入力装置、外部装置、外部ストレージ部、外部センサ、カメラ等から情報を取得するためのインターフェースや、出力装置、外部装置、外部ストレージ部等に情報を出力するためのインターフェースなどを含む。入力装置は、例えばタッチパネル、キーボード、マウス、マイク、カメラ等である。出力装置は、例えばディスプレイ、スピーカ、プリンタ、ランプ等である。 The input / output interface 3A includes an interface for acquiring information from an input device, an external device, an external storage unit, an external sensor, a camera, etc., an interface for outputting information to an output device, an external device, an external storage unit, etc. including. The input device is, for example, a touch panel, a keyboard, a mouse, a microphone, a camera, or the like. The output device is, for example, a display, a speaker, a printer, a lamp, or the like.
　プロセッサ１Ａは、各モジュールに指令を出し、それらの演算結果をもとに演算を行うことができる。 Processor 1A can issue commands to each module and perform calculations based on the calculation results.
　通信インターフェース５Ａは処理装置１００、撮像装置２００、ユーザ端末３００が外部装置と相互に通信することを実現する他、処理装置１００、撮像装置２００、ユーザ端末３００が相互に通信することを実現する。なお、処理装置１００、撮像装置２００、ユーザ端末３００の一部の機能をコンピュータで構成してもよい。 The communication interface 5A realizes that the processing device 100, the image pickup device 200, and the user terminal 300 communicate with each other with the external device, and also realizes that the processing device 100, the image pickup device 200, and the user terminal 300 communicate with each other. A computer may configure some functions of the processing device 100, the imaging device 200, and the user terminal 300.
＜変形例＞
　上述の実施形態に対して適用可能な変形例を説明する。度数推定部１４０が対象の撮像画像においてレンズの度数の推定に失敗した場合、当該撮像画像の前後に撮影された撮像画像に対して再度処理を行ってもよい。また、画像処理部１５０は、撮像画像に対し眼鏡の縁を除去するように画像処理を行ってもよい。 <Modification example>
An example of modification applicable to the above-described embodiment will be described. If the power estimation unit 140 fails to estimate the power of the lens in the captured image of the target, the captured images taken before and after the captured image may be processed again. Further, the image processing unit 150 may perform image processing on the captured image so as to remove the edges of the spectacles.
　他の変形例を説明する。撮像部２１０は三次元的な深度情報（奥行き情報）を取得する装置を備えてもよい。その場合、計算部１４１は、二次元画像から撮像部２１０と人物（または人物の顔）との距離を計算する必要はなく、推定部１４３は、撮像部２１０が取得した深度情報を用いて、レンズの度数を推定してもよい。また、照合部１７０は、抽出部１３０が抽出した二次元画像に映る人物と、記憶部１６０が格納する複数の三次元登録画像との照合を行う他、画像処理部１５０が処理した補正後画像と三次元登録画像とを照合するように設計してもよい。 Other modified examples will be explained. The imaging unit 210 may include a device that acquires three-dimensional depth information (depth information). In that case, the calculation unit 141 does not need to calculate the distance between the image pickup unit 210 and the person (or the face of the person) from the two-dimensional image, and the estimation unit 143 uses the depth information acquired by the image pickup unit 210 to use the depth information. The power of the lens may be estimated. Further, the collation unit 170 collates the person reflected in the two-dimensional image extracted by the extraction unit 130 with the plurality of three-dimensional registered images stored in the storage unit 160, and the corrected image processed by the image processing unit 150. It may be designed to collate with the three-dimensional registered image.
　上記の実施形態に対して適用可能な他の変形例を説明する。撮像装置２００は、処理装置１００が実行する処理の一部あるいは全てを実行してもよい。撮像装置２００は、例えば、図１８に示すように、撮像部２１０に加えて検出部１２０と抽出部１３０と度数推定部１４０とを備えていてもよい。この場合、撮像装置２００において、レンズの度数を推定し、処理装置１００が備える画像処理部１５０は、レンズの度数に関する情報を撮像装置２００から取得してもよい。この場合、撮像装置２００がＳ１０１～Ｓ１０７の処理を実行してもよい。また、撮像装置２００は、撮像画像に映る人物の輪郭差が閾値以上である撮像画像のみを抽出し、レンズの度数を推定する処理や、処理装置１００に送信する処理の対象としてもよい。上記の変形例によれば、処理装置１００の処理負荷軽減や、本発明における照合補助システムの送信処理に係る負荷の軽減を図ることができる。 Other modifications applicable to the above embodiment will be described. The image pickup apparatus 200 may execute a part or all of the processes executed by the processing apparatus 100. For example, as shown in FIG. 18, the image pickup apparatus 200 may include a detection unit 120, an extraction unit 130, and a power estimation unit 140 in addition to the image pickup unit 210. In this case, the image processing device 200 may estimate the power of the lens, and the image processing unit 150 included in the processing device 100 may acquire information regarding the power of the lens from the image pickup device 200. In this case, the image pickup apparatus 200 may execute the processes of S101 to S107. Further, the imaging device 200 may be a target of a process of extracting only an captured image in which the contour difference of a person reflected in the captured image is equal to or greater than a threshold value and estimating the power of the lens, or a process of transmitting the image to the processing device 100. According to the above modification, it is possible to reduce the processing load of the processing device 100 and the load related to the transmission processing of the collation assisting system in the present invention.
　なお、前述の実施形態の構成は、組み合わせたり或いは一部の構成部分を入れ替えたりしてもよい。また、本発明の構成は前述の実施形態のみに限定されるものではなく、本発明の要旨を逸脱しない範囲内において種々変更を加えてもよい。上記の実施形態の一部又は全部は、以下の付記のようにも記載されうるが、以下には限られない。 Note that the configurations of the above-described embodiments may be combined or some components may be replaced. Further, the configuration of the present invention is not limited to the above-described embodiment, and various modifications may be made without departing from the gist of the present invention. Some or all of the above embodiments may also be described, but not limited to:
　（付記１）
　人物を撮像した少なくとも一つの入力画像において、レンズを装着した人物を含む入力画像を検出する検出部と、
　前記入力画像において推定した前記レンズの度数を用いて当該レンズに映る目の位置及び大きさを補正した画像である補正後画像と、前記補正後画像との照合対象である三次元登録画像を出力する出力部と、
を備える照合補助装置。 (Appendix 1)
A detection unit that detects an input image including a person wearing a lens in at least one input image of a person.
Outputs a three-dimensional registered image that is a collation target between the corrected image, which is an image in which the position and size of the eyes reflected on the lens are corrected using the power of the lens estimated in the input image, and the corrected image. Output section and
A collation assisting device comprising.
　（付記２）
　前記補正後画像及び前記三次元登録画像を同一画面上に表示する表示部、
をさらに備える付記１に記載の照合補助装置。 (Appendix 2)
A display unit that displays the corrected image and the three-dimensional registered image on the same screen.
The collation assisting device according to Appendix 1, further comprising.
　（付記３）
　前記表示部は、前記補正後画像に対応する前記入力画像と、複数の登録者の前記三次元登録画像との照合により得られた類似度が閾値以上の三次元登録画像を表示する、
付記２に記載の照合補助装置。 (Appendix 3)
The display unit displays a three-dimensional registered image having a similarity equal to or higher than a threshold value obtained by collating the input image corresponding to the corrected image with the three-dimensional registered image of a plurality of registrants.
The collation assisting device according to Appendix 2.
　（付記４）
　人物を撮像した少なくとも一以上の入力画像において、レンズを有する眼鏡を装着した人物を含む入力画像を検出する検出部と、
　前記入力画像において推定した前記レンズの度数を用いて前記入力画像との照合対象である三次元登録画像の目の位置及び大きさを補正した三次元補正後画像と、前記入力画像と、を出力する出力部と、
を備える照合補助装置。 (Appendix 4)
A detection unit that detects an input image including a person wearing eyeglasses having a lens in at least one input image of a person.
The three-dimensional corrected image obtained by correcting the eye position and size of the three-dimensional registered image to be collated with the input image using the power of the lens estimated in the input image, and the input image are output. Output section and
A collation assisting device comprising.
　（付記５）
　前記三次元補正後画像と、前記入力画像とを、同一画面上に表示する表示部と、を備える付記４に記載の照合補助装置。 (Appendix 5)
The collation assisting device according to Appendix 4, further comprising a display unit that displays the three-dimensional corrected image and the input image on the same screen.
　（付記６）
　前記表示部は、前記入力画像との照合により得られた類似度が閾値以上である三次元登録画像に対応する前記三次元補正後画像を表示する、
付記５に記載の照合補助装置。 (Appendix 6)
The display unit displays the three-dimensional corrected image corresponding to the three-dimensional registered image whose similarity obtained by collation with the input image is equal to or higher than the threshold value.
The collation assisting device according to Appendix 5.
　（付記７）
　前記入力画像における前記レンズの内側に映る輪郭と当該レンズの外側に映る輪郭との位置関係と、当該レンズを装着する人物を撮影した撮像部と当該人物との位置関係と、を用いて、当該レンズの度数を推定する推定部、をさらに備える付記１から６の何れか１項に記載の照合補助装置。 (Appendix 7)
The positional relationship between the contour reflected inside the lens and the contour reflected outside the lens in the input image, and the positional relationship between the imaging unit and the person who photographed the person wearing the lens are used. The collation assisting device according to any one of Appendix 1 to 6, further comprising an estimation unit for estimating the power of the lens.
　（付記８）
　前記推定部は、人物を撮像した複数の入力画像において、当該人物が装着する前記レンズの内側と外側に映る輪郭の距離が閾値以上である画像を用いて、当該レンズの度数を推定する、付記７に記載の照合補助装置。 (Appendix 8)
The estimation unit estimates the power of the lens by using an image in which the distance between the contours reflected on the inside and the outside of the lens worn by the person is equal to or more than a threshold value in a plurality of input images obtained by capturing the person. 7. The collation assisting device according to 7.
　（付記９）
　前記推定部は、複数の登録者の前記三次元登録画像の何れかとの照合により得られた類似度が閾値以上である前記入力画像を対象に、当該入力画像に映る前記レンズの度数を推定する、付記７または８に記載の照合補助装置。 (Appendix 9)
The estimation unit estimates the power of the lens reflected in the input image with respect to the input image whose similarity obtained by collation with any of the three-dimensional registered images of a plurality of registrants is equal to or more than a threshold value. , The collation assisting device according to Appendix 7 or 8.
　（付記１０）
　人物を撮像した少なくとも一つの入力画像において、レンズを装着した人物を含む入力画像を検出し、
　前記入力画像において推定した前記レンズの度数を用いて当該レンズに映る目の位置及び大きさを補正した画像である補正後画像と、前記補正後画像との照合対象である三次元登録画像を出力する、
照合補助方法。 (Appendix 10)
In at least one input image obtained by capturing a person, an input image including a person wearing a lens is detected, and the input image is detected.
Outputs a three-dimensional registered image that is a collation target between the corrected image, which is an image in which the position and size of the eyes reflected on the lens are corrected using the power of the lens estimated in the input image, and the corrected image. do,
Matching assistance method.
　（付記１１）
　コンピュータに、
　人物を撮像した少なくとも一つの入力画像において、レンズを装着した人物を含む入力画像を検出する処理、
　前記入力画像において推定した前記レンズの度数を用いて当該レンズに映る目の位置及び大きさを補正した画像である補正後画像と、前記補正後画像との照合対象である三次元登録画像を出力する処理、
を実行させるためのプログラムを記憶するプログラム記憶媒体。 (Appendix 11)
On the computer
A process of detecting an input image including a person wearing a lens in at least one input image obtained by capturing a person.
Outputs a three-dimensional registered image that is a collation target between the corrected image, which is an image in which the position and size of the eyes reflected on the lens are corrected using the power of the lens estimated in the input image, and the corrected image. Processing to do,
A program storage medium that stores a program for executing a program.
　（付記１２）
人物を撮像した少なくとも一以上の入力画像において、レンズを有する眼鏡を装着した人物を含む入力画像を検出し、
　前記入力画像において推定した前記レンズの度数を用いて前記入力画像との照合対象である三次元登録画像の目の位置及び大きさを補正した三次元補正後画像と、前記入力画像と、を出力する、
照合補助方法。 (Appendix 12)
In at least one input image of a person, an input image including a person wearing spectacles with a lens is detected.
The three-dimensional corrected image obtained by correcting the eye position and size of the three-dimensional registered image to be collated with the input image using the power of the lens estimated in the input image, and the input image are output. do,
Matching assistance method.
　（付記１３）
　コンピュータに、
　人物を撮像した少なくとも一以上の入力画像において、レンズを有する眼鏡を装着した人物を含む入力画像を検出する処理、
　前記入力画像において推定した前記レンズの度数を用いて前記入力画像との照合対象である三次元登録画像の目の位置及び大きさを補正した三次元補正後画像と、前記入力画像と、を出力する処理、
を実行させるためのプログラムを記憶するプログラム記憶媒体。 (Appendix 13)
On the computer
A process of detecting an input image including a person wearing spectacles having a lens in at least one input image obtained by capturing a person.
The three-dimensional corrected image obtained by correcting the eye position and size of the three-dimensional registered image to be collated with the input image using the power of the lens estimated in the input image, and the input image are output. Processing to do,
A program storage medium that stores a program for executing a program.
　１　　　　照合補助システム
　１００　　処理装置
　１１０　　入力部
　１２０　　検出部
　１３０　　抽出部
　１４０　　度数推定部
　１４１　　計算部
　１４２　　学習モデル記憶部
　１４３　　推定部
　１５０　　画像処理部
　１６０　　記憶部
　１７０　　照合部
　１８０　　出力部
　１９０　　三次元画像処理部
　２００　　撮像装置
　２１０　　撮像部
　３００　　ユーザ端末
　３１０　　表示部
　３２０　　操作受付部
　１Ａ　　　プロセッサ
　２Ａ　　　メモリ
　３Ａ　　　入出力インターフェース
　４Ａ　　　周辺回路
　５Ａ　　　通信インターフェース
　６Ａ　　　バス
　Ｅ１　　　第１走査距離
　Ｅ２　　　第２走査距離
　Ｅ３、Ｅ４、ＦＬ　　顔の輪郭
　ＶＦＬ　　レンズ内の顔の輪郭
　ＧＬ１、ＧＬ２　　レンズ
　Ψ　　　　入射角
　Θ　　　　人物の顔向き
　Φ　　　　画角
　ｘｓ　　　第１ピクセル距離
　ｃ　　　　第２ピクセル距離
　ＸＬ　　　撮像画像の画素数
　Ｐ　　　　カメラレンズ
　ＣＳ　　　ＣＣＤセンサ 1 Collation assistance system 100 Processing device 110 Input unit 120 Detection unit 130 Extraction unit 140 Frequency estimation unit 141 Calculation unit 142 Learning model storage unit 143 Estimating unit 150 Image processing unit 160 Storage unit 170 Matching unit 180 Output unit 190 Three-dimensional image processing unit 200 Imaging device 210 Imaging unit 300 User terminal 310 Display unit 320 Operation reception unit 1A Processor 2A Memory 3A Input / output interface 4A Peripheral circuit 5A Communication interface 6A Bus E1 First scanning distance E2 Second scanning distance E3, E4, FL Face contour Contour of face in VFL lens GL1, GL2 Lens Ψ Incident angle Θ Face direction of person Φ Angle of view xs 1st pixel distance c 2nd pixel distance XL Number of pixels of captured image P Camera lens CS CCD sensor

Claims (13)

  1. 　人物を撮像した少なくとも一つの入力画像において、レンズを装着した人物を含む入力画像を検出する検出部と、
    　前記入力画像において推定した前記レンズの度数を用いて当該レンズに映る目の位置及び大きさを補正した画像である補正後画像と、前記補正後画像との照合対象である三次元登録画像を出力する出力部と、
    を備える照合補助装置。     A detection unit that detects an input image including a person wearing a lens in at least one input image of a person.
    Outputs a three-dimensional registered image that is a collation target between the corrected image, which is an image in which the position and size of the eyes reflected on the lens are corrected using the power of the lens estimated in the input image, and the corrected image. Output section and
    A collation assisting device comprising.
  2. 　前記補正後画像及び前記三次元登録画像を同一画面上に表示する表示部、
    をさらに備える請求項１に記載の照合補助装置。     A display unit that displays the corrected image and the three-dimensional registered image on the same screen.
    The collation assisting device according to claim 1.
  3. 　前記表示部は、前記補正後画像に対応する前記入力画像と、複数の登録者の前記三次元登録画像との照合により得られた類似度が閾値以上の三次元登録画像を表示する、
    請求項２に記載の照合補助装置。     The display unit displays a three-dimensional registered image having a similarity equal to or higher than a threshold value obtained by collating the input image corresponding to the corrected image with the three-dimensional registered image of a plurality of registrants.
    The collation assisting device according to claim 2.
  4. 　人物を撮像した少なくとも一以上の入力画像において、レンズを有する眼鏡を装着した人物を含む入力画像を検出する検出部と、
    　前記入力画像において推定した前記レンズの度数を用いて前記入力画像との照合対象である三次元登録画像の目の位置及び大きさを補正した三次元補正後画像と、前記入力画像と、を出力する出力部と、
    を備える照合補助装置。     A detection unit that detects an input image including a person wearing eyeglasses having a lens in at least one input image of a person.
    The three-dimensional corrected image obtained by correcting the eye position and size of the three-dimensional registered image to be collated with the input image using the power of the lens estimated in the input image, and the input image are output. Output section and
    A collation assisting device comprising.
  5. 　前記三次元補正後画像と、前記入力画像とを、同一画面上に表示する表示部と、を備える請求項４に記載の照合補助装置。 The collation assisting device according to claim 4, further comprising a display unit that displays the three-dimensional corrected image and the input image on the same screen.
  6. 　前記表示部は、前記入力画像との照合により得られた類似度が閾値以上である三次元登録画像に対応する前記三次元補正後画像を表示する、
    請求項５に記載の照合補助装置。     The display unit displays the three-dimensional corrected image corresponding to the three-dimensional registered image whose similarity obtained by collation with the input image is equal to or higher than the threshold value.
    The collation assisting device according to claim 5.
  7. 　前記入力画像における前記レンズの内側に映る輪郭と当該レンズの外側に映る輪郭との位置関係と、当該レンズを装着する人物を撮影した撮像部と当該人物との位置関係と、を用いて、当該レンズの度数を推定する推定部、をさらに備える請求項１から６の何れか１項に記載の照合補助装置。 The positional relationship between the contour reflected inside the lens and the contour reflected outside the lens in the input image, and the positional relationship between the imaging unit and the person who photographed the person wearing the lens are used. The collation assisting device according to any one of claims 1 to 6, further comprising an estimation unit for estimating the power of the lens.
  8. 　前記推定部は、人物を撮像した複数の入力画像において、当該人物が装着する前記レンズの内側と外側に映る輪郭の距離が閾値以上である画像を用いて、当該レンズの度数を推定する、請求項７に記載の照合補助装置。 The estimation unit estimates the power of a plurality of input images of a person by using an image in which the distance between the contours reflected on the inside and outside of the lens worn by the person is equal to or greater than a threshold value. Item 7. The collation assisting device according to item 7.
  9. 　前記推定部は、複数の登録者の前記三次元登録画像の何れかとの照合により得られた類似度が閾値以上である前記入力画像を対象に、当該入力画像に映る前記レンズの度数を推定する、請求項７または８に記載の照合補助装置。 The estimation unit estimates the power of the lens reflected in the input image with respect to the input image whose similarity obtained by collation with any of the three-dimensional registered images of a plurality of registrants is equal to or more than a threshold value. , The collation assisting device according to claim 7 or 8.
  10. 　人物を撮像した少なくとも一つの入力画像において、レンズを装着した人物を含む入力画像を検出し、
    　前記入力画像において推定した前記レンズの度数を用いて当該レンズに映る目の位置及び大きさを補正した画像である補正後画像と、前記補正後画像との照合対象である三次元登録画像を出力する、
    照合補助方法。     In at least one input image obtained by capturing a person, an input image including a person wearing a lens is detected, and the input image is detected.
    Outputs a three-dimensional registered image that is a collation target between the corrected image, which is an image in which the position and size of the eyes reflected on the lens are corrected using the power of the lens estimated in the input image, and the corrected image. do,
    Matching assistance method.
  11. 　コンピュータに、
    　人物を撮像した少なくとも一つの入力画像において、レンズを装着した人物を含む入力画像を検出する処理、
    　前記入力画像において推定した前記レンズの度数を用いて当該レンズに映る目の位置及び大きさを補正した画像である補正後画像と、前記補正後画像との照合対象である三次元登録画像を出力する処理、
    を実行させるためのプログラムを記憶するプログラム記憶媒体。     On the computer
    A process of detecting an input image including a person wearing a lens in at least one input image obtained by capturing a person.
    Outputs a three-dimensional registered image that is a collation target between the corrected image, which is an image in which the position and size of the eyes reflected on the lens are corrected using the power of the lens estimated in the input image, and the corrected image. Processing to do,
    A program storage medium that stores a program for executing a program.
  12. 　人物を撮像した少なくとも一以上の入力画像において、レンズを有する眼鏡を装着した人物を含む入力画像を検出し、
    　前記入力画像において推定した前記レンズの度数を用いて前記入力画像との照合対象である三次元登録画像の目の位置及び大きさを補正した三次元補正後画像と、前記入力画像と、を出力する、
    照合補助方法。     In at least one input image of a person, an input image including a person wearing spectacles with a lens is detected.
    The three-dimensional corrected image obtained by correcting the eye position and size of the three-dimensional registered image to be collated with the input image using the power of the lens estimated in the input image, and the input image are output. do,
    Matching assistance method.
  13. 　コンピュータに、
    　人物を撮像した少なくとも一以上の入力画像において、レンズを有する眼鏡を装着した人物を含む入力画像を検出する処理、
    　前記入力画像において推定した前記レンズの度数を用いて前記入力画像との照合対象である三次元登録画像の目の位置及び大きさを補正した三次元補正後画像と、前記入力画像と、を出力する処理、
    を実行させるためのプログラムを記憶するプログラム記憶媒体。     On the computer
    A process of detecting an input image including a person wearing spectacles having a lens in at least one input image obtained by capturing a person.
    The three-dimensional corrected image obtained by correcting the eye position and size of the three-dimensional registered image to be collated with the input image using the power of the lens estimated in the input image, and the input image are output. Processing to do,
    A program storage medium that stores a program for executing a program.
PCT/JP2020/010980 2020-03-13 2020-03-13 Collation assistance device, collation assistance method, and program storage medium WO2021181642A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/JP2020/010980 WO2021181642A1 (en) 2020-03-13 2020-03-13 Collation assistance device, collation assistance method, and program storage medium
JP2022505677A JP7439899B2 (en) 2020-03-13 2020-03-13 Verification auxiliary device, verification auxiliary method and program

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/010980 WO2021181642A1 (en) 2020-03-13 2020-03-13 Collation assistance device, collation assistance method, and program storage medium

Publications (1)

Publication Number Publication Date
WO2021181642A1 true WO2021181642A1 (en) 2021-09-16

Family

ID=77671055

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/010980 WO2021181642A1 (en) 2020-03-13 2020-03-13 Collation assistance device, collation assistance method, and program storage medium

Country Status (2)

Country Link
JP (1) JP7439899B2 (en)
WO (1) WO2021181642A1 (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014059819A (en) * 2012-09-19 2014-04-03 Toshiba Corp Display device
JP2015025859A (en) * 2013-07-24 2015-02-05 富士通株式会社 Image processing apparatus, electronic device, eye-glasses characteristics determination method, and eye-glasses characteristics determination program
JP2015138333A (en) * 2014-01-21 2015-07-30 富士通株式会社 Display control program, display control apparatus, and display control system
JP3204175U (en) * 2016-03-01 2016-05-19 株式会社メディックエンジニアリング Image matching device
JP2019121143A (en) * 2017-12-29 2019-07-22 京セラドキュメントソリューションズ株式会社 Image processing apparatus, image processing method and image processing program

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014059819A (en) * 2012-09-19 2014-04-03 Toshiba Corp Display device
JP2015025859A (en) * 2013-07-24 2015-02-05 富士通株式会社 Image processing apparatus, electronic device, eye-glasses characteristics determination method, and eye-glasses characteristics determination program
JP2015138333A (en) * 2014-01-21 2015-07-30 富士通株式会社 Display control program, display control apparatus, and display control system
JP3204175U (en) * 2016-03-01 2016-05-19 株式会社メディックエンジニアリング Image matching device
JP2019121143A (en) * 2017-12-29 2019-07-22 京セラドキュメントソリューションズ株式会社 Image processing apparatus, image processing method and image processing program

Also Published As

Publication number Publication date
JP7439899B2 (en) 2024-02-28
JPWO2021181642A1 (en) 2021-09-16

Similar Documents

Publication Publication Date Title
US10564446B2 (en) Method, apparatus, and computer program for establishing a representation of a spectacle lens edge
US20210287026A1 (en) Method and apparatus with liveness verification
US9971920B2 (en) Spoof detection for biometric authentication
JP5061645B2 (en) Information extraction method, information extraction device, program, registration device, and verification device
US9361681B2 (en) Quality metrics for biometric authentication
US11657133B2 (en) Systems and methods of multi-modal biometric analysis
RU2431190C2 (en) Facial prominence recognition method and device
CN109918982B (en) Secure face authentication system using active infrared light source and RGB-IR sensor
CN104933389B (en) Identity recognition method and device based on finger veins
CN106598221A (en) Eye key point detection-based 3D sight line direction estimation method
US20120275665A1 (en) Method of generating a normalized digital image of an iris of an eye
KR101202448B1 (en) Apparatus and method for recognizing iris
TW200905577A (en) Iris recognition system
KR20210030230A (en) Data collection device for augmented reality
US8780116B2 (en) Object-shape generation method, object-shape generation apparatus, and program
WO2021181642A1 (en) Collation assistance device, collation assistance method, and program storage medium
KR102316587B1 (en) Method for biometric recognition from irises
EP3893147B1 (en) Liveliness detection using a device comprising an illumination source
WO2023175939A1 (en) Information processing device, information processing method, and recording medium
US20240153136A1 (en) Eye tracking
US11335123B2 (en) Live facial recognition system and method
Ntawiniga Head motion tracking in 3D space for drivers

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20923755

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022505677

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20923755

Country of ref document: EP

Kind code of ref document: A1