JP2021135970A - Image collation device, image collation system, and image collation method - Google Patents

Abstract

To provide an image collation device that, when frames meeting a criteria are not obtained or frames meeting a criteria are insufficient, can prevent a reduction in collation accuracy.SOLUTION: In an image collation system, an acquisition unit of an image collation device in which software is mounted on a computer continuously acquires images to be collated from an input device. A detection unit detects one or more objects of collation from a plurality of input frames included in the images to be collated. A tracking unit tracks a predetermined object of collation based on a result of detection of the objects of collation or extracts a set of frames including the objects of collation. In accordance with the number of conforming frames conforming to the criteria, of a plurality of predetermined frames including the predetermined objects of collation, a frame selection unit dynamically selects one or more frames for collation from the plurality of predetermined frames. A collation unit collates a registered image registered in a database with predetermined images to be collated included in the frames for collation one to one or one to plurality. An output unit outputs a collation result.SELECTED DRAWING: Figure 1

Description

Embodiments of the present invention relate to an image matching device, an image matching system, and an image matching method.

An image matching device is known that collates a face image (its feature information) included in an input image with a pre-registered face image (its feature information) to identify a person included in the input image. For example, the input image is an image from a surveillance camera that captures a pedestrian or the like. In order to improve the collation accuracy, the image collation device narrows down a plurality of frames included in the input image to frames that satisfy the criteria and executes the collation process.

JP-A-2019-57815

In cases where frames that meet the criteria cannot be obtained or frames that meet the criteria are insufficient, it is conceivable to forgo the collation process. However, in an environment where a sufficient number of frames satisfying the criteria cannot be obtained, the collation process is often postponed. Further, in such a case, it is conceivable to select an arbitrary one frame and execute the collation process, but there is a concern that the collation accuracy will be significantly reduced.

The problem to be solved by the present invention is an image matching device, an image matching system, and an image matching system that can prevent a decrease in matching accuracy when a frame satisfying the standard cannot be obtained or a frame satisfying the standard is insufficient. It is to provide an image matching method.

An image matching device in which software is mounted on a computer according to an embodiment includes acquisition means, detection means, tracking means, frame selection means, matching means, and output means. The acquisition means continuously acquires images to be collated from the image pickup apparatus. The detection means detects one or more collation targets from a plurality of input frames included in the collation target image. The tracking means tracks a predetermined collation target based on the detection result of the collation target, or extracts a set of frames including the collation target. The frame selection means dynamically selects one or more matching frames from the plurality of predetermined frames according to the number of matching frames that meet the criteria among the plurality of predetermined frames including the predetermined matching target. The collation means collates a registered image registered in a database with a predetermined collation target image included in the collation frame on a one-to-one or one-to-many basis. The output means outputs a collation result.

FIG. 1 is a configuration diagram showing an example of an image matching system according to the first embodiment. FIG. 2 is a diagram showing an example of frame selection according to the first embodiment. FIG. 3 is a flowchart showing an example of collation processing by the image collation system according to the first embodiment. FIG. 4 is a configuration diagram showing an example of the image matching system according to the second embodiment. FIG. 5 is a diagram showing an example of the parameter setting screen according to the second embodiment. FIG. 6 is a configuration diagram showing an example of the image matching system according to the third embodiment. FIG. 7 is a configuration diagram showing an example of the image matching system according to the fourth embodiment.

The first embodiment will be described below with reference to the drawings.
FIG. 1 is a configuration diagram showing an example of an image matching system according to the first embodiment.
In the present embodiment, one or more collation frames are dynamically selected from a plurality of predetermined frames according to the number of matching frames that meet the collation criteria among a plurality of (N) predetermined frames including a predetermined collation target. select. For example, when the number of conforming frames is a specified number (n, n ≦ N) or more, a collation frame is selected from a plurality of predetermined frames by the first selection method (selection method S1), and the number of conforming frames is specified. When the number is less than the number, a collation frame is selected from a plurality of predetermined frames by the second selection method (selection method S2). For example, according to the selection method S1, a first predetermined number (m1 pieces, m1 ≦ n) of matching frames, which is equal to or less than a specified number, is selected as a collation frame. According to the selection method S2, a second predetermined number (m2, m2 ≦ N) of frames is selected as the collation frame.

In the present embodiment, as an example, a system in which a detection target is a person's face and a registered person (= person) is collated using a face image will be described. The detection target may be a part or all of a moving object such as a person or an animal, or a part or all of an artificial moving object such as a vehicle, an aircraft, or a drone.

As shown in FIG. 1, the image matching system according to the first embodiment includes a registration device 100, an input device 200, and an image matching device 300. The registration device 100 can be composed of one or more computers, and includes a registration unit 111 and a registration image database 112. The input device 200 includes an image input unit (imaging device) 201. The image collating device 300 can be composed of one or more computers, and includes a processor 301, a main memory 302, an auxiliary storage device 303, and interfaces 304 and 305. In other words, the image matching device 300 is a device in which software is installed in a computer.

The processor 301 performs processing such as calculation and control necessary for the image matching process. The processor 301 executes control to realize various functions based on a program such as system software, application software, or firmware stored in at least one of the main memory 302 and the auxiliary storage device 303. The processor 301 is, for example, a CPU (central processing unit), an MPU (micro processing unit), or a DSP (digital signal processor). Alternatively, the processor 301 is a combination of a plurality of these.

The main memory 302 is a computer-readable storage medium, and includes a non-volatile memory (non-temporary storage medium) used exclusively for reading data and a volatile memory used for reading and writing data. The non-volatile memory plays a role as a program memory, and stores a part or all of programs, data, various setting values, and the like used by the processor 301 to perform various processes. The volatile memory plays a role as a work memory and stores data temporarily used by the processor 301 for performing various processes.

The auxiliary storage device 303 is a computer-readable storage medium, for example, a non-temporary storage medium such as an EEPROM (electric erasable programmable read-only memory) (registered trademark), an HDD (hard disk drive), or an SSD (solid state drive). be. The auxiliary storage device 303 stores a part or all of a program, data, or various set values used by the processor 301 to perform various processes.

The interface 304 functions as an acquisition unit (acquisition means), continuously acquires an input image from the image input unit 201, and outputs the input image to the processor 301. The interface 305 functions as an output unit and outputs a collation result or the like to an external device such as a display.

Further, the processor 301 will be described. The processor 301 includes a detection unit (detection means) 3011, a tracking unit (tracking means) 3012, a frame selection unit (frame selection means) 3013, a collation unit (matching means) 3014, and the like. The detection unit 3011 uses the input image as the collation target image, and detects one or more collation targets from a plurality of input frames included in the collation target image. The tracking unit 3012 tracks a predetermined matching target based on the detection result of the matching target, or extracts a set of frames including the matching target. The frame selection unit 3013 collates one or more from the tracking frame set according to the number of matching frames that meet the collation criteria among a plurality of (N) predetermined frames (that is, tracking frame sets) including the predetermined collation target. Dynamically select the frame for. For example, the frame selection unit 3013 selects a matching frame that meets the face matching criteria from the tracking frame set. The frame selection unit 3013 selects a collation frame from the tracking frame set by the selection method S1 when the number of matching frames is the specified number (n) or more, and when the number of matching frames is less than the specified number, the tracking frame. A collation frame is selected from the set by the selection method S2. The collation unit 3014 collates the registered image (or its feature information) registered in the database with the predetermined collation target image (or its feature information) included in the collation frame on a one-to-one or one-to-many basis.

A more detailed explanation is supplemented below.
The image input unit 201 can be installed at one or a plurality of locations, and inputs a person image of a person to be photographed, and includes, for example, an ITV camera. The ITV camera digitizes the optical information obtained through the lens of the camera by an A / D converter and outputs it as image data. It should be noted that the search target may be digital human image data as a result. Therefore, the image input unit 201 may be an interface for capturing an image file or a moving image file taken by a digital still camera via a medium. A device for inputting ID (Identification) information used for identity verification may be provided together with the image input unit 201. Any device may be used as long as it can read the information recorded from various media, such as an IC card reader, a QR code (registered trademark) reader, and a magnetic information reader. It is possible to use it for the collation process by reading the attached information about the passerby from these devices.

The detection unit 3011 detects the face image of the object from the input image from the image input unit 101. In face detection, for example, an average image of a large number of rectangular regions of a face is created as a face template, and the face template is scanned while being parallel-moved, rotated, and enlarged / reduced on the input image from the image input unit 201, and the face template is detected. It can be realized by a method in which the position where the difference between the brightness value and the face is the smallest is set as the face detection area.

The tracking unit 3012 tracks the face image of the detected object between frames. Since it is assumed that the detected face images of the same person are continuously displayed over a plurality of frames, the face image of the person is tracked so that they can be associated with each other as the same person. As a means for realizing this, there are a method of associating the position of the face image detected from one frame with the next frame by using optical flow, and a method of using the technique disclosed in Japanese Patent Application Laid-Open No. 2011-170711. .. In the face feature extraction means described later, a method of selecting at least one image suitable for searching from the images of the face area of a plurality of frames associated with the same person, and the maximum number of detected frames. There is a method of using an arbitrary number of images.

The frame selection unit 3013 has n (1 ≦ n ≦ N) used by the collation unit 3014 from among the tracking frame sets (N frames) associated with the same person in the detection unit 102 and the tracking unit 3012. Select and output the frame of.

The frame selection unit 3013 first selects n frames based on an index for determining whether or not the N frames meet the collation criteria. An example of the index is shown below.
・ Object size (resolution)
-Score of certainty of the object-Brightness distribution in the image area of the object-Position in the image area of the object-Orientation of the object The size (resolution) of the object in the present embodiment is determined by the detection unit 3011. Since the area when the target face is detected is obtained as a rectangular area, the number of pixels in the horizontal direction may be used as an index. Further, the number of pixels in the vertical direction may be used as an index, or the area obtained by multiplying the number of pixels in the vertical direction by the number of pixels in the horizontal direction may be used as an index.

As the score of the certainty of the object, the score indicating the consistency with the template when detected by the detection unit 3011 can be used as it is.

As the brightness distribution in the image area of the object, it is generally said that it is desirable that the contrast of the face image is large and that there is no bias. Therefore, the dispersion value and standard deviation of the brightness values in the face area are used. .. It is also possible to use as an index the number or ratio of pixels that are too bright or too dark, such as the number of pixels having a brightness value above a predetermined value (overexposure) or a brightness value below a predetermined value (blackout). The lower the ratio of the luminance value above or below the predetermined value, the higher the index. Therefore, the ratio is converted to the reciprocal or the value obtained by subtracting the ratio from 1.

As for the position of the object in the image area, the image quality deteriorates due to lens distortion when approaching the top, bottom, left, and right edges of the input image, and part of the face image or the image around the face becomes out of the area and information is lost. Therefore, it is also possible to use the distance from the center coordinates as an index so that the score becomes higher toward the center of the image (again, the closer the distance from the center coordinates is and the smaller the value is, the better the index is. It is possible to adopt the reciprocal so that

As the orientation of the object, the orientation of the object can be detected by using the technique disclosed in Japanese Patent Application Laid-Open No. 2007-4767, and the closer to the front, the higher the index. In addition to the method of calculating by the calculation formula in this way, it is also possible to obtain an image suitable for collation by learning the correlation between a plurality of images of the subject and the difference from others by using Japanese Patent Application Laid-Open No. 2019-57815. Is.

The frame selection unit 3013 evaluates whether or not each frame meets the collation criteria based on at least one index of each frame. When a plurality of indexes are used, the frame selection unit 3013 evaluates whether or not each frame meets the collation criteria using the weighted average value obtained by multiplying each index by a predetermined coefficient and adding them as an index. You may. Further, the frame selection unit 3013 pays attention to each index of each frame, extracts frames having a high index at a predetermined ratio, pays attention to a plurality of indexes of the extracted frames, and conforms to the collation criteria. Narrow down the applicable frames. Alternatively, the frame selection unit 3013 sets a threshold value for each index of each frame, and narrows down the frames that are equal to or greater than the threshold value as conforming frames. Further, the judgment criteria may be switched for each index, and the frame selection unit 3013 narrows down the frames that exceed a predetermined threshold value with the first index, and then further sets the frames in descending order of the second index. You may narrow down and select a suitable frame.

The frame selection unit 3013 selects n frames from the frames that satisfy the collation criteria. For example, the frame selection unit 3013 selects n frames in descending order of index. Alternatively, the frame selection unit 3013 may select n frames depending on whether or not each index is equal to or higher than a preset reference value. The number of selected frames may be less than n.

FIG. 2 is a diagram showing an example of frame selection according to the first embodiment.
For example, the tracking unit 3012 tracks a predetermined collation target and identifies a plurality of (N) predetermined frames including the predetermined collation target. As shown in FIG. 2, it is assumed that six frames F1 to F6 including a predetermined collation target are specified. In addition, as indexes, it is assumed that three indexes of "face certainty score (face score)", "face size", and "face orientation" are used.

The frame selection unit 3013 selects a frame based on the threshold value of each index. Frame F1 with a small face size and a face size index less than the threshold does not meet the collation criteria, and the face orientation is sideways, so the face orientation index is less than the threshold. The frames F3 and F4 also do not meet the collation criteria, and the three frames F2, F5 and F6 in which all the indexes are above the threshold value meet the collation criteria.

If the number of conforming frames is equal to or greater than the specified number, for example, if the number of conforming frames is 3 and the specified number is 2, the frame selection unit 3013 is among the three frames F2, F5, and F6 that meet the collation criteria. Two frames are selected in descending order of face score (selection method S1-1). Alternatively, the frame selection unit 3013 may multiply each of the three indexes by a coefficient and select two frames from the three frames that meet the criteria based on the total value of the indexes (selection method S1-2). ).

On the other hand, when the number of conforming frames is less than the specified number, for example, when the number of conforming frames is 3 and the specified number is 4, the frame selection unit 3013 has a plurality of (N) predetermined frames including faces to be collated. Is selected as it is as a collation frame (selection method S2-1). As a result, when there are not enough frames that match the matching, the amount of information can be increased to improve the matching accuracy.

Alternatively, the frame selection unit 3013 preferentially selects the frames excluded by the low priority index based on the priority specified for the above three indexes (selection method S2-2), and determines the number of frames. Make up for the shortage. The number of insufficient frames may be supplemented, or the number of frames larger than the number of insufficient frames may be compensated. If the collation time is prioritized, make up for the shortage of frames. When giving priority to collation accuracy, the number of frames that is larger than the number of insufficient frames is compensated. For example, the frame selection unit 3013 preferentially selects, in addition to selecting the conforming frame, the frame excluded by the index having a low priority among the frames that do not conform to the collation criteria. As a result, it is possible to add information according to the weight of the index, and it is possible to improve the collation accuracy more effectively.

Alternatively, the frame selection unit 3013 selects extraction frames extracted at regular intervals from a plurality of (N) predetermined frames including faces to be collated as collation frames (selection method S2-3). When the collation time is prioritized, a fixed interval is set as the first interval, and when the collation accuracy is prioritized, the constant interval is set as the second interval (first interval <second interval). Further, the frame selection unit 3013 may select frames to be extracted at regular intervals in addition to selecting matching frames. In the case of FIG. 2, in addition to the selection of the matching frames F2, F5, F6, some of the frames F1, F3, F5 extracted at regular intervals may be selected. For example, when giving priority to the collation time, four frames (= specified number n) including the matching frames F2, F5, F6 and the extracted frames F1 or F3 are selected. When the collation accuracy is prioritized, five (> specified number n) frames including the matching frames F2, F5, F6 and the extracted frames F1 and F3 are selected. As a result, information can be added, and the collation accuracy can be improved more effectively.

Alternatively, the frame selection unit 3013 may select a frame to be extracted based on the time from a plurality of (N) predetermined frames including the face to be collated (selection method S2-4). Further, the frame selection unit 3013 may select a frame to be extracted based on the time in addition to selecting a matching frame. As a result, information can be added, and the collation accuracy can be improved more effectively.

For example, a new upper predetermined number (= specified number n) of frames is extracted based on the time, and the extracted frames are selected (selection method S2-41). In the case of FIG. 2, it is assumed that the time of the frame F1 is the oldest, and the time is newer in the order of F2, F3, F4, F5, and F6. In this case, the frame selection unit 3013 selects frames F3, F4, F5, and F6. Further, in addition to the selection of the conforming frames F2, F5, and F6, the frame lacking in the specified number may be supplemented from the new frames F3, F4, F5, and F6 based on the time. For example, the frame selection unit 3013 selects the matching frames F2, F5, F6 and the newest frame F4 based on the time.

Alternatively, the old upper predetermined number (= specified number n) of frames is extracted based on the time, and the extracted frames are selected (selection method S2-42). In the case of FIG. 2, frames F1, F2, F3, and F4 are selected. Further, in addition to selecting the matching frames F2, F5, and F6, the oldest frame F1 among the old frames F1, F2, F3, and F4 based on the time may be selected.

Alternatively, when the number of matching frames is less than the specified number, the frame selection unit 3013 groups the frames into frame groups based on the amount of change between the frames of a plurality of predetermined frames, and includes a representative face image from each frame group. A frame is extracted, and the extracted frame is selected as a collation frame (selection method S2-5). For example, the frame selection unit 3013 clusters the facial images into similar groups and selects a representative image from each group. For example, all combinations of face images are used to determine the degree of face similarity (processing equivalent to the collation process of the collation unit 3014), and k-mean clustering is performed to group them. The index used for clustering may be a correlation value between facial images, or an index used in frame selection may be used. The frame selection unit 3013 selects a representative image of each group after performing clustering. As a method of selecting the representative image, the most average image of each group may be selected, or the images may be simply selected in the order of oldest or newest in time.

The registration unit 111 is a scanner or a camera. The scanner scans a face image for registration (for example, a still image such as a photograph), extracts facial feature information from the acquired registered image, and registers the registered image and the feature information in the registered image database 112. Alternatively, the camera captures a face image (moving image) for registration, extracts facial feature information from the acquired registered image, and registers the registered image and the feature information in the registered image database 112. Extraction of facial feature information is the same as the method of extracting facial feature information from an image by the collating unit 3014. The registration unit 111 may be an interface for capturing the face image file via the medium.

The registered image database 112 is a database used when the collation unit 3014 calculates the similarity index, and stores facial feature information for each individual to be searched. Further, in order to make the collation result easy to understand for users and the like, the registered image database 112 associates the face feature information with the corresponding person such as ID information, face image, name, gender, age, and height. Stores attribute information about. The facial feature information may be the data itself extracted by the facial feature extraction process, an m × n feature vector, or a correlation matrix immediately before performing subspace or KL expansion.

The collation unit 3014 collates the feature information of the predetermined collation target image included in the collation frame selected by the frame selection unit 3013 with the feature information of the registered image registered in the registered image database 112. When one or more IC cards or the like are used in the image input unit 201, the collation unit 3014 is included in a predetermined collation frame selected from the input images obtained by shooting the image input unit 201. The collation target image is collated with the registered image of the person associated with the information obtained from the IC card. When the IC card or the like is not used, the collation unit 3014 registers the predetermined collation target image included in the collation frame selected from the input images obtained by the shooting of the image input unit 201 and the registered image database 112. Check against all the registered images.

The collation unit 3014 generates a collation result including a degree of similarity indicating the certainty of the person. For example, the collation unit 3014 generates collation results sorted in descending order of similarity. The collation unit 3014 may generate a collation result in association with the ID information and the attribute information such as the face image stored in the registered image database 112. As a process for obtaining facial similarity, the technique disclosed in Japanese Patent Application Laid-Open No. 2007-4767 may be used. This can be realized by creating an image in which the orientation and state of the face are intentionally changed by using a model for one person image information, and collating the face feature information. In addition, literature that calculates multiple frames of the same person using moving images using multiple images (Kazuhiro Fukui, Osamu Yamaguchi, Kenichi Maeda: "Face recognition system using moving images" IEICE Technical Report PRMU, The technique disclosed in vol97, No.113, pp17-24 (1997)) may be utilized. This can be realized by using the feature information such as the subspace output by such a method as the personal feature information for the face detected in the input image. In this case, the similarity index is the degree of similarity between the subspaces managed as facial feature information. In addition, as a similarity index other than the degree of similarity, distances on the feature space and Mahalanobis distances can also be used. When the distance is used, the larger the numerical value, the lower the similarity. Therefore, as a method of comparing with the threshold value, the smaller the value, the higher the similarity with the registered person.

In the present embodiment, a collation method using facial features is used, but in any collation using features obtained by images other than the face, the present embodiment can be applied, and other than face collation. Also, by changing the camera to near infrared, it is possible to collate using the iris and retinal information of the eyes, and collation using veins such as hands and face. In this embodiment, a collation based on a full-body image using features, possessions, etc., or a collation technique based on actions using gaits or predetermined movements may be applied. These methods can be realized based on the following literature information.

<Literature information>
・ 2007, Patent application technology trend survey report, input / recognition of biometric collation (summary version), April 2008, JPO, Internet
<URL: https://www.jpo.go.jp/resources/report/gidou-houkoku/tokkyo/document/index/vaio_youyaku.pdf>
-Patent No. 3307936, Iridescent / Japanese Patent Application Laid-Open No. 11-213164, Retinal / Japanese Patent Application Laid-Open No. 2006-0949493, Venous / JP-A-2006-011988, Vine-Japanese Patent Application Laid-Open No. 2018-82408, Whole Body / Strategy Creative Research Promotion Project, CREST, Research Area, "Construction of Human-Harmonized Information Technology for a Symbiotic Society", Research Projects, "Personal Behavior Analysis Based on the Intentional Behavior Model and Construction of an Information Environment that Captures the Mind" , Research Completion Report, Research Period October 2010-March 2017, Principal Investigator: Yasushi Yagi, (Osaka University, Director / Vice President), Internet
<URL: https://www.jst.go.jp/kisoken/crest/evaluation/s-houkoku/sh_h28/JST_1111071_10101452_2016_PER.pdf>

The interface 305 outputs a collation result or the like to an external device. The external device is a display device such as a liquid crystal display, a speaker that outputs a notification or an alarm, and a traffic control device that controls traffic by opening and closing a gate. For example, the interface 305 outputs display information including the collation result to the display device. Based on the display information, the display device displays the success or failure of the collation, the image input by the image input unit 201, and the collation target (face area) detected by the detection unit 3011. Further, the display device (monitoring display device) is based on the display information so that the input image (face image) can be known to which person among the persons registered in the registered image database 112 and how similar it is. , The input image, the image of a person similar to the input image, and the degree of similarity to the input image may be displayed side by side.

Further, the interface 305 outputs an audio signal indicating a warning sound to the speaker according to the degree of similarity. The interface 305 outputs an audio signal indicating a warning sound to the speaker when a collation result having a similarity equal to or higher than the threshold value cannot be obtained. The speaker outputs a warning sound based on the audio signal.

Further, the interface 305 outputs an unlocking signal for unlocking the gate according to the degree of similarity to the passage control device. The interface 305 outputs an unlock signal when a collation result having a similarity equal to or higher than the threshold value is obtained, and does not output an unlock signal when a collation result having a similarity equal to or higher than the threshold value cannot be obtained. .. When the traffic control device receives the unlocking signal, the traffic control device unlocks the gate based on the unlocking signal.

As described above, even in an environment in which an image unsuitable for collation is input, it is possible to suppress a decrease in collation accuracy and improve the reliability of control such as entry / exit based on the collation result.

FIG. 3 is a flowchart showing an example of collation processing by the image collation system according to the first embodiment.
The image input unit 201 photographs the admission or entry monitoring area and outputs the input image obtained by the imaging. The interface 304 of the image collation device 300 acquires the input image as the collation target image (ST101). The processor 301 of the image collation device 300 executes a collation process on the predetermined collation target image included in the acquired collation target image (ST102 to ST109).

The processor 301 realizes various functions based on a program stored in at least one of the main memory 302 and the auxiliary storage device 303, and executes the collation process. For example, the detection unit 3011 detects one or more matching targets from a plurality of input frames included in the matching target image (ST102). The tracking unit 3012 tracks a predetermined collation target based on the detection result of the collation target (ST103). When the tracking of the predetermined collation target by the tracking unit 3012 is completed (ST104, YES), the frame selection unit 3013 calculates the index of a plurality (N) predetermined frames including the predetermined collation target (ST105). The frame selection unit 3013 dynamically selects one or more collation frames from a plurality of predetermined frames according to the number of frames whose index is equal to or greater than the reference value, that is, the number of frames that meet the collation criteria.

If the number of frames conforming to the collation criteria is equal to or greater than the threshold value (= specified number) (ST106, YES), the frame selection unit 3013 uses the collation frame based on the first frame selection process (frame selection method S1). (ST107), the collation unit 3014 collates the feature information of the registered image registered in the registered image database 112 with the feature information of the predetermined collation target image included in the collation frame (ST109). The interface 305 outputs the collation result from the collation unit 3014 to the external device (ST110).

If the number of frames conforming to the collation standard is less than the threshold value (= specified number) (ST106, NO), the frame selection unit 3013 is the collation frame based on the second frame selection process (frame selection method S2). (ST108), the collation unit 3014 collates the feature information of the registered image registered in the registered image database 112 with the feature information of the predetermined collation target image included in the collation frame (ST109). The interface 305 outputs the collation result from the collation unit 3014 to the external device (ST110).

The second embodiment will be described below with reference to FIG.
FIG. 4 is a configuration diagram showing an example of the image matching system according to the second embodiment. The main difference between the image matching system according to the first embodiment and the image matching system according to the second embodiment is that the image matching system according to the second embodiment includes the interface 306. In the image matching system according to the first embodiment and the image matching system according to the second embodiment, the same reference numerals are given to common configurations, and the description thereof will be omitted as appropriate.

The interface 306 accepts the processing parameter settings of the frame selection unit 3013 and the collation unit 3014, and inputs the accepted settings to the frame selection unit 3013 and the collation unit 3014. For example, the interface 306 is connected to the display device and the input device, outputs a parameter setting screen to the display device, and accepts the parameters input by the input device on the parameter setting screen. As a result, parameters related to the collation processing time and the collation accuracy can be adjusted from the outside. For example, administrator and maintenance personnel parameters can be adjusted via interface 306.

FIG. 5 is a diagram showing an example of the parameter setting screen according to the second embodiment.
As shown in FIG. 5, the parameter setting screen has a setting unit for setting face matching parameters (denoted as “authentication threshold”) for each gate. In this setting unit, a value of 0 to 1 can be set. The value may be adjusted with the slider bar or the value may be directly input.

Further, the parameter setting screen has a setting unit for setting parameters related to frame selection. This setting unit accepts the setting of the number of predetermined frames (N) including the predetermined collation target. For example, as shown in FIG. 5, five (5 frames) are set. The number of predetermined frames is not limited to the number of predetermined frames, and the ratio of predetermined frames in all frames may be set.

In addition, this setting unit also accepts parameters for each of the selection method S1 (frame selection mode 1) and the selection method S2 (frame selection mode 2). When the number of conforming frames is the specified number or more, the frame selection mode 1 is executed, and when the number of conforming frames is less than the specified number, the frame selection mode 2 is executed. Parameters related to frame selection mode 1 and frame selection mode 2 are set. The frame selection method of the frame selection mode 1 and the frame selection mode 2 is set. The frame selection method can be selected from "best order (highest index order)", "thinning out", "time newest order", "time oldest order", "clustering", and "learning automatic selection". .. Further, the number of frames output in the frame selection mode 1 and the number of frames output in the frame selection mode 2 can also be set.

For example, when the total input frame is 10 frames and the conforming frame is 7 frames, the frame selection mode 1 is selected because it is larger than the set threshold value of 5, and the index suitable for matching is high among the 7 frames. Three frames are output in order. On the other hand, when there are only two conforming frames, mode 2 is selected, 10 frames are evenly thinned out, and 3 frames are selectively output.

On the setting screen, other setting parameter items required for the system may be added. In addition, it is clear that it may be possible to set it from a mobile terminal via a network, or it may be reflected in the system by writing it in a setting file.

The third embodiment will be described below with reference to FIG.
FIG. 6 is a configuration diagram showing an example of the image matching system according to the third embodiment. In the image matching system according to the first embodiment and the image matching system according to the second embodiment, the same reference numerals are given to common configurations, and the description thereof will be omitted as appropriate. In the first embodiment and the second embodiment, it is assumed that the registered image is registered in the passerby registration image database 1121 in advance, but in the third embodiment, the image is taken at the time of passing through the entrance / exit gate or the like. Registered images are registered from the captured images to be captured, and the captured images captured at a plurality of points are used to associate them with each other to see if they are the same person. For example, it is possible to register a face image at the time of entry into a predetermined area, associate the face image at the time of entry with the face image at the time of entry, and manage which person entered and entered. .. Furthermore, it is possible to check that a person other than the person who entered has participated.

Therefore, in the third embodiment, a plurality of passage gates, for example, passage gate G1 (entrance) and passage gate G2 (entry) are assumed. As shown in FIG. 6, the image matching system according to the third embodiment includes a registration device 100, an input device 210 provided corresponding to the passage gate G1, an input device 220 provided corresponding to the passage gate G2, and an input device 220. An image collation device 300 is provided.

The registration device 100 can be configured by one or more computers and includes a processor 101, a main memory 102, an auxiliary storage device 103, an interface 104, an interface 105, and a passerby registration image database 1121. The processor 101 includes a detection unit 1011, a tracking unit 1012, a frame selection unit 1013, a registration unit 1014, and the like. The input device 210 includes an image input unit (imaging device) 211. The input device 220 includes an image input unit (imaging device) 221. The image collating device 300 can be configured by one or more computers and includes a processor 301, a main memory 302, an auxiliary storage device 303, and interfaces 304, 305, 306.

The registration device 100 registers the image to be registered included in the image captured from the input device 210. In the present embodiment, it is assumed that the passage gate G1 and the passage gate G2 are dedicated gates in the direction of entering and exiting the area, instead of the gates that can pass in both directions. By installing two image input units 211 and 221 in opposite directions, it can be applied to a gate that can pass in both directions.

The image input unit 211 of the input device 210 photographs the passage area of the passage gate G1 for visitors and outputs the captured image. The interface 104 of the registration device 100 acquires the captured image, and the detection unit 1011 detects the registration target from the captured image. The tracking unit 1012 tracks the detected registration target in association with each other between frames. The frame selection unit 1013 selects a matching frame suitable for face matching from a plurality of predetermined frames (that is, a tracking frame set) including a registration target. The registration unit 1014 registers the selected frame in the passerby registration image database 1121. The interface 105 outputs display information indicating a registration result or the like as information at the time of passage to the display device, and outputs an unlocking signal or the like to the passage control device according to the registration result.

The image input unit 221 of the input device 220 photographs the passage area of the passage gate G2 for the contestants and outputs the captured image. The interface 304 of the image collation device 200 acquires a captured image, and the detection unit 3011 detects a predetermined collation target from the captured image. The tracking unit 3012 tracks the detected predetermined collation target in association with each other between frames. The frame selection unit 3013 selects a matching frame suitable for face matching from a plurality of predetermined frames (that is, a tracking frame set) including a predetermined matching target. The frame selection unit 3013 dynamically selects one or more collation frames from the tracking frame set according to the number of matching frames among a plurality of predetermined frames including a predetermined collation target. The collation unit 3014 collates the feature information of the predetermined collation target image included in the collation frame selected by the frame selection unit 3013 with the feature information of the registered image registered in the passerby registration image database 1121. The interface 305 outputs a collation result or the like to an external device. Further, the interface 306 accepts the processing parameter settings of the frame selection unit 3013 and the collation unit 3014, and inputs the accepted settings to the frame selection unit 3013 and the collation unit 3014.

The detection unit 1011 performs the same processing as the detection unit 3011, and the tracking unit 1012 performs the same processing as the tracking unit 3012. The frame selection unit 1013 may select all of a plurality of predetermined frames (that is, a tracking frame set) including a registration target, or may select all of matching frames that match face matching. Alternatively, the frame selection unit 1013 may thin out a certain number of frames from a plurality of predetermined frames including the registration target in order to shorten the processing time or reduce the amount of information.

Similar to the collation unit 3014, the registration unit 1014 extracts facial feature information from the face image and outputs the result to the passerby registration database 112.

According to the third embodiment, it is possible to manage people who enter and leave a specific area in association with each other.

The fourth embodiment will be described below with reference to FIG. 7.
FIG. 7 is a configuration diagram showing an example of the image matching system according to the fourth embodiment. The main difference between the image matching system according to the third embodiment and the image matching system according to the fourth embodiment is that the image matching system according to the fourth embodiment has an ID registration unit (registration means) 113. The point is that the ID information database 114, the ID confirmation unit 1015, and the ID confirmation unit 3015 are provided. In the image matching system according to the third embodiment and the image matching system according to the fourth embodiment, the same reference numerals are given to common configurations, and the description thereof will be omitted as appropriate.

In the fourth embodiment, the face image at the time of entry and the face image at the time of entry are associated with each other to manage which person entered and participated, and the ID information is managed by the ID information database 114. It is possible to manage the entry / exit of a person by associating the ID information with the person who enters or exits.

For example, the image input unit 211 or 221 reads the ID information from the IC card or the like in parallel with the shooting of the face image, the passerby registration image database 1121 stores the facial feature information and the attribute information, and the ID information database 114 Stores ID information. The feature information and attribute information stored in the passerby registration image database 1121 and the ID information stored in the ID information database 114 are associated with each other by management identification information or the like.

Alternatively, the ID registration unit 113 reads the ID information and the face image information (feature information) from the registration medium, and registers the ID information and the face image information in association with each other. That is, the passerby registration image database 1121 stores the face image information (feature information), and the ID information database 114 stores the ID information. The face image information (feature information) stored in the passerby registration image database 1121 and the ID information stored in the ID information database 114 are associated with each other by management identification information or the like.

The image input unit 211 of the input device 210 photographs the passage area of the passage gate G1 for visitors and outputs the captured image. The interface 104 of the registration device 100 acquires the captured image, and the detection unit 1011 detects the registration target from the captured image. The tracking unit 1012 tracks the detected registration target in association with each other between frames. The frame selection unit 1013 selects a matching frame suitable for face matching from a plurality of predetermined frames (that is, a tracking frame set) including a registration target. The registration unit 1014 registers the selected frame in the passerby registration image database 1121. The interface 105 outputs display information indicating a registration result or the like as information at the time of passage to the display device, and outputs an unlocking signal or the like to the passage control device according to the registration result.

When it is necessary to read the ID information at the time of admission, for example, the image input unit 211 reads the ID information from the ID card or the like, and the ID information database 114 stores the read ID information. Further, the ID confirmation unit 1015 confirms whether or not the read ID information is the registered ID information stored in the ID information database 114, and the interface 105 outputs the confirmation result of the ID information.

The image input unit 221 of the input device 220 photographs the passage area of the passage gate G2 for the contestants and outputs the captured image. The interface 304 of the image collation device 200 acquires a captured image, and the detection unit 3011 detects a predetermined collation target from the captured image. The tracking unit 3012 tracks the detected predetermined collation target in association with each other between frames. The frame selection unit 3013 selects a matching frame suitable for face matching from a plurality of predetermined frames (that is, a tracking frame set) including a predetermined matching target. The frame selection unit 3013 dynamically selects one or more collation frames from the tracking frame set according to the number of matching frames among a plurality of predetermined frames including a predetermined collation target. The collation unit 3014 collates the feature information of the predetermined collation target image included in the collation frame selected by the frame selection unit 3013 with the feature information of the registered image registered in the passerby registration image database 1121. The interface 305 outputs a collation result or the like to an external device. Further, the interface 306 accepts the processing parameter settings of the frame selection unit 3013 and the collation unit 3014, and inputs the accepted settings to the frame selection unit 3013 and the collation unit 3014.

When it is necessary to read the ID information at the time of participation, for example, the image input unit 221 reads the ID information from the ID card or the like, and the ID information database 114 stores the read ID information. Further, the ID confirmation unit 3015 confirms whether or not the read ID information is the registered ID information stored in the ID information database 114, and the interface 305 outputs the confirmation result of the ID information.

Here, the person identification based on the ID information is supplemented. The ID registration unit 113 acquires ID information that identifies a person passing through the area. For example, the ID registration unit 113 acquires ID information via a medium such as an IC card, a QR code, or a magnetic card. Alternatively, the ID registration unit 113 acquires a face image taken by a camera, acquires a face image read by a scanner, acquires a face image file from the outside via media or a network, and registers at least one piece of personally identifiable information. .. The ID registration unit 113 acquires feature information from the face image by the same processing as that of the collation unit 3014. Further, the ID information may be managed by associating information such as name, age, and gender, and credit card information for billing.

The ID confirmation unit 1015 uses the ID information input from the image input unit 211 to match with the ID information database 407 and identifies who is currently passing by. Further, the ID confirmation unit 1015 may collate the facial feature information obtained by the registration unit 1014 with the passerby registration image database 1121 to identify who is currently passing by. By identifying a passerby, information on which person is in which area can be output from the interface 105, the passage history of the identified passerby can be managed, and the identified passerby can be charged. It is possible.

Similarly, the ID confirmation unit 3015 uses the ID information input from the image input unit 221 to match with the ID information database 407 and identifies who is currently passing by. Further, the ID confirmation unit 3015 may collate the facial feature information obtained by the collation unit 3014 with the passerby registration image database 1121 to identify who is currently passing by.

The collation processing according to each of the above-described embodiments is as follows, and according to each embodiment, when a frame satisfying the standard cannot be obtained or when a frame satisfying the standard is insufficient, a decrease in collation accuracy is prevented. It is possible to provide an image matching device, an image matching system, and an image matching method that can be used.

In the image matching system according to each embodiment, when there are n (n ≦ N) or more matching frames satisfying a predetermined condition among the input N frames, m1 frame (m1 ≦ n) among them is present. Is selected as the frame to be used for face matching.

Further, in the image matching system, when the number of conforming frames satisfying a predetermined condition among the input N frames is less than n (n ≦ N) frames, all N frames or m2 frames (m2 ≦ N) are used. Is selected as the frame to be used for face matching.
The predetermined conditions are one or more of the face size (resolution), the score of the certainty of the face, the brightness distribution of the face, the position of the face in the image, the orientation of the face, and the matching optimality index learned from the matching score. Is a condition to be judged by combining. The matching optimum index is an automatic learning of the best shot evaluation formula such that the best shot is selected from the images having a high degree of similarity between the images of the person himself / herself.

That is, whether or not the image matching system according to each embodiment is suitable for face matching for each frame in face matching using face images of a plurality of frames obtained by tracking the face in the image. Score the degree (the higher the score, the closer to the match), and select the frame to be used for matching as follows.

1) If there is a frame whose score is higher than the predetermined standard, the frame with the lower score is excluded based on the predetermined standard, and the remaining frames are used for collation.

2) If there is no frame whose score is higher than the predetermined standard, the frame is selected by a method different from 1). For example, the majority or all frames are used for matching, even if the frames are lower than the reference.

As a result, it is possible to ensure the matching accuracy by increasing the number of images that can be used for matching even in an environment where it is difficult to shoot a good image or in the case of a subject. In addition, there is an effect that it becomes easy to secure the accuracy even when both the images at the time of registration and the images at the time of collation are in a bad environment.

Applications of this embodiment include entry / exit control devices for security areas or buildings, immigration control devices, entrance / exit gates (ticket gates) for public transportation, and access control for devices that can only be used by a predetermined person.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

100 ... Registration device 101 ... Image input unit 101 ... Processor 102 ... Detection unit 102 ... Main memory 103 ... Auxiliary storage device 104 ... Interface 105 ... Interface 111 ... Registration unit 112 ... Registered image database 113 ... ID registration unit 114 ... Information database 200 ... Image collation device 200 ... Input device 201 ... Image input unit 210 ... Corresponding input device 210 ... Input device 211 ... Image input unit 220 ... Corresponding input device 220 ... Input device 221 ... Image input unit 300 ... Image collation device 300 ... and image collation device 301 ... Processor 302 ... Main memory 303 ... Auxiliary storage device 303 ... And auxiliary storage device 304 ... Interface 305 ... Interface 306 ... Interface 407 ... Information database 1011 ... Detection unit 1012 ... Tracking unit 1013 ... Frame selection unit 1014 ... Registration unit 1015 ... ID confirmation unit 1121 ... Passerby registration image database 3011 ... Detection unit 3012 ... Tracking unit 3013 ... Frame selection unit 3014 ... Verification unit 3015 ... ID confirmation unit

Claims (15)

An image matching device in which software is installed in a computer, and this image matching device is
An acquisition means for continuously acquiring images to be collated from an image pickup device, and
A detection means for detecting one or more collation targets from a plurality of input frames included in the collation target image, and
A tracking means for tracking a predetermined collation target based on the detection result of the collation target, or extracting a set of frames including the collation target.
A frame selection means for dynamically selecting one or more matching frames from the plurality of predetermined frames according to the number of conforming frames that meet the criteria among the plurality of predetermined frames including the predetermined collation target.
A collation means for collating a registered image registered in a database with a predetermined collation target image included in the collation frame on a one-to-one or one-to-many basis.
Output means to output the collation result and
An image matching device comprising. The frame selection means selects the collation frame from the plurality of predetermined frames by the first selection method when the number of conforming frames is equal to or more than the specified number, and when the number of conforming frames is less than the specified number. The image collation apparatus according to claim 1, wherein the collation frame is selected from the plurality of predetermined frames by a second selection method. The image matching device according to claim 1 or 2, wherein the frame selecting means selects the matching frames having the specified number or less as the matching frame when the number of matching frames is equal to or more than the specified number. The image matching device according to any one of claims 1 to 3, wherein the frame selecting means selects the plurality of predetermined frames as the matching frame when the number of matching frames is less than a specified number. Any one of claims 1 to 3, wherein the frame selection means selects an extraction frame extracted from the plurality of predetermined frames at regular intervals as the collation frame when the number of conforming frames is less than a specified number. Two image matching devices. Any one of claims 1 to 3, wherein the frame selection means selects an extraction frame extracted from the plurality of predetermined frames based on time when the number of conforming frames is less than the specified number as the matching frame. Two image matching devices. The image matching device according to claim 6, wherein the frame selection means extracts a new or old upper predetermined number of frames based on a time. When the number of matching frames is less than the specified number, the frame selection means groups the frames into frame groups based on the amount of change of the plurality of predetermined frames, and the extracted frames extracted from each frame group are used as the matching frames. The image matching device according to any one of claims 1 to 3 to be selected. The frame selection means determines the number of matching frames determined to meet the criteria based on at least one of the size of the matching target, the score of certainty, the brightness distribution, the position, the orientation, and the matching optimum index. An image matching device according to any one of claims 1 to 8, which dynamically selects the matching frame accordingly. The image collation device according to any one of claims 1 to 9, wherein the collation target is a part or an entire image of a person, an animal, a vehicle, an aircraft, or a drone. The collation means is an image collation device according to any one of claims 1 to 10, which collates the whole body, face, eyes, or gait of a person specified based on the registered image and the predetermined collation target image. The image matching device according to any one of claims 1 to 11 and
A registration device that registers the registered image from a still image or moving image, and
An input device for inputting the collation target image from a moving image, and
An image matching system. The image matching device according to any one of claims 1 to 11 and
A registration device that registers the registered image from the captured image of the traffic area corresponding to the entrance gate, and
An input device that inputs the collation target image from the captured image of the traffic area corresponding to the entry gate, and
An image matching system. The image matching system according to claim 12 or 13, wherein the registration device includes a registration means for registering the registered image and ID information in association with each other. Get the image to be collated and
One or more collation targets are detected from a plurality of input frames included in the collation target image, and the collation target is detected.
A predetermined collation target is tracked based on the detection result of the collation target, and the collation target is tracked.
One or more matching frames are dynamically selected from the plurality of predetermined frames according to the number of conforming frames that meet the criteria among the plurality of predetermined frames including the predetermined collation target.
The registered image registered in the database is collated with the predetermined collation target image included in the collation frame, and the image is collated.
An image matching method that outputs the matching result.

Images