JP2016105293A - User detection device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress interruption of continuous detection of an area corresponding to a user when the luminance of an image to be used for the detection of the area corresponding to a user has rapidly changed.SOLUTION: In the beginning of continuous authentication, color histogram authentication (96) is performed, and while the similarity of the color histogram is equal to or larger than a threshold (Y in 98), the color histogram authentication is continued. When the similarity of the color histogram becomes less than the threshold (N in 98), it is determined if a rapid change in luminance has occurred to an image (102), and if the change in luminance has occurred, the continuous authentication is switched to optical flow authentication (110). While the optical flow authentication is performed, it is determined whether or not the rapid change in luminance of the image has stopped (114 and 116), if the change in luminance is determined to have stopped (Y in 116), the color histogram authentication is resumed.SELECTED DRAWING: Figure 4

Description

  The disclosed technology relates to a user detection device, a user detection method, and a user detection program.

  In recent years, with the rapid spread of information equipment, various kinds of information are held in the information equipment. For this reason, establishment of security technology that suppresses leakage of information held in information devices has become an extremely important issue, and various authentication technologies (for example, password authentication, biometric authentication, card authentication, etc.) It has been proposed and put to practical use. However, many of the existing authentication technologies perform authentication processing only at the time of login. For example, while a legitimate user is away from the installation location of the information device, a third party illegally uses the information device. When used for the above, there is a common problem that this cannot be detected.

  In order to solve the above problems, a continuous authentication technique for continuously performing authentication processing on a user even after login has been proposed. Among the continuous authentication technologies, color histogram authentication that uses the color histogram of the image taken of the user is continuous authentication that is more robust against changes in the user's posture than other methods that use face authentication or keystrokes. It has the feature that it is possible. In color histogram authentication, for example, a color histogram of an area corresponding to the user in the image is registered, and the user corresponding area in the image is detected based on the similarity of the color histogram with the registered color histogram. By following the movement, continuous authentication is realized.

K. Niinuma, U. Park, A. K. Jain, "Soft Biometric Traits For Continuous User Authentication", IEEE Transactions on Information Forensics and Security, Vol. 5, No. 4, pp. 771-780, 2010 P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in Proc. IEEE Computer Vision and Pattern Recognition, pp. I_511-I_518, 2001 B.D.Lucas and T.Kanade, An iterative image registration technique with an application to stereo vision, in Proc.7th IJCAI, Vancouver, B.C., Canada, pp.674-679,1981

  However, in the detection of the user corresponding area using the color histogram, for example, when the brightness of the image used for authentication changes suddenly due to a rapid change in lighting conditions, the color histogram changes and there is a user. In spite of this, it may happen that the user corresponding area cannot be specified. In this case, there are problems that the authentication is interrupted or that the user is erroneously determined to have left the information device.

  An abrupt change in the brightness of an image can be caused by, for example, when indoor lighting is turned on or off while the information device is used indoors, or when the information device is used indoors or when the information device is used indoors. This occurs when the amount of external light incident on the abruptly changes. In addition, for example, when an exposure control function that keeps the brightness of the entire image approximately constant is provided in the imaging unit that captures the user, it is within the imaging range such as a person passing behind the user. Even when the exposure is changed according to the brightness change, the brightness of the user corresponding area changes rapidly. If the shooting unit is a camera built in an information device such as a PC (Personal Computer), and the camera is also used in other applications, the exposure control function is turned off and the user corresponding area It is difficult to suppress a sudden change in luminance.

  An object of the disclosed technology is to prevent the continuous detection of the user corresponding area from being interrupted when the luminance of the image used for the detection of the user corresponding area rapidly changes.

The disclosed technology includes an image acquisition unit that sequentially acquires images captured by the imaging unit. In the first disclosed technique, detection of a user corresponding area corresponding to a user in each of the images based on the similarity of color histograms for the images sequentially acquired by the image acquisition unit is repeated. Includes a detector. In addition, the disclosed technology detects a user corresponding area in each of the images by sequentially comparing feature points existing in the images with respect to the images sequentially acquired by the image acquisition unit. It includes a second detector that repeats. The disclosed technique allows the second detection unit to detect the user corresponding area while the luminance of the images sequentially acquired by the image acquisition unit is changed by a predetermined value or more, and during other periods, that it includes a switching determination unit to perform the detection of the user corresponding regions by the first detection unit.

  The disclosed technology has an effect that the continuous detection of the user corresponding area can be prevented from being interrupted when the brightness of the image used for the detection of the user corresponding area is rapidly changed.

It is a functional block diagram of the authentication apparatus demonstrated in 1st Embodiment. It is a schematic block diagram of the computer which functions as an authentication apparatus. It is a flowchart of an authentication process. It is a flowchart of the continuous authentication process demonstrated in 1st Embodiment. It is a flowchart of an initial registration process. (A) is a color histogram information, (B) is a schematic diagram showing an example of feature point information. It is a flowchart of a color histogram authentication process. It is a flowchart of an optical flow authentication process. It is explanatory drawing for demonstrating discrimination | determination of the correspondence of the feature point in an optical flow authentication process. It is explanatory drawing for demonstrating the calculation of the face area | region in an optical flow authentication process. It is a functional block diagram of the authentication apparatus demonstrated in 2nd and 3rd embodiment. It is a flowchart of the continuous authentication process demonstrated in 2nd Embodiment. (A) is a timing chart showing switching of authentication methods in the first embodiment, and (B) is a second embodiment. It is a timing chart which shows the other example of the switching of the authentication method in 2nd Embodiment. It is a flowchart of the continuous authentication process demonstrated in 3rd Embodiment. It is a flowchart of a combined authentication mode process. It is a timing chart for demonstrating the stop conditions of the optical flow authentication in combined authentication mode. 6 is a timing chart for explaining updating of a reference color histogram in a combined authentication mode.

  Hereinafter, an example of an embodiment of the disclosed technology will be described in detail with reference to the drawings. In the following, an aspect in which the disclosed technique is applied to continuous authentication will be mainly described.

[First Embodiment]
FIG. 1 shows an authentication device 10 according to the first embodiment. The authentication device 10 is a device that performs authentication of a user who uses an information device such as a PC or a mobile terminal, and includes a login authentication device 12 that performs authentication (login authentication) of a user who starts using the information device, and an information device. Is provided with a continuous authentication device 14 for performing authentication (continuous authentication) of a user who is using the device. For example, the authentication device 10 can be built in an information device used by a user (the information device can also function as the authentication device 10). Instead, the authentication device 10 is connected to the information device via a communication line. This can also be realized by causing the computer to function as the authentication device 10. Further, the login authentication device 12 may have a configuration in which any of various known authentication methods (for example, password authentication, biometric authentication, card authentication, etc.) is applied as a user authentication method.

  The continuous authentication device 14 is an example of a user detection device in the disclosed technology, and includes an image acquisition unit 16, a color histogram authentication unit 18, an optical flow authentication unit 20, an authentication switching determination unit 22, a notification unit 24, and a face detection unit 26. I have. In addition, the continuation authentication device 14 includes a first storage unit 30 that stores the color histogram information 28 and a second storage unit 34 that stores the feature point information 32. The color histogram authentication unit 18 is an example of a first detection unit in the disclosed technology, the optical flow authentication unit 20 is an example of a second detection unit in the disclosed technology, and the authentication switching determination unit 22 is in the disclosed technology. It is an example of a switching determination part.

  The image acquisition unit 16 periodically takes the face and part of the user (authenticated person) who uses the information device through login authentication by the login authentication device 12 and the background of the user as subjects. The image data of the image obtained by performing shooting is periodically acquired. For example, when the authentication device 10 is built in the information device, the image acquisition unit 16 may include the above-described photographing unit. For example, when a computer connected to an information device via a communication line functions as the authentication device 10, the image acquisition unit 16 transmits image data of an image photographed by a photographing unit provided in the information device from the information device. It may be a communication unit acquired by receiving.

  The color histogram authentication unit 18 applies the facial region and body corresponding to the user's face existing in the image acquired by the image acquisition unit 16 at the timing immediately after the user succeeds in the login authentication by the login authentication device 12. A color histogram of the corresponding body region is calculated. The color histograms of the face area and body area calculated here are stored in the first storage unit 30 as the color histogram information 28 of the reference color histogram. Further, the color histogram authentication unit 18 performs color histogram authentication (detection of a user corresponding area based on the similarity of the color histograms) on the image acquired by the image acquisition unit 16 using the reference color histogram. That is, for the image acquired by the image acquisition unit 16, the face region and the body region are searched for regions where the similarity between the color histogram and the reference color histogram is maximum. If the similarity of the color histogram in the region extracted in the search is equal to or greater than the threshold, it is determined that the user exists in the extracted region. If the similarity of the color histogram is less than the threshold, the user is informed It is determined that the user has left the equipment.

  At the beginning of authentication, the optical flow authentication unit 20 extracts feature points (for example, edges, preferably edges having luminance changes in multiple directions) from the face area in the image acquired by the image acquisition unit 16, and the extracted features. The point is stored in the second storage unit 34 as the feature point information 32. Further, the optical flow authentication unit 20 performs optical flow authentication (detection of a user corresponding region by comparing feature points) on the image acquired by the image acquisition unit 16 using the feature point information 32 thereafter. That is, from the image acquired by the image acquisition unit 16, feature points are extracted from a past face region and a region including the periphery thereof, and information is stored as feature point information 32 for the extracted feature points. A process for discriminating the correspondence relationship with the feature points is performed. Then, if the number of feature points whose correspondence with past feature points can be determined in the process is greater than or equal to a predetermined number, the user (face) is identified as a face region identified from a predetermined number or more of feature points. Is determined to exist. Also, if the number of feature points whose correspondence with past feature points can be determined is less than a predetermined number (for example, the number of feature points decreased is a predetermined value or more), the user has left the information device. judge.

  The authentication switching determination unit 22 determines whether to apply the color histogram authentication by the color histogram authentication unit 18 or the optical flow authentication by the optical flow authentication unit 20 to the continuous authentication (detection of the user corresponding area) for the user. judge. For the authentication method determination by the authentication switching determination unit 22, at least one of the luminance change of the image acquired by the image acquisition unit 16 and the similarity of the color histogram is used.

  When the continuous authentication for the user is switched from the color histogram authentication to the optical flow authentication, the notification unit 24 uses the optical flow authentication to recognize the face area information recognized by the color histogram authentication by the color histogram authentication unit 18 until then. Notification to the unit 20. Further, when the continuous authentication for the user is switched from the optical flow authentication to the color histogram authentication, the notification unit 24 notifies the color histogram authentication unit 18 of information on the face area recognized by the optical flow authentication so far. To do.

  The face detection unit 26 detects a face region corresponding to the user's face existing in the image acquired by the image acquisition unit 16 at a timing immediately after the user succeeds in login authentication by the login authentication device 12. A process of extracting the detected face area is performed. The face detection unit 26 outputs the detection result of the face area to the color histogram authentication unit 18. The detection result of the face area output from the face detection unit 26 to the color histogram authentication unit 18 is used by the color histogram authentication unit 18 to specify the face area and the body area in the image.

  The authentication device 10 can be realized, for example, by a computer 42 shown in FIG. In the following, a mode in which the computer 42 is included in the information device used by the user will be described, but the computer 42 may be a computer connected to the information device via a communication line. The computer 42 includes a CPU 44, a memory 46, a nonvolatile storage unit 48, a keyboard 50, a mouse 52, a display 54, and a camera 56, which are connected to each other via a bus 58. The camera 56 is an example of a photographing unit, and is arranged so as to be able to photograph a face or part of a user who uses the information device (computer 42) and the background of the user as subjects.

  The storage unit 48 can be realized by an HDD (Hard Disk Drive), a flash memory, or the like. The storage unit 48 as a recording medium stores a login authentication program 60 and a continuous authentication program 62 for causing the computer 42 to function as the authentication device 10, the color histogram information 28, and the feature point information 32. The CPU 44 reads out the login authentication program 60 and the continuous authentication program 62 from the storage unit 48 and expands them in the memory 46, and sequentially executes the processes of the programs 60 and 62.

  The CPU 44 operates as the login authentication device 12 shown in FIG. 1 by executing the process of the login authentication program 60. The continuous authentication program 62 includes an image acquisition process 64, a color histogram authentication process 66, an optical flow authentication process 68, an authentication switching determination process 70, a notification process 72, and a face detection process 74. The CPU 44 operates as the image acquisition unit 16 illustrated in FIG. 1 by executing the image acquisition process 64. The CPU 44 operates as the color histogram authentication unit 18 shown in FIG. 1 by executing the color histogram authentication process 66. The CPU 44 operates as the optical flow authentication unit 20 shown in FIG. 1 by executing the optical flow authentication process 68. Further, the CPU 44 operates as the authentication switching determination unit 22 illustrated in FIG. 1 by executing the authentication switching determination process 70. The CPU 44 operates as the notification unit 24 illustrated in FIG. 1 by executing the notification process 72. The CPU 44 operates as the face detection unit 26 shown in FIG. 1 by executing the face detection process 74. The continuous authentication program 62 is an example of a user detection program in the disclosed technology.

  When the authentication device 10 is realized by the computer 42, the storage unit 48 that stores the color histogram information 28 is used as the first storage unit 30, and a partial region of the memory 46 is used as a storage region of the color histogram information 28. The storage unit 48 that stores the feature point information 32 is also used as the second storage unit 34, and a partial region of the memory 46 is used as a storage region of the feature point information 32. As a result, the computer 42 that has executed the login authentication program 60 and the continuous authentication program 62 functions as the authentication device 10.

  The authentication device 10 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit) or the like.

  Next, the operation of the first embodiment will be described. As explained earlier, color histogram authentication allows continuous authentication that is robust against changes in the posture of the recognition target, but the brightness of the image used for authentication changes suddenly due to a sudden change in lighting conditions, etc. In addition, the continuous authentication may be interrupted. On the other hand, optical flow authentication is robust against sudden changes in the brightness of the image due to changes in lighting, etc., but the number of feature points and correspondences such as the user's face facing sideways, etc. If a posture change that causes a relationship change occurs, it is difficult to continue authentication. Further, the optical flow authentication has a drawback that it has a large processing load and is not suitable for long-time authentication. As described above, in view of the fact that the color histogram authentication and the optical flow authentication have different characteristics, in the present embodiment, the optical flow authentication is applied during the period in which the luminance of the image is changed, and the other periods. Applies color histogram authentication to perform continuous authentication.

  Hereinafter, the authentication process by the authentication apparatus 10 according to the present embodiment will be described first with reference to the flowchart of FIG. The authentication process shown in FIG. 3 is executed when the use of the information device is started by the user. First, in step 80, the login authentication device 12 performs the login authentication process. The authentication method in this login authentication process may be any of various known authentication methods (for example, password authentication, biometric authentication, card authentication, etc.) as described above. In the next step 82, the login authentication device 12 determines whether or not the user has been confirmed to be a regular user by the login authentication.

  If the determination in step 82 is negative, the authentication process ends. This prevents others other than the authorized user from using the information device. On the other hand, if the determination in step 82 is affirmed, the login authentication device 12 activates the continuous authentication device 14 and proceeds to step 84. In step 84, the continuous authentication device 14 performs continuous authentication processing. As a result, a user who has been confirmed to be a legitimate user by login authentication can use the information device while the continuous authentication device 14 is performing the continuous authentication process.

  Next, details of the continuous authentication process performed by the continuous authentication apparatus 14 in step 84 of the authentication process shown in FIG. 3 will be described with reference to FIG. In the continuous authentication process shown in FIG. 4, an initial registration process is first performed at step 90.

  This initial registration process will be described with reference to FIG. 5. First, in step 130, the image acquisition unit 16 acquires image data of the latest image taken by the imaging unit (camera 56). In the next step 132, the face detection unit 26 detects a user's face area present in the image represented by the image data acquired by the image acquisition unit 16. Note that any method such as a method using a Haar classifier (see Non-Patent Document 2) can be applied to the detection of the face area by the face detection unit 26. In the next step 134, the face detection unit 26 determines whether the face area corresponding to the user's face has been extracted by the detection of the face area in step 132. For example, when the user's face is not facing the front of the imaging unit (camera 56) at the time of shooting, the determination in step 134 is negative, the process returns to step 130, and the determination in step 134 is affirmed. Step 130 to Step 134 are repeated.

  On the other hand, if the determination in step 134 is affirmative, the process proceeds to step 136. In step 136, the color histogram authentication unit 18 roughly corresponds to the user's face based on the face region extraction result by the face detection unit 26. A face region having a certain shape (for example, an elliptical shape or another shape) is set. Further, below the set face area, a body area having a fixed shape (for example, a rectangular shape) approximately corresponding to a part of the user's body is set. Note that other methods may be used for setting the body region. In FIG. 6A, an example of the face area set by the color histogram authentication unit 18 is indicated by a solid line, and an example of the body area is indicated by a broken line.

  In the next step 138, the color histogram authentication unit 18 generates a color histogram of the user's face area and body area set in step 136, respectively. For example, when the image data representing the color of each pixel of the image as a combination of R, G, B3 colors is acquired by the image acquisition unit 16, the color histogram generated above is shown in FIG. As shown, a color histogram representing the frequency of each combination of R, G, and B colors is obtained. A color space other than the RGB color space may be used.

  In step 140, the color histogram authentication unit 18 causes the first storage unit 30 to store the color histograms of the user's face area and body area generated in step 138 as reference color histograms (part of the color histogram information 28). In step 142, the color histogram authentication unit 18 stores information representing the position and size of the user's face area set in step 136 in the first storage unit 30 as part of the color histogram information 28. As a result, information as shown in FIG. 6A is stored in the first storage unit 30 as the color histogram information 28. In step 144, the authentication switching determination unit 22 sets the color histogram authentication mode as the continuous authentication operation mode, and ends the initial registration process.

  The “target region” in the color histogram information 28 shown in FIG. 6A has the center of the face region located at the coordinates (200, 100) in the image, the width is 100 pixels, and the height is 150. This represents the size of the pixel. Also, “Histogram (R, G, B)” in the color histogram information 28 shown in FIG. 6A divides a numerical range of 0 to 255 for each of R, G, and B into 16 numerical ranges, The color histogram is represented by the proportion of colors that fall within each numerical range. For example, the histogram (15,15,15) shown in FIG. 6 (A) shows the number of pixels of the color in the numerical range of (R, G, B) = (0,0,0) to (15,15,15). It represents that the ratio to the whole is 0.01 = 1%.

  When the initial registration process described above ends, the process proceeds to step 92 in the continuous authentication process of FIG. In step 92, the image acquisition unit 16 acquires image data of the latest image captured by the imaging unit (camera 56). In the next step 94, the authentication switching determination unit 22 determines whether or not the current operation mode of continuous authentication is the color histogram authentication mode. If the determination in step 94 is affirmative, the authentication switching determination unit 22 calls the color histogram authentication unit 18 and proceeds to step 96. Thus, in step 96, the color histogram authentication unit 18 performs color histogram authentication processing.

  The color histogram authentication process will be described with reference to FIG. 7. First, in step 150, the color histogram authentication unit 18 uses the color histogram information 28 stored in the first storage unit 30 to identify the face area of the user. Get information (position and size). In the next step 152, the color histogram authentication unit 18 uses the face area acquired in step 150 as a reference, and the similarity of the color histogram to the reference color histogram of the face area in the image acquired in the previous step 92. The region where the maximum is (the current face region) is searched. This search is based on the position of the face area acquired in step 150 as a reference, and the similarity of the color histogram is calculated for each of the plurality of candidate areas existing at the reference position and a plurality of positions around the reference position. This can be done by selecting a candidate region having the maximum similarity.

  In the next step 154, the color histogram authentication unit 18 first sets a corresponding body region having a size corresponding to the size of the face region below the face region for which information has been acquired in step 150. Then, using the set body region as a reference, a region (current body region) in which the similarity of the color histogram with respect to the reference color histogram of the body region is maximized is searched for in the image acquired in the previous step 92. In this search as well, the color histogram similarity is calculated for each of a plurality of candidate regions existing at the reference position and a plurality of positions around it based on the position of the body region set in advance, and the similarity is determined from the plurality of candidate regions. This can be done by selecting the candidate region with the maximum degree.

In the next step 156, the color histogram authentication unit 18 determines the final similarity from the similarity of the color histogram of the face region extracted in the search of step 152 and the similarity of the color histogram of the body region extracted in the search of step 154. Calculate the degree. When the similarity of the color histogram of the face region is Sface and the similarity of the color histogram of the body region is Sbody, the final similarity Sfinal can be calculated by the following equation (1).
Sfinal = t × Sface + (1−t) × Sbody (1)
However, t is a weighting factor for the similarity Sface of the color histogram of the face region, and 0 ≦ t ≦ 1.

  In step 158, the color histogram authentication unit 18 stores the extracted information (position and size) of the user's face area and the color histograms of the face area and the body area as color histogram information 28 in the first storage unit 30. The color histogram authentication process is terminated.

  When the above-described color histogram authentication process ends, the process proceeds to step 98 of the continuous authentication process of FIG. In step 98, the authentication switching determination unit 22 determines whether or not the color histogram similarity (final similarity Sfinal) obtained in the color histogram authentication process (step 96) by the color histogram authentication unit 18 is equal to or greater than a threshold value. It is then determined whether or not to continue color histogram authentication. If the similarity between the color histograms is equal to or greater than the threshold value, the determination in step 98 is affirmed and the process proceeds to step 100. The authentication switching determination unit 22 sets the color histogram authentication mode as the operation mode of continuous authentication, and step 92 Return to.

  Thereby, while the similarity of the color histogram obtained by the color histogram authentication process is equal to or greater than the threshold value, Step 92 to Step 100 are repeated, and continuous authentication is performed by color histogram authentication.

On the other hand, when the similarity of the color histogram is less than the threshold value, the determination in step 98 is negative and the process proceeds to step 102. In step 102, the authentication switching determination unit 22 determines whether or not the cause of the similarity of the color histogram being less than the threshold value is that a sudden luminance change has occurred in the entire image, whereby the authentication method for continuous authentication is optical. Authentication method switching determination is performed to determine whether to switch to flow authentication. Specifically, the difference I ₁ (x, y) −I ₂ (x, y) between the luminance values of the corresponding pixels for the images I ₁ and I ₂ continuously acquired by the image acquisition unit 16 is calculated. Then, the number N of pixels whose difference is equal to or greater than the threshold Th is counted. When the number of pixels N is equal to or greater than a certain value, it can be determined that an abrupt luminance change due to large illumination on / off or the like has occurred in the image. Note that the luminance change calculation target is not limited to the entire image. For example, a partial region of the image may be the luminance change calculation target, such as a region excluding the face region and the body region.

  In the next step 104, the authentication switching determination unit 22 switches the authentication method to optical flow authentication and continues the continuous authentication based on whether or not an abrupt luminance change of the image is detected in the authentication method switching determination in step 102. Judge whether to do. If a sudden change in brightness of the image is not detected in the authentication method switching determination in step 102, it can be determined that the cause of the similarity of the color histogram being less than the threshold value is not because of a sudden change in brightness in the image. The determination at step 104 is negative and the routine proceeds to step 124. In step 124, the authentication switching determination unit 22 outputs a determination result that the user has left the table and ends the continuous authentication process.

  In addition, when a sudden brightness change is detected in the authentication method switching determination in step 102, the similarity of the color histogram may have become less than the threshold because the sudden brightness change has occurred in the image. The determination at step 104 is affirmed and the routine proceeds to step 106. In step 106, the authentication switching determination unit 22 switches the continuous authentication operation mode from the color histogram authentication mode to the optical flow authentication mode. In the next step 108, the notification unit 24 extracts face area information (position and size) from the color histogram information 28 stored in the first storage unit 30, and uses the extracted face area information as an optical flow authentication unit. 20 is notified. In the next step 110, the optical flow authentication unit 20 performs an optical flow authentication process.

  In this way, when the similarity of the color histogram is less than the threshold in color histogram authentication and a sudden change in brightness of the image is detected, the rapid change in brightness of the image can be achieved by switching the authentication method for continuous authentication to optical flow authentication. It is possible to suppress the interruption of continuous authentication due to the above.

  The optical flow authentication process in step 110 will be described with reference to FIG. 8. First, in step 160, the optical flow authentication unit 20 has not registered the feature point information 32 in the second storage unit 34 (in the previous cycle, the optical flow authentication process). It was determined whether or not. If the determination in step 160 is affirmed, the process proceeds to step 162. In step 162, the optical flow authentication unit 20 receives the information on the user's face area (from the notification unit 24 in step 108 of FIG. 4) ( Get position and size).

  In the next step 164, the optical flow authentication unit 20 extracts a feature point (for example, an edge or the like) from the face area acquired in step 162 from the image acquired by the image acquisition unit 16. In the next step 172, as shown in FIG. 6B as an example, the optical flow authentication unit 20 uses information (position and size) of the user's face area and information (position) of the feature point as feature point information. 32 is stored in the second storage unit 34. When the process of step 172 is performed, the optical flow authentication process is terminated.

  Note that the “target region” in the feature point information 32 shown in FIG. 6B has the center of the face region located at the coordinates (200, 100) in the image, the width is 100 pixels, and the height is 150. This represents the size of the pixel. Further, the “feature point group” in the feature point information 32 shown in FIG. 6B represents the position of each feature point. For example, the feature point 1 is located at the coordinates (130, 70) in the image. It represents that.

  As described above, when the feature point information 32 is stored in the second storage unit 34, the determination in the previous step 160 is denied in the next cycle, and the process proceeds to step 166. In step 166, the optical flow authentication unit 20 extracts face area information (position and size) from the feature point information 32 stored in the second storage unit 34. Then, the optical flow authentication unit 20 extracts feature points from the extraction target area including the reference position and the surrounding area with reference to the position of the face area from which the information is extracted.

  In the next step 168, the optical flow authentication unit 20 calculates the optical flow (movement vector), and thereby the feature points extracted in step 166 and the features stored as the feature point information 32 in the second storage unit 34. The correspondence between the points is determined (see also FIG. 9). For the calculation of the optical flow, any method such as the method described in Non-Patent Document 3 can be applied.

  In this embodiment, as an example of the optical flow authentication process, an example is described in which feature points are extracted only for face regions and correspondences are determined (same for FIG. 9), but the same applies to body regions. Processing may be performed.

  In the next step 170, the optical flow authentication unit 20 determines the face area of the user based on the distribution of the feature point group for which the correspondence relationship can be determined. The determination of the face area includes, for example, calculating the average position (centroid position) of the feature point group for which the correspondence relationship can be determined, and including all the feature point groups for which the correspondence relationship can be determined with the calculated center position as the center. This can be realized by setting an area having a fixed shape.

In addition, the determination of the face area is not limited to the above. For example, when the optical flow authentication process is first executed, the distance from the center of gravity (center) position of the face area to the feature point, and from the feature point to the face area. It is also possible to determine using the ratio of the distance to the boundary. That is, as shown in FIG. 10, when the optical flow authentication process is first executed, the distance d1 between the center of gravity of the face area and each feature point and the distance d2 between each feature point and the boundary of the face area are stored. If this is done, the distance D from the center of gravity of the face area to the boundary of the face area can be calculated by the following equation (2).
D = e1 × (d1 + d2) / d1 (2)

  In equation (2), e1 is the distance from the gravity center position of the face area to the feature point obtained when determining the face area. Although only a single feature point is shown in FIG. 10, the above processing is performed for all feature points, and after obtaining the average position of the boundary of the face region, the face region is fitted by fitting a region having a fixed shape. Can be determined.

  Further, depending on the actual face position of the user when the feature points are extracted, it may be possible that some of the feature points corresponding to the background region are included in the extracted feature points. In consideration of this, the face area may be determined by adding reliability to each extracted feature point individually. For example, the reliability can be given based on the distance from the center of gravity of the feature point group so that the reliability increases as the distance decreases. When determining the face area, only the feature points with high reliability are used, or the individual feature points are weighted according to the reliability, so that the accuracy of determining the face area can be improved.

  In the next step 172, the optical flow authentication unit 20 uses the information (position and size) of the user's face area and the information (position) of the feature points obtained by the above processing as the feature point information 32 in the second storage unit. 34, and the optical flow authentication process is terminated.

  When the optical flow authentication process described above ends, the process proceeds to step 112 of the continuous authentication process in FIG. In step 112, the authentication switching determination unit 22 determines whether or not to continue the authentication based on the result of the optical flow authentication process. That is, in the optical flow authentication process, if the number of feature points whose correspondence relationship can be determined is less than a predetermined value, the user is likely to have left the seat, so the determination in step 112 is negative and the process proceeds to step 124. Transition. In step 124, the authentication switching determination unit 22 outputs a determination result that the user has left the table and ends the continuous authentication process.

If the determination at step 112 is affirmative, the routine proceeds to step 114. In step 114, the authentication switching determination unit 22 performs authentication method switching determination for determining whether or not to switch the authentication method for continuous authentication to color histogram authentication by determining whether or not the sudden brightness change of the image has subsided. Do. Specifically, the difference I ₁ (x, y) −I ₂ (x, y) between the luminance values of the corresponding pixels for the images I ₁ and I ₂ continuously acquired by the image acquisition unit 16 is calculated. Then, the number N of pixels whose difference is equal to or greater than the threshold Th is counted. When the number of pixels N is less than a certain value, it can be determined that a sudden change in luminance of the image caused by turning on / off a large illumination has been settled. Note that the luminance change calculation target is not limited to the entire image. For example, a partial region of the image may be the luminance change calculation target, such as a region excluding the face region and the body region.

Note that the authentication method switching determination in step 114 may be performed using the similarity of color histograms. That is, the face region color histograms C ₁ and C ₂ are obtained from the images I ₁ and I ₂ successively acquired by the image acquisition unit 16, and when the similarity of these color histograms is sufficiently high, You may make it determine with the rapid brightness | luminance change having settled.

  In the next step 116, the authentication switching determination unit 22 determines whether or not to end the continuous authentication based on the optical flow authentication based on whether or not the rapid brightness change of the image has been settled in the authentication method switching determination in step 114. . If it is determined in step 114 that the sudden change in luminance of the image is not settled, the continuous authentication is interrupted due to the sudden change in luminance of the image when the authentication method for continuous authentication is switched to color histogram authentication. there is a possibility. For this reason, if the determination in step 116 is negative, the process proceeds to step 118, and the authentication switching determination unit 22 sets the optical flow authentication mode as the operation mode of continuous authentication, and returns to step 92.

  As a result, while a sudden luminance change occurs in the image, the determinations in step 94 and step 116 are respectively negated, so that step 92, step 94, step 110 to step 118 are repeated, and optical flow authentication is performed. Continuous authentication is performed.

  On the other hand, as described above, the optical flow authentication is not suitable for long-time authentication because the processing load is large and the robustness with respect to the user's posture change is lower than the color histogram authentication. Further, if the sudden change in luminance of the image is settled, the possibility that the continuous authentication is interrupted by the continuous authentication by the color histogram authentication is also reduced. Therefore, if the determination in step 116 is affirmed, the process proceeds to step 120, and the authentication switching determination unit 22 switches the operation mode of continuous authentication from the optical flow authentication mode to the color histogram authentication mode. In the next step 122, the notification unit 24 extracts face area information (position and size) from the feature point information 32 stored in the second storage unit 34, and uses the extracted face area information as a color histogram authentication unit. 18 and returns to step 92.

  As a result, in the subsequent processing, the determinations in step 94 and step 98 are affirmed, so that steps 92 to 100 are repeated, and continuous authentication by color histogram authentication is resumed.

[Second Embodiment]
Next, a second embodiment of the disclosed technique will be described. In addition, the same code | symbol is attached | subjected to the part same as 1st Embodiment, description is abbreviate | omitted and hereafter, only a different part from 1st Embodiment is demonstrated.

  FIG. 11 shows an authentication device 10 according to the second embodiment. The second embodiment is different from the first embodiment in that the continuous authentication device 14 further includes a past image storage unit 36 and a third storage unit 40 for storing past image information 38. . Each time an image is acquired by the image acquisition unit 16, the past image storage unit 36 stores the image data of the acquired image in the third storage unit 40 as past image information 38. In the second embodiment, an upper limit is set for the number of images that can be stored as the past image information 38 in the third storage unit 40. When the number of images stored in the third storage unit 40 reaches the upper limit, the past image storage unit 36 stores the image data stored in the third storage unit 40 when storing the image data of the new image. Among them, the image data of the image having the oldest storage time is deleted from the third storage unit 40.

  Further, when the authentication device 10 according to the second embodiment is realized by the computer 42 illustrated in FIG. 2, the continuous authentication program 62 according to the second embodiment includes the past image as illustrated by a broken line in FIG. 2. A storage process 76 is further included. In this case, the CPU 44 operates as the past image storage unit 36 illustrated in FIG. 11 by executing the past image storage process 76. When the authentication apparatus 10 according to the second embodiment is realized by the computer 42 illustrated in FIG. 2, the past image information 38 is further stored in the storage unit 48. In this case, the storage unit 48 that stores the past image information 38 is also used as the third storage unit 40, and a partial area of the memory 46 is used as a storage area of the past image information 38.

  Next, the operation of the second embodiment will be described. In the continuous authentication process (FIG. 4) described in the first embodiment, when a sudden change in the luminance of the image occurs during the period of continuous authentication by color histogram authentication, the latest face area is switched to the authentication method. Information was notified to the optical flow authentication unit 20 (step 108). However, there is a slight time lag from when the luminance change of the image starts to occur until it is detected that the similarity of the color histogram becomes less than the threshold value and a sudden change in the luminance of the image has occurred. For this reason, the optical flow authentication unit 20 is also notified of low-accuracy face area information detected after the processing accuracy of the color histogram authentication unit 18 starts to decrease due to the start of image brightness change. There is a possibility (see also FIG. 13A). In this case, the optical flow authentication may be affected and the processing accuracy may be reduced.

  The second embodiment considers the above, and hereinafter, the continuous authentication process according to the second embodiment is different from the continuous authentication process (FIG. 4) described in the first embodiment with reference to FIG. Only the part will be described.

  In the continuous authentication process according to the second embodiment, after the image acquisition unit 16 acquires an image in step 92, the image data of the image acquired by the image acquisition unit 16 is acquired by the past image storage unit 36 in step 180. The past image information 38 is stored in the third storage unit 40. Further, when the authentication method of continuous authentication is switched from color histogram authentication to optical flow authentication (when the determination in step 104 is affirmative), after the authentication switch determination unit 22 switches the operation mode of continuous authentication in step 106, the step Move to 182.

  In the second embodiment, the color histogram authentication unit 18 stores the color histogram information 28 of images for x + 1 frames (where x ≧ 1) in the first storage unit 30. In step 182, the notification unit 24 extracts the facial area information included in the color histogram information 28 of the image x frames before the latest image at the time when the luminance change is detected from the first storage unit 30 and extracts it. The optical flow authentication unit 20 is notified of the information on the face area that has been processed.

  As a result, the optical flow authentication unit 20 detects the face area of the image before x frames from the latest image and the image before x frames from the latest image among the past image information 38 stored in the third storage unit 40. The optical flow authentication is performed based on the image data. As described above, when switching the authentication method from color histogram authentication to optical flow authentication, as shown in FIG. 13B as (1), the information before the processing accuracy of the color histogram authentication unit 18 is lowered is used. Optical flow authentication is started. Therefore, it is possible to avoid a decrease in the processing accuracy of the optical flow authentication due to the influence of the time lag described above.

  In the above, when switching from color histogram authentication to optical flow authentication, optical flow authentication is started using image information (face area information and image data) x frames before the latest image. It is not limited to. For example, as shown in FIG. 14, when a change in luminance is detected, the transition of the similarity of the color histogram up to that point is referred to and before the similarity of the color histogram starts to decrease due to the influence of the luminance change. Optical flow authentication may be started using image information. That is, the number of frames x may be dynamically changed according to the timing at which the luminance change starts to affect the similarity of the color histogram. Thereby, the fall of the processing precision of optical flow authentication can be avoided more reliably.

[Third Embodiment]
Next, a third embodiment of the disclosed technique will be described. Since the third embodiment has the same configuration as that of the second embodiment, the same reference numerals are given to the respective parts, description of the configuration is omitted, and the operation of the third embodiment will be described below. .

  As shown in FIGS. 13A and 13B, in the first and second embodiments described above, when it is detected that a sudden change in luminance of an image has occurred during execution of optical flow authentication, the optical flow is detected. At the same time authentication was stopped, color histogram authentication was restarted. However, if the sudden brightness change of the image occurs again immediately after resuming the color histogram authentication, there is a concern that the continuous authentication is interrupted. As a state in which a sudden change in brightness of an image frequently occurs as described above, for example, a situation where lighting conditions change every moment because a portable information device is used in a moving body such as a train or a car. Etc.

  The third embodiment considers the above, and hereinafter, the continuous authentication process according to the third embodiment is different from the continuous authentication process (FIG. 12) described in the second embodiment with reference to FIG. Only the part will be described.

  In the continuous authentication process according to the third embodiment, after the image data is stored in the third storage unit 40 in step 180, in the next step 184, the authentication switching determination unit 22 determines that the operation mode of continuous authentication is a color histogram. -It is determined whether or not the optical flow combination authentication mode is set. If the determination in step 184 is negative, the process proceeds to step 94, and the processing after step 94 is performed as in the second embodiment.

  Further, in the continuous authentication process according to the third embodiment, when the determination in step 116 is affirmed in a state where the continuous authentication is performed by the optical flow authentication (the optical flow authentication is performed because the sudden change in luminance of the image is settled. If completed, the process proceeds to step 118. In step 188, the authentication switching determination unit 22 switches the operation mode of continuous authentication from the optical flow authentication mode to the color histogram / optical flow combined authentication mode. As a result, the determination in step 184 described above is affirmed, the process proceeds to step 186, and after the combined authentication mode process is performed, the process returns to step 92.

  This combined authentication mode process will be described with reference to FIG. 16. First, in step 190, the color histogram authentication unit 18 performs a color histogram authentication process (see also FIG. 7). In the next step 192, the authentication switching determination unit 22 determines whether or not the similarity of the color histogram obtained by the color histogram authentication process (step 190) by the color histogram authentication unit 18 is equal to or greater than a threshold value. Determine whether to continue authentication. If the similarity of the color histogram is equal to or greater than the threshold, the determination in step 192 is affirmed and the process proceeds to step 194. The authentication switching determination unit 22 continues the state where the similarity of the color histogram is equal to or greater than the threshold for a certain period or more. It is determined whether or not.

  If the determination in step 194 is negative, the process proceeds to step 196, and in step 196, the optical flow authentication unit 20 performs an optical flow authentication process (see also FIG. 8). In the next step 198, the authentication switching determination unit 22 sets the color histogram / optical flow combined authentication mode as the operation mode of continuous authentication, thereby continuing the mode, and temporarily ends the combined authentication mode process. Thus, as shown in FIG. 17 as an example, the color histogram authentication and the optical flow authentication are performed in parallel until the similarity of the color histogram obtained by the color histogram authentication continues for a certain period or longer. Each is executed.

  If the state where the similarity of the color histogram is equal to or higher than the threshold is maintained for a certain period or longer, the determination in step 194 is affirmed and the routine proceeds to step 200. In step 200, the authentication switching determination unit 22 stops the optical flow authentication by the optical flow authentication unit 20 and switches the operation mode of continuous authentication from the color histogram / optical flow combined authentication mode to the color histogram authentication mode. As described above, when it is determined that the sudden change in luminance of the image has subsided, the optical flow authentication is terminated after the state where the similarity of the color histogram is equal to or higher than the threshold value continues for a certain period or more, so the processing load becomes excessive or continued. It is possible to avoid a decrease in the stability of authentication.

  If the similarity of the color histogram becomes less than the threshold before the state where the similarity of the color histogram is equal to or higher than the threshold continues for a certain period or longer, the determination of step 192 is made before the determination of step 194 is affirmed. If the result is negative, the process proceeds to step 202. In step 202, the notification unit 24 extracts face area information (position and size) from the feature point information 32 stored in the second storage unit 34, and sends the extracted face area information to the color histogram authentication unit 18. Notice. In the next step 204, the color histogram authentication unit 18 generates color histograms of the user's face region and body region based on the face region information notified from the notification unit 24.

  In step 206, the color histogram authentication unit 18 stores the color histograms of the user's face area and body area generated in step 204 in the first storage unit 30 as reference color histograms. As a result, the reference color histogram stored in the first storage unit 30 as part of the color histogram information 28 is updated. In step 208, the authentication switching determination unit 22 sets the color histogram / optical flow combined authentication mode as the operation mode for continuous authentication, thereby continuing the mode.

  As described above, during the period in which the color histogram authentication and the optical flow authentication are performed in parallel, a sudden luminance change occurs in the image due to a change in the illumination condition, and the similarity of the color histogram decreases accordingly. In that case, the reference color histogram is updated. Thereby, as shown in FIG. 18, the similarity of the color histogram is restored, and the accuracy of the color histogram authentication is also restored accordingly. After that, when the state where the similarity of the color histogram is equal to or greater than the threshold value continues for a certain period or longer, as described above, the optical flow authentication is stopped and the operation mode of continuous authentication is switched to the color histogram authentication mode. Thus, frequent stop / restart of the optical flow authentication is suppressed, so that it is possible to prevent the continuous authentication from being interrupted when a sudden change in luminance of the image occurs frequently.

  In the above description, when the continuous authentication process is started, the aspect in which the face area extracted by the face detection unit 26 and the color histogram of the body area specified from the face area are registered as the reference histogram has been described. It is not limited to. For example, if the login authentication device 12 is configured to perform face authentication for determining whether or not the user's face corresponds to a regular user's face registered in advance as login authentication, The reference color histogram may be registered based on the face region extraction result. In this case, the configuration of the continuous authentication apparatus 14 can be simplified by omitting the face detection unit 26.

  In the above description, the face detection is performed after the login authentication is completed, and the reference color histogram is registered immediately after the face area is detected. However, the present invention is not limited to this. For example, during the continuous authentication by color histogram authentication, the detection of the face area and the reference periodically or at the timing when a gradual brightness change of the image due to a change in the solar radiation angle is detected, etc. The color histogram may be updated. Also, the part for registering the reference color histogram is not limited to the user's face area and body area. For example, only one of the face area and body area may be registered, or a part of the face or body may be registered. Only the color histogram may be registered.

  In the above description, after the login authentication, the aspect in which the disclosed technology is applied to the continuous authentication for authenticating whether the same user who has undergone the login authentication is continuously using the information device has been described. The technique is not limited to the above embodiment. For example, video (moving image) and audio playback and application programs are executed only while the user is using the information device, and video is played while the user is away from the installation location of the information device. In the aspect of stopping the above, the present invention may be applied to detection of the presence / absence of the user.

  In the above description, the continuous authentication program 62 is stored (installed) in the storage unit 48 in advance. However, the continuous authentication program 62 is recorded on a recording medium such as a CD-ROM or a DVD-ROM. Can also be provided.

  All documents, patent applications and technical standards mentioned in this specification are to the same extent as if each individual document, patent application and technical standard were specifically and individually stated to be incorporated by reference. Incorporated by reference in the book.

DESCRIPTION OF SYMBOLS 10 Authentication apparatus 14 Continuous authentication apparatus 16 Image acquisition part 18 Color histogram authentication part 20 Optical flow authentication part 22 Authentication switching determination part 24 Notification part 26 Face detection part 30 1st memory | storage part 34 2nd memory | storage part 36 Past image preservation | save part 40 1st 3 storage unit 42 computer 44 CPU
48 storage unit 56 camera

Claims (16)

An image acquisition unit that sequentially acquires images taken by the imaging unit;
A first detection unit that repeatedly detects a user corresponding region corresponding to a user in each of the images based on the similarity of color histograms for images sequentially acquired by the image acquisition unit;
A second detector that repeats detecting user-corresponding areas in each of the images by sequentially comparing feature points in the images of the images sequentially acquired by the image acquisition unit; ,
The second detection unit detects the user corresponding area while the luminance of the images sequentially acquired by the image acquisition unit changes by a predetermined value or more, and the other detection period is used by the first detection unit. A switching determination unit that performs detection of the person corresponding area;
A user detection device including:   The switching determination unit calculates the difference value of the luminance of the images sequentially acquired by the image acquisition unit, and when the difference value of the luminance of the whole image or a part of the image becomes a predetermined value or more, the first detection unit The user detection apparatus according to claim 1, wherein the detection is switched from detection of a user corresponding area to detection of a user corresponding area by the second detection unit.   The switching determination unit determines whether the luminance of images sequentially acquired by the image acquisition unit has changed by a predetermined value or more when the similarity of the color histogram is determined to be less than a threshold value by the first detection unit. The user detection device according to claim 2, wherein the user detection device is determined.   The switching determination unit continues a state where the similarity of the color histogram of the user corresponding region in the image is equal to or higher than a threshold during a period in which the user corresponding region is detected by the second detection unit. 4. The user detection device according to claim 1, wherein the user detection area is switched from detection of the user corresponding area by the second detection unit to detection of the user corresponding area by the first detection unit.   The switching determination unit calculates the difference value of the luminance of the images sequentially acquired by the image acquisition unit, and when the difference value of the luminance of the whole image or a part of the image becomes less than a predetermined value, the second detection unit The user detection device according to claim 1, wherein the detection is switched from detection of a user corresponding area to detection of a user corresponding area by the first detection unit. A first storage unit that stores a color histogram of the user corresponding area in the image acquired by the image acquisition unit as a reference color histogram;
The first detection unit calculates a similarity of a color histogram of a plurality of regions in the image acquired by the image acquisition unit with respect to the reference color histogram, and the similarity is equal to or greater than a threshold among the plurality of regions. The user detection device according to claim 1, wherein the area is determined as a user corresponding area in the image. The user detected by the first detection unit when the detection of the user corresponding region by the second detection unit is started from the state in which the user detection region is detected by the first detection unit. A notification unit for notifying the position information of the corresponding area to the second detection unit;
The user detection device according to claim 6, wherein the second detection unit detects a user corresponding area based on a feature point existing in an area specified by the position information notified from the notification unit. A second storage unit that stores information on feature points extracted by the second detection unit from within the user corresponding area in the image acquired by the image acquisition unit;
The second detection unit extracts a feature point from the image newly acquired by the image acquisition unit, and determines a correspondence relationship between the extracted feature point and the feature point whose information is stored in the second storage unit And determining a region identified from the feature point determined to correspond to the feature point whose information is stored in the second storage unit as a user corresponding region in the image. The user detection device according to any one of the preceding claims. The user detected by the second detection unit when the detection of the user corresponding region by the first detection unit is started from the state in which the user corresponding region is detected by the second detection unit. A notification unit for notifying the first detection unit of position information of the corresponding region;
The user detection device according to claim 8, wherein the first detection unit detects a user-corresponding region based on a color histogram in a region specified by position information notified from the notification unit. A past image storage unit for storing images acquired in the past by the image acquisition unit in a third storage unit;
When the detection of the user corresponding area by the second detection unit is started from the state in which the user corresponding area is detected by the first detection unit, the second detection unit is configured to store the third storage unit. The user detection device according to claim 1, wherein detection of a user corresponding area is started using an image stored in the image. The first detection unit calculates a similarity of a color histogram of a plurality of regions in the image acquired by the image acquisition unit with respect to the reference color histogram, and the similarity is equal to or greater than a threshold among the plurality of regions. Is determined as a user-corresponding area in the image,
When the user detection area is detected by the second detection unit from the state where the user detection area is detected by the first detection unit, the user detection area is detected by the first detection unit. And a notification unit for notifying the second detection unit of the position information of the region when the region having the similarity equal to or greater than a threshold exists.
When the detection of the user corresponding area by the second detection unit is started from the state in which the user corresponding area is detected by the first detection unit, the second detection unit is configured to store the third storage unit. The detection of the user corresponding area is started by using the image when the area having the similarity equal to or higher than the threshold is detected by the detection of the user corresponding area by the first detection unit. Item 10. The user detection device according to Item 10.   In the state where the user detection area is detected by the second detection unit, the switching determination unit continues a state where the similarity of the color histogram in the user corresponding area of the image is equal to or more than a threshold for a certain period or more. Or when the difference value of the luminance of the images sequentially acquired by the image acquisition unit becomes less than a predetermined value, the detection of the user corresponding area by the first detection unit is started and the first The detection of the user corresponding area by the second detection unit is continued until a predetermined period elapses after the detection of the user corresponding area by the detecting unit is started. 12. The user detection device described.   The switching determination unit determines that the similarity of the color histogram is greater than or equal to a threshold value by the first detection unit within a period in which the first detection unit and the second detection unit each detect a user corresponding area. The user detection device according to claim 12, wherein when the state that has been performed continues for a certain period or longer, the detection of the user corresponding area by the second detection unit is stopped. A first storage unit that stores a color histogram of the user corresponding area in the image acquired by the image acquisition unit as a reference color histogram;
When the first detection unit determines that the similarity of the color histogram is less than a threshold value within a period in which each of the first detection unit and the second detection unit detects a user corresponding area, A notification unit for notifying the first detection unit of positional information of the user corresponding area detected by the second detection unit;
The first detection unit calculates the similarity of the color histogram using the reference color histogram, and the first detection unit and the second detection unit each detect a user corresponding area. In addition, when the position information is notified by the notification unit, a color histogram in an area specified by the notified position information is stored in the first storage unit as the reference color histogram. Item 14. The user detection device according to Item 13. An image acquisition step for sequentially acquiring images captured by the imaging unit;
A first detection step for repeatedly detecting a user corresponding area corresponding to a user in each of the images based on the similarity of color histograms for the images sequentially acquired by the image acquisition step;
A second detection step of repeatedly detecting a user corresponding region in each of the images by sequentially comparing feature points existing in each of the images sequentially acquired by the image acquisition step; ,
While the brightness of the images sequentially acquired by the image acquisition step changes by a predetermined value or more, the user corresponding area is detected by the second detection step, and other periods are used by the first detection step. Switching determination step for detecting the person corresponding area;
User detection method including On the computer,
An image acquisition step for sequentially acquiring images captured by the imaging unit;
A first detection step for repeatedly detecting a user corresponding area corresponding to a user in each of the images based on the similarity of color histograms for the images sequentially acquired by the image acquisition step;
A second detection step of repeatedly detecting a user corresponding region in each of the images by sequentially comparing feature points existing in each of the images sequentially acquired by the image acquisition step; ,
While the brightness of the images sequentially acquired by the image acquisition step changes by a predetermined value or more, the user corresponding area is detected by the second detection step, and other periods are used by the first detection step. Switching determination step for detecting the person corresponding area;
A user detection program for executing processing including