JP6927495B2 - Person evaluation equipment, programs, and methods - Google Patents

Description

The present invention relates to a person evaluation device used for evaluating a person, a person evaluation program and a person evaluation method used in such a person evaluation device, and the like.

For example, when a company hires a new employee, the hiring manager spends a great deal of effort and time in order to evaluate the individuality of a large number of applicants and select the applicants to be hired. In addition, the evaluation criteria may differ depending on the hiring manager, and it is required to establish objective evaluation criteria for evaluating the applicant's personality. Therefore, if information that can be used as a reference when evaluating an applicant's person can be obtained based on the applicant's audio or moving image, the selection work of the hiring manager will be streamlined and the objectivity of the evaluation criteria will be improved. Can be made to.

On the other hand, even when communicating with an unspecified number of people via telephone or the Internet, there is a demand for a tool that can judge whether or not the communication partner can be trusted. As a related technique, Patent Document 1 discloses a person evaluation device for users to communicate with peace of mind in a community without a mechanism for determining whether or not a communication partner can be trusted.

This person evaluation device is a dictionary construction means for constructing a dictionary in which a feature word included in the learning data and a score for the feature word are associated with each evaluation content according to a unified evaluation scale, and for learning. By referring to the data (for example, articles) of a plurality of users constituting the data and the above dictionary, it is provided with an evaluation means for evaluating each user based on the identification information and the basic value of the plurality of users.

In the dictionary, feature words in learning data and scores according to their frequency of appearance are registered for each evaluation content using a unified evaluation scale. Further, the evaluation of the data constituting the learning data is performed based on the user's identification information while referring to the above dictionary. Therefore, according to Patent Document 1, it is possible to perform highly accurate evaluation not only on a data-by-data basis but also on a person-by-person basis with respect to a user of the Internet community in which data input by the user is collected.

Japanese Unexamined Patent Publication No. 2006-190196 (paragraph 0002-0008, FIG. 1)

In order to evaluate a person in Patent Document 1, articles and the like created by a plurality of users are required. However, for example, if the article created by the user does not reflect the user's original thoughts and thoughts, or if the user plagiarizes another person's article when creating the article, accurate person evaluation is performed. Can't.

Therefore, in view of the above points, a first object of the present invention is to obtain a subject's voice based on the voice of the subject to be evaluated, or based on the moving image and voice of the subject. It is to provide a person evaluation device that can provide information that can be used as a reference when performing a person evaluation. A second object of the present invention is to provide a person evaluation program, a person evaluation method, and the like used in such a person evaluation device.

In order to solve at least a part of the above problems, the person evaluation device according to the first aspect of the present invention converts the voice data obtained by recording the voice of the subject into Fourier transform for each data block per unit time. Then, the voice processing unit that generates voiceprint data representing the sound pressure distribution in a plurality of frequency bands for each data block and the data blocks are classified according to the magnitude and spread of the sound pressure in the plurality of frequency bands, and a predetermined number of data blocks are classified. When evaluating the voice of a subject based on the classification result of the data block, the sound pressure is below the threshold in all frequency regions for the data block whose sound pressure exceeds the threshold in any frequency region. For a data block that gives a score higher than the score of the data block and the number of frequency bands where the sound pressure exceeds the threshold and reaches the maximum exceeds a predetermined value, the frequency band where the sound pressure exceeds the threshold and reaches the maximum With a voice analysis unit that gives a score higher than the score of the data block whose number is less than or equal to a predetermined value and determines the rank of the subject's voice based on the total value or the average value of the scores of the predetermined number of data blocks. To be equipped with.

Further, in the person evaluation program according to the first aspect of the present invention, the voice data obtained by recording the voice of the subject is Fourier-converted for each data block per unit time, and a plurality of frequency bands for each data block. Data blocks are classified according to the procedure (a) for generating voiceprint data representing the sound pressure distribution in the above and the magnitude and spread of sound pressure in a plurality of frequency bands, and the data blocks are classified based on the classification result of a predetermined number of data blocks. When evaluating the examiner's voice, for a data block whose sound pressure exceeds the threshold in any frequency region, a score higher than the score of the data block whose sound pressure is below the threshold in all frequency regions is given. Data in which the number of frequency bands in which the sound pressure exceeds the threshold and becomes the maximum is equal to or less than the predetermined value for a data block in which the number of frequency bands in which the sound pressure exceeds the threshold and becomes the maximum exceeds a predetermined value. A score higher than the score of the block is given, and the CPU is made to execute the procedure (b) of determining the rank of the subject's voice based on the total value or the average value of the scores of a predetermined number of data blocks.

Further, in the person evaluation method according to the first aspect of the present invention, the voice data obtained by recording the voice of the subject is Fourier-converted for each data block per unit time, and a plurality of frequency bands for each data block. Data blocks are classified according to the step (a) of generating voiceprint data representing the sound pressure distribution in the above and the magnitude and spread of sound pressure in a plurality of frequency bands, and the data blocks are classified based on the classification result of a predetermined number of data blocks. When evaluating the examiner's voice, for a data block whose sound pressure exceeds the threshold in any frequency region, a score higher than the score of the data block whose sound pressure is below the threshold in all frequency regions is given. Data in which the number of frequency bands in which the sound pressure exceeds the threshold and becomes the maximum is equal to or less than the predetermined value for a data block in which the number of frequency bands in which the sound pressure exceeds the threshold and becomes the maximum exceeds a predetermined value. A step (b) is provided in which a score higher than the score of the block is given, and the rank of the subject's voice is determined based on the total value or the average value of the scores of a predetermined number of data blocks.

According to the first aspect of the present invention, voiceprint data representing the sound pressure distribution in a plurality of frequency bands is generated for each data block from the voice data obtained by recording the voice of the subject, and the plurality of frequency bands. Data blocks are classified according to the magnitude and spread of the sound pressure in, and when the voice of the subject is evaluated based on the classification result of a predetermined number of data blocks, the test is performed based on the score given to the data blocks. By determining the rank of the person's voice, it is possible to provide information that can be used as a reference when evaluating the person of the subject based on the voice of the subject to be evaluated.

The person evaluation device according to the second aspect of the present invention recognizes each frame of the moving image data obtained by imaging the face of the subject in the person evaluation device according to the first aspect of the present invention. By performing the processing, a plurality of feature points recognized on the subject's face are extracted, and the image processing unit for obtaining the coordinates of the plurality of feature points and the coordinates of the plurality of feature points in a predetermined number of frames are used as the basis for the processing. Evaluation results by the image analysis unit and the voice analysis unit that calculate the amount of facial movement of the subject and visually evaluate the subject based on the statistical processing of the amount of facial movement of the subject during the evaluation period. It is further provided with a comprehensive evaluation unit that evaluates the person of the subject based on the evaluation result by the image analysis unit.

In addition, the person evaluation program according to the second aspect of the present invention is the person evaluation program according to the first aspect of the present invention for each frame with respect to the moving image data obtained by imaging the face of the subject. The procedure (c) of extracting a plurality of feature points recognized on the subject's face by performing face recognition processing and obtaining the coordinates of the plurality of feature points, and the coordinates of the plurality of feature points in a predetermined number of frames. The procedure (d) and the procedure (d) in which the amount of movement of the subject's face is calculated based on the above and the visual evaluation of the subject is performed based on the statistical processing of the amount of movement of the subject's face during the evaluation period. Further, the CPU is made to execute the procedure (e) for evaluating the person of the subject based on the evaluation result in b) and the evaluation result in the procedure (d).

Further, the person evaluation method according to the second aspect of the present invention is the person evaluation method according to the first aspect of the present invention for each frame with respect to the moving image data obtained by imaging the face of the subject. By performing face recognition processing, a plurality of feature points recognized on the subject's face are extracted, and the step (c) of obtaining the coordinates of the plurality of feature points and the coordinates of the plurality of feature points in a predetermined number of frames are obtained. The step (d) and the step (d) in which the amount of movement of the subject's face is calculated based on the above and the visual evaluation of the subject is performed based on the statistical processing of the amount of movement of the subject's face during the evaluation period. Further, the step (e) for evaluating the person of the subject based on the evaluation result in b) and the evaluation result in step (d) is further provided.

According to the second aspect of the present invention, the coordinates of a plurality of feature points recognized on the subject's face are obtained from the moving image data obtained by imaging the subject's face, and the coordinates are obtained in a predetermined number of frames. By calculating the amount of movement of the subject's face based on the coordinates of multiple feature points and performing a visual evaluation of the subject based on the statistical processing of the amount of movement of the subject's face during the evaluation period. , Information that can be used as a reference when evaluating a subject's person can be provided based on the moving image and sound of the subject to be evaluated.

The block diagram which shows the structural example of the person evaluation apparatus which concerns on one Embodiment of this invention. The figure which shows the example of the voice waveform represented by the voice data. The figure which shows the example of the sound pressure distribution represented by the voiceprint data. The figure for demonstrating the evaluation example of the voice based on the voiceprint data. The figure which shows the example of the image used for calculating the amount of movement of the face of a subject about the 1st axis as a rotation center. The figure for demonstrating the change of the height ratio of the 1st triangle and the 2nd triangle by the movement of the subject's face about the 1st axis of rotation. The figure which shows the example of the image used for calculating the amount of movement of a subject's face about the 2nd axis as a rotation center. The figure for demonstrating the change of the area ratio between the 1st triangle and the 2nd triangle by the movement of the subject's face about the 2nd axis as a rotation center. The figure which shows the example of the image used for calculating the amount of movement of a subject's face about the 3rd axis as a rotation center. The figure which shows the example of the probability distribution of the variance value of the quantity which shows the direction of the face of a subject in the evaluation period. The figure which shows the example of the mapping area used for person evaluation of a subject. The flowchart which shows the person evaluation method which concerns on one Embodiment of this invention. A flowchart showing an example of a processing flow of moving image data (first half). A flowchart showing an example of a processing flow of moving image data (second half).

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The same components are designated by the same reference numerals, and duplicate description will be omitted.
<Person evaluation device>
FIG. 1 is a block diagram showing a configuration example of a person evaluation device according to an embodiment of the present invention. As the person evaluation device, for example, a personal computer, a tablet terminal, a smartphone, or the like can be used. In the following, as an example, a case where a personal computer is used as a person evaluation device will be described.

As shown in FIG. 1, the person evaluation device includes an operation unit 10, a display unit 20, an input / output interface 30, a network interface 40, a CPU (central processing unit) 50, a memory 60, and a storage unit 70. And is included. The input / output interfaces 30 to the storage unit 70 are connected to each other via a bus line. A part of the component shown in FIG. 1 may be omitted or changed, or another component may be added to the component shown in FIG.

The operation unit 10 is composed of a keyboard, a mouse, and the like, and is used for inputting various commands and data. The display unit 20 includes, for example, an LCD (liquid crystal display) and the like, and displays an operation screen, an evaluation screen, and the like. The input / output interface 30 is connected to the operation unit 10 and the display unit 20, supplies various commands and data input using the operation unit 10 to the CPU 50 or the memory 60, and outputs the display data generated by the CPU 50. It is supplied to the display unit 20.

Further, the input / output interface 30 can input audio data or moving image data from the outside, and serially transfer data to and from a peripheral device such as a USB (universal serial bus) memory. Further, the input / output interface 30 may include an analog / digital converter that converts an analog audio signal or image signal into digital audio data or moving image data.

The network interface 40 connects the CPU 50 to a network such as a LAN or the Internet. The CPU 50 performs various calculations and data processing according to the software stored in the storage unit 70. The memory 60 temporarily stores various instructions and data supplied from the input / output interface 30, data supplied from the network interface 40, data generated or processed by the CPU 50, and the like.

The storage unit 70 stores various data, various software for causing the CPU 50 to operate, and the like in a recording medium. As the recording medium, in addition to the built-in hard disk, an external hard disk, a flexible disk, MO, MT, CD-ROM, DVD-ROM, various memories, or the like can be used.

Here, the voice processing unit 51, the voice analysis unit 52, the image processing unit 53, the image analysis unit 54, and the comprehensive evaluation unit 55 are used as functional blocks by the CPU 50 and software (including the person evaluation program). It is composed.

The person evaluation device shown in FIG. 1 is supplied with voice data obtained by recording the voice of a subject to be evaluated. Alternatively, moving image data obtained by imaging the subject's face is supplied together with audio data. In that case, the moving image data and the audio data may be combined. Alternatively, the analog / digital converter of the input / output interface 30 may convert the analog audio signal or image signal supplied to the person evaluation device into digital audio data or moving image data.

For example, voice data or moving image data obtained by using a microphone, a mobile phone (smartphone or the like), a tablet terminal, a video camera, or Skype (Skype) or the like may be supplied to the person evaluation device in real time. Alternatively, the voice data or the moving image data recorded in advance in the voice recorder, the video movie, or the like may be supplied to the person evaluation device by batch processing.

<Voice processing>
The audio data or moving image data supplied from the input / output interface 30 or the network interface 40 or the like is stored in the raw data storage unit 71. The voice processing unit 51 reads the voice data from the raw data storage unit 71 and acquires the voice data.

FIG. 2 is a diagram showing an example of a voice waveform represented by voice data. In FIG. 2, the horizontal axis represents time [seconds], and the vertical axis represents the amplitude of the voice waveform. For example, for voice evaluation, voice data representing voice in an evaluation period of 30 seconds from 5 seconds to 35 seconds after the start of recording is used. The amplitude of the voice waveform in the voice data may be normalized based on the peak value or the like.

The voice processing unit 51 shown in FIG. 1 Fourier transforms the voice data obtained by recording the voice of the subject to be evaluated as a person for each data block per unit time (for example, about 0.07 seconds). , Voiceprint data representing the sound pressure distribution in a plurality of frequency bands is generated for each data block. The data block of audio data generally corresponds to a frame of audio data, but in the present application, the term data block is used for audio data in order to distinguish it from a frame of image data. Be done.

FIG. 3 is a diagram showing an example of the sound pressure distribution represented by the voiceprint data. In FIG. 3, the horizontal axis is the time axis representing 2 × time [seconds], and the vertical axis is the frequency axis representing the frequency in logarithmic representation. In FIG. 3, the brightness in each frequency region represents the sound pressure [dB], and the higher the sound pressure in the frequency region, the closer to white it is displayed. Alternatively, a three-dimensional display in which the sound pressure axis orthogonal to the time axis and the frequency axis represents the sound pressure [dB] may be used.

Here, an example of a method for generating voiceprint data based on voice data will be described. The voice processing unit 51 shown in FIG. 1 divides time-series voice data at predetermined time intervals by applying a humming window to the voice waveform represented by the voice data, and divides a plurality of data blocks along the time axis. create. For example, if the sampling frequency is about 44 kHz, one data block contains 2048 samples of audio data. It should be noted that each of the two consecutive data blocks may include a plurality of overlapping samples.

Next, the voice processing unit 51 extracts a plurality of frequency components by Fourier transforming the voice data for each data block. For example, the voice processing unit 51 may perform a fast Fourier transform (FFT) process on the voice data. Since the frequency component obtained by the Fourier transform is a complex number, the voice processing unit 51 obtains the absolute value of each frequency component.

The voice processing unit 51 integrates the absolute values of those frequency components by multiplying the windows of the frequency domain for each octave or the windows of the frequency domain determined based on the mel scale (perceptual scale of pitch). As a result, the integrated value in the frequency band of each window is obtained, and further, the logarithm of the integrated value is taken to obtain the sound pressure [dB]. As a result, if there are 20 windows in the frequency domain, sound pressure in the 20 frequency bands can be obtained.

<Voice analysis>
The voice processing unit 51 stores the voiceprint data generated in this way in the voiceprint data storage unit 72. The voice analysis unit 52 reads the voiceprint data from the voiceprint data storage unit 72, classifies the data blocks according to the magnitude and spread of the sound pressure in a plurality of frequency bands, and based on the classification result of a predetermined number of data blocks. Evaluate the subject's voice.

For example, the voice analysis unit 52 classifies the data blocks according to whether or not the sound pressure exceeds the threshold in any frequency region, and further classifies the data blocks in which the sound pressure exceeds the threshold in any frequency region. Classify according to the number of frequency bands in which the sound pressure exceeds the threshold and becomes maximum.

FIG. 4 is a diagram for explaining an example of voice evaluation based on voiceprint data. 4 (A) to 4 (D) show an example of the sound pressure distribution represented by four types of voiceprint data. In FIGS. 3 and 4, the black frequency region in each data block is a frequency region in which the sound pressure is equal to or less than a threshold value (for example, 15 dB), and it is determined that the frequency component is silent or noise.

As shown in FIGS. 4 (A) and 4 (B), when the subject speaks fluently and there is little interruption in voice, the proportion of data blocks in which the sound pressure exceeds the threshold value in any frequency region is high. growing. In particular, as shown in FIG. 4 (A), in the case of a voice quality in which the overtones are richly extended, the voiceprint has many striped patterns, the outline is clear, and the voice quality passes well, the sound pressure sets a threshold value. The frequency band that exceeds the maximum becomes large.

On the other hand, as shown in FIGS. 4 (C) and 4 (D), when the subject is clogged with words and the voice tends to be interrupted, the sound pressure of the data block is below the threshold value in all frequency regions. The ratio increases. In particular, as shown in FIG. 4 (D), in the case of a dark and muffled voice quality such as a blurred outline, the sound pressure is insufficient in the voice of the subject due to insufficient extension of overtones and few stripes of the voiceprint. Exceeds the threshold and the maximum frequency band is reduced.

Therefore, the voice analysis unit 52 gives a score S0 to the data blocks whose sound pressure is equal to or lower than the threshold value in all frequency regions, and all the data blocks whose sound pressure exceeds the threshold value in any frequency region. A score higher than the score S0 of the data block whose sound pressure is below the threshold in the frequency domain is given.

Further, the voice analysis unit 52 obtains the number of frequency bands in which the sound pressure exceeds the threshold value and becomes maximum for the data block in which the sound pressure exceeds the threshold value in any of the frequency regions. Referring to FIGS. 3 and 4, when the brightness of a certain frequency domain is brighter than the brightness of the frequency domains on both the upper and lower sides of each data block, the sound pressure is maximized in that frequency domain.

Alternatively, when a three-dimensional display is used in which the sound pressure axis orthogonal to the time axis and the frequency axis represents the sound pressure [dB], in each data block, the sound pressure in a certain frequency region is the sound in the frequency regions on both the upper and lower sides thereof. When the sound pressure is higher than the pressure and the sound pressure is convex toward the high sound pressure side, the sound pressure is maximized in the frequency region.

The voice analysis unit 52 refers to a data block in which the number of frequency bands in which the sound pressure exceeds the threshold and becomes maximum is equal to or less than a predetermined value among the data blocks in which the sound pressure exceeds the threshold in any of the frequency regions. A score S1 is given, and a score S2 higher than the score S1 is given to a data block in which the number of frequency bands in which the sound pressure exceeds the threshold value and becomes maximum exceeds a predetermined value.

Next, the voice analysis unit 52 determines the rank of the subject's voice based on the total value or the average value of the scores of a predetermined number of data blocks. For example, using the number N0 of data blocks with a score S0, the number N1 of data blocks with a score S1, and the number N2 of data blocks with a score S2, the average value AVE of the scores of a predetermined number (N) of data blocks. Is expressed by the following equation (1).
AVE = (S0 x N0 + S1 x N1 + S2 x N2) / N ... (1)
Here, N0 to N2 are integers of zero or more, N is an integer of 3 or more, and the following equation (2) holds.
N = N0 + N1 + N2 ... (2)
For example, in the formula (1), S0 = 0, S1 = 1, and S2 = 3 to 5 may be set.

The voice analysis unit 52 may compare the total value or the average value of the scores of a predetermined number of data blocks with at least one preset reference value to determine the rank of the subject regarding the voice. Therefore, in the learning data storage unit 73, for example, the voice data obtained by recording the voice of a pseudo subject such as an internship applicant, the rank actually evaluated by the evaluator, and the like, etc. The evaluation data representing the above is stored in advance as the determination learning data. The voice analysis unit 52 sets at least one reference value so that a judgment result close to the judgment learning data can be obtained by functioning as an AI (artificial intelligence) that performs machine learning using the judgment learning data. The rank of the examiner's voice may be determined.

For example, the voice analysis unit 52 compares the average value AVE with the first to third reference values when evaluating the voice of the subject in four stages. When the average value AVE is equal to or less than the first reference value, the voice analysis unit 52 determines that the subject's voice is rank RA0 (extremely bad voice), and the average value AVE is larger than the first reference value. When it is equal to or less than the second reference value, the voice of the subject is determined to be rank RA1 (bad voice). Further, the voice analysis unit 52 determines that the voice of the subject is rank RA2 (ordinary voice) when the average value AVE is larger than the second reference value and equal to or less than the third reference value, and the average value. When the AVE is larger than the third reference value, the voice of the subject is determined to be rank RA3 (good voice).

If the voice data corresponding to the evaluation period does not reach a certain period (for example, 15 seconds), the voice analysis unit 52 ranks the subject's voice at rank RA0. May be determined. The voice analysis unit 52 stores the voice evaluation data representing the voice evaluation result thus obtained in the evaluation data storage unit 74.

<Image processing>
When the moving image data is stored in the raw data storage unit 71, the image processing unit 53 reads the moving image data from the raw data storage unit 71 to acquire the moving image data. For example, the moving image data represents an image of 24 frames per second, and for visual evaluation, moving image data representing an image in an evaluation period of 55 seconds from 5 seconds to 60 seconds after the start of imaging is used. ..

The image processing unit 53 extracts a plurality of feature points recognized on the subject's face by performing face recognition processing on each frame for the moving image data obtained by imaging the subject's face. , Find the coordinates of multiple feature points. Here, an example of face recognition processing, which is a kind of image processing, will be described.

First, the image processing unit 53 detects the position of the subject's face in an image represented by one frame of moving image data (hereinafter, also referred to as an “input image”). For example, the image processing unit 53 can detect the position, region, and the like of the face of the subject by using software such as OpenCV.

Next, the image processing unit 53 recognizes the face of the subject in the input image by using the moving image data of one frame and the face recognition learning data stored in advance in the learning data storage unit 73. In this face recognition process, for example, an active appearance model (AAM) is used. After that, the image processing unit 53 determines whether or not the face of the subject can be recognized.

When the face of the subject can be recognized, the image processing unit 53 extracts a plurality of feature points recognized in the face of the subject and obtains the coordinates of the plurality of feature points in the input image. Further, the image processing unit 53 stores the coordinates of the plurality of feature points in the coordinate data storage unit 75 together with the frame number. The image processing unit 53 may obtain the coordinates of a plurality of feature points in the input image as pixel numbers.

<Details of face recognition processing>
In the learning data storage unit 73, for example, image data representing an image taken in advance using a standard human face or a model thereof, and coordinates of a plurality of feature points set in the image are stored in the face. It is stored in advance as recognition learning data. The image processing unit 53 extracts a plurality of feature points from the face of the subject by performing face recognition processing on the image data obtained by imaging the face of the subject based on the face recognition learning data. Find the coordinates of those feature points.

The active appearance model that can be used in the above face recognition process divides the image of the target object into a shape and an appearance, and each of them is subjected to principal component analysis. By dimensional compression, it is a model that can express changes in the shape and texture of an object with a small number of parameters. Shape and texture information can be represented by low-dimensional parameters.

In the active appearance model, the shape vector x in which all the feature points are arranged is an eigenvector matrix obtained by principal component analysis of the average shape vector u obtained in advance from the face recognition learning data and the deviation from the average shape vector u. by using the P _s, it is represented by the following formula (3).
x = u + P _s b _s ... (3)
Here, b _s are parameters vector, called shape parameter.

Further, the appearance vector g in which the brightness values of the normalized textures are arranged is an eigenvector matrix obtained by principal component analysis of the average appearance vector v obtained from the face recognition learning data in advance and the deviation from the average appearance vector v. It is expressed by the following equation (4) using _{P g.}
g = v + P _g b _g ... (4)
Here, b _g is a parameter vector and is called an appearance parameter. The shape parameter b _s and the appearance parameter b _g are parameters representing changes from the average, and the shape and appearance can be changed by changing these.

In addition, since there is a correlation between the shape and the appearance, a low-dimensional parameter vector that controls both the shape and the appearance by further principal component analysis of the _{shape parameter b s} and the appearance parameter b _{g (in the following).} The shape vector x (c) and the texture vector g (c) are represented by the following equations (5) and (6) using (also referred to as “combination parameter”) c.
x (c) = u + P _s W _s ^-1 Q _s c ... (5)
g (c) = v + P _g Q _g c ... (6)
Here, W _s is a matrix that normalizes the difference in units between the shape vector and the appearance vector, Q _s is an eigenvector matrix related to the shape, and Q _g is an eigenvector matrix related to the appearance. By controlling the coupling parameter c in this way, it is possible to handle the shape and appearance at the same time and express the change of the object.

Next, consider a parameter (hereinafter, also referred to as “posture parameter”) q relating to a wide range of changes in where, in what size, and in what direction the object exists in the image. The posture parameter q is expressed by the following equation (7).
q = [roll scale trans_x trans_y] ・ ・ ・ (7)
Here, roll represents the rotation angle of the model with respect to the image plane, scale represents the size of the model, and trans_x and trans_y represent the amount of translation of the model in the x-axis direction and the y-axis direction, respectively.

In the active appearance model, the model search is to generate the target image by changing the model locally and widely by the coupling parameter c and the posture parameter q, and compare the generated image with the input image. Therefore, the coupling parameter c and the attitude parameter q are obtained so that the error is minimized. According to the active appearance model, it is possible to extract feature points robustly and at high speed in response to changes in the direction of the target.

Specifically, for a certain coupling parameter c'and posture parameter q', a function that transforms the shape X by the shape _{parameter b s'obtained from the coupling parameter c'and the posture parameter q'is W (X; q).} ', B _s '). Further, assuming that the function for obtaining the luminance value in the shape X when the input image Img and the shape X are given is I (Img, X), the error value Er in the model search is expressed by the following equation (8). Will be done.
Er = [(v + P _g Q _g c')-I (Img, W (X; q', b _s '))] ²
... (8)

For example, when the error values of the K shapes X (1), X (2), ..., X (K) constituting the subject's face are obtained (K is a natural number), each of them. Assuming that the error values are Er (1), Er (2), ..., Er (K), the fit ratio Fr, which is an index representing the recognition error in the face recognition process, is expressed by the following equation (9).
Fr = (Er (1) + Er (2) + ... + Er (K)) / K ... (9)
Therefore, highly accurate face recognition processing can be performed by determining the coupling parameter c and the posture parameter q such that the error value Er or the fit rate Fr becomes small.

Next, the image processing unit 53 determines whether or not the fit rate of the subject's face obtained as a result of face recognition in the input image is equal to or less than a preset threshold value. The image processing unit 53 determines that the face of the subject could be recognized when the fit rate is equal to or less than the threshold value, and determines that the face of the subject could not be recognized when the fit rate exceeds the threshold value. ..

<Image analysis>
The image analysis unit 54 reads out the coordinates of a plurality of feature points in a predetermined number of frames from the coordinate data storage unit 75, and calculates the amount of movement of the subject's face based on the coordinates of the plurality of feature points in the predetermined number of frames. The calculation is performed, and the subject is visually evaluated based on the statistical processing of the amount of movement of the subject's face during the evaluation period. When the moving image data represents an image of 24 frames per second, the predetermined number of frames may be 24 frames corresponding to 1 second.

For example, the image analysis unit 54 has a first axis determined based on the position of a specific part in the image of the subject represented by the moving image data, a second axis substantially orthogonal to the first axis, or a second axis. , The amount of movement of the subject's face with the third axis substantially orthogonal to the first and second axes as the center of rotation may be calculated. In that case, the amount of movement of the subject's face is calculated based on the change in the orientation of the subject's face in a predetermined number of frames.

FIG. 5 is a diagram showing an example of an image used for calculating the amount of movement of the subject's face with the first axis as the center of rotation. In this example, the first axis is an axis parallel to the line connecting the right and left inner corners of the subject (X-axis in the figure). For example, in order to calculate the amount of movement of the subject's face with the first axis as the center of rotation, the feature point P0 located at the midpoint between the right and left inner corners of the subject and the subject's A feature point P1 located at the right end of the nose, a feature point P2 located at the left end of the nose of the subject, and a feature point P3 located at the midpoint between the right and left mouth corners of the subject are used.

As shown in FIG. 5, the feature points P0 to P2 form the first triangle on the upper side in the figure, and the feature points P1 to P3 form the second triangle on the lower side in the figure. The value of the area or height ratio of the first triangle and the second triangle as seen from an image sensor such as a video camera represents the orientation of the subject's face in the movement centered on the first axis. Used as a quantity.

FIG. 6 is a diagram for explaining a change in the height ratio of the first triangle and the second triangle due to the movement of the subject's face with the first axis as the center of rotation. As shown on the left side of FIG. 6, when the subject faces the front surface of the image sensor such as a video camera, the heights of the first triangle H1 and the heights of the second triangles seen from the image sensor are high. It is assumed that the value H1 / H2 of the ratio with H2 is A.

On the other hand, as shown on the right side of FIG. 6, when the subject nods and faces the face below the front surface of the image sensor such as a video camera, the second triangle is the first triangle. Since the angle changes as the distance from the front surface of the image sensor increases, the value of the ratio between the height H1'of the first triangle and the height H2' of the second triangle seen from the image sensor H1'/ H2' Becomes A'(A'> A). The image analysis unit 54 may obtain the heights of the first triangle and the second triangle as the number of pixels. This simplifies the calculation of the distance.

FIG. 7 is a diagram showing an example of an image used for calculating the amount of movement of the subject's face with the second axis as the center of rotation. In this example, the second axis is an axis parallel to the line (Y-axis in the figure) connecting the midpoint between the right and left inner corners of the subject and the midpoint between the right and left corners of the mouth. .. For example, in order to calculate the amount of movement of the subject's face with the second axis as the center of rotation, the feature point P1 located at the right end of the subject's nose and the feature point P1 located at the left end of the subject's nose are located. The feature point P2, the feature point P4 located at the right mouth corner of the subject, the feature point P5 located at the left mouth corner of the subject, the feature point P6 located at the right inner corner of the subject, and the subject. The feature point P7 located at the inner corner of the left eye is used.

As shown in FIG. 7, the feature points P1, P4, and P6 form the first triangle on the left side of the figure, and the feature points P2, P5, and P7 form the second triangle on the right side of the figure. The value of the ratio of the area of the first triangle and the area of the second triangle as seen from the image sensor of a video camera or the like is used as a quantity indicating the orientation of the subject's face in the movement centered on the second axis. Be done.

FIG. 8 is a diagram for explaining a change in the area ratio between the first triangle and the second triangle due to the movement of the subject's face with the second axis as the center of rotation. As shown on the left side of FIG. 8, when the subject faces the front surface of the image sensor such as a video camera, the area S1 of the first triangle and the area S2 of the second triangle seen from the image sensor It is assumed that the value S1 / S2 of the ratio with and is B.

On the other hand, as shown on the right side of FIG. 8, when the subject faces the right side of the front surface of the image sensor such as a video camera, the first triangle is the image sensor rather than the second triangle. Since the angle changes as the distance from the front surface increases, the ratio value S1'/ S2'of the area S1'of the first triangle and the area S2' of the second triangle seen from the image sensor is B'(B'< B).

FIG. 9 is a diagram showing an example of an image used for calculating the amount of movement of the subject's face with the third axis as the center of rotation. In this example, the third axis is an axis parallel to the Z axis orthogonal to the X and Y axes in the figure. For example, in order to calculate the amount of movement of the subject's face with the third axis as the center of rotation, the feature point P6 located at the right inner corner of the subject and the feature point located at the left inner corner of the subject. P7 and is used.

As shown in FIG. 9, a line connecting the feature point P6 located at the inner corner of the right eye of the subject and the feature point P7 located at the inner corner of the left eye (solid line in the figure) and a line parallel to the X axis (in the figure). The angle θ formed by the broken line) or the trigonometric function value of the angle θ (sinθ, cosθ, tanθ, etc.) is used as a quantity representing the orientation of the subject's face in the movement centered on the third axis. ..

Referring to FIG. 1 again, the image analysis unit 54 calculates a dispersion value of an amount representing the orientation of the subject's face in a predetermined number of frames as the amount of movement of the subject's face, and determines the dispersion value of the dispersion value during the evaluation period. Determine the rank of the subject for visual evaluation based on the probability distribution. For example, when 24 frames are treated as one block for calculating one dispersion value, each of two consecutive blocks may include 12 overlapping frames.

_{For a predetermined number (L pieces) of frames, the dispersion value V X} of the quantity X (i) representing the orientation of the subject's face in the movement centered on the first axis parallel to the X axis is given by the following equation ( It is defined in 10) (L is an integer of 2 or more).
_{V X = (1 / L)} Σ (X (i) -E X) 2 ··· (10)
Here, a i = 1 to L, _{E X} is the mean value of X (i) in the L frames.

_{For example, a plurality of variance values V X} can be obtained based on the moving image data of 24 × 55 frames in the evaluation period. The image analysis unit 54 _{classifies the plurality of variance values V X} in the evaluation period into M classes according to their size (M is an integer of 2 or more), and thereby the variance values belonging to the jth class. determine the V _X existence probability _P X of the (j) (j) (j = 1~M).

_{Similarly, for L frames, the variance value V Y} of the quantity Y (i) representing the orientation of the subject's face in the movement centered on the second axis parallel to the Y axis is given by the following equation (11). ).
_{V Y = (1 / L)} Σ (Y (i) -E Y) 2 ··· (11)
Here, a i = 1 to L, _{E Y} is the average value of Y (i) in the L frames. The image analysis unit 54 _{classifies the plurality of variance values V Y} in the evaluation period into M classes according to their size, and thereby, the existence probability of the _{variance value V Y (j) belonging to the jth class.} P _Y (j) is obtained (j = 1 to M).

_{Further, for the L frames, the variance value V Z} of the quantity Z (i) representing the direction of the subject's face in the movement centered on the third axis parallel to the Z axis is given by the following equation (12). Defined in.
V _Z = (1 / L) Σ (Z (i) -E _Z ) ² ... (12)
Here, i = 1 to _{L, and E Z} is the average value of Z (i) in L frames. The image analysis unit 54 _{classifies the plurality of variance values V Z} in the evaluation period into M classes according to their size, and thereby, the existence probability of the _{variance value V Z (j) belonging to the jth class.} P _Z (j) is obtained (j = 1 to M).

FIG. 10 is a diagram showing an example of a probability distribution of a variance value of an amount representing the orientation of the subject's face during the evaluation period. 10, the horizontal axis three variance _{V X (j), V Y} (j), represents in class value 50 increments the _V Z (j), the vertical axis, the existence probability _P X ( It represents j), P _Y (j), and P _Z (j). In order to display the three types of dispersion values in one figure, the three types of dispersion values are displayed with their positions shifted. At least one of them is used as the amount of facial movement of the subject.

In the image analysis unit 54 shown in FIG. 1, for example, the ratio at which the coordinates of the feature points of the subject's face can be obtained in the moving image data corresponding to the evaluation period does not reach a certain ratio (for example, 60%). In this case, the image of the subject is determined to be rank RV0 (unevaluated). On the other hand, when the ratio at which the coordinates of the feature points of the subject's face can be obtained is equal to or more than a certain ratio, the image analysis unit 54 examines the subject according to the amount of movement of the subject's face. The image of the person is classified into one of a plurality of ranks of rank RV1 or higher.

In general, when the subject speaks while thinking in words, the movement of the face tends to stop, and when the subject speaks convincingly and fluently, the movement of the face becomes active. Therefore, if the amount of movement of the subject's face is generally smaller than the predetermined reference amount, the image analysis unit 54 determines that the image of the subject is rank RV1 (small movement), and determines that the subject's image has a rank RV1 (small movement). If the amount of movement of the face is generally larger than the reference amount, the image of the subject may be determined to be rank RV2 (large movement).

For example, the image analysis unit 54 may compare the probability distribution of at least one type of dispersion value with the probability distribution of a preset reference amount to determine the rank regarding the visual evaluation of the subject. Therefore, in the learning data storage unit 73, for example, moving image data obtained by imaging the face of a pseudo-subject such as an internship applicant and a rank determined by the evaluator by actually evaluating the image. Evaluation data representing the above is stored in advance as determination learning data. The image analysis unit 54 sets the probability distribution of the reference amount and the comparison method so that the judgment result close to the judgment learning data can be obtained by functioning as an AI (artificial intelligence) that performs machine learning using the judgment learning data. Then, the rank regarding the visual evaluation of the subject may be determined.

Alternatively, the image analysis unit 54 _{obtains the total value or the average value of the variance values V X} (j), V _Y (j), and V _Z (j) belonging to at least one class as the amount of movement of the subject's face. If the amount of movement of the subject's face is smaller than the predetermined reference amount, the image of the subject is judged to be rank RV1 (small movement), and the amount of movement of the subject's face is larger than the reference amount. For example, the image of the subject may be determined to be rank RV2 (large movement).

If the period in which the subject's face is recorded does not reach a certain period (for example, 15 seconds) in the moving image data corresponding to the evaluation period, the image analysis unit 54 determines the subject. The image of may be determined to be rank RV0. The image analysis unit 54 stores the visual evaluation data representing the visual evaluation result thus obtained in the evaluation data storage unit 74.

<Comprehensive evaluation>
When the voice evaluation data and the visual evaluation data of the same subject are stored in the evaluation data storage unit 74, the comprehensive evaluation unit 55 reads out the voice evaluation data and the visual evaluation data from the evaluation data storage unit 74, and then reads the voice evaluation data and the visual evaluation data. The person of the subject is evaluated based on the evaluation result by the voice analysis unit 52 and the evaluation result by the image analysis unit 54. For example, the comprehensive evaluation unit 55 evaluates the person of the subject using a plurality of mapping areas arranged two-dimensionally based on a plurality of ranks in the voice evaluation and a plurality of ranks in the visual evaluation.

FIG. 11 is a diagram showing an example of a mapping area used for character evaluation of a subject. As shown in FIG. 11, the voice evaluation is divided into rank RA0 (extremely bad voice), rank RA1 (bad voice), rank RA2 (normal voice), and rank RA3 (good voice). On the other hand, the visual evaluation is divided into rank RV0 (unevaluated), rank RV1 (small movement), and rank RV2 (large movement).

For example, areas 0 to 5 having a voice evaluation of rank RA0 or RA1 and areas 7 having a voice evaluation of rank RA2 and a visual evaluation of rank RV1 are set as rejected areas. The area 6 in which the visual evaluation is rank RV0 even if the voice evaluation is rank RA2 is set as a human check area in which further human check is required. On the other hand, the area 8 in which the audio evaluation is rank RA2 and the visual evaluation is rank RV2 and the areas 9 to 11 in which the audio evaluation is rank RA3 are set as pass areas.

The comprehensive evaluation unit 55 shown in FIG. 1 is based on the rank related to the voice of the subject represented by the voice evaluation data and the rank related to the visual evaluation of the subject represented by the visual evaluation data. By selecting one of the areas 0 to 11 shown in the above, the person of the subject is evaluated. The comprehensive evaluation unit 55 stores the person evaluation data representing the person evaluation result thus obtained in the evaluation data storage unit 74.

<Person evaluation method>
Next, the person evaluation method used in the person evaluation device according to the embodiment of the present invention will be described with reference to FIGS. 1 to 12. FIG. 12 is a flowchart showing a person evaluation method according to an embodiment of the present invention. For processes that are independent of each other, they may be performed in parallel.

In step S11 shown in FIG. 12, the voice processing unit 51 Fourier transforms the voice data obtained by recording the voice of the subject for each data block per unit time, and sounds in a plurality of frequency bands for each data block. Generate voiceprint data representing the pressure distribution.

In step S12, the voice analysis unit 52 classifies data blocks according to the magnitude and spread of sound pressure in a plurality of frequency bands, and evaluates the voice of the subject based on the classification results of a predetermined number of data blocks. conduct. Thereby, the rank regarding the voice of the subject is determined.

In step S13, the image processing unit 53 performs face recognition processing on the moving image data obtained by imaging the face of the subject for each frame, so that a plurality of features recognized on the face of the subject are recognized. Extract points and obtain the coordinates of multiple feature points.

In step S14, the image analysis unit 54 calculates the amount of movement of the subject's face based on the coordinates of a plurality of feature points in a predetermined number of frames, and statistically processes the amount of movement of the subject's face during the evaluation period. A visual evaluation of the subject is made based on. Thereby, the rank regarding the visual evaluation of the subject is determined.

In step S15, the comprehensive evaluation unit 55 evaluates the person of the subject based on the evaluation result in step S12 and the evaluation result in step S14. At that time, the comprehensive evaluation unit 55 uses, for example, a mapping area as shown in FIG. 11, and uses the subject's person based on the subject's voice rank and the subject's visual evaluation rank. Make an evaluation.

<Processing flow of moving image data>
13 and 14 are flowcharts showing an example of the processing flow of moving image data. In this example, the moving image data represents an image of 24 frames per second.
In step S21 shown in FIG. 13, the image processing unit 53 receives the raw data storage unit 71 that stores the moving image data supplied to the person evaluation device for the visual evaluation of the subject from 5 seconds after the start of imaging. The moving image data representing the image in the evaluation period of 60 seconds and 55 seconds is acquired, and the frame number n is set to 1.

In step S22, the image processing unit 53 performs face recognition processing on the moving image data of the nth frame to extract a plurality of feature points recognized on the face of the subject, and of these feature points. Find the coordinates. Further, in step S23, the image processing unit 53 stores the coordinates of the plurality of feature points in the coordinate data storage unit 75 together with the frame numbers.

In step S24, the image processing unit 53 determines whether the frame number n is 1320 (= 24 × 55) or the nth frame is the final frame of the moving image data. If the frame number n is smaller than 1320 and the nth frame is not the final frame of the moving image data, the image processing unit 53 increments the frame number n to (n + 1), and the process returns to step S22. On the other hand, when the frame number n is 1320 or the nth frame is the final frame of the moving image data, the process proceeds to any one of steps S25 to S27. Alternatively, steps S25 to S27 may be sequentially processed or may be processed in parallel.

In step S25, the image analysis unit 54 reads out the coordinates of a plurality of feature points in each frame from the coordinate data storage unit 75, and calculates the amount of movement of the subject's face with the first axis as the rotation center. The area or height of the first triangle and the second triangle required for the above is obtained for each frame. After that, the process proceeds to step S28 (FIG. 14).

In step S26, the image analysis unit 54 reads out the coordinates of a plurality of feature points in each frame from the coordinate data storage unit 75, and calculates the amount of movement of the subject's face with the second axis as the rotation center. The areas of the first triangle and the second triangle required for the above are obtained for each frame. After that, the process proceeds to step S28 (FIG. 14).

In step S27, the image analysis unit 54 reads out the coordinates of a plurality of feature points in each frame from the coordinate data storage unit 75, and calculates the amount of movement of the subject's face with the third axis as the rotation center. In addition, the angle of the line connecting the left and right inner corners of the eye is obtained for each frame as an amount indicating the direction of the subject's face. After that, the process proceeds to step S29 (FIG. 14).

In step S28 shown in FIG. 14, the image analysis unit 54 obtains the value of the area or height ratio of the first triangle and the second triangle for each frame as an amount representing the direction of the face of the subject. .. After that, the process proceeds to step S29.

In step S29, the image analysis unit 54 calculates the dispersion value of the amount representing the orientation of the subject's face in 24 frames (1 second) as the amount of movement of the subject's face. Further, in step S30, the image analysis unit 54 classifies the plurality of variance values obtained in the evaluation period (55 seconds) into a plurality of classes according to their size, thereby determining the existence probability of each variance value. Ask.

In step S31, the image analysis unit 54 determines the rank of the subject regarding the visual evaluation based on the probability distribution of the variance values during the evaluation period. As a result, when the ratio at which the coordinates of the feature points of the subject's face can be obtained in the moving image data corresponding to the evaluation period is a certain ratio or more, the movement amount of the subject's face is adjusted. , The subject's image is classified into one of multiple ranks.

As described above, according to the embodiment of the present invention, voiceprint data representing the sound pressure distribution in a plurality of frequency bands is generated for each data block from the sound data obtained by recording the sound of the subject. , Data blocks are classified according to the magnitude and spread of sound pressure in a plurality of frequency bands, and the voice of the subject is evaluated based on the classification results of a predetermined number of data blocks. Based on the voice of the subject, it is possible to provide information that can be used as a reference when evaluating the person of the subject.

Further, the coordinates of a plurality of feature points recognized in the subject's face are obtained from the moving image data obtained by imaging the subject's face, and based on the coordinates of the plurality of feature points in a predetermined number of frames. By calculating the amount of movement of the subject's face and performing a visual evaluation of the subject based on the statistical processing of the amount of movement of the subject's face during the evaluation period, the subject to be evaluated as a person. It is possible to provide information that can be used as a reference when evaluating a person of a subject based on the moving image and sound of the person.

The determination method in the embodiment described above is an example. The present invention is not limited to those embodiments, and many modifications can be made within the technical idea of the present invention by a person having ordinary knowledge in the art.

The present invention can be used in a person evaluation device or the like used for evaluating a person.

10 ... Operation unit, 20 ... Display unit, 30 ... Input / output interface, 40 ... Network interface, 50 ... CPU, 51 ... Voice processing unit, 52 ... Voice analysis unit, 53 ... Image processing unit, 54 ... Image analysis unit, 55 ... Comprehensive evaluation unit, 60 ... Memory, 70 ... Storage unit, 71 ... Raw data storage unit, 72 ... Voiceprint data storage unit, 73 ... Learning data storage unit, 74 ... Evaluation data storage unit, 75 ... Coordinate data storage unit

Claims (9)

A voice processing unit that Fourier transforms the voice data obtained by recording the voice of the subject for each data block per unit time and generates voiceprint data representing the sound pressure distribution in a plurality of frequency bands for each data block.
When classifying data blocks according to the magnitude and spread of sound pressure in the plurality of frequency bands and evaluating the voice of the subject based on the classification results of a predetermined number of data blocks , one of the frequencies For data blocks whose sound pressure exceeds the threshold in the region, a score higher than the score of the data block whose sound pressure is below the threshold is given in all frequency regions, and the sound pressure exceeds the threshold and is maximized in the frequency band. A predetermined number of data blocks are given a score higher than the score of the data block in which the number of frequency bands in which the sound pressure exceeds the threshold value and becomes maximum is equal to or less than the predetermined value for the data blocks whose number exceeds the predetermined value. A sound analysis unit that determines the rank of the subject regarding the sound based on the total value or the average value of the block scores, and the sound analysis unit.
A person evaluation device equipped with. By performing face recognition processing for each frame on the moving image data obtained by imaging the face of the subject, a plurality of feature points recognized on the face of the subject are extracted, and the plurality of feature points are extracted. An image processing unit that obtains the coordinates of feature points,
The amount of movement of the subject's face is calculated based on the coordinates of the plurality of feature points in a predetermined number of frames, and the amount of movement of the subject's face during the evaluation period is statistically processed. Image analysis unit that visually evaluates
The person evaluation device according to claim 1, further comprising. A first axis defined by the image analysis unit based on the position of a specific part in the image of the subject represented by the moving image data, a second axis substantially orthogonal to the first axis, Alternatively, the person evaluation device according to claim 2 , wherein the amount of movement of the subject's face is calculated with the third axis substantially orthogonal to the first and second axes as the center of rotation. The image analysis unit calculates a dispersion value of an amount representing the orientation of the subject's face in the predetermined number of frames as the amount of movement of the subject's face, and a probability distribution of the dispersion value in the evaluation period. The person evaluation device according to claim 2 or 3 , which determines the rank of the subject with respect to the visual evaluation based on the above. The person evaluation device according to any one of claims 2 to 4 , further comprising a comprehensive evaluation unit that evaluates the person of the subject based on the evaluation result by the voice analysis unit and the evaluation result by the image analysis unit. .. The procedure (a) and the procedure (a) in which the voice data obtained by recording the voice of the subject is Fourier-transformed for each data block per unit time to generate voiceprint data representing the sound pressure distribution in a plurality of frequency bands for each data block. ,
When classifying data blocks according to the magnitude and spread of sound pressure in the plurality of frequency bands and evaluating the voice of the subject based on the classification results of a predetermined number of data blocks , one of the frequencies For data blocks whose sound pressure exceeds the threshold in the region, a score higher than the score of the data block whose sound pressure is below the threshold is given in all frequency regions, and the sound pressure exceeds the threshold and is maximized in the frequency band. A predetermined number of data blocks are given a score higher than the score of the data block in which the number of frequency bands in which the sound pressure exceeds the threshold value and becomes maximum is equal to or less than the predetermined value for the data blocks whose number exceeds the predetermined value. The procedure (b) of determining the rank of the subject regarding the sound based on the total value or the average value of the block scores, and
A person evaluation program that causes the CPU to execute. By performing face recognition processing for each frame on the moving image data obtained by imaging the face of the subject, a plurality of feature points recognized on the face of the subject are extracted, and the plurality of feature points are extracted. The procedure (c) for obtaining the coordinates of the feature points and
The amount of movement of the subject's face is calculated based on the coordinates of the plurality of feature points in a predetermined number of frames, and the amount of movement of the subject's face during the evaluation period is statistically processed. Procedure (d) for visual evaluation of
The procedure (e) for evaluating the person of the subject based on the evaluation result in the procedure (b) and the evaluation result in the procedure (d), and
6. The person evaluation program according to claim 6, wherein the CPU is further executed. Step (a): Fourier transform the voice data obtained by recording the voice of the subject for each data block per unit time, and generate voiceprint data representing the sound pressure distribution in a plurality of frequency bands for each data block. ,
When classifying data blocks according to the magnitude and spread of sound pressure in the plurality of frequency bands and evaluating the voice of the subject based on the classification results of a predetermined number of data blocks , one of the frequencies For data blocks whose sound pressure exceeds the threshold in the region, a score higher than the score of the data block whose sound pressure is below the threshold is given in all frequency regions, and the sound pressure exceeds the threshold and is maximized in the frequency band. A predetermined number of data blocks are given a score higher than the score of the data block in which the number of frequency bands in which the sound pressure exceeds the threshold value and becomes maximum is equal to or less than the predetermined value for the data blocks whose number exceeds the predetermined value. Step (b) of determining the rank of the subject regarding the sound based on the total value or the average value of the block scores, and
Person evaluation method including. By performing face recognition processing for each frame on the moving image data obtained by imaging the face of the subject, a plurality of feature points recognized on the face of the subject are extracted, and the plurality of feature points are extracted. Step (c) to find the coordinates of the feature points and
The amount of movement of the subject's face is calculated based on the coordinates of the plurality of feature points in a predetermined number of frames, and the amount of movement of the subject's face during the evaluation period is statistically processed. Step (d) for visual evaluation of
In step (e), the person of the subject is evaluated based on the evaluation result in step (b) and the evaluation result in step (d).
8. The person evaluation method according to claim 8.

Images