KR101984283B1 - Automated Target Analysis System Using Machine Learning Model, Method, and Computer-Readable Medium Thereof - Google Patents

Abstract

The present invention relates to an automated evaluated person analysis system using a machine learning model, a method thereof, and a computer readable medium thereof capable of automatically providing a more accurate analysis result using a machine learning model while taking into account an evaluation situation and a state of an evaluated person. According to one embodiment of the present invention, the system comprises: a feature information generating part for generating voice feature information and image feature information from input image information and voice information; a speech state inference part for generating speech state information on whether the evaluated person is speaking from the information included in the image feature information; and an analysis result inference part for deriving the analysis result information of the evaluated person from the information including the voice feature information, the image feature information, and the speech state information.

Description

Technical Field [0001] The present invention relates to an automated evaluator analysis system, method, and computer readable medium using a machine learning model,

More particularly, the present invention relates to a system and method for automatically analyzing an evaluated guest using a machine learning model, and more particularly, To an automated assessor analysis system, method, and computer readable medium using a machine learning model that can automatically provide a machine-readable model.

Many people have been self-taught or trained in professional schools to prepare interviews for communication skills or job hunting. This reflects the fact that people are becoming accustomed to the online environment such as SNS, rather than offline, and that social skills in face-to-face situations are lowered when they use messengers such as KakaoTalk or SMS to communicate in the same way can do.

Various systems and researches are being conducted in an online environment to help such people, but conventional technologies have the following limitations.

First, conventional technologies were unable to evaluate corresponding responses according to the types of questions. For example, suppose you have a system that automatically evaluates online interview results for recruitment. In the prior art, it is basically evaluated based on expression, tone, tone and gesture when answering it regardless of a question. In such a conventional technique, there arises a situation where irregular evaluation results can be derived regardless of the question and the situation. For example, when asked a humorous question based on personality evaluation criteria, it would be a good evaluation for the interviewer to answer in a relaxed mood and tone with a smiling face, ask questions for evaluation of technical expertise in technical interviews, In this case, as described above, it would be better to maintain a facial expression and tone in a more gentle manner than to answer with a smile. However, the conventional technology does not reflect the factor of such evaluation situation, and there is a problem that the voice and expression of the subject to be evaluated are evaluated by a consistent standard. That is, in the conventional technology, it is impossible to evaluate different responses according to the situation, and it has a limitation that it derives a generalized evaluation result by learning the whole interview or its parts as a whole.

Second, while listening to communication skills is very important, conventional technologies lack diagnosis. The ability to communicate is also important, but the attitude of listening to the other person's words is also very important. For example, suppose that A and B are talking to each other for a video chat or a video interview. In the case where the camera is illuminating B and the microphone is turned on in the existing system, in a situation where A is talking to B, the information of B is inputted in the image information, but the information of A is inputted in the audio information. In this state B is not talking, the input audio information may be input with different noise information. In this case, in order to accurately diagnose B, voice information should be discarded and evaluated in the direction of maximizing the use of image information. However, the conventional technology has a limitation in that it can not adapt to such situation.

Third, communication should express emotions or words in different ways depending on the place and situation, and should be evaluated differently when analyzing it. In addition to the above-mentioned questions, communication methods should be changed depending on the place and situation. The conversation at the funeral hall will be more sad, and the conversation at the amusement park will be a little more active. However, the conventional technology does not take into consideration such a situation, so that it is difficult to adequately diagnose and feedback it.

Finally, in the conventional art, most of the feature information to be judged when the person evaluates the evaluator is hand-crafted and the machine learning is performed based on the feature information. In this case, the feature information for deriving the final result is missing There is a danger. When evaluating the evaluation through facial expression, landmark information is derived and learned using the corresponding position point. It is difficult for people to find clear criteria for whether the actual information about the image information and the voice information should be used should be similar to the actual person's evaluation, and the latent information should be used unconsciously at the time of evaluation The higher the performance of the results of diagnosing the social skills.

delete

Korean Patent No. 10-1198445 (Apparatus and method for analyzing interview image, registered on October 31, 2012)

An object of the present invention is to provide an automated assessor analysis system, a method, and a computer readable medium using a machine learning model capable of automatically providing a more accurate analysis result by using a machine learning model, taking into consideration the evaluation situation and the condition of an assessor Media.

According to an aspect of the present invention, there is provided an automated evaluator analysis system comprising at least one processor and at least one memory, wherein the system comprises voice feature information and image feature information from input image information and voice information, A feature information generating unit to generate the feature information; A horse state reasoning unit for generating horse state information on whether an evaluated person is speaking from information included in the image feature information; And an analysis result inferring unit for deriving analysis result information of the subject from the information including the speech feature information, the image feature information, and the speech state information.

In an embodiment of the present invention, the feature information generating unit includes: a voice feature information generating unit that generates voice feature information; And an image feature information generating unit for generating image feature information, wherein the image feature information generating unit comprises: a first image feature information extracting unit for extracting at least one element in a face outside the mouth area from the input image information; And a second image feature information extraction unit for extracting an element of the input area from the input image information.

In one embodiment of the present invention, the horse state inferring unit may generate the horse state information from the second image feature information.

In one embodiment of the present invention, the analysis result inferring unit includes: a voice feature map generating unit that generates a voice feature map from the voice feature information; A first image feature map generator for generating a first image feature map from the first image feature information; And a second image feature map generator for generating a second image feature map from the second image feature information.

In one embodiment of the present invention, the analysis result inferring unit assigns weights to the voice feature map and the second image feature map, respectively, based on the speech state information, so that the speech feature map, And a complex feature map generator for generating a complex feature map from the second image feature map.

In an embodiment of the present invention, the automated assessor analysis system further includes a context-aware parameter generation unit for generating context-aware parameters including input voice information and information on an evaluation status of an evaluator from the speech status information .

In one embodiment of the present invention, the situation recognition parameter generation unit may include a situation determination unit for extracting the situation audio information, which is audio information of a person other than the subject of evaluation, of the audio information from whether the audio information is output or not have.

In one embodiment of the present invention, the context recognition parameter generation unit may include: a content feature map generation unit for generating a content feature map for the content of speech from the context audio information; And a non-content feature map generation unit for generating a non-content feature map for the speech characteristic from the context speech information.

In one embodiment of the present invention, the analysis result inferring unit may derive analysis result information from the speech feature information, the image feature information, the speech state information, and the context recognition parameter.

In an embodiment of the present invention, the analysis result inference unit may apply the context awareness parameter to the complex feature map generated based on the speech feature information, the image feature information, and the speech state information to derive analysis result information have.

According to an aspect of the present invention, there is provided an automated evaluator analysis method implemented by a computing device including at least one processor and at least one memory, And a feature information generation step of generating image feature information; A word state inference step of generating word state information on whether the assessor is speaking from information included in the image characteristic information; And an analysis result inference step of deriving analysis result information of the subject from the information including the speech feature information, the image feature information, and the speech state information.

In one embodiment of the present invention, the characteristic information generating step includes: a voice characteristic information generating step of generating voice characteristic information; And an image feature information generating step of generating image feature information, wherein the image feature information generating step includes: a first image feature information extracting step of extracting at least one element inside a face outside the mouth area from the input image information; And a second image feature information extracting step of extracting an element of the input area from the input image information.

In one embodiment of the present invention, the horse state inferring step may generate the horse state information from the second image feature information.

In an embodiment of the present invention, the analysis result inference step may include: a voice feature map generation step of generating a voice feature map from the voice feature information; A first image feature map generation step of generating a first image feature map from the first image feature information; And a second image feature map generation step of generating a second image feature map from the second image feature information.

In one embodiment of the present invention, the analysis result inferring step assigns weights to the voice feature map and the second image feature map, respectively, based on the speech state information, so that the speech feature map, A feature map, and a composite feature map generation step of generating a composite feature map from the second image feature map.

In one embodiment of the present invention, the automated assessor analysis method further includes a situation recognition parameter generation step of generating a situation recognition parameter including information on an evaluation situation of an assessor from the input speech information and the speech state information can do.

In one embodiment of the present invention, the context recognition parameter generation step includes a situation determination step of extracting context audio information, which is speech information of a person other than the subject of evaluation, of the speech information from whether the speech information is audibly output or from the speech state information can do.

In an embodiment of the present invention, the context recognition parameter generation step may include: a content feature map generation step of generating a content feature map for the contents of speech from the context speech information; And generating a non-content feature map for the utterance characteristic from the context speech information.

In an embodiment of the present invention, the analysis result inference step may derive analysis result information from the speech feature information, the image feature information, the speech state information, and the context recognition parameter.

In an embodiment of the present invention, the analysis result inference step may include applying the context aware parameter to a complex feature map generated based on the speech feature information, the image feature information, and the speech state information to derive analysis result information .

One embodiment of the present invention is a computer-readable medium having stored thereon instructions for causing a computing device to perform the following steps: : A feature information generating step of generating speech feature information and image feature information from input image information and speech information; A word state inference step of generating word state information on whether the assessor is speaking from information included in the image characteristic information; And an analysis result inferring step of deriving an analyzed result information of the subject from the information including the speech feature information, the image feature information, and the speech state information.

According to the embodiment of the present invention, when the diagnosis is made differently according to the interview or the question, it is possible to obtain the analysis result of the accurate assessor according to the question.

According to one embodiment of the present invention, it is possible to train and improve how a job seeker responds to an interview or an evaluation question.

According to the embodiment of the present invention, it is possible to simply input the script information to unilaterally obtain the diagnosis result based on the evaluator, and to listen to the result through the interactive conversation.

According to an embodiment of the present invention, communication ability according to a place or atmosphere can be improved. We provide a structure that allows us to learn and diagnose various environmental situations such as funeral parlor, amusement park, service worker who has to convince customers, salesperson who has to convince customers, and conversation with strict bosses. And provides an environment in which the user can identify his / her deficient parts and improve his / her ability through feedback.

According to an embodiment of the present invention, it is possible to obtain a more accurate analysis result by eliminating the noise according to the mouth shape in the image evaluation.

According to the embodiment of the present invention, in the voice evaluation, it is possible to filter out only the voice of the actual evaluator and to obtain a more accurate analysis result.

According to an embodiment of the present invention, by using a complex feature map and a machine learning model in which an image and a voice are combined, it is possible to exhibit an effect of providing a similar or higher recognition result to that of an actual person.

According to an embodiment of the present invention, when evaluating a combination of video and audio, it is possible to provide a more accurate analysis result by considering the mouth shape of the subject.

FIG. 1 is a schematic diagram illustrating an operating environment of an automated system for analyzing an evaluated system according to an exemplary embodiment of the present invention. Referring to FIG.
FIG. 2 is a schematic diagram illustrating an internal configuration of an automated system for analyzing an evaluated subject matter according to an exemplary embodiment of the present invention. Referring to FIG.
FIG. 3 schematically shows the operation of the analysis result speculation unit according to an embodiment of the present invention.
FIG. 4 schematically shows an internal configuration of a feature information generating unit according to an embodiment of the present invention.
FIG. 5A schematically shows an internal configuration of an analysis result speculation unit according to an embodiment of the present invention.
5B schematically shows the internal structure of the analysis result inferring unit according to an embodiment of the present invention. FIG. 5C schematically shows the internal structure of the analysis result inferring unit according to an embodiment of the present invention.
FIG. 6 schematically illustrates generation of a feature map and feature map of voice feature information and image feature information according to an embodiment of the present invention.
7 schematically illustrates the operation of the complex feature map generator according to an embodiment of the present invention.
FIG. 8 schematically illustrates the operation of the context-aware adaptive reasoning unit according to an embodiment of the present invention.
FIG. 9 schematically shows an internal configuration of a context-aware parameter generation unit according to an embodiment of the present invention.
10 schematically shows the operation of the situation determining unit according to an embodiment of the present invention.
FIG. 11 schematically illustrates an internal configuration of a non-content feature map generating unit and a content feature map generating unit according to an embodiment of the present invention.
Figure 12 schematically illustrates the steps of an automated assessor analysis method in accordance with an embodiment of the present invention.
FIG. 13 schematically illustrates a feature map generation process according to an embodiment of the present invention.
Figure 14 schematically shows the overall configuration of an automated assessor analysis system in accordance with one embodiment of the present invention.
15 schematically shows a user screen for an analysis result of an automated assessor analysis system according to an embodiment of the present invention.
16 illustrates an exemplary internal configuration of a computing device according to an embodiment of the present invention.

In the following, various embodiments and / or aspects are now described with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it will also be appreciated by those of ordinary skill in the art that such aspect (s) may be practiced without these specific details. The following description and the annexed drawings set forth in detail certain illustrative aspects of one or more aspects. It is to be understood, however, that such aspects are illustrative and that some of the various ways of practicing various aspects of the principles of various aspects may be utilized, and that the description set forth is intended to include all such aspects and their equivalents.

In addition, various aspects and features will be presented by a system that may include multiple devices, components and / or modules, and so forth. It should be understood that the various systems may include additional devices, components and / or modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in connection with the drawings Must be understood and understood.

As used herein, the terms " an embodiment, " " an embodiment, " " an embodiment, " " an embodiment ", etc. are intended to indicate that any aspect or design described is better or worse than other aspects or designs. . The terms 'component', 'module', 'system', 'interface', etc. used in the following generally refer to a computer-related entity, And a combination of software and software.

It is also to be understood that the term " comprises " and / or " comprising " means that the feature and / or component is present, but does not exclude the presence or addition of one or more other features, components and / It should be understood that it does not.

Also, terms including ordinal numbers such as first, second, etc. may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

Furthermore, in the embodiments of the present invention, all terms used herein, including technical or scientific terms, unless otherwise defined, are intended to be inclusive in a manner that is generally understood by those of ordinary skill in the art to which this invention belongs. Have the same meaning. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and, unless explicitly defined in the embodiments of the present invention, are intended to mean ideal or overly formal .

1 is a schematic diagram illustrating an operating environment of an automated assessor analysis system 1000 according to an embodiment of the present invention.

An embodiment of the present invention provides a system, a method, and a computer readable medium considering the problems of the prior art described above. In the present invention, the video information and the audio information audio are basically received, and analysis result information is derived through a learned model through a machine learning technique such as deep learning.

Here, the module that derives the overall or detailed results of the subject analyzer system 1000 of the present invention may correspond to the learned model. In addition, the voice and image information can be extracted in the form of extracting voice and image information from the moving image or receiving the voice and the image separately.

The automated assessor analysis system 1000 according to an embodiment of the present invention includes a model for inferring whether or not the state is speaking through a mouth shape. Specifically, in the present invention, a model for distinguishing whether or not speech is being made using a circular neural network technique including temporal concepts such as RNN, LSTM, and GRU is separately detected for the mouth portion of the face included in the image information .

Based on this information, the degree of weighting of the image information and the audio information is automatically adjusted so as to influence the evaluation. Preferably, the model can also be configured to learn about such weights.

In the case where the image information and the audio information are inputted in such a configuration, the image information is processed more importantly (a relatively higher weight is given than the normal state) in the case where the subject or the speaker does not speak through the mouth shape The speech information will be treated less importantly (giving a relatively lower weight than the normal state).

In addition, the automated assessor analysis system 1000 according to an exemplary embodiment of the present invention may be configured to further input context information such as a question type, a place, a situation, or the like, And to diagnose and evaluate social skills based on this information. Therefore, even if the video and audio information of the same subject is received, different diagnosis results can be derived according to the situation information.

In addition, since the evaluation of social skills such as communication techniques must utilize the potential information that is difficult for humans to characterize, the automated assessor analysis system 1000 according to an embodiment of the present invention can be used for deep learning Based on a neural network technology.

According to an embodiment of the present invention, feedback can be provided to a user using a statistical method to utilize the derived data in a direction to provide an environment that can identify and train its own strengths and weaknesses.

1 (A), audio and video information is input. The system 1000 analyzes the information about the situation automatically from the audio and video information. Based on the information about the situation, The result information can be derived.

1B, context information is input in addition to voice and image information (the context information may be input by a user directly or in a manner of loading predetermined context information) ) Can derive analysis result information of the evaluator in consideration of the situation information in the voice and image information.

Such an evaluator analysis system 1000 may derive analysis result information from previously stored voice and image information or moving image data or derive analysis result information from voice and image information input in real time.

FIG. 2 is a schematic diagram illustrating an internal configuration of an automated assessor analysis system 1000 according to an embodiment of the present invention.

The assessor analysis system 1000 according to this embodiment may be implemented by a computing device having one or more processors and one or more memories.

Such a computing device may include a processor A, a bus (corresponding to a bi-directional arrow between the processor, memory, and the network interface), a network interface B, and a memory C. The memory C may store inference unit learning data used in the inference or prediction module of the present invention as learning data learned in implementing the operating system and the artificial neural network. Alternatively, the inference unit learning data may refer to the modeling information itself in which the deep learning is performed. In the processor A, the feature information generation unit 100, the analysis result reasoning unit 200, the horse state reasoning unit 300, and the context awareness parameter generation unit 400 may be executed. In other embodiments, automated assessor analysis system 1000 may include more components than the components of FIG.

The memory may be a computer-readable recording medium and may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), and a disk drive. These software components may be loaded from a computer readable recording medium separate from the memory using a drive mechanism (not shown). Such a computer-readable recording medium may include a computer-readable recording medium (not shown) such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, or a memory card. In other embodiments, the software components may be loaded into the memory via the network interface (B) rather than from a computer readable recording medium.

The bus may enable communication and data transfer between components of the computing device. The bus may be configured using a high-speed serial bus, a parallel bus, a Storage Area Network (SAN), and / or other suitable communication technology.

Network interface (B) may be a computer hardware component for connecting a computing device that implements an automated assessor analysis system 1000 to a computer network. The network interface (B) can connect the automated assessor analysis system (1000) to the computer network via a wireless or wired connection.

Processor A may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations that implement an automated assessor analysis system 1000. The command may be provided by the memory C or the network interface B and via the bus to the processor. The processor may be configured to execute program codes for the feature information generator 100, the analysis result speculator 200, the horse state speculator 300, and the context aware parameter generator 400. Such program codes may be stored in a recording device such as a memory.

The feature information generating unit 100, the analysis result reasoning unit 200, the state-state reasoning unit 300, and the context-awareness parameter generating unit 400 may be configured to perform the automated assessor analysis method . The above-mentioned processor may omit some components according to an automated evaluator analysis method, or may further include additional components not shown, or two or more components may be combined.

Preferably, the computing device corresponds to a personal computer or a server, and may be a smart phone, a tablet, a mobile phone, a videophone, an e-book reader e a notebook PC, a netbook PC, a personal digital assistant (PDA), a portable personal computer (PC) A mobile multimedia device, a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, a wearable device (e.g., a head- Electronic devices such as a head-mounted device (HMD), an electronic apparel, an electronic bracelet, an electronic necklace, an electronic app apparel, an electronic tattoo, or a smart watch ) And the like.

That is, the automated assessor analysis system 1000 according to an embodiment of the present invention includes a feature information generation unit 100 for generating speech feature information and image feature information from input image information and speech information; A word state inference unit (300) for generating word state information on whether the subject is speaking based on the information included in the image characteristic information; And an analysis result speculation unit (200) for deriving analysis result information of the subject from the information including the speech feature information, the image feature information, and the speech state information.

Preferably, the automated assessor analysis system 1000 further includes a context-aware parameter generator 400 for generating context-aware parameters including input voice information and information on the evaluation status of the evaluator from the speech status information .

Hereinafter, detailed components of the system 1000 will be described in detail.

FIG. 3 schematically shows the operation of the analysis result speculation unit 200 according to an embodiment of the present invention.

The feature information generating unit 100 generates the speech feature information, the image feature information, and the speech state information from the video information and the audio information.

The image feature information includes first image feature information and second image feature information corresponding to feature information on an area outside the mouth and feature information on the mouth area.

The feature information generated by the feature information generating unit 100 is input to the analysis result estimating unit 200. The analysis result estimating unit 200 estimates the speech feature information, And the analysis result information is derived from the speech state information and the context recognition parameter.

The system 1000 for analyzing the automated system according to an embodiment of the present invention does not directly apply the artificial neural network to the video and audio information itself but extracts the feature information from the artificial neural network to process the data first, By applying it to the artificial neural network, it is possible to reduce the computational load and implement a system that meets the objective of analyzing the evaluator.

FIG. 4 schematically shows an internal configuration of a feature information generating unit 100 according to an embodiment of the present invention.

As shown in FIG. 4, the feature information generation unit 100 includes a speech feature information generation unit 110 for generating speech feature information; And an image characteristic information generating unit 120 for generating image characteristic information.

That is, according to an embodiment of the present invention, the voice information and the video information are separated first, the feature information is derived from each of the voice information and the video information, and the feature map is generated by matching the sink information.

The voice feature information generation unit 110 includes a voice preprocessor 111 for removing noise from voice information or extracting a part to be processed from the entire voice information; And a speech feature information extraction unit 112 for extracting feature information such as raw signal or prosody, MFCC, eGeMAPs, frequency information, etc., from which noise has been removed from the speech information preprocessed by the speech preprocessing unit 111.

The image feature information generation unit 120 includes a video pre-processing unit 121 for removing noise from the image information, removing the effect of illumination, or adjusting brightness, saturation, or color values according to the surrounding environment. A face detection unit 122 for detecting a face from the preprocessed image; A first image feature information extracting unit (123) for extracting at least one element in a face outside the input area from the detected face area of the input image information; And a second image characteristic information extraction unit (124) for extracting an input area element from the detected face area of the input image information.

In the case where the evaluator speaks, when the mouth shape changes and is applied as it is to the model according to the artificial neural network, such a mouth shape may affect the feature map or analysis information related to the information about the facial expression have.

In one embodiment of the present invention, in order to optimize the performance of the artificial neural network and provide a more accurate analysis result, the feature information or the image of the elements of the mouth area and the mouth area are respectively extracted and the feature map And then determines the weight of the feature map of the mouth area according to whether the speaker is talking or not, and merges the two feature maps.

Here, the image feature information may correspond to the image itself or the processing information extracted from the image. For example, the second image feature information may include characteristic information derived from a predetermined rule or algorithm from an image of a cropped input area itself, an image of a cropped input area subjected to a predetermined image process or an image of a cropped input area .

FIG. 5A schematically shows an internal configuration of an analysis result speculation unit 200 according to an embodiment of the present invention.

The feature information generating unit 100 generates speech feature information, first image feature information, and second image feature information from the inputted image information and audio information.

The speech feature information, the first image feature information, and the second image feature information are input to the analysis result speculative unit 200, and the second image feature information is input to the horse state speculative unit 300.

The horse state inference unit 300 generates horse state information on whether the assessor is to be ignited based on the information included in the second image feature information and outputs the result of the analysis to the reasoning unit 200 and the context recognition parameter generator 400. [ .

The word state inference unit 300 derives a probability value ([0, 1]) P _{s as} to whether the assessor is talking or not when the mouth image is sequentially input according to the time flow. For example, if the talking state is close to 1, it is close to 1. If not speaking, a value close to 0 is derived, which is used in the reasoning unit 200 and the context-awareness parameter generating unit 400.

Specifically, the horse state inferring unit 300 deduces the horse state from one or more mouth-shaped images existing in a sampling interval to judge whether or not the horse is currently being spoken. For example, in order to judge whether or not speech is being made within 2 to 3 seconds, the evaluator speaks in a period of 2 to 3 seconds from 5 mouth shape information (second image feature information) of 2 to 3 seconds Or not. Or whether the assessor speaks within a period of 2 to 3 seconds from one mouth shape information (second image feature information) between 2 and 3 seconds.

According to an embodiment of the present invention, the speech state reasoning unit 300 may include a ConvNet (feature map generation from a single image without consideration of temporal elements), an LSTM (considering two or more images existing in the sampling interval, Feature map generation), Sigmoid (LSTM value normalized to a value between 0 and 1), or a combination of both. Preferably, a feature map is derived from ConvNet, and a plurality of such feature maps are derived to derive a secondary feature map through the LSTM in consideration of the time concept, and then it is referred to as FC (Fully-Connected) or Global Average Pooling After connecting through a layer, the probability value is obtained through a function such as Sigmoid. It is preferable that the speech state reasoning unit 300 performs learning through various learning images.

The horse state information derived by the horse state inference unit 300 is input to the compound feature map generation unit 240 of the analysis result inference unit 200. Since the voice feature map may be a noise or a voice of a person who is not the subject of the evalua- tion, the influence of the voice feature map is reduced and an evaluation thereof is performed. This can be implemented by lowering the weight of the voice feature map when the feature map is merged in the compound feature map generator 240. [ As described above, when the voice feature map is adjusted according to the speech state information, it is possible to diagnose the attitude of the audience to be listened to.

On the other hand, in the case where the subject is speaking, the influence of the second image feature map corresponding to the feature map on the mouth shape is reduced and evaluated in the feature map related to the image. This can be implemented by lowering the influence of the second image feature map when the feature map is merged in the compound feature map generator 240. [ When evaluating social skills, various information of the face is used, but emotion information plays a very important role. Since the mouth shape contains a lot of information about it, it is difficult to distinguish between talking and surprising or smiling. . In order to solve such a problem, in the present invention, it is determined whether the subject is speaking by a predetermined fixed or variable interval, and the complex feature map generator 240 adjusts the weight of each feature map, Create a map.

Meanwhile, the analysis result inference unit 200 derives analysis result information of the subject from the information including the speech feature information, the first image feature information, the second image feature information, and the speech state information, The inference unit 300 generates the speech state information from the second image feature information.

The analysis result inference unit 200 may include a voice feature map generation unit 210 for generating a voice feature map from the voice feature information. A first image feature map generating unit 220 for generating a first image feature map from the first image feature information; A second image feature map generator 230 for generating a second image feature map from the second image feature information; A complex feature map generator 240 for generating a complex feature map from the voice feature map, the first image feature map, and the second image feature map; And a context-aware adaptive reasoning unit 250 for applying context aware parameters to the complex feature map to derive analysis result information.

Here, the feature map may include data as an analysis result of a known artificial neural network, data as intermediate results generated in the analysis process of the artificial neural network, analysis result values derived through a model, or feature information data derived by a predetermined method .

The voice feature map generating unit 210, the first image feature map generating unit 220, and the second image feature map generating unit 230 may be implemented using at least one artificial neural network model used for image analysis CNN-based artificial neural network model may be used in one embodiment of the present invention.

Meanwhile, the compound feature map generator 240 assigns weights to the voice feature map and the second image feature map, respectively, based on the horse state information of the horse state inference unit 300, And generates a composite feature map from the feature map, the first image feature map, and the second image feature map.

In an embodiment of the present invention, it is possible to use the voice feature map, and the second probability value as the end of the image feature map status information operation that multiplies in series (P _s).

On the other hand, the context-aware parameter generation unit 400 generates a context-awareness parameter including information on an evaluation status of an evaluated person from the input speech information and the speech state information. This context awareness parameter corresponds to information about the current assessor's assessment situation and may have the form of a matrix or vector column according to a predetermined rule.

Preferably, the context-aware parameter generation unit 400 converts information such as the type of question, context, place, and atmosphere into appropriate permutation or hashing, or generates an appropriate parameter vector using a machine learning model do.

Preferably, the context-aware parameter generator 400 may input appropriate information such as the type of question, context, location, atmosphere, etc., according to user settings, without the input voice information and the speech state information. 5B and an internal configuration for the same embodiment.

When the speech is being generated, the context-aware parameter generation unit 400 determines whether the speech is evaluated by the speech state inference unit 300. If it is determined that the speech is speech from a person other than the subject, Information can be determined as question information, and a feature map can be generated therefrom. A feature map is generated for each of the content information and the vocal information (intonation, height, intensity, etc.) of the uttered voice, and the generated feature maps are classified into FC (Fully Connected) and Global Average Pooling layers After combining, we can generate a parameter vector from it.

In an embodiment of the present invention, instead of generating the context-aware parameters automatically from the voice and image information as described above, the context-aware parameters may be directly input from the user to the context-aware parameters, It is possible to generate a final context-aware parameter from the information on the input status and the automatically generated context-aware parameters as described above.

Meanwhile, the complex feature map and context-aware parameters generated by the complex feature map generator 240 are input to the context-aware adaptive reasoning unit 250, and the context- The analysis result information can be generated from the map and the situation recognition parameter, and the evaluation result information of each of the evaluators is included in the analysis result information.

In particular, the context-aware adaptive reasoning unit 250 may include a cyclic neural network module (e.g., a cyclic neural network module) capable of receiving a plurality of inputs so as to generate a first final feature map considering a temporal concept for a plurality of compound feature maps A dynamic parameter layer that receives context aware parameters and can generate a second final feature map to which context aware parameters are applied from the final feature map, and an FC layer that links the second final feature maps can do.

FIG. 5B schematically shows the internal structure of the analysis result speculation unit 200 according to an embodiment of the present invention.

As shown in FIG. 5B, the context-aware parameter generation unit generates context-aware parameters based on user input information.

In an embodiment of the present invention, the user input information may include script and status information such as a question type or location. In this case, the context-aware parameter generation unit may generate a first content feature map from the character string included in the script with the same configuration as that of the content feature map generation unit 430 of FIG. 11, Generate context aware parameters. In addition, the context awareness parameter generator may generate a second context awareness parameter from contextual information such as the question type or location, and generate context awareness parameters from the first context awareness parameter and the second context aware parameter. This corresponds to the semi-automatic mode of Fig.

In an embodiment of the present invention, the user input information may include status information such as a question type or a place. In this case, the context awareness parameter generator may generate context awareness parameters directly from context information such as the question type or location. This corresponds to the manual mode of Fig.

5C schematically shows the internal structure of the analysis result speculation unit 200 according to an embodiment of the present invention.

The embodiment of FIG. 5C differs from the embodiment of FIG. 5A in that the input information of the inverse state inference unit is facial image information, not the second image feature information.

Specifically, as shown in FIG. 5C, the horse state inference unit extracts horse state information from the facial image information.

Preferably, the horse state inference unit receives the image of the face region pre-processed by the face preprocessing unit 121 and the face detection unit 122 of the feature information generation unit 100, Generates the speech state information on whether the subject is speaking from the information including the face image information, and transmits the speech state information to the analysis result inference unit 200 and the context recognition parameter generation unit 400. [ As in the situation recognition parameter generation unit 400 of FIG. 5B, the horse state information may be input to the complex feature map generation unit 240 only.

The word state inference unit 300 derives a probability value ([0, 1]) Ps of whether the assessor is talking or not when the mouth image is sequentially input according to the time flow. For example, if the speech state is in the talking state, the value is close to 1. If the speech state is not speech, a value close to 0 is derived, which is used in the reasoning unit 200 and / or the context-awareness parameter generation unit 400. This is the same as the description in FIGS. 5A and 5B, and a duplicate description thereof will be omitted.

FIG. 6 schematically illustrates generation of a feature map and feature map of voice feature information and image feature information according to an embodiment of the present invention.

The analysis result inference unit 200 includes a voice feature map generation unit 210 that generates a voice feature map from the voice feature information; A first image feature map generating unit 220 for generating a first image feature map from the first image feature information; And a second image feature map generator 230 for generating a second image feature map from the second image feature information.

Here, the audio information and the video information are sampled at predetermined intervals. For example, as shown in FIG. 6, in the case of image information, there is image information at 8 frames per second. When 1 second is set as the sampling timing, 0 to 1 second interval, 1 to 2 second interval, The voice feature map can be generated from the voice information of the section.

On the other hand, in the case of the image feature map, for example, one frame is extracted in the interval of 0 to 1 second, one frame in the interval of 1 to 2 seconds, and one frame in the interval of 2 to 3 seconds, A feature map can be generated.

On the other hand, the speech state inference unit 300 extracts speech state information about whether speech is being spoken in 0 to 1 second intervals from the entire 8 frames of 0 to 1 second interval, It is possible to extract the speech state information as to whether or not speech is being spoken within a period of 0 to 1 second in one frame of the speech signal (Fig. 6).

On the other hand, each of the voice feature map, the first image feature map, and the second image feature map may have time information synchronized with the horse state information. That is, in a predetermined time interval, the voice feature map, the first image feature map, the second image feature map, and the horse state information are temporally synchronized, or the voice feature map, the first image feature map, And may have a data field for time information as data data type.

FIG. 7 schematically illustrates the operation of the complex feature map generator 240 according to an embodiment of the present invention.

The compound feature map generator 240 assigns weights to the voice feature map and the second image feature map based on the speech state information, and outputs the voice feature map, the first image feature map, A composite feature map is generated from the second image feature map. That is, the complex feature map generator 240 includes a feature mappers 241 for merging the weighted feature maps.

FIG. 7 specifically shows a time period or timing of the # 1 to #Nth time periods, and a combination of the voice feature map, the first image feature map, the second image feature map, and the horse state information In the feature map generation, an embodiment in which the horse state information has a probability value P _s is shown.

Specifically, the complex feature map generator 240 performs a process of merging, combining, or concatenating or hashing each feature map after assigning a weight to the input feature map. As a method of merging such a feature map, various feature map merging algorithms known in the field of artificial neural networks can be used.

According to one embodiment of the invention, the end of the status information is the probability value for that and the end of speech feature map in the case that is implemented by (P _s between 0 and 1), in the case of speech feature map before it is input to the feature maepmeo jingbu to become a weight P _s applied, and a second map applied when the image feature weight is P _s-1 for the mouth. Thus, when the subject is speaking, the influence of the voice feature map is increased, and the composite feature map is generated with the influence of the second image feature map on the mouth shape being small. On the other hand, if the assessor is not speaking, the influence of the voice feature map is reduced, and the composite feature map is generated with the influence of the second image feature map on the mouth shape becoming larger. Such application of the state information may be implemented by multiplying the feature map with the state information as the probability value in the feature map.

FIG. 8 schematically illustrates the operation of the context-aware adaptive reasoning unit 250 according to an embodiment of the present invention.

In one embodiment of the present invention, the analysis result inference unit 200 derives analysis result information from the speech feature information, the image feature information, the speech state information, and the context recognition parameter.

Specifically, the analysis result estimating unit 200 derives analysis result information by applying the context awareness parameter to the complex feature map generated based on the speech feature information, the image feature information, and the speech state information.

As shown in Fig. 8, a composite feature map is generated for each sampling period. For example, if there are 1 to #Nth sampling intervals, N complex feature maps can be generated.

Here, the sampling interval from 1 to #N may correspond to the entire interval of the video and audio information, or may correspond to a partial interval of the video and audio information.

Meanwhile, the N compound feature maps are input to the context-aware adaptive reasoning unit 250. Specifically, the context-aware adaptive reasoning unit 250 includes a plurality of input machine learning model units 242.1 such as the above-described circular neural network, a context-aware parameter application unit 242.2 such as a dynamic parameter hierarchy, And an FC module 242.3 that processes the feature map derived through the FC layer 242.2.

As such, each of the N composite feature maps (including information about voices and images) in the entire sampling interval is input to the multiple-input machine learning model unit 242.1 to generate a first final feature map, The feature map is converted into a second final feature map according to the context-aware parameters input by the context-aware parameter application 242.2. The second final feature map may then be FC and output in the form of diagnostic result information.

FIG. 9 schematically shows the internal configuration of the context-aware parameter generation unit 400 according to an embodiment of the present invention.

The situation recognition parameter generation unit 400 includes a situation determination unit 410 for extracting situation audio information, which is audio information of a person other than the subject who is the subject of the audio information, based on whether or not the audio information is output and the speech state information. A non-content feature map generation unit 420 for generating a non-content feature map for a speech feature from the speech information; A content feature map generation unit 430 for generating a content feature map for the actual content from the audio information; A feature map matching unit 440 for matching or merging the non-content feature map and the content feature map; And a context-aware parameter deriving unit 450 for generating context-aware parameters from the matched feature map.

That is, the context awareness parameter generation unit 400 includes a content feature map generation unit 430 for generating a content feature map for the content of speech from the context audio information; And a non-content feature map generation unit (420) for generating a non-content feature map for the speech characteristic from the context speech information.

The other In the embodiment , remind The context-aware parameters  Or may be generated directly by the user's input.

The other In the embodiment , remind The context-aware parameters  remind Matched Feature map  ≪ / RTI > and user input information.

Meanwhile, the context awareness parameter generator 400 may be generated from speech information for a preset interval or an entire interval. In this case, the speech state information # 1 to #N for the section is additionally considered. Such speech state information corresponds to information for extracting the voice of the evaluator from the entire voice, not the subject of the evaluation.

FIG. 10 schematically shows the operation of the situation determination unit 410 according to an embodiment of the present invention.

As shown in FIG. 10, the situation determination unit 410 includes a situation determination unit 410 for extracting the situation information, which is the voice information of a person other than the subject who is the subject of the voice information, .

Specifically, when the audio information for the entire interval is given, the situation determination unit 410, and speech is output, the end state information pipyeonggaja does not have a say in or end of a probability value P _S is a predetermined reference or less , It is determined that the section is the context voice information.

The non-content feature map generation unit 420 and the content feature map generation unit 430 generate feature maps from the context audio information.

FIG. 11 schematically shows an internal configuration of a non-content feature map generation unit 420 and a content feature map generation unit 430 according to an embodiment of the present invention.

The context awareness parameter generation unit 400 includes a content feature map generation unit 430 for generating a content feature map for the content of speech from the context speech information; And a non-content feature map generation unit (420) for generating a non-content feature map for the speech characteristic from the context speech information.

The content feature map generation unit converts the voice uttered by the subject to text into a text and generates a feature map for the substantive contents. For example, if an affirmative answers a given question, then the feature map for the substantive content of the answer corresponds to the content feature map. As an example of such substantive contents, the lexical feature may be applicable. For example, a feature map according to characteristics such as a frequency with respect to a specific expression (positive expression, negative expression, avoidance expression, defensive expression, and unspecified expression, etc.) can be implemented as an example of a content feature map have.

On the other hand, the non-content feature map corresponds to the utterance characteristic of the speech uttered by the subject. For example, feature maps according to characteristics of one or more of speech accents, tones, pitches, speeds, breaths, and rhyme can be implemented as an example of a non-content feature map.

Specifically, the non-content feature map generation unit 420 includes a context-aware feature extraction unit 421 for extracting a speech characteristic of a speech of a partial sampling period; A non-content single input machine learning model unit 422 for generating a spare feature map from the extracted utterance characteristic; And a non-content multiple input machine learning model unit 423 for generating a non-content feature map from the feature maps in the partial sampling periods.

The non-content multiple input machine learning model unit 422 may be implemented as an artificial neural network model that extracts analysis results from a single input such as DNN or CNN, And extracting the analysis result from a plurality of inputs of the input device.

On the other hand, the content feature map generation unit 430 includes a text extraction unit 431 for converting speech into text information; And a content multiple input machine learning model unit 432 for generating a content feature map from the text information extracted by the text extraction unit.

The content multiple input machine learning model unit 432 may be implemented as an artificial neural model that extracts analysis results from a plurality of inputs using a circular neural network.

Figure 12 schematically illustrates the steps of an automated assessor analysis method in accordance with an embodiment of the present invention.

The automated evaluator analysis method can be performed by the automated evaluator analysis system described above, and some of the parts overlapping with the description of the system will be omitted for convenience.

An automated assessor analysis method in accordance with an embodiment of the present invention is implemented in a computing device that includes one or more processors and one or more memories.

Specifically, the method includes a feature information generation step (S100) of generating speech feature information and image feature information from input image information and speech information; A horse state inferring step (S200) of generating horse state information on whether the assessor is speaking from information included in the image feature information; And an analysis result inferring step S400 of deriving analysis result information of the subject from the information including the speech feature information, the image feature information, and the speech state information.

Preferably, the method further includes a context-aware parameter generation step (S300) of generating context-aware parameters including the input speech information and the evaluation status of the assessor from the speech status information, The analysis result information can be derived by further considering the context awareness parameter.

Preferably, the characteristic information generating step includes: a voice characteristic information generating step of generating voice characteristic information; And an image feature information generating step of generating image feature information, wherein the image feature information generating step includes: a first image feature information extracting step of extracting at least one element inside a face outside the mouth area from the input image information; And a second image feature information extracting step of extracting an element of the input area from the input image information.

Advantageously, the horse state inference step may generate the horse state information from the second image feature information.

Preferably, the analysis result inferring step includes: a voice feature map generating step of generating a voice feature map from the voice feature information; A first image feature map generation step of generating a first image feature map from the first image feature information; And a second image feature map generation step of generating a second image feature map from the second image feature information.

Preferably, the analysis result inference step may assign each of the weights to the voice feature map and the second image feature map based on the speech state information to determine the voice feature map, the first image feature map, And a composite feature map generation step of generating a composite feature map from the second image feature map.

Preferably, the automated assessor analysis method may further include a context-aware parameter generation step of generating context-aware parameters including input voice information and information on an evaluation status of an assessor from the speech status information.

Preferably, the context recognition parameter generation step may include a situation determination step of extracting context audio information, which is speech information of a person other than the subject of evaluation, of the speech information from whether the speech information is audible output or from the speech state information.

Preferably, the context recognition parameter generation step includes: a content feature map generation step of generating a content feature map for the content of speech from the context audio information; And generating a non-content feature map for the utterance characteristic from the context speech information.

Preferably, the analysis result inference step may derive analysis result information from the speech feature information, the image feature information, the speech state information, and the context awareness parameter.

Preferably, the analysis result inference step may derive analysis result information by applying the context awareness parameter to a complex feature map generated based on the speech feature information, the image feature information, and the speech state information.

FIG. 13 schematically illustrates a feature map generation process according to an embodiment of the present invention.

FIG. 13 exemplarily shows a process of generating a feature map for an input area using a CNN artificial neural network model. The method illustrated in FIG. 13 may be performed by the second image feature map generation unit 230. FIG.

For example, referring to FIG. 3, the second image feature information having a size of 64 × 64 is convolved with 32 filters having a size of 5 × 5 to output 32 outputs of 60 × 60 size , It is reduced to 30 X 30 by using 3 X 3 max pooling. 32 results of 30 X 30 are convolved with 32 filters of 5 X 5 size, and 3 X 3 max pooling is performed again to output 32 results of 13 X 13 size. Here, 64 convolutions of 5 x 5 size are convolved and 64 results of 9 x 9 size are displayed. The boundary is filled with 0 to make 10 x 10 and 3 x 3 max pooling is applied to 5 x 5 You get 64 results of size.

We then obtain a feature map consisting of the last 128 vectors through FC with 512 weights and FC with 128 weights. For example, the number of final output values may be equal to the number of characteristic information for the mouth shape to be obtained.

The second image feature map may be the final 128 output values itself, or may correspond to a feature map convolved at the middle of the phrase. The configuration of the CNN-based artificial neural network module can be formed by combining at least one convolution layer, a pooling layer, an activation function such as ReLu, and an FC layer or a global average pooling layer.

The first feature feature map generation unit 220, the second feature feature map generation unit 230, and the second feature feature map generation unit 230 may be configured to generate the feature map, for example, a voice feature map generation unit 210, The learning model unit 242.1, the non-content feature map generation unit 420, and the content feature map generation unit 430 may be implemented by various modules capable of implementing an artificial neural network model. However, And that the output information is derived based on the learned result.

The first image feature map generator 220, the second image feature map generator 230, the multiple input machine learning model unit 242.1, the non-content feature map generator 230, (420) and the content feature map generation unit (430) are assumed to have been learned by various learning data, and may be continuously learned.

In the present invention, instead of using a single artificial neural network model that learns a given moving picture information or video / audio information as a whole, a feature map is generated using an artificial neural network model, The generated feature map is merged according to the horse state information. After that, the situation recognition parameter is applied to analyze the analyzed subject using another artificial neural network model. Therefore, unlike the artificial neural network model that is applied to the whole process, the artificial neural network model is applied to each of the detailed processes, and after data processing is performed again, it is possible to obtain a more accurate result by applying it to the artificial neural network model again .

Figure 14 schematically illustrates the overall configuration of an automated assessor analysis system 1000 in accordance with one embodiment of the present invention.

In the embodiment shown in FIG. 14, the image and audio information for the face portion are extracted and extracted from the given moving picture information or voice and image information, the voice information is converted into the feature map through the CNN artificial neural network model, After the mouth area and the outside mouth area are respectively converted into the feature map through the CNN artificial neural network model, the three feature maps are converted into the complex feature map according to the horse state information as the probability value.

Then, the complex feature map is input to a circular neural network model such as LSTM. Then, the feature map output by the circumstantial parameter and the circular neural network is input to the dynamic parameter hierarchy, and the final result is generated through the FC hierarchy.

On the other hand, in the case of the automatic mode in FIG. 14, context recognition parameters in all or partial sections of the moving picture are derived with reference to the speech state information, and in the semi-automatic mode, Based feature map is generated and context-aware parameters are derived, and in the passive mode, the user's input is directly converted into context-aware parameters.

FIG. 15 schematically illustrates a user screen for an analysis result of an automated assessor analysis system 1000 according to an embodiment of the present invention.

As shown in FIG. 15, in the embodiment of the present invention, the diagnosis result information includes the following social skill evaluation items: lively, smiling face, sight alignment, familiarity, reliability, confidence, concentration, , Recommendation level, calmness, authenticity, and so on. Such an evaluation item may correspond to the value of each item of the vector obtained through the FC layer in the context-aware adaptive reasoning unit 250.

In addition, as in the emotional characteristic, the diagnostic result information may include surprise, happiness, love, contempt, angry, fear, sadness, and discomfort. And the value of each item of < RTI ID = 0.0 >

According to the present invention, the potential items that can be an important factor in the evaluation of the evaluator are detected by the artificial neural network model and reflected thereby, and the resultant value for the social skill and emotion can be accurately derived.

In the prior art, such potential factors may not be reflected when automatically evaluating the assessor based on the predetermined feature information (for example, the length-width ratio in the shape of the eye). On the other hand, when an artificial neural network module is constructed for the entire moving image, the computation load is overloaded, and accurate results may not be obtained.

On the other hand, in the automated assessor analysis system 1000 according to the embodiment of the present invention, while the artificial neural network model is optimized, the noise load in the learning model is further reduced by performing other data processing on the horse state, Can be lowered.

16 illustrates an exemplary internal configuration of a computing device according to an embodiment of the present invention.

16, computing device 11000 includes at least one processor 11100, a memory 11200, a peripheral interface 11300, an input / output subsystem I / Osubsystem) 11400, a power circuit 11500, and a communication circuit 11600. At this time, the computing device 11000 may correspond to the user terminal A connected to the tactile interface device or the computing device B described above.

Memory 11200 can include, for example, a high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory or non-volatile memory. have. The memory 11200 may include software modules, a set of instructions, or various other data required for operation of the computing device 11000.

At this point, accessing memory 11200 from other components, such as processor 11100 or peripheral device interface 11300, may be controlled by processor 11100. The processor 11100 may be configured as a single or a plurality of processors and may include a processor in the form of a GPU and a TPU in order to improve an arithmetic processing speed.

Peripheral device interface 11300 may couple the input and / or output peripheral devices of computing device 11000 to processor 11100 and memory 11200. The processor 11100 may execute a variety of functions and process data for the computing device 11000 by executing a software module or set of instructions stored in the memory 11200.

The input / output subsystem 11400 may couple various input / output peripherals to the peripheral interface 11300. For example, input / output subsystem 11400 may include a controller for coupling a peripheral, such as a monitor, keyboard, mouse, printer, or a touch screen or sensor, as needed, to peripheral interface 11300. According to another aspect, the input / output peripheral devices may be coupled to the peripheral device interface 11300 without going through the input / output subsystem 11400.

Power circuitry 11500 may provide power to all or a portion of the components of the terminal. For example, the power circuit 11500 may include one or more power supplies, such as a power management system, a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, And may include any other components for creation, management, distribution.

Communication circuitry 11600 may enable communication with other computing devices using at least one external port.

Or as described above, communication circuitry 11600 may, if necessary, enable communications with other computing devices by sending and receiving RF signals, also known as electromagnetic signals, including RF circuitry.

16 is merely an example of the computing device 11000, and the computing device 11000 may have the additional components omitted in Fig. 16, or further components not shown in Fig. 16, Lt; RTI ID = 0.0 > components. ≪ / RTI > For example, in addition to the components illustrated in FIG. 16, a computing device for a mobile communication terminal may further include a touch screen, a sensor, and the like. The communication device 1160 may be connected to various communication methods (WiFi, 3G, LTE , Bluetooth, NFC, Zigbee, etc.). The components that may be included in computing device 11000 may be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing or application specific integrated circuits.

The methods according to embodiments of the present invention may be implemented in the form of a program instruction that can be executed through various computing devices and recorded in a computer-readable medium. In particular, the program according to the present embodiment can be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied can be installed in a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transfer unit (not shown) for transferring the file according to a request from the user terminal.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computing device and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (23)

An automated assessor analysis system,
A feature information generation unit for generating speech feature information and image feature information from input image information and speech information;
A horse state reasoning unit for generating horse state information on whether an evaluated person is speaking from information included in the image feature information; And
And an analysis result speculation unit for deriving analysis result information of the subject from the information including the speech feature information, the image feature information, and the speech state information,
Wherein the feature information generating unit comprises:
A voice feature information generation unit for generating voice feature information; And
And an image characteristic information generating unit for generating image characteristic information,
Wherein the image feature information generating unit comprises:
A first image feature information extracting unit for extracting at least one element inside the face from the input image information from the input image information; And
And a second image feature information extraction unit for extracting an element of the input area from the input image information,
As a result of the analysis,
A voice feature map generation unit for generating a voice feature map from the voice feature information;
A first image feature map generator for generating a first image feature map from the first image feature information;
A second image feature map generator for generating a second image feature map from the second image feature information; And
Assigning a weight to each of the voice feature map and the second image feature map based on the horse state information to generate a composite feature map from the voice feature map, the first image feature map, and the second image feature map, And a composite feature map generation unit for generating a composite feature map,
The complex feature map generation unit relatively increases the weight of the voice feature map as compared with the case where the probability value of whether the assessor is speaking is low when the probability value of whether the assessor is speaking in the speech state information is high, Wherein a weight of the second image feature map is relatively lowered to generate a composite feature map.
delete The method according to claim 1,
The word-
And generates the speech state information from the face image information of the input image information or the second image feature information.
delete delete The method according to claim 1,
The automated assessor analysis system,
Further comprising a context-aware parameter generating unit for generating context-aware parameters including input voice information and information on an evaluation status of an evaluator from the speech status information.
The method according to claim 1,
The automated assessor analysis system,
And a situation-aware parameter generating unit for generating a situation-aware parameter including information on an evaluation situation of the evaluator according to a user's input.
The method of claim 6,
Wherein the context-aware parameter generator comprises:
And a situation discriminating section for extracting situation audio information, which is audio information of a person other than the subject who is the subject of the audio information, based on whether or not the audio information is output from the speech and the speech state information.
The method of claim 8,
Wherein the context-aware parameter generator comprises:
A content feature map generation unit for generating a content feature map for the contents of speech from the context audio information; And
And a non-content feature map generator for generating a non-content feature map for the speech characteristic from the context speech information.
delete delete What is claimed is: 1. An automated assessor analysis method implemented in a computing device comprising at least one processor and at least one memory,
A feature information generating step of generating speech feature information and image feature information from input image information and audio information;
A word state inference step of generating word state information on whether the assessor is speaking from information included in the image characteristic information; And
And an analysis result inferring step of deriving analysis result information of the subject from the information including the speech feature information, the image feature information, and the speech state information,
The feature information generating step may include:
A voice feature information generation step of generating voice feature information; And
An image feature information generating step of generating image feature information,
The image feature information generating step may include:
A first image feature information extracting step of extracting one or more elements in a face outside the input area from the input image information; And
And a second image feature information extraction step of extracting an element of the input area from the input image information,
As a result of the analysis,
A voice feature map generation step of generating a voice feature map from the voice feature information;
A first image feature map generation step of generating a first image feature map from the first image feature information;
A second image feature map generation step of generating a second image feature map from the second image feature information; And
Assigning a weight to each of the voice feature map and the second image feature map based on the horse state information to generate a composite feature map from the voice feature map, the first image feature map, and the second image feature map, And a composite feature map generation step of generating a composite feature map,
The complex feature map generating step may increase the weight of the voice feature map relative to the case where the probability value of whether the assessor is talking is low or not when the probability value of whether the assessor is speaking in the speech state information is high And relatively weighting the second image feature map to produce a composite feature map.
delete The method of claim 12,
The verbal state inference step comprises:
And generates the speech state information from the face image information of the input image information or the second image feature information.
delete delete The method of claim 12,
The automated assessor analysis method comprises:
Further comprising: a context-aware parameter generation step of generating context-aware parameters including input voice information and information on an evaluation situation of an evaluator from the speech state information.
The method of claim 12,
The automated assessor analysis method comprises:
Further comprising a situation-aware parameter generation step of generating a situation-aware parameter including information on an evaluation situation of the evaluator according to a user's input.
18. The method of claim 17,
The context-aware parameter generation step includes:
And a situation determination step of extracting, from the speech state information, whether or not speech of the speech information is outputted, and situation speech information which is speech information of a person other than the subject of the speech.
The method of claim 19,
The context-aware parameter generation step includes:
A content feature map generation step of generating a content feature map for the contents of speech from the context audio information; And
And a non-content feature map generation step of generating a non-content feature map for a speech characteristic from the context speech information.
delete delete 22. A computer-readable medium,
The computer-readable medium storing instructions that cause a computing device to perform the steps of:
A feature information generating step of generating speech feature information and image feature information from input image information and audio information;
A word state inference step of generating word state information on whether the assessor is speaking from information included in the image characteristic information; And
And an analysis result inferring step of deriving analysis result information of the subject from the information including the speech feature information, the image feature information, and the speech state information,
The feature information generating step may include:
A voice feature information generation step of generating voice feature information; And
An image feature information generating step of generating image feature information,
The image feature information generating step may include:
A first image feature information extracting step of extracting one or more elements in a face outside the input area from the input image information; And
And a second image feature information extraction step of extracting an element of the input area from the input image information,
As a result of the analysis,
A voice feature map generation step of generating a voice feature map from the voice feature information;
A first image feature map generation step of generating a first image feature map from the first image feature information;
A second image feature map generation step of generating a second image feature map from the second image feature information; And
Assigning a weight to each of the voice feature map and the second image feature map based on the horse state information to generate a composite feature map from the voice feature map, the first image feature map, and the second image feature map, And a composite feature map generation step of generating a composite feature map,
The complex feature map generating step may increase the weight of the voice feature map relative to the case where the probability value of whether the assessor is talking is low or not when the probability value of whether the assessor is speaking in the speech state information is high And relatively weighting the second image feature map to produce a composite feature map.

Images