KR20220037129A - Method for providing realtime assessment for interview with interviewee and apparatus using the same - Google Patents

Abstract

Disclosed in the present disclosure is a method for providing a real-time interview evaluation for an interviewee and a computing device using the same. Specifically, according to the method of the present disclosure, the method performs: a non-linguistic analysis process in which a computing device acquires video data and sound data of an interviewee, and non-linguistic analysis data for the face of the interviewee detected over a first time unit including a time interval of at least one frame of the video data is calculated; and a linguistic analysis process in which linguistic analysis data for the interviewee's voice detected over a second time unit among the acoustic data is calculated. Evaluation information provided to or having been provided to an external entity is updated in real time. Hourly evaluation information including at least the advice data among the video data, the sound data, the non-linguistic analysis data, the linguistic analysis data, and the advice data is displayed through a predetermined user interface.

Description

A method for providing real-time interview evaluation for an interviewee and a computing device using the same

Disclosed in the present disclosure are a method for providing real-time interview evaluation for an interviewee and a computing device using the same. Specifically, according to the method of the present disclosure, the computing device acquires image data and sound data of the interviewee, and the image data is detected over a first time unit including a time interval of at least one frame among the image data. performing a non-verbal analysis process of calculating non-verbal analysis data of the interviewee's face, and a verbal analysis process of calculating verbal analysis data of the interviewee's voice detected over a second time unit among the acoustic data; generating advice data for the interviewee based on the non-verbal analysis data and the verbal analysis data; The time-based evaluation information is provided to an external entity through a predetermined user interface, or the evaluation information provided to the external entity is updated in real time.

As competition for employment intensifies, such as diversification of interview types due to recent job difficulties, the demand for preparing for and studying for a job interview in advance is on the rise. Various mock interview services are conventionally provided to prospective interviewees so that they can properly respond to the interview accompanying employment.

For example, Japanese Patent Application Laid-Open No. 2016-153883 discloses a personality evaluation support system capable of appropriately evaluating a personality or the like by using a technique for detecting emotions based on a subject's voice in an employment test or the like. While it is premised on the premise that the interview evaluation of the present disclosure discussed below has room for improvement by appropriate training, the personality evaluation in this patent document is based on a judgment about the subject's immutable nature. In addition to this, the method described in this document is limited to the analysis of voice and has a disadvantage in that it is difficult to accurately grasp emotions because non-verbal factors such as facial expressions and gestures of the subject are not reflected together.

For example, cross-validation with other factors is not necessary to more accurately determine whether a subject actually corresponds to a specific emotional state because, for example, the subject is more likely to mistake the subject for an angry emotional state for a voice with the characteristic of an angry voice. you will need In short, depending on the characteristics of each individual's different voices, a kind of prejudice may be involved in recognizing emotions, and it is difficult to accurately grasp the personality of a subject by recognizing a temporary emotional state.

In preparation for the above document that considers speech-language characteristics, Japanese Patent Application Laid-Open No. 2018-060374 specifies an interviewee's body part from video data, detects the motion of the specified part, and detects the detected body part. An interview evaluation support system based on non-verbal information is disclosed that extracts a behavior based on the behavior of , classifies it according to whether it is a behavior during utterance or not, and evaluates it based on a predetermined evaluation criterion. Also, there is a difficulty in that verbal and non-verbal information are not evaluated together.

As in these prior patent documents, in order to overcome the limitations caused by considering only one of verbal information and non-verbal information, not only voice but also images about expression information based on face or attitude for interview participants can be considered, for example, For example, Japanese Patent Application Publication No. 6042015 includes a video data analysis unit that acquires expression information based on the face or attitude of an interview participant, and an audio data analysis unit that obtains expression information based on speech content or sound of an interview participant. An online interview evaluation device is presented. The evaluation criteria are, at least, composed of semantic features of speech, vocal features, facial features, and behavioral features, and are improved compared to the above-mentioned Japanese Patent Laid-Open Publications.

Even though the evaluation method has been improved so that it can be based on both video and audio, it is still difficult to know which non-verbal response or verbal response was evaluated positively or negatively at what point in time because the evaluation based on video and audio is still given as a whole. Bar, this has the disadvantage that these interview evaluation methods are not very helpful to the prospective interviewees. This seems to be because such prior patent documents are mainly aimed at evaluation rather than learning of the interviewee.

As such, the conventional mock interview evaluation service does not evaluate the user's interview results in real time, but rather comprehensively evaluates the overall evaluation. In addition, it is possible to obtain advice (coaching) information on how the evaluation of the interviewee can be improved, and also provides practical advice on how to improve the behavior, facial expression, or voice response at each point in time series. This is to suggest a technical way to continuously improve the interview evaluation by providing an improved response to the interviewee by providing positive feedback.

Japanese Patent Laid-Open No. 2016-153883 Japanese Patent Laid-Open No. 2018-060374 Japanese Patent Publication No. 6042015

Therefore, the present disclosure solves the problems of the prior art, so that the evaluation of the image and audio of how the person conducted the interview can be grasped for each time period, as well as the evaluation information about the interview and this so that it can be grasped in real time An object of the present invention is to provide a method for receiving advice about and an apparatus for using the same.

The characteristic configuration of the present invention for achieving the object of the present invention as described above and for realizing the characteristic effects of the present invention to be described later is as follows.

According to one aspect of the present disclosure, there is provided a method of providing a real-time interview evaluation for an interviewee, the method comprising: a computing device, acquiring image data and sound data of the interviewee, or other methods linked to the computing device a data acquisition step of supporting a device to acquire the image data and the sound data; A non-verbal analysis process in which the computing device calculates, by the computing device, non-verbal analysis data of the face of the interviewee detected over a first time unit including a time interval of at least one frame among the image data, and a second of the sound data an analysis step of performing a linguistic analysis process of calculating linguistic analysis data of the interviewee's voice detected over time or supporting the other device to perform; a coaching data generating step of generating, by the computing device, coaching data for the interviewee based on the non-verbal analysis data and the verbal analysis data or supporting the other device to generate the coaching data; and the computing device provides time-wise evaluation information including at least the coaching data among the image data, the sound data, the non-verbal analysis data, the verbal analysis data, and the coaching data to an external entity through a predetermined user interface. or performing an output process of updating the evaluation information provided to the external entity in real time, or providing evaluation information for supporting the other device to perform the output process.

In an embodiment, the real-time data acquisition step to the evaluation information providing step are repeatedly performed for the image data and the sound data acquired in real time, or the analysis step for the image data and the sound data acquired at one time to the coaching data generation step may be repeatedly performed.

Preferably, the non-verbal analysis process includes: obtaining an image unit by dividing the image data into the first time unit; and analyzing the separated image unit.

More preferably, the analyzing of the separated image unit may include: detecting the face of the interviewee from the image unit, and performing an analysis on the facial expression of the interviewee on the detected face; detecting a feature point of the interviewee from the image unit, and analyzing the motion of the interviewee based on the detected feature point; and detecting the gaze of the interviewee from the image unit, and performing analysis on the detected gaze.

Advantageously, the linguistic analysis process comprises: separating the sound data into the second time unit to obtain a sound unit; and analyzing the separated acoustic units to analyze acoustic information including at least one of pitch, tone, jitter, and shimmer of the interviewee's voice.

According to another aspect of the present invention, there is also provided a computer program comprising instructions (instructions) implemented to perform the method according to the present invention.

According to another aspect of the present invention, there is provided a computing device for providing a real-time interview evaluation for an interviewee, the apparatus comprising: a communication unit for acquiring image data and sound data of the interviewee; and a non-verbal analysis process of calculating non-verbal analysis data of the face of the interviewee detected over a first time unit including a time interval of at least one frame of the image data, and detection over a second time unit of the sound data and a processor for performing an analysis process including a linguistic analysis process for calculating linguistic analysis data for the interviewee's voice, or for supporting another device interworking through the communication unit to perform, the processor comprising: a coaching data generation process for generating coaching data for the interviewee based on the non-verbal analysis data and the verbal analysis data or supporting the other device to generate the coaching data; and providing timely evaluation information including at least the coaching data among the image data, the sound data, the non-verbal analysis data, the verbal analysis data, and the coaching data to an external entity using a predetermined user interface through the communication unit Alternatively, the evaluation information providing process of updating the evaluation information being provided to the external entity in real time is performed or the other device is supported to perform.

According to an embodiment of the present disclosure, it is possible to grasp the evaluation of interview performance in real time and to receive advice thereon, so that effective learning about the interview is possible.

According to an embodiment of the present disclosure, there is an effect that not only evaluation results based on various interview evaluation factors, including non-verbal factors such as gaze, facial expression, movement, etc., but also appropriate advice can be evaluated in real time together with an image.

Therefore, according to an embodiment of the present disclosure, it is possible to receive one's interview evaluation information in real time along with the video, so that it is possible to determine at what point in time a good evaluation or a bad evaluation was received, so that you can prepare for an effective interview. there is an effect

The accompanying drawings for use in the description of the embodiments of the present invention are only a part of the embodiments of the present invention, and for those of ordinary skill in the art to which the present invention pertains (hereinafter referred to as "those skilled in the art") Other drawings may be obtained based on these drawings without an effort leading to the invention.
1 is a conceptual diagram schematically illustrating an exemplary configuration of a computing device performing a method for providing a real-time interview evaluation for an interviewee (hereinafter referred to as a “real-time interview evaluation providing method”) according to an embodiment of the present disclosure. .
2 is a conceptual diagram exemplarily illustrating a hardware and software architecture of a computing device performing a method for providing a real-time interview evaluation according to an embodiment of the present disclosure.
3 is a flowchart exemplarily illustrating a method for providing a real-time interview evaluation according to an embodiment of the present disclosure.
4 is a diagram exemplarily showing a user interface for providing hourly evaluation information including advice data in a method for providing a real-time interview evaluation according to an embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following detailed description of the present invention refers to the accompanying drawings, which show by way of illustration a specific embodiment in which the present invention may be practiced, in order to clarify the objects, technical solutions and advantages of the present invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. In the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the embodiments are not limited to the specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Although terms such as “first” or “second” may be used to describe various elements, these terms should be interpreted only for the purpose of distinguishing one element from another, and no order is implied. because it doesn't For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected to” another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features, number, step , it should be understood that it does not preclude the possibility of the existence or addition of an operation, a component, a part, or a combination thereof.

In this disclosure, 'learning' or 'learning' is a term that refers to performing machine learning through procedural computing, and is not intended to refer to a mental action such as human educational activity. A person skilled in the art will understand.

In addition, the term 'interviewee' used in this disclosure refers to a person subject to interview evaluation, and should be construed to include not only those who participate in the actual interview, but also those who participate in mock interview evaluation or interview evaluation practice. .

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Moreover, the present invention encompasses all possible combinations of the embodiments indicated in the present disclosure. It should be understood that various embodiments of the present invention are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it should be understood that the position or arrangement of individual components in each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description given below is not intended to be taken in a limiting sense. Like reference numerals in the drawings refer to the same or similar functions throughout the various aspects.

In this specification, unless indicated otherwise or clearly contradicted by context, items referred to in the singular encompass the plural unless the context requires otherwise. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

Hereinafter, in order to enable those skilled in the art to easily practice the present invention, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram schematically illustrating an exemplary configuration of a computing device performing a method for providing a real-time interview evaluation according to an embodiment of the present disclosure.

Referring to FIG. 1 , a computing device 100 according to an embodiment of the present disclosure includes a communication unit 110 and a processor 120 , and communicates with an external computing device (not shown) through the communication unit 110 . Can communicate directly or indirectly.

Specifically, the computing device 100 may include typical computer hardware (eg, a computer, a processor, a memory, storage, an input device and an output device, a device that may include other components of a conventional computing device; a router; , electronic communication devices such as switches, etc.; electronic information storage systems such as network-attached storage (NAS) and storage area networks (SANs)) and computer software (ie, enabling the computing device to operate in a specific way). to achieve the desired system performance by using a combination of instructions that make it function as The storage may include a storage device such as a hard disk and a Universal Serial Bus (USB) memory, as well as a storage device based on a network connection such as a cloud server.

The communication unit 110 of such a computing device may transmit and receive a request and a response between another computing device, for example, a mobile terminal, etc. that are interlocked. As an example, such a request and a response are the same TCP (Transmission Control Protocol) session ( session), but is not limited thereto, and may be transmitted and received as, for example, User Datagram Protocol (UDP) datagrams.

Specifically, the communication unit 110 may be implemented in the form of a communication module including a communication interface. For example, the communication interface is WLAN (Wireless LAN), WiFi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMax (World interoperability for Microwave access), HSDPA (High Speed Downlink Packet Access), etc. Including wireless Internet interface of wireless internet interface and short-distance communication interface such as Bluetooth™, RFID (Radio Frequency IDentification), Infrared Data Association (IrDA), UWB (Ultra-WideBand), ZigBee, NFC (Near Field Communication) can do. In addition, the communication interface may indicate any interface (eg, a wired interface) capable of performing communication with the outside.

For example, the communication unit 110 may transmit/receive data to and from other computing devices through the appropriate communication interface as described above. In addition, in a broad sense, the communication unit 110 may include or interwork with a keyboard, a mouse, other external input devices, printing devices, displays, and other external output devices for receiving commands or instructions. In order to enable interaction with a user by displaying and providing an appropriate user interface to a user of a computing device, for example, a portable terminal, the computing device 100 has a built-in display device or an external display device through the communication unit 110 . It is known that it can be linked with For example, such a display device may be a touch screen capable of a touch input.

In addition, the processor 120 of the computing device includes a micro processing unit (MPU), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU) or a tensor processing unit (TPU), and a cache memory. ), and may include a hardware configuration such as a data bus. In addition, the operating system may further include a software configuration of an application for performing a specific purpose.

2 is a conceptual diagram exemplarily illustrating a hardware and software architecture of a computing device performing a method for providing a real-time interview evaluation according to an embodiment of the present disclosure.

An overview of the configuration of the method and apparatus according to the present invention with reference to FIG. 2 , the computing device 100 may include a data acquisition module 210 as a hardware component and/or a software component thereof. The data acquisition module 210 acquires image data and sound data of the interviewee from an external data source or storage device such as a general camera, a 3D camera, or a depth camera.

The non-verbal analysis module 220a receiving the acquired image data is configured to calculate non-verbal analysis data for the face of the interviewee detected from the image data over a first time unit, the data acquisition module 210 and the non-verbal analysis The individual modules shown in FIG. 2 including the module 220a may be implemented by, for example, the communication unit 110 included in the computing device 100 or the interworking of the communication unit 110 and the processor 120 . will be understood by those of ordinary skill in the art.

Here, the first time unit may have various time intervals such as several seconds, several minutes, several hours, etc., but of course, the time interval corresponds to at least one image frame.

Meanwhile, the linguistic analysis module 220b receiving the acquired sound data is configured to calculate linguistic analysis data for the interviewee's voice detected over the second time unit. Like the first time unit, the second time unit may also have various time intervals, such as several seconds, several minutes, and several hours, but of course it may be the same as or different from the first time unit.

The setting of the first time unit and the second time unit is because the interview is a dynamic process and the evaluation thereof also has a time series nature.

Next, the advice data generating module 230 is configured to generate advice data for the interviewee based on the non-verbal analysis data and the verbal analysis data, and the generated advice data is transmitted to the evaluation information providing module 240, The evaluation information providing module 240 functions as an output module to output or update the timely evaluation information including the advice data and provide it to an external entity.

Here, the external entity includes the interviewee or the interviewee, a user of the computing device 100, an administrator, a third party who guides or assists the interviewee's learning, etc., but in addition to this, interview evaluation and appropriate advice data are required. It should be understood that any subject is included if it is a subject. For example, the external entity may be an external AI device including a separate artificial intelligence (AI) hardware and/or software module that utilizes the time-based evaluation information. In addition, 'external' in the external entity is not intended to exclude embodiments in which AI hardware and/or software modules using the information are integrated into the computing device 100, but performs the method of the present invention It should be interpreted as a term to encompass the case where time-based evaluation information, which is the result of hardware and/or software modules, is used as input data for other methods. That is, of course, the external entity may be another computing device itself.

Now, specific functions and effects that can be achieved by each of the components schematically described with reference to FIG. 2 will be described in detail with reference to FIGS. 3 and 4 . Although the components shown in FIG. 2 are exemplified as being realized in one computing device for convenience of description, it will be understood that the computing device 100 performing the method of the present disclosure may be configured such that a plurality of devices may interwork with each other. . For example, the data acquisition module 210 , the non-verbal analysis module 220a , and the verbal analysis module 220b are performed in the first computing device, and the advice data generation module 230 and the evaluation information providing module 24 are It may be configured to be performed on a second computing device that is linked to the first computing device, and a person skilled in the art will be able to configure a plurality of devices in various ways to perform the method of the present disclosure in conjunction with each other. Invar, the terms 'computing device' or 'the computing device' referred to in the method according to the present disclosure should be construed to encompass not only the same device but also different devices.

3 is a flowchart exemplarily illustrating a method for providing a real-time interview evaluation according to an embodiment of the present disclosure.

Referring to FIG. 3 , in the method for providing real-time interview evaluation according to the present disclosure, first, the data acquisition module 210 implemented by the computing device 100 receives image data and sound data of an interviewee from a photographing device or a storage device. and a data acquisition step (S100) of obtaining support or assisting another device interworking with the computing device 100 to acquire the image data and the sound data through the communication unit 110 (S100).

Additional information regarding identification information unique to the interviewee, for example, name, gender, age, interview purpose, etc., may be further input before or after step S100 or simultaneously.

Next, in the method for providing real-time interview evaluation according to the present disclosure, the non-verbal analysis module 220a implemented by the computing device 100 includes a time interval of at least one frame among the image data in a first time unit. A non-verbal analysis process (S210a to S220a) for calculating non-verbal analysis data for the interviewee's face detected over the course of the non-verbal analysis process (S210a to S220a) or the other device interworking through the communication unit 110 is supported to perform the verbal analysis process , a linguistic analysis process (S210b to S210b) in which the linguistic analysis module 220b implemented by the computing device 100 calculates linguistic analysis data for the interviewee's voice detected over a second time unit among the sound data It further includes an analysis step (S200) of performing S220b) or supporting the other device to perform it through the communication unit 100 .

In an embodiment, the non-verbal analysis process (S210a to S220a) includes the steps of obtaining an image unit by dividing the image data into the first time unit (S210a) and analyzing the separated image unit (S220a) include

There may be various methods of analyzing the image unit. For example, in the step of analyzing the image unit ( S220a ), the face of the interviewee is detected from the image unit, and the detected face is Performing an analysis on the facial expression (S222a), detecting a feature point for the interviewee from the image unit, and performing an analysis on the movement of the interviewee based on the detected feature point (S224a), and the image It may include detecting the gaze of the interviewee from the unit, and performing an analysis on the detected gaze (S226a).

Individual artificial neural networks for performing each step (S222a, S224a, S226a) can be utilized. A deep neural network (DNN), which refers to the latest artificial neural network, is a plurality of hidden layers in addition to the input layer and the output layer. layers), and if you use this deep neural network, you can figure out the potential structure of image data.

Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), Q networks, U It may include a network, a Siamese network, a Single Shot Detector (SSD), You Only Look Once (YOLO), and the like.

Each of these deep neural networks specifically detects the face of the interviewee in the method of the present disclosure, performs classification of facial expressions, and performs classification on movement based on feature points detected in the interviewee, eyes of the interviewee can be used to first detect and detect a gaze from that eye.

A deep learning (deep learning) technique such as a convolutional neural network (CNN) may be used for the analysis of the image required in the method of the present disclosure, but when performance is prioritized over quality, that is, short analysis It is known to those skilled in the art that artificial neural network technology such as MobileNetV2 can be utilized when time is of the essence.

For example, the analysis on the facial expression of the interviewee may include a ratio value for each of the facial expression classification items including positive (favorable), non-expressive, negative (dislike), embarrassed, and the like. However, this is only an example, and it goes without saying that the step of performing analysis on criteria other than gaze, expression, and movement may be included as an alternative or additionally.

On the other hand, in another embodiment that can be carried out together with or independently of the above-described embodiment, the linguistic analysis process (S210b to S220b) separating the sound data into the second time unit to obtain a sound unit (S210b) and analyzing the separated acoustic units to perform an analysis on acoustic information including at least one of pitch, tone, jitter, and shimmer of the interviewee's voice (S220b) include This is also only an example, and in addition to the voice of the interviewee, the step of performing an analysis on other criteria such as other sounds (coughing, gasping, trembling, etc.) and the interviewee's voice in addition to the interviewee's voice may be included as an alternative or additionally.

It is known to those skilled in the art that, for example, an open source library such as Praat may be used in the analysis of sound and voice.

Analysis step (S200) Next, in the real-time interview evaluation providing method according to the present disclosure, the advice data generation module 230 implemented by the computing device 100 is based on the non-verbal analysis data and the verbal analysis data. The method further includes an advice data generation step (S300) of generating advice data for the interviewee or supporting the other device to generate the advice data.

Preferably, such advice data may be in the form of a sentence or may correspond to it. For example, if the face of the interviewee detected in a certain image unit in step S222a is not facing the front, it is reflected in the non-verbal analysis data. Based on the time information corresponding to the video unit, 'I am not looking straight ahead. (Please see the front.)' can be generated, and the ratio of expressionless or negative elements in the facial expression analyzed in (S222a) in the video unit that would be appropriate if you smile in the context of an interview question or response If this is high, advice data such as 'I would like you to answer with a smile more' may be generated.

Continuing, for example, if the voice of the interviewee detected in a certain acoustic unit in step S220b is excessively high outside the range expected to be appropriate, based on the linguistic analysis data reflecting this, the time information corresponding to the voice unit together with the time information 'Your voice is too loud. (Please lower your voice a little.)' advice data may be generated.

When the advice data is generated in the advice data generating step (S300), reflecting this, the real-time interview evaluation providing method according to the present disclosure, the evaluation information providing module 240 implemented by the computing device 100 is the image Time-based evaluation information including at least the advice data among data, the acoustic data, the non-verbal analysis data, the verbal analysis data, and the advice data is provided to an external entity through a predetermined user interface or is currently provided to the external entity The method further includes an evaluation information providing step (S400) of performing an output process for updating the evaluation information in real time or supporting the other device to perform the output process.

Here, the term 'renewing in real time' is not only applied to the case where the method according to the present disclosure confirms the evaluation while playing back image and sound data after recording about the interview, but the method is performed with real-time shooting and It is intended to cover a case in which the advice data displayed in real time as feedback along with the photographed image data is updated (or changed).

4 shows an example of a user interface that can be used to provide timely evaluation information including advice data in the evaluation information providing step (S400) of the real-time interview evaluation providing method according to an embodiment of the present disclosure; It is a drawing.

Referring to FIG. 4 , image data is indicated by identification number 410 on the exemplary user interface, and advice data for this is displayed by identification number 420 for each time period.

For example, in the evaluation information providing step ( S400 ), the ratio numerical value 412 for each of the above-described expression classification items may be displayed and provided together in response to the image data 410 as a part of the non-verbal analysis data.

Meanwhile, although the advice data 420 in FIG. 4 is exemplified in the form of a sentence, a person skilled in the art is not limited to the form of such a sentence, and time-based evaluation information including advice data is automatically generated voice, graph, symbol, photo, etc. It will be appreciated that it may be provided in many other forms.

Further, the advice data 420 may be displayed together with information 422 about a time zone to which the advice data is applied, and such time zone information 422 is a time range corresponding to the first time unit or the second time unit. , for example, may be displayed in a range such as 1 minute 20 seconds to 1 minute 40 seconds. It can also be represented as a single point in time corresponding to the middle value of a time range.

In particular, when the method of the present disclosure is applied to real-time shooting for the interviewee or to playback after recording, the advice data 424 corresponding to the current time of shooting or the time during playback will be highlighted as illustrated in FIG. can

As such, according to the performance of the above-described steps of the present disclosure, the interviewee can be provided with evaluation information in real time along with image and sound data, and it is possible to identify at what point in time a good evaluation or a bad evaluation was received by time period. Not only is it possible, but it has the effect of effectively preparing for an interview through the advice provided.

The steps (or corresponding processes) described so far do not mean that they must be performed in the order described unless otherwise logically contradicted or otherwise specified in the context, and each step (or corresponding processes) It will be understood by those skilled in the art that may be performed simultaneously or at different times.

In addition, at least some of the above-described steps may be performed in real time and/or iteratively in the case of playback after recording or when the interviewee obtains advice data at each point in real-time shooting. . For example, in the former case, after the analysis step (S200) to the advice data generation step (S300) are repeatedly performed on the temporarily acquired image data and the sound data, the evaluation information providing step (S400) may be performed, In the latter case, the real-time data acquisition step ( S100 ) to the evaluation information providing step ( S400 ) may be repeatedly performed for the image data and the sound data acquired in real time.

In other words, in the latter case, steps (S100) to (S400) may be repeated for each image unit or sound unit of the image data and sound data obtained in real time, so that the evaluation information for each time period is determined when the interview or mock interview is finished. It can be calculated and provided until

The components shown in FIGS. 1 to 3 are exemplified as being realized in one computing device, for example, a user terminal for convenience of description, but as described above, the computing device 100 performing the method of the present invention includes a plurality of devices. It will be understood that they may be configured as interlocking with each other. Therefore, each step of the method of the present invention described above may be performed by one computing device directly performing or by supporting the one computing device to cause another computing device interlocked with the one computing device to perform. self-evident

Above, based on the description of various embodiments of the present disclosure, those of ordinary skill in the art will recognize that the method and/or processes of the present invention, and the steps thereof, are hardware, software, or hardware and software suitable for a specific application. It can be clearly understood that they can be realized in any combination. The hardware may include general purpose computers and/or dedicated computing devices or specific computing devices or special features or components of specific computing devices. The processes may include one or more processors, eg, microprocessors, controllers, eg, microcontrollers, embedded microcontrollers, microcomputers, arithmetic logic units (ALUs), digital signal processors, with internal and/or external memory. , for example, by a programmable digital signal processor or other programmable device. Additionally, or alternatively, the processes may be implemented using an application specific integrated circuit (ASIC), a programmable gate array, such as a field programmable gate array (FPGA), a programmable logic unit (PLU) or programmable array logic. (Programmable Array Logic; PAL) or any other device capable of executing and responding to instructions, any other device or combination of devices that can be configured to process electronic signals. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of these, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device for interpretation by or providing instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more machine-readable recording media.

Moreover, the objects of the technical solution of the present invention or parts contributing to the prior arts may be implemented in the form of program instructions that can be executed through various computer components and recorded in a machine-readable medium. The machine-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the machine-readable recording medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the machine-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs, DVDs, and Blu-rays, and floppy disks. ), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include any one of the devices described above, as well as heterogeneous combinations of processors, processor architectures, or combinations of different hardware and software, or stored and compiled or interpreted for execution on a machine capable of executing any other program instructions. Machine code, which can be created using a structured programming language such as C, an object-oriented programming language such as C++, or a high-level or low-level programming language (assembler, hardware description languages and database programming languages and techniques) This includes not only bytecode, but also high-level language code that can be executed by a computer using an interpreter or the like.

Accordingly, in one aspect according to the present invention, when the above-described method and combinations thereof are performed by one or more computing devices, the methods and combinations of methods may be implemented as executable code for performing respective steps. In another aspect, the method may be implemented as systems performing the steps, the methods may be distributed in various ways across devices or all functions may be integrated into one dedicated, standalone device or other hardware. In another aspect, the means for performing the steps associated with the processes described above may include any of the hardware and/or software described above. All such sequential combinations and combinations are intended to fall within the scope of this disclosure.

For example, the hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa. The hardware device may include a processor such as an MPU, CPU, GPU, TPU coupled with a memory such as ROM/RAM for storing program instructions and configured to execute the instructions stored in the memory, and an external device and signal It may include a communication unit that can send and receive. In addition, the hardware device may include a keyboard, a mouse, and other external input devices for receiving instructions written by developers.

In the above, the present invention has been described with specific matters such as specific components and limited embodiments and drawings, but these are provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments, Those of ordinary skill in the art to which the present invention pertains can devise various modifications and variations from these descriptions.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and not only the claims appended to the present disclosure but also all modifications equivalently or equivalently to the claims attached to the present disclosure are the spirit of the present invention will fall into the category of For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Such equivalent or equivalent modifications will include, for example, logically equivalent methods capable of producing the same results as practiced by the methods according to the present invention, the spirit and spirit of the present invention. The scope should not be limited by the above examples, and should be understood in the broadest sense permitted by law.

Claims (7)

In the method of providing a real-time interview evaluation for an interviewee,
a data acquisition step of, by a computing device, acquiring the image data and sound data of the interviewee or assisting another device interworking with the computing device to acquire the image data and the sound data;
A non-verbal analysis process in which the computing device calculates, by the computing device, non-verbal analysis data of the face of the interviewee detected over a first time unit including a time interval of at least one frame among the image data, and a second of the sound data an analysis step of performing a linguistic analysis process of calculating linguistic analysis data of the interviewee's voice detected over time or supporting the other device to perform;
an advice data generating step of generating, by the computing device, advice data for the interviewee based on the non-verbal analysis data and the verbal analysis data or supporting the other device to generate the advice data; and
The computing device provides, to an external entity, time-based evaluation information including at least the advice data from among the image data, the sound data, the non-verbal analysis data, the verbal analysis data, and the advice data, to an external entity through a predetermined user interface; An evaluation information providing step of performing an output process of updating the evaluation information provided to the external entity in real time or supporting the other device to perform the output process
A method of providing real-time interview assessments, including According to claim 1,
The real-time data acquisition step to the evaluation information providing step are repeatedly performed for the image data and the sound data acquired in real time, or
A method for providing a real-time interview evaluation in which the analysis step or the advice data generation step are repeatedly performed on the temporarily acquired image data and the sound data. According to claim 1,
The non-verbal analysis process is
obtaining an image unit by dividing the image data into the first time unit; and
analyzing the separated image unit
A method of providing real-time interview evaluation, including. 4. The method of claim 3,
The step of analyzing the separated image unit comprises:
detecting the face of the interviewee from the image unit, and performing an analysis on the facial expression of the interviewee on the detected face;
detecting a feature point of the interviewee from the image unit, and analyzing the motion of the interviewee based on the detected feature point; and
Detecting the gaze of the interviewee from the image unit, and performing an analysis on the detected gaze
A method of providing real-time interview evaluation, including. According to claim 1,
The linguistic analysis process is
obtaining a sound unit by dividing the sound data into the second time unit; and
analyzing the separated acoustic units to analyze acoustic information including at least one of pitch, tone, jitter, and shimmer of the interviewee's voice;
A method of providing real-time interview evaluation, including. A computer program, stored in a machine-readable recording medium, comprising instructions implemented to cause a computing device to perform the method of any one of claims 1 to 5. In the computing device for providing real-time interview evaluation for the interviewee,
a communication unit for acquiring image data and sound data of the interviewee; and
A non-verbal analysis process for calculating non-verbal analysis data of the face of the interviewee detected over a first time unit including a time interval of at least one frame of the image data, and a second time unit of the sound data detected over a second time unit A processor that performs an analysis process including a linguistic analysis process for calculating linguistic analysis data for the interviewee's voice or supports another device interworking through the communication unit to perform it
including,
The processor is
an advice data generating process for generating advice data for the interviewee based on the non-verbal analysis data and the verbal analysis data or supporting the other device to generate the advice data; and
Time-based evaluation information including at least the advice data among the image data, the sound data, the non-verbal analysis data, the verbal analysis data, and the advice data is provided to an external entity through the communication unit using a predetermined user interface, or A real-time interview evaluation providing apparatus for performing an evaluation information providing process of updating the evaluation information being provided to the external entity in real time, or for causing the other device to perform.

Images