KR20230067445A - Method for analyzing user shot video, apparatus and program thereof - Google Patents

Abstract

Disclosed is a device for analyzing user captured images based on artificial intelligence. The device includes: a communication unit, an input unit, a display, one or more processors, and a memory electrically connected to the processor and storing at least one code executed by the processor. By providing the device, user's actions and facial expressions in the images can be effectively recognized.

Description

Method for analyzing user-photographed video based on artificial intelligence, device and program therefor

The present disclosure relates to a method and apparatus for analyzing an image captured by a user based on artificial intelligence, and more particularly, to a method and apparatus for analyzing an image based on a user's behavior and expression.

AI technologies related to computer vision that can be used as existing open sources include object tracking, action recognition, and facial expression recognition. Using existing technology, it is possible to obtain behavioral recognition results and facial expression recognition results for each specific unit (k frames) of a human object appearing in a video.

It is possible to know the result of what action a specific object performed and what expression it had, but the result is information that can be known even if a person directly watches the video, and it is not at all useful to those who need to analyze the video in a special way (such as counselors). This is unhelpful information.

In addition, in the existing technology, when an object reappears after leaving the same image, it is not recognized as the same object, and subsequent actions are interpreted as being performed by different objects, which hinders image analysis.

By interlocking object tracking, action recognition, and facial expression recognition technologies and utilizing Re-ID technology, all actions performed by each object and facial expressions in the same image are tracked and collected, and the results are visually displayed in time series. A proposal is needed.

Registered Patent Publication No. 10-2157313, 2020.09.11.

Embodiments disclosed in the present invention are aimed at providing a data processing method and apparatus using computer vision AI technology related to object recognition in an image.

Another embodiment disclosed in the present invention aims to provide a method of selectively extracting a user's specific behavior and facial expression from an image by determining in advance.

Another embodiment disclosed in the present invention aims to generate data related to a user interest keyword that can be derived from a user's specific behavior and facial expression.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

A method of analyzing a user-captured image based on artificial intelligence according to an aspect of the present disclosure for achieving the above-described technical problem is a label to be output by a recognition model that recognizes a user's behavior and expression in the image when the user-captured image is input ( label), setting a user interest keyword that can be derived from at least one of the user's behavior and facial expression, specifying the user when a user photographed image to be analyzed is input, based on the label, the specified Recognizing the user's behavior and facial expression through the recognition model, generating data related to the user's interest keyword based on the recognized user's behavior and facial expression, and displaying the generated data. .

Setting the user interest keyword may include outputting the user interest keyword by inputting the set label to a pre-learned interest keyword recommendation model.

The step of specifying the user includes mapping unique identification information for each user recognized from the input user-captured image, and the step of recognizing through the recognition model includes mapping unique identification information to each user based on the mapped unique identification information. It may include collecting behaviors and facial expressions of the chronologically.

The generating of the data may include generating data related to a keyword of interest to the user based on the recognized behavior and expression of each user collected in the chronological order.

Generating the data may include generating data related to the keyword of interest to the user by using a pre-learned data generation model.

The displaying may include providing the recognized behaviors and facial expressions of each user collected in the chronological order in chronological order.

The displaying may include providing data related to the keyword of interest in chronological order for each user.

The displaying may include a step of comparing and displaying a user's action or facial expression that may predict a situation or relationship between users.

An apparatus for analyzing a user-captured image based on artificial intelligence according to an aspect of the present disclosure is electrically connected to a communication unit, an input unit, a display, one or more processors, and the processor, and includes at least one code executed by the processor. It may include a memory to be stored.

The memory, when executed by the processor, sets a label to be output by a recognition model recognizing the user's behavior and expression in the image when the processor obtains a user-captured image through the communication unit or the input unit, and the user's behavior and set a user interest keyword derived by at least one of facial expressions, specify a user when a user-captured image to be analyzed is input, and recognize behaviors and facial expressions of the specified user through the recognition model based on the label. and a code that generates data related to the keyword of interest to the user based on the recognized behavior and expression of the user, and causes the generated data to be displayed through the display.

In addition to this, a computer program stored in a computer readable recording medium for execution to implement the present disclosure may be further provided.

In addition to this, a computer readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

In the data processing method using computer vision AI technology related to object recognition in an image according to an aspect of the present invention for solving the above problems, setting a behavior label and a facial expression label according to a user input, Constructing a unique processing logic according to the method, acquiring an image for analysis, tracking an object in the image and assigning a unique ID, recognizing the behavior and expression of the object, and converting the recognized result to the unique ID. It includes interworking with ID, processing data based on the interworking result and the unique processing logic, and visualizing the processed result.

In addition, the recognizing step may include recognizing behavior and facial expression changes of each object in the image over time.

According to the above-mentioned problem solving means of the present disclosure, a user's specific behavior and facial expression can be determined and extracted in advance, and data related to a user's interest keyword that can be derived from the user's specific behavior and facial expression can be generated, so that user convenience is improved. can be improved.

In addition, it is possible to analyze what actions and expressions (emotions) the object performed over time in the video, so it is possible to understand the nature of the video and understand the object in the video, compared to simply presenting the results according to the frame. and facilitate analysis.

In addition, since the information necessary for video analysis can be processed according to one's own situation, if the processed results are summarized and visually confirmed, only the corresponding result can be checked without the need to review the entire video, resulting in the same effect as performing video analysis directly. can be obtained. Through this, it is possible to drastically reduce the time required for image analysis and increase work efficiency.

In addition, the person performing the video analysis can select the result labels output by the AI and set the order of causal and causal by combining specific results. In other words, rather than simply using only the information analyzed by AI, a person can directly set and combine AI analysis logic to actively derive information suitable for his or her situation.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram for schematically explaining an apparatus for analyzing a user-captured image based on artificial intelligence according to an embodiment of the present invention;
2 is a block diagram showing the configuration of a device for analyzing a user-captured image according to an embodiment of the present invention;
3 is a sequence diagram illustrating a method of analyzing a user-captured image based on artificial intelligence according to an embodiment of the present invention;
4 is a diagram for explaining a process of a recognition model according to an embodiment of the present invention;
5 is a diagram for explaining a process of a keyword derivation model and a process of a data generation model according to an embodiment of the present invention;
6 is a diagram for explaining an analysis device that maps and outputs a unique identification number for each user according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

The term "unit" or "module" used in the specification means a hardware component such as software, FPGA or ASIC, and "unit" or "module" performs certain roles. However, "unit" or "module" is not meant to be limited to software or hardware. A “unit” or “module” may be configured to reside in an addressable storage medium and may be configured to reproduce one or more processors. Thus, as an example, a “unit” or “module” may refer to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, datastructures, tables, arrays and variables. Functions provided within components and "units" or "modules" may be combined into smaller numbers of components and "units" or "modules" or may be combined into additional components and "units" or "modules". can be further separated.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram schematically illustrating an apparatus 100 (hereinafter, referred to as “analysis apparatus”) for analyzing a user-captured image based on artificial intelligence according to an embodiment of the present invention.

The analysis device 100 may analyze user-captured images captured in various places using various artificial intelligence algorithms. For example, the analysis device 100 may analyze a captured image 10 taken at a predetermined place (PL, for example, a daycare center), and the analysis device 100 may identify users U1 and U2 in the captured image, , the user's action and/or expression may be recognized, and the user's emotional state may be estimated based on the user's action and/or expression.

As shown in FIG. 1 , the analysis device 100 may recognize specific behaviors (a hitting behavior, a hitting behavior) of the first user U1 and the second user U2, and the second user U2 A distorted facial expression and a tearful facial expression may be recognized, and furthermore, the emotional state of each user U1 or U2 may be estimated based on the behavior and facial expression.

Here, the camera C1 may take a picture of a shooting permitted range of a predetermined place (PL, for example, a daycare center), and may be included as a component of the analysis device 100, but the embodiment is not limited thereto.

As shown, the analysis device 100 may identify a user in a user-captured image, recognize a behavior and/or facial expression of the specified user, and estimate an emotional state of the user.

The above-described analysis device 100 may include all of various devices capable of providing results to users by performing calculation processing. For example, the analysis device 100 may be implemented to include all of a computer, a server device, and a portable terminal, or may be implemented in any one form.

Here, the computer may include, for example, a laptop computer, a desktop computer, a laptop computer, a tablet PC, a slate PC, and the like equipped with a web browser.

The server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

The portable terminal is, for example, a wireless communication device that ensures portability and mobility, and includes a Personal Communication System (PCS), a Global System for Mobile communications (GSM), a Personal Digital Cellular (PDC), a Personal Handyphone System (PHS), and a PDA. (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, smart phone ) and wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted-devices (HMDs). can include

2 is a block diagram showing the configuration of an analysis device 100 according to an embodiment of the present invention.

The analysis device 100 may include a communication unit 110, an input unit 120, a display 130, a memory 150, and a controller 190. The components shown in FIG. 2 are not essential to implement the analysis device 100 according to the present disclosure, so the analysis device 100 described in this specification has more or fewer components than the components listed above. can have elements.

The communication unit 110 may include one or more components enabling communication with an external device, and for example, at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-distance communication module, and a location information module. can include

The input unit 120 is for inputting image information (or signals), audio information (or signals), data, or information input from a user, and includes at least one of at least one camera, at least one microphone, and a user input unit. can do. Voice data or image data collected by the input unit 120 may be analyzed and processed as a user's control command.

A camera processes an image frame such as a still image or a moving image obtained by an image sensor in a photographing mode. The processed image frame may be displayed on a display unit (or a screen of the analysis device of the present disclosure) or stored in a memory.

On the other hand, when there are a plurality of cameras, they can be arranged to form a matrix structure, and a plurality of image information having various angles or focal points can be input through the cameras forming the matrix structure, and the cameras form a three-dimensional image. It may be arranged in a stereo structure so as to obtain left and right images for realizing a stereoscopic image.

A microphone processes an external acoustic signal into electrical voice data. The processed voice data can be used in various ways according to the function (or application program being executed) being performed in the present device. Meanwhile, various noise cancellation algorithms for removing noise generated in the process of receiving an external sound signal may be implemented in the microphone.

The user input unit is for receiving information from a user, and when information is input through the user input unit, the control unit can control the operation of the device to correspond to the input information. The user input unit includes hardware physical keys (for example, a button located on at least one of the front, rear, and side surfaces of the device, a dome switch, a jog wheel, a jog switch, etc.) and a software touch key. can include As an example, the touch key is composed of a virtual key, soft key, or visual key displayed on a touch screen type display unit through software processing, or the touch screen It may be made of a touch key (touch key) disposed in a part other than the part. On the other hand, the virtual key or visual key can be displayed on the touch screen while having various forms, for example, graphic (graphic), text (text), icon (icon), video (video) or these can be made of a combination of

The display unit 130 displays (outputs) information processed by the present device. For example, the display unit may display execution screen information of an application program (eg, application) driven in the present device, or UI (User Interface) and GUI (Graphic User Interface) information according to such execution screen information.

The memory 150 may store data supporting various functions of the device and programs for operation of the control unit, and may store input/output data (eg, music files, still images, moving images, etc.) , a plurality of application programs (application programs or applications) running in the present device, data for the operation of the present device, and instructions can be stored. At least some of these application programs may be downloaded from an external server through wireless communication.

The memory 150 may include an artificial intelligence-based recognition model M1, a keyword derivation model M2, and a data generation model M3.

The memory 150 may be a flash memory type, a hard disk type, a solid state disk type, a silicon disk drive type, or a multimedia card micro type. micro type), card type memory (eg SD or XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable EEPROM (EEPROM) It may include a storage medium of at least one type of a programmable read-only memory (PROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. In addition, the memory 150 is separate from the present device, but may be a database connected by wire or wirelessly.

The control unit 190 performs the above-described operation using the memory 150 that stores data for an algorithm or a program that reproduces the algorithm for controlling the operation of the components in the device, and the data stored in the memory 150. It may be implemented with at least one processor (not shown). Processors may be implemented in plural even when described in the singular. The memory 150 may be electrically connected to the processor and store at least one code executed by the processor. In this case, the memory and the processor may be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.

In addition, the control unit 190 may control any one or a combination of the components described above in order to implement various embodiments according to the present disclosure described in FIGS. 3 to 6 below on the present device.

3 is a sequence diagram illustrating a method of analyzing a user-captured image based on artificial intelligence according to an embodiment of the present invention. 4 is a diagram for explaining a process of a recognition model M1 according to an embodiment of the present invention, and FIG. 5 is a process of a keyword derivation model M2 and a data generation model M3 according to an embodiment of the present invention. ) is a diagram for explaining the process, and FIG. 6 is a diagram for explaining the analysis device 100 that maps and outputs a unique identification number for each user according to an embodiment of the present invention. 3 will be described with reference to the contents of FIGS. 4 to 6 together.

First, when a user-captured image is input, the controller 190 may set a label to be output by the recognition model M1 recognizing the user's behavior and expression in the image (S310).

Here, the label is set to output only desired results among result values output by the artificial intelligence pipeline, and may be a value actually output by the recognition model M1 and an intermediate output value of the network, but the embodiment is not limited thereto. .

The recognition model M1 may set a label to recognize a specific action (eg, sitting, walking, running, standing, lying down, specific movement of a hand or foot, bending a customer, changing direction, etc.) among user actions. A label may be set to recognize a specific facial expression (eg, various facial expressions including a smiling expression, a crying expression, a cute expression, shedding tears, and eye blinking) among facial expressions. A specific action and a specific facial expression may vary according to embodiments. In addition, the recognition model M1 may estimate the user's emotional state based on the recognized behavior and/or facial expression.

Referring to FIG. 4 , the recognition model M1 receives a user-captured image 410, recognizes the action in the case of a user action (420), and recognizes a facial expression in the case of a facial expression (430), and based on this recognizes the emotion. It can be estimated (430). However, the recognition model M1 may first identify and recognize a user before recognizing a behavior and/or facial expression.

As an optional embodiment, the recognition model M1 may include a plurality of networks based on neural networks. A first network is a network for recognizing a user's behavior, a second network is a network for recognizing a user's facial expression, and a third network is a network for recognizing a user's facial expression. The network may be a network for estimating a user's emotional state based on the first network and information collected from the network. All of the plurality of networks may be trained based on supervised learning, and may be repeatedly learned based on ground truth until a predetermined goal is reached.

After step S310 of FIG. 3 , the controller 190 may set a keyword of user interest that can be derived by at least one of the user's behavior and expression (S320).

The user interest keyword may correspond to a unique processing logic for each user that can be derived from at least one of the user's behavior and facial expression, and for example, calm (specific expression or emotion) and sitting (specific behavior) are repeated according to a predetermined sequence. In this case, the user interest keyword may be set to “focus”, but the embodiment is not limited thereto and may be diverse.

In addition, the keyword of interest to the user may be set using the keyword derivation model M2 of the analysis device 100, and will be described with reference to FIG. 5.

The keyword derivation model M2 may derive a user interest keyword by receiving the recognized behavior 510 and the recognized facial expression 520 . The keyword derivation model M2 may set a keyword of interest to the user based on repetition of a specific action, repetition of a specific facial expression, repetition sequence of a specific action and/or specific facial expression, and the like.

The user interest keyword may be a term that can be immediately derived when a certain action is followed by a certain facial expression, and may correspond to a unique processing logic for each user.

That is, the keyword derivation model M2 may be trained in advance, and may output a user interest keyword when a set label is input.

After step S320 of FIG. 3 , the controller 190 may specify a user when a user photographed image as an analysis target is input (S330).

At this time, the controller 190 may map unique identification information for each user recognized from the input user-captured image, which will be described with reference to FIG. 6 .

As shown in FIG. 6 , the controller 190 may map and output unique identification numbers ID5 to ID7 corresponding to the users U5 to U7 in the user-captured image.

The controller 190 may also recognize a case where a user is screened out of a captured image and then screened in, and may identify a user even if the user's appearance changes within a predetermined range.

After step S330 of FIG. 3 , the controller 190 may recognize the behavior and expression of the specified user through the recognition model M1 based on the above-described label (S340).

The recognition model M1 may collect each user's behavior and expression in chronological order based on the mapped unique identification information, and may store corresponding images in chronological order.

After step S340 of FIG. 3 , the controller 190 may generate data related to a keyword of interest to the user based on the recognized behavior and expression of the user (S350).

The controller 190 may generate data related to a keyword of interest to the user based on the recognized behavior and expression of each user collected in chronological order. The controller 190 may calculate and store a new result in the memory 150 according to the unique processing logic set by the user for each frame or predetermined unit.

In addition, the controller 190 may generate data related to the keyword of interest to the user by using a pre-learned data generation model M3, which will be described with reference to FIG. 5 .

Referring to FIG. 5 , the data generation model M3 may receive a user interest keyword and generate data (530).

After step S350 of FIG. 3 , the controller 190 may display the generated data through the display 130 .

The controller 190 may provide the recognized behaviors and facial expressions of each user collected in chronological order in chronological order.

The controller 190 may provide data related to the keyword of interest for each user in chronological order.

The controller 190 may compare the user's action or facial expression, which may predict a situation or relationship between users, and output the comparison to the display 130 .

On the other hand, in the above-described analysis device 100, AI analyzes an image in which a person appears, synthesizes actions and facial expressions (emotions) performed by a specific object over time in the image, and visually displays the result of analyzing them according to time series. If the user selects and recognizes the desired action and facial expression, and designs his or her own special processing logic by combining the action and facial expression, AI summarizes the results according to the designed processing logic when performing image analysis and visually conveys the image. It is about an image analysis AI assistant system that increases analysis efficiency.

The user may set conditions in advance to receive results before performing the analysis using a web UI or the like.

The controller 190 may set an action label and a facial expression label according to the user input.

Here, the action label relates to the action of the object and may include sitting, walking, running, standing up, lying down, and the like. The facial expression label relates to the facial expression of the object and may include calmness, sadness, joy, and the like.

The controller 190 may set to output only desired results among result values output by the AI pipeline. When setting, analysis results can be output by filtering only the values set inside the AI pipeline.

For example, if the action label is preset to “walking” and the expression label is preset to “sadness”, the result value output by the AI pipeline thereafter may be a behavior related to walking and a facial expression related to sadness.

The control unit 190 may configure a unique processing logic according to the user input.

Settings can be made to obtain the results the user wants to analyze. For example, if the expression label "calm" and the action label "sitting" last for more than a few frames, it is "concentrate", and you can configure your own processing logic by combining the labels to obtain the processed result value. can

The controller 190 may obtain an image for analysis.

Depending on the embodiment, a user may upload an image to be analyzed using a user's terminal through a web UI or the like. The controller 190 may acquire the uploaded image through a web UI or the like.

The controller 190 may assign a unique ID by tracking the object in the image.

The controller 190 tracks each of the human objects appearing in the image using AI. By assigning a unique ID to each object appearing in the video, AI can know the object location of each ID for each frame and extract the action recognition and facial expression recognition results for each unit time. These results can be stored in memory.

When a rectangle (hereinafter referred to as a bounding box) containing an object is drawn in an image frame, the control unit 190 takes the leftmost, topmost, rightmost, and bottommost pixel positions of the rectangle as vector values and corresponds them with object IDs. object location is known.

The controller 190 may recognize the object's behavior and expression.

The control unit 190 extracts feature points, etc. using a series of algorithms from the object tracking result, the object from the bounding box including the known object, and the frame cut out around the object, and the AI model that can predict the behavior using this extracts the object ID of each object ID. Action recognition results for each unit can be extracted. These results can be stored in memory.

Expression recognition proceeds in the order of face recognition → expression recognition, and an AI model that extracts feature points using a series of algorithms from frames cut out of the face and face surroundings from the bounding box of the recognized face, and using these, an AI model that can predict facial expressions results in facial expression recognition can be extracted. These results can be stored in memory.

The result of performing action and facial expression recognition until the end of the video is synthesized with the existing video so that the analyzed video can be visually checked, and can also be saved as a text log file.

The controller 190 may link the recognized result with the unique ID.

The control unit 190 performs a task of interlocking the facial expression recognition result with the ID for each object calculated during object tracking. As a result of object tracking, the controller 190 checks whether the bounding box of the object being tracked overlaps with the bounding box of the recognized face as a result of facial expression recognition, and interlocks them.

The controller 190 may process data based on the linked result and the unique processing logic.

The control unit 190 calculates a new result according to the unique processing logic set by the user for each frame or predetermined unit from the log file as a result of the interlocking process analysis and stores it in a file system or DB.

The controller 190 may visualize the processed result.

Actions performed by a specific object over time and facial expression recognition result values can be visualized and displayed in a graph or the like.

The method of the present invention described above can be applied in various spaces.

In one embodiment, the present invention can be applied to an indoor space where monitoring is possible, such as a daycare center and a kids cafe.

When this system analyzes images stored through CCTVs obligatory installed in daycare centers or kids cafes, it is possible to perform action and facial expression recognition for each active child object. Based on the recognized results, it is possible to detect incidents and accidents that occur in the proposed indoor space.

For example, when 'hit sb' and 'sad' are recognized at the same time in a specific time period in the recognized action and facial expression, it can be determined that there is a quarrel. If a user configures a unique processing logic by combining actions and facial expressions with such logic, it is possible to maximize efficiency by referring only to the time period of the analysis result without the need for a person to view the video individually even in a video recorded for a long time.

In addition, it can be applied to detect safety accidents with a combination of 'fall down' and 'sad'.

In another embodiment, the present invention can be applied to a learning space.

In order to identify the user's concentration time in the learning space, 'sit', 'read', 'calm' or 'sit', 'write', 'calm' (calm)', etc., it is possible to determine the concentration time from the detected result by constructing a unique processing logic that combines behavior and emotion. Alternatively, by detecting a combination of 'sit' and 'use phone', it is possible to determine how often a mobile phone is used when sitting in a learning space.

As above, if the user configures a specific unique processing logic, more accurate analysis results can be obtained compared to existing behavior recognition AI analysis techniques. For example, if you use the existing behavioral recognition AI analysis technique that simply detects 'sit' when you sit in a seat in a learning space and talk to an acquaintance or laugh without concentrating on learning, the analysis can be performed properly. don't lose

In another embodiment, the present invention is applicable to a counseling center.

In counseling centers for infants, teenagers, and adults, there are cases in which a counselor conducts counseling by analyzing an image of a predetermined subject. In the case of counseling for infants, the frequency of interaction between the child and parent in the video, such as looking at each other, speaking and responding, etc., acts as an important index.

Therefore, a combination of 'watch somebody' and 'talk' is detected, or a combination of 'listen' and 'happy' is detected and visualized and provided to the counselee, for example. A counselor can maximize efficiency by delivering meaningful information or analyzing the video around the time zone when the result is detected.

In addition, the controller 190 may be used with various analysis factors (F), and an additional function (U) may be implemented using the corresponding factors.

Here, the analysis factor (F) is the pixel coordinates of each object appearing in the image, the reliability (probability value) of the recognized behavior and facial expression, and the estimated age range (estimated minimum age - estimated maximum age pair) at face-expression recognition. It may include, but is not limited to.

Depending on the embodiment, the present invention may also detect a change in average distance between objects in the image through pixel coordinates of each object appearing in the image among the analysis factors (F).

Depending on the embodiment, the present invention can also detect a change in expression (emotion) of a specific object over time through the reliability (probability value) of the recognized behavior and facial expression among the analysis factors (F), and the detected change is a graph can be visualized with

100: A device for analyzing user-captured images based on artificial intelligence
190: control unit

Claims (10)

A method performed by an apparatus for analyzing a user-captured image based on artificial intelligence, the method comprising:
setting a label to be output by a recognition model that recognizes a user's behavior and expression in the image when a user-captured image is input;
setting a user interest keyword that can be derived by at least one of a user's behavior and expression;
specifying a user when a user-captured image to be analyzed is input;
Recognizing the specified behavior and expression of the user based on the label through the recognition model;
generating data related to the user's interest keyword based on the recognized user's behavior and expression; and
An analysis method comprising displaying the generated data. According to claim 1,
The step of setting the user interest keyword,
and outputting a user interest keyword by inputting the set label to a pre-learned interest keyword recommendation model. According to claim 2,
The step of specifying the user includes mapping unique identification information for each user recognized from the input user-captured image,
Wherein the recognizing through the recognition model includes collecting each user's behavior and expression in chronological order based on the mapped unique identification information. According to claim 3,
The step of generating the data is,
And generating data related to a keyword of interest to the user based on the recognized behavior and expression of each user collected in the chronological order. According to claim 4,
The step of generating the data is,
And generating data related to the user interest keyword using a pre-learned data generation model. According to claim 5,
The displaying step is
And providing the recognized behaviors and facial expressions of each user collected in the chronological order in chronological order. According to claim 6,
The displaying step is
And providing data related to the keyword of interest in chronological order for each user. According to claim 7,
The displaying step is
An analysis method comprising the step of comparing and displaying a user's behavior or expression that can predict a situation or relationship that has occurred between users. An artificial intelligence-based user-captured image analysis program combined with a hardware computer and stored in a medium to execute the method of any one of claims 1 to 8. A device for analyzing user-photographed images based on artificial intelligence,
communications department;
input unit;
display;
one or more processors; and
A memory electrically connected to the processor and storing at least one code executed by the processor,
When the memory is executed by the processor, when the processor obtains a user-captured image through the communication unit or the input unit, a recognition model that recognizes the user's behavior and expression in the image sets a label to be output, and the user's behavior and set a user interest keyword derived by at least one of facial expressions, specify a user when a user-captured image to be analyzed is input, and recognize behavior and facial expression of the specified user through the recognition model based on the label. and a code for generating data related to the keyword of interest to the user based on the recognized behavior and expression of the user and displaying the generated data through the display is stored.

Images