KR20220093642A - Apparatus for labeling data for early screening of developmental disabilities based on learning model and method therefor - Google Patents

Abstract

A device for labeling is provided. The device comprises: a label deriving unit which calculates the probability that the behavior of an examiner or a subject shown in each of a plurality of frames belongs to each of multiple types of behavior through a learning model which has been completed with training, when a streaming video including the plurality of frames is input, and estimates the behavior of the examiner or the subject shown in each of the plurality of frames according to the calculated probability; and a label processing unit which detects a start time and an end time of a continuous frame having the same behavior estimated by the label deriving unit, from the plurality of frames, and labels the estimated behavior in response to the detected start time and end time for the streaming video.

Description

Apparatus for labeling data for early screening of developmental disabilities based on learning model and method therefor

The present invention relates to a labeling technique for data, and more particularly, to an apparatus and method for labeling data for early detection of a walking disorder based on a learning model.

Developmental Disability is a set of developmental disorders of the brain and nervous system starting from infancy, and refers to a disability that is severely delayed or not achieved in the aspects of language, communication, cognition, and sociality. Developmental disabilities are defined as intellectual disabilities and autistic disabilities. In Korea, the total number of people with disabilities is decreasing every year, but it is estimated that only a part of people diagnosed with developmental disabilities are registered as disabilities. It is estimated that more

Autism Spectrum Disorders (ASD) can be diagnosed around 2 years of age, lasts for a lifetime, and causes abnormalities in many of the most basic areas of development, resulting in children's independent development, education, and quality of family life. It can be considered as an obstacle that has a large impact on Early detection and early intervention are very important issues in both clinical and research aspects of ASD. Infancy and early childhood are a period of high brain plasticity, which provides an opportunity to change close to a normal form, as well as to prevent secondary neurological damage. It is possible to prevent the secondary serious behavioral problems from gradually accumulating in advance.

ASD diagnosis includes direct observation, information provided by caregivers and teachers, detailed history of growth, objective/quantitative evaluation of cognitive ability or other psychological functions, tests for differential diagnosis, neurological evaluation, brain function tests, etc. in need. Existing ASD screening tools take a very long time to train for use, there is inconsistency in diagnosis depending on the experiences and abilities of individual experts, and it takes at least 6-7 hours of testing time and resources to diagnose one child. very voluminous In addition, in the case of infants/children, the behaviors generally seen in daily life may differ greatly depending on where, with whom, the diagnostic test is performed. Therefore, the development of an ASD detection screening test tool for infants/children for the convergence of artificial intelligence technology and an AI-based solution that can efficiently evaluate the contents and cognitive abilities or psychological changes to which it is applied need. In other words, problem solving through the ASD early screening system such as non-verbal communication recognition through automatic analysis of multisensory data collection and automatic analysis of infants/children based on a new screening test tool, recognition of abnormal/homologous target behavior, and psychological prediction based on complex information need.

Korean Patent Publication No. 2020-0085766 published on July 15, 2020 (Title: Cognitive dysfunction diagnosis device and cognitive dysfunction diagnosis program)

An object of the present invention is to provide an apparatus for labeling data for early screening of developmental disabilities based on a learning model and a method therefor.

In an apparatus for labeling according to a preferred embodiment of the present invention for achieving the object as described above, when streaming image data including a plurality of frames including a color frame, a depth frame, and a voice is input, learning is completed learning model Calculates the probability that the behavior of the inspector or examinee displayed in each of the plurality of frames belongs to each of a plurality of types of behavior through Detects a start time and an end time of consecutive frames having the same behavior estimated by the label derivation unit among the plurality of frames, and the estimated behavior corresponding to the detected start time and end time for the streaming image data It includes a label processing unit for labeling.

For each timeline, a manual label indicating the type of behavior of the examiner or examinee is given, and video data for training, which is streaming video data including a plurality of frames, is prepared, and a plurality of frames of the video data for training are sequentially provided to the learning model. input, and when the learning model calculates, as an output value, the probability that the behavior of the examiner or examinee displayed in each of the plurality of frames belongs to each of the plurality of types of behavior through a plurality of operations to which a plurality of inter-layer weights are applied, the calculated output value and a model generator for performing optimization of modifying the weight of the learning model so that a difference between it and the manual label is minimized.

The learning model includes an input layer that sequentially receives a plurality of frames of a streaming image, at least one convolutional layer, at least one pooling layer, at least one fully connected layer, and a plurality of types of actions of an examiner or examinee It includes an output layer including an output node corresponding to each.

의 값인 손실이 최소가 되도록 학습모델의 가중치를 수정하는 최적화를 수행하며, 상기 L은 손실을 나타내고, 상기 n은 학습용 영상 데이터의 프레임의 수이고, 상기 t는 프레임의 인덱스이고, 상기 w는 학습모델이 분류할 수 있는 피검자의 행위의 수에 비례하여 증가하는 하이퍼파라미터이며, 상기 s는 검사자의 행위인지 혹은 피검자의 행위인지 여부를 구분하는 매뉴얼 레이블이고, 상기 a는 피검자의 행위의 유형을 나타내는 매뉴얼 레이블이고, 상기 e(t)는 출력값 중 검사자의 행위를 나타내는 출력노드의 출력값의 합과, 피검자의 행위를 나타내는 출력노드의 출력값의 합을 나타내며, 상기 b(t)는 출력값 중 피검자의 행위의 유형에 대응하는 출력노드 각각의 출력값을 나타내는 것을 특징으로 한다. The model generator is a loss function

Optimization is performed to correct the weight of the learning model so that the loss, which is the value of , is minimized, where L represents the loss, n is the number of frames of image data for training, t is the index of the frame, and w is the learning It is a hyperparameter that increases in proportion to the number of behaviors of the subject that the model can classify, where s is the inspector's behavior or a manual label that distinguishes whether the behavior is the subject's behavior, and a is the type of behavior of the subject. manual label, wherein e(t) represents the sum of the output values of the output nodes indicating the behavior of the examiner among the output values and the sum of the output values of the output nodes indicating the behavior of the examinee, and b(t) is the behavior of the examinee among the output values It is characterized in that it represents the output value of each output node corresponding to the type of .

In the method for labeling according to a preferred embodiment of the present invention for achieving the object as described above, when streaming image data including a plurality of frames including a color frame, a depth frame and a voice is input to the label derivation unit, learning is Calculating a probability that an inspector or examinee's behavior shown in each of the plurality of frames through the completed learning model belongs to each of a plurality of types of behavior; estimating the behavior of an examiner or subject; detecting, by the label processing unit, start time and end time of consecutive frames having the same behavior estimated by the label derivation unit among the plurality of frames, the label processing unit detecting the streaming image; and labeling the estimated behavior corresponding to a detected start time and an end time for the data.

According to the present invention, during an examination for early screening of developmental disabilities, the examiner's question and the subject's response behavior can be identified through a learning model, and the corresponding image can be labeled. Accordingly, it is possible to efficiently collect the data required for the development of the ASD detection screening test tool, the contents to which it is applied, and the study of the AI-based solution technique that can efficiently perform objective and quantitative evaluation of cognitive abilities or psychological changes.

1 is a diagram for explaining the configuration of an apparatus for labeling data for early screening of developmental disabilities based on a learning model according to an embodiment of the present invention.
2 is a diagram for explaining the detailed configuration of an apparatus for labeling data for early screening of developmental disabilities based on a learning model according to an embodiment of the present invention.
3 is a diagram for explaining the configuration of a learning model for labeling data according to an embodiment of the present invention.
4 is a diagram for explaining data calculated by a learning model for labeling data according to an embodiment of the present invention.
5 is a flowchart illustrating a method of preparing learning data for learning a learning model (ML) according to an embodiment of the present invention.
6 is a flowchart illustrating a method of learning a learning model using image data for training according to an embodiment of the present invention.
7 is a flowchart illustrating a method of labeling data for early screening of developmental disabilities using a learning model (ML) according to an embodiment of the present invention.

Prior to the detailed description of the present invention, the terms or words used in the present specification and claims described below should not be construed as being limited to their ordinary or dictionary meanings, and the inventors should develop their own inventions in the best way. It should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be appropriately defined as a concept of a term for explanation. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all the technical spirit of the present invention, so various equivalents that can be substituted for them at the time of the present application It should be understood that there may be water and variations.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this case, it should be noted that in the accompanying drawings, the same components are denoted by the same reference numerals as much as possible. In addition, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted. For the same reason, some components are exaggerated, omitted, or schematically illustrated in the accompanying drawings, and the size of each component does not fully reflect the actual size.

First, an apparatus for labeling data for early screening of developmental disabilities based on a learning model according to an embodiment of the present invention will be described. 1 is a diagram for explaining the configuration of an apparatus for labeling data for early screening of developmental disabilities based on a learning model according to an embodiment of the present invention. 2 is a diagram for explaining the detailed configuration of an apparatus for labeling data for early screening of developmental disabilities based on a learning model according to an embodiment of the present invention.

First, referring to FIG. 1 , a label device 10 according to an embodiment of the present invention includes a camera unit 11 , a sensor unit 12 , a voice processing unit 13 , an input unit 14 , a display unit 15 , and a storage unit. It includes a unit 16 and a control unit 17 .

The camera unit 11 is for capturing an image including a plurality of color frames. The camera unit 11 includes a plurality of cameras. The plurality of cameras may generate images including a plurality of color frames by photographing the examiner and the examinee in different directions, respectively.

The sensor unit 12 includes a plurality of Kinect sensors. The plurality of Kinect sensors radiate infrared rays to the examiner and the subject in different directions, and capture a depth image including a plurality of IR depth frames therefrom.

The voice processing unit 13 includes at least one microphone, and collects voices of the examiner and the examinee through the at least one microphone.

The input unit 14 may receive a user's manipulation for controlling the label apparatus 10 , generate an input signal, and transmit it to the control unit 17 . The input unit 14 includes various keys and buttons for controlling the label device 10 .

The display unit 15 is for screen display, and may visually provide a menu of the label device 10, input data, function setting information, and other various information to the user. The display unit 15 may be formed of a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), or the like. Meanwhile, the display unit 15 may be implemented as a touch screen. In this case, the display unit 15 includes a touch sensor. The touch sensor detects a user's touch input. The touch sensor may be composed of a touch sensing sensor such as a capacitive overlay, a pressure type, a resistive overlay, or an infrared beam, or may be composed of a pressure sensor. . In addition to the above sensors, all types of sensor devices capable of sensing contact or pressure of an object may be used as the touch sensor of the present invention. The touch sensor may detect a user's touch input, generate a detection signal including input coordinates indicating the touched position, and transmit it to the controller 17 .

The storage unit 16 serves to store programs and data necessary for the operation of the label device 10 . The storage unit 16 may store a color image photographed by a plurality of cameras of the camera unit 11 , a depth image photographed by a plurality of kinetic sensors of the sensor unit 12 , and a voice collected by the audio processing unit 13 . have. Each type of data stored in the storage unit 16 may be deleted, changed, or added according to a user's operation.

The control unit 17 may control the overall operation of the label apparatus 10 and the signal flow between internal blocks of the label apparatus 10 , and perform a data processing function of processing data. Also, the control unit 17 basically serves to control various functions of the label apparatus 10 . The control unit 17 may be exemplified by a central processing unit (CPU), a digital signal processor (DSP), or the like. In particular, the control unit 17 performs labeling on streaming image data. In an embodiment of the present invention, streaming image data includes a color image including a plurality of color frames photographed by a plurality of cameras of the camera unit 11 , and a plurality of depth frames photographed by a plurality of Kinect sensors of the sensor unit 12 . It is data in which a plurality of color frames, a plurality of depth frames, and audio are synchronized according to a timeline in the depth image and audio collected by the audio processing unit 13 .

Referring to FIG. 2 , the control unit 17 includes a data processing unit 100 , a model generation unit 200 , a label derivation unit 300 , and a label processing unit 400 for labeling.

The data processing unit 100 collects a plurality of color frames, a plurality of depth frames, and voices through the camera unit 11 , the sensor unit 12 , and the voice processing unit 13 , and a plurality of color frames according to the collected timeline. , a plurality of depth frames and audio are synchronized to generate streaming video data. The generated streaming image data may be provided to the model generating unit 200 , the label deriving unit 300 , and the label processing unit 400 .

The model generating unit 200 is for learning a learning model (LM: Leaning Model). A learning model (LM) may be a deep neural network. In particular, the learning model LM may be representative of a Convolution Neural Network (CNN). This learning model (LM) will be described in more detail below.

The label deriving unit 300 is provided with the learning model LM in which learning is completed from the model generating unit 200 . The label extracting unit 300 may receive streaming image data. The streaming image data includes a color frame, a depth frame, and a plurality of frames including audio corresponding to the frame. The label derivation unit 300 inputs streaming image data to the learning model LM for each frame. Then, the probability that the behavior of the examiner or the examinee displayed in each of the plurality of frames belongs to each of the plurality of types of behavior is calculated through the learning model LM on which the learning has been completed. Then, the label derivation unit 300 estimates the behavior of the examiner or the examinee appearing in each of the plurality of frames according to the probability calculated by the learning model LM.

The label processing unit 400 detects a start time and an end time of consecutive frames having the same behavior estimated by the label deriving unit 300 among a plurality of frames of streaming image data, and detects a start time and an end time for streaming image data. Label the estimated behavior corresponding to time.

Next, a configuration of a learning model for labeling data according to an embodiment of the present invention will be described. 3 is a diagram for explaining the configuration of a learning model for labeling data according to an embodiment of the present invention. 4 is a diagram for explaining data calculated by a learning model for labeling data according to an embodiment of the present invention.

Referring to FIG. 3 , the learning model LM includes an input layer (IL), at least a pair of alternately repeated convolution layers (CL) and a pooling layer (PL), at least one a fully-connected layer (FL) and an output layer (OL). As shown, the deep neural network 400 according to an embodiment of the present invention sequentially includes an input layer (IL), a convolution layer (CL), a pooling layer (PL), a fully connected layer (FL), and an output layer (OL). ) is included.

The convolution layer CL and the pooling layer PL include at least one feature map (FM). The feature map FM is derived as a result of receiving a value to which a weight and a threshold are applied to the operation result of the previous layer, and performing an operation on the input value. These weights are applied through a filter or kernel W that is a weight matrix of a predetermined size. In the embodiment of the present invention, the first filter W1 is used for the convolution operation of the convolutional layer CL, and the second filter W2 is used for the pooling operation of the pooling layer PL.

When any one frame (including a color frame, a depth frame, and an audio) of streaming image data is input to the input layer IL, the convolution layer CL performs a first filter on the frame input to the input layer IL. At least one first feature map FM1 is derived by performing a convolution operation using (W1) and an operation by an activation function. Next, the pooling layer PL performs a pooling or sub-sampling operation using the second filter W2 on at least one first feature map FM1 of the convolution layer CL to obtain at least one A second feature map FM2 is derived.

As shown in FIG. 4 , the final connection layer FL includes a plurality of operation nodes F1 to Fm. The plurality of operation nodes F1 to Fm of the final connection layer CL calculates a plurality of operation values through an operation by an activation function with respect to at least one second feature map FM2 of the pooling layer PL.

The output layer OL includes a plurality of output nodes O1 to On. Each of the plurality of operation nodes F1 to Fm of the final connection layer FL is connected to the output nodes O1 to On of the output layer OL through a channel having a weight (W). In other words, a weight W is applied to the plurality of operation values of the plurality of operation nodes F1 to Fm and is input to each of the plurality of output nodes O1 to On. Accordingly, the plurality of output nodes O1 to On of the output layer OL calculates an output value through an activation function operation with respect to a plurality of calculated values to which the weight W of the final connection layer FL is applied.

Each of the plurality of output nodes O1 to On of the output layer OL corresponds to the type of behavior of the examiner or examinee. For example, the first output node O1 responds to a question during the examiner's actions, the second output node O2 corresponds to eye contact during the examinee's response actions, and the third output node O2 responds to the examinee's response actions. Corresponds to shaking the head, and the nth output node (On) may correspond to handing among the response actions of the subject. Accordingly, for example, the output value of the first output node O1 is the probability that the behavior of the inspector or examinee in the frame is the questioning behavior of the inspector, and the output value of the second output node O2 indicates that the behavior of the inspector or examinee in the frame is the examinee. is the probability of eye contact among the response actions of the third output node O2, the output value of the third output node O2 is the probability that the examiner or examinee’s action within the frame is a head shake among the examinee’s response actions, and the output value of the nth output node (On) is within the frame It may represent the probability that the behavior of the examiner or the examinee is handing among the response actions of the examinee.

For example, if the output values of the plurality of output nodes O1, O2, O3, ..., On are 0.026, 0.712, 0.111, ..., 0.007, since the output value of the first output node O1 is 0.026, within the frame The probability that the examiner's or subject's action is the examiner's questioning action is 2%, and since the output value of the second output node (O2) is 0.712, there is a 71% probability that the examiner's or examinee's action within the frame is eye contact among the examinee's response actions. , because the output value of the third output node O2 is 0.111, the probability that the examiner’s or examinee’s action is a head shake among the examinee’s response actions is 11%, and since the output value of the nth output node (On) is 0.007, the examiner in the frame Or, it indicates that the probability that the subject's behavior is handing among the respondent's response behaviors is 1%.

In this way, when the learning model (ML) calculates the probability that the behavior of the examiner or the examinee belongs to each of the plurality of types of behavior, the label derivation unit 300 determines the number of the examiner or the examinee appearing in each of the plurality of frames according to the calculated probability. behavior can be inferred.

For example, if the output values of the plurality of output nodes O1, O2, O3, ..., On are 0.026, 0.712, 0.111, ..., 0.007, the response behavior of the examinee according to the output value of the second output node O2 Since the probability of eye contact is the highest at 71%, the label derivation unit 300 estimates that it is eye contact among the subject's response actions.

The activation functions used in the convolutional layer (CL), final connection layer (FL) and output layer (OL) described above are Sigmoid, Hyperbolic tangent (tanh), Exponential Linear Unit (ELU), and ReLU. (Rectified Linear Unit), Leakly ReLU, Maxout, Minout, Softmax, etc. can be exemplified. Any one of these activation functions may be selected and applied to the convolutional layer CL, the finite connection layer FL, and the output layer OL.

Next, a method for labeling data for early screening of developmental disabilities based on a learning model according to an embodiment of the present invention will be described. In order to train the learning model (ML) to automatically perform labeling, it is necessary to prepare training data for training the learning model (ML). Accordingly, first, a method for preparing initial learning data will be described. 5 is a flowchart illustrating a method of preparing learning data for learning a learning model (ML) according to an embodiment of the present invention.

First, the examiner can input basic information such as, for example, study number, age, ADOS (Autism Diagnostic Observation Schedule) format, BeDevel (Behavior Development Screening for Toddler) format, test date, and final diagnosis result through the input unit 14 . have. Then, the data processing unit 100 of the control unit 17 of the label apparatus 10 receives this basic information through the input unit 14 in step S110 and temporarily stores it in the storage unit 16 .

Subsequently, in step S120 , imaging is started according to the input of the examiner, the examiner makes a question about the subject, and an examination for observing the subject's behavior is started. That is, while the inspection is in progress, the data processing unit 100 continuously uses the camera unit 11 , the sensor unit 12 , and the voice processing unit 13 to continuously capture a plurality of colors captured by a plurality of cameras of the camera unit 11 . A color image including a frame, a depth image including a plurality of depth frames captured by a plurality of Kinect sensors of the sensor unit 12 , and a plurality of color frames and a plurality of depths in the voice collected by the audio processing unit 13 . Frames and audio are collected, and a plurality of color frames, a plurality of depth frames, and audio are synchronized according to the collected timeline to generate streaming image data.

Next, when the examination is finished in step S130 , the photographing is terminated according to the examiner's input, and the data processing unit 100 stores the continuously collected and generated streaming image data.

Then, according to the examiner's input, in step S140 , the data processing unit 100 of the control unit 17 may reproduce at least a portion of the streaming image data, and output it through the display unit 15 and the audio processing unit 13 . The examiner may attach a manual label to the streaming image data reproduced through the input unit 14 . At this time, the examiner determines the type of action so that the type of response is distinguished in the case of a manual label that separates the timeline for the questioning action of the examiner and the timeline for the response action of the examinee, and the timeline for the response action of the examinee A manual label indicating Accordingly, training image data, which is streaming image data including a plurality of frames and a manual label indicating the type of the examiner's or examinee's behavior, is provided for each timeline. The data processing unit 100 stores the image data for learning in the storage unit 16 in step S150 .

As described above, when the image data for learning is provided, the learning model LM may be trained. These methods will be described. 6 is a flowchart illustrating a method of learning a learning model using image data for learning according to an embodiment of the present invention.

Referring to FIG. 6 , the model generating unit 200 is provided with a manual label indicating the type of behavior of the examiner or examinee for each timeline from the storage unit 16 in step S210 and streaming image data including a plurality of frames Load image data for in-learning.

Then, the model generator 200 sequentially inputs a plurality of frames of image data for training to the learning model ML in step S220 . Accordingly, the learning model (ML) performs a plurality of operations in which a plurality of inter-layer weights are applied to a plurality of frames in step S230 so that the behavior of the examiner or the examinee displayed in each of the plurality of frames is applied to each of the plurality of types of behavior. The probability of belonging is calculated as an output value.

Then, the model generator 200 performs optimization to correct the weight of the learning model so that the difference between the output value calculated in step S240 and the manual label is minimized. At this time, the model generating unit 200 performs optimization using a loss function that obtains a difference between the output value and the manual label as shown in Equation 1 below.

Here, L represents a loss, n is the number of frames of image data for training, and t is an index of the frame. w is a preset value, a hyperparameter, and increases in proportion to the number of actions of the subject. s is a manual label that distinguishes whether it is the examiner's action or the examinee's action, and a is a manual label indicating the type of the examinee's action. e(t) represents the sum of the output values of the output nodes indicating the behavior of the examiner among the output values and the sum of the output values of the output nodes indicating the behavior of the examinee. b(t) represents the output value of each output node corresponding to the type of behavior of the examinee among the output values. That is, the model generator 200 performs optimization of correcting the weight of the learning model so that the value of the loss function of Equation 1, that is, the loss is minimized.

The optimization as described above may be repeatedly performed using different image data for training. These iterations may be performed until the accuracy is calculated through the evaluation index and the desired accuracy is reached.

Next, a method for labeling data for early screening of developmental disabilities using a learning model (ML) that has been trained according to the method as described above will be described. 7 is a flowchart illustrating a method of labeling data for early screening of developmental disabilities using a learning model (ML) according to an embodiment of the present invention.

Referring to FIG. 7 , according to a user's input, the data processing unit 100 receives basic information in step S310 and stores it in the storage unit 16 . For example, basic information includes study number, age, ADOS (Autism Diagnostic Observation Schedule) format, BeDevel (Behavior Development Screening for Toddler) format, test date, final diagnosis result, and the like.

Then, the data processing unit 100 continuously receives a plurality of color frames photographed by the plurality of cameras of the camera unit 11 through the camera unit 11, the sensor unit 12, and the voice processing unit 13 in step S320. a plurality of color frames, a plurality of depth frames, and a depth image including a plurality of depth frames captured by a plurality of Kinect sensors of the sensor unit 12 Audio is collected, and a plurality of color frames, a plurality of depth frames, and audio are synchronized according to the collected timeline to generate streaming video data. A plurality of frames of such streaming image data are sequentially provided to the label deriving unit 300 .

The label deriving unit 300 inputs streaming image data including a plurality of frames to the learning model LM for each frame in step S330. Then, the probability that the behavior of the examiner or the examinee displayed in each of the plurality of frames belongs to each of the plurality of types of behavior is calculated through the learning model LM on which the learning has been completed.

Then, the label derivation unit 300 estimates the behavior of the examiner or the examinee appearing in each of the plurality of frames according to the probability calculated by the learning model LM in step S340 . Next, the label processing unit 400 individually assigns a label to the corresponding frame according to the action estimated in step S350 . Steps S320 to S350 described above are repeated until the test is finished according to the determination of step S360.

When the inspection is finished, the label processing unit 400 detects a start time and an end time of a continuous frame having the same behavior estimated by the label deriving unit 300 among a plurality of frames of streaming image data in step S370, and streaming image data A label corresponding to the estimated behavior is given in response to the detected start time and end time.

According to the method as described above, when testing for early screening of developmental disabilities, the examiner's question and the subject's response behavior can be identified through the learning model, and a label corresponding to the identified behavior can be automatically assigned to the image. In this way, it is possible to efficiently collect data necessary for the development of the ASD detection screening test tool, the contents to which it is applied, and the study of the AI-based solution technique that can efficiently evaluate objective and quantitative cognitive abilities or psychological changes.

Meanwhile, the method according to the embodiment of the present invention described above may be implemented in the form of a program readable by various computer means and recorded in a computer readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks ( magneto-optical media) and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language wires such as those generated by a compiler, but also high-level language wires that can be executed by a computer using an interpreter or the like. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although the present invention has been described above using several preferred embodiments, these examples are illustrative and not restrictive. As such, those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made in accordance with the doctrine of equivalents without departing from the spirit of the present invention and the scope of rights set forth in the appended claims.

10: label device 11: camera unit
12: sensor unit 13: voice collection unit
14: input unit 15: display unit
16: storage unit 17: control unit
100: model generation unit 200: label derivation unit
300: label processing unit

Claims (5)

A device for labeling, comprising:
When streaming image data including a plurality of frames including a color frame, a depth frame, and an audio is input, the behavior of the examiner or the examinee displayed in each of the plurality of frames through the learning model on which the learning is completed is applied to each of the plurality of types of behavior a label derivation unit for calculating a probability of belonging and estimating an examiner's or subject's behavior appearing in each of the plurality of frames according to the calculated probability; and
Detecting the start time and end time of consecutive frames having the same behavior estimated by the label derivation unit among the plurality of frames, and labeling the estimated behavior in response to the detected start time and end time for the streaming image data label processing unit;
characterized in that it comprises
Device for labeling. According to claim 1,
Prepare video data for training, which is streaming video data that includes a plurality of frames and is given a manual label indicating the type of behavior of the examiner or examinee for each timeline;
A plurality of frames of the image data for training are sequentially input to the learning model,
When the learning model calculates, as an output value, the probability that an examiner or examinee's behavior displayed in each of the plurality of frames belongs to each of a plurality of types of behavior through a plurality of operations to which a plurality of inter-layer weights are applied, the calculated output value and the manual a model generator that optimizes the weight of the learning model so that the difference with the label is minimized;
characterized in that it further comprises
Device for labeling. 3. The method of claim 2,
The learning model is
an input layer that sequentially receives a plurality of frames of a streaming image;
at least one convolutional layer;
at least one pooling layer;
at least one fully connected layer; and
an output layer including an output node corresponding to each of a plurality of types of actions of the examiner or the examinee;
characterized in that it comprises
Device for labeling. 3. The method of claim 2,
The model generation unit
loss function

Optimization is performed to modify the weight of the learning model so that the loss, which is the value of , is minimized.
L represents the loss,
Where n is the number of frames of image data for training,
Where t is the index of the frame,
Where w is a hyperparameter that increases in proportion to the number of subject actions that the learning model can classify,
Wherein s is a manual label that distinguishes whether it is an action of an examiner or an action of a subject,
wherein a is a manual label indicating the type of behavior of the subject,
e(t) represents the sum of the output values of the output nodes indicating the behavior of the examiner among the output values and the sum of the output values of the output nodes indicating the behavior of the examinee,
wherein b(t) represents an output value of each output node corresponding to the type of behavior of the examinee among the output values
Device for labeling. A method for labeling, comprising:
When streaming image data including a plurality of frames including a color frame, a depth frame, and an audio is input to the label derivation unit, the behavior of the examiner or the examinee displayed in each of the plurality of frames through the learning model on which the learning is completed is performed in a plurality of types. calculating a probability of belonging to each action;
estimating, by the label derivation unit, the behavior of the examiner or the examinee appearing in each of the plurality of frames according to the calculated probability;
detecting, by the label processing unit, start time and end time of consecutive frames having the same behavior estimated by the label derivation unit among the plurality of frames; and
labeling, by the label processing unit, the estimated action in response to the detected start time and end time for the streaming image data;
characterized in that it comprises
Method for labeling.

Images