KR20230093116A - Protocol/tool for early detection of autism spectrum disorder by measuring the response to joint attention - Google Patents

Abstract

The present invention relates to an apparatus and method for diagnosing developmental disorders in infants and young children, comprising the steps of acquiring at least one image including activity of infants and young children; analyzing motion including eye tracking of the target infant based on the obtained at least one image; Determining whether there is passive joint attention (Response to joint attention, RJA) or voluntary joint attention (IJA) according to the analysis result; and outputting a diagnosis result for the target infant or toddler based on an artificial intelligence machine learning model, wherein the machine learning model uses learning data obtained by combining image data including eye tracking data and diagnosis data. may have been learned.

Description

Early Detection Protocol/Tool for Autism Spectrum Disorder Using Joint Attention Response Measurement

The present invention relates to an artificial intelligence-based apparatus and method for diagnosing developmental disorders in infants and young children, and in particular, to a protocol/tool for early detection of autism spectrum disorder through joint attention response measurement for diagnosing developmental disorders based on eye tracking of infants and young children.

Developmental disabilities is a comprehensive term that includes ADHD, autism spectrum disorder, cerebral palsy, developmental delay, language impairment, learning disability, intellectual disability, and sensory impairment (hearing and vision). The number of children diagnosed with developmental disabilities is on the rise worldwide. According to a paper summarizing studies from 1990 to 2016, 100,000 children in Korea were diagnosed with attention deficit hyperactivity syndrome (ADHD), autism spectrum disorder (ASD), or intellectual disability (ID). , 81 million in the United States, 2.35 million in Europe, and 4.28 million in the Middle East. In India, the number of children with developmental delays quadrupled within 5 years, in Japan, a 7.4-fold increase in 20 years, and in Korea, the prevalence of developmental disabilities more than quadrupled from 0.6% to 2.5% between 2003 and 2017. The incidence of autism spectrum disorder, developmental delay, and language disorder increased by 13.75, 817.6%, and 30.7%, respectively. (Rah et al., 2020) Judging from the above research results, it can be said that the severity of child developmental disabilities is increasing both at home and abroad.

Diagnosis of developmental disabilities in children costs 300,000 to 1.50 million won, and treatment is performed for about 10 hours a week, so about 2.2 million won (50,000 to 120,000 won / 1 hour required) is spent per month, and children from low-income families cannot even receive diagnosis/treatment. Treatment by pediatric psychiatrists, psychological evaluation and treatment including autism spectrum disorder tests are concentrated in Seoul and the metropolitan area. Accessibility for children is significantly reduced.

Children with developmental disabilities face difficulties in various aspects of life. When children with intellectual disabilities are compared with normal children of the same age, they score lower on phonological-loop capacity (maintaining and manipulating sound information) and central-executive tests (planning attention and decision-making), and have lower scores than children with developmental disabilities. 81.8% scored in the bottom 16% in terms of motor skills. Psychosocial problems and deterioration of mental health are also indispensable problems, and up to 60% of children with borderline intellectual disability and mild developmental disabilities are confirmed to have behavioral and emotional problems. There is a significant difference from that of normal children.

Childhood developmental disabilities persist throughout life. Some changes can be seen in the developmental stage, and it is known that early diagnosis, family support, and education or training programs received while growing up have a positive effect on long-term prognosis, including independent living in adulthood (Szymanski, 1999). ). According to the US National Institute for Early Education Research, 85% of children's brain development occurs before age 5. In addition, expression language and overall cognitive function level around the age of 5-6 determine the child's lifelong prognosis. Based on the language and cognitive functions learned up to this point, they acquire new knowledge and skills at a rapid pace and grow into independent adults. It gets bigger. According to the research conducted by the Institute for Health and Social Affairs, the lifetime loss per patient is 350 million won, and the parent's work loss amounts to 1.6 trillion won. Despite these high socio-economic costs and lifelong effects, national R&D support is very limited compared to other major mental disorders such as dementia, adult depressive disorder, and psychotic disorder.

Therefore, it is necessary to develop a technology that can diagnose developmental disabilities by analyzing movements including eye tracking of infants and young children through images taken of activities of infants and young children.

The present invention to solve the problems described above is an artificial intelligence-based diagnosis of developmental disorders in infants and young children, which enables diagnosis of developmental disorders by analyzing movements including eye-tracking of the infants and young children through images taken of activities of the infants and young children. Devices and methods can be provided.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

An apparatus for diagnosing developmental disorders in infants and young children based on artificial intelligence according to an embodiment of the present invention for solving the above problems includes acquiring at least one image including an activity of a target infant or young child; analyzing motion including eye tracking of the target infant based on the obtained at least one image; Determining whether there is passive joint attention (Response to joint attention, RJA) or voluntary joint attention (IJA) according to the analysis result; and outputting a diagnosis result for the target infant or toddler based on an artificial intelligence machine learning model, wherein the machine learning model uses learning data obtained by combining image data including eye tracking data and diagnosis data. may have been learned.

In addition to this, another method for implementing the present invention, another system, and a computer readable recording medium recording a computer program for executing the method may be further provided.

According to the present invention as described above, the present invention makes it possible to diagnose developmental disabilities by analyzing movements including eye-tracking of infants and young children through images taken of activities of infants and young children.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram showing the network structure of an artificial intelligence-based infant development disorder diagnosis system according to an embodiment of the present invention.
2 is a diagram showing the configuration of an artificial intelligence-based diagnosis apparatus for developmental disorders in infants and young children according to an embodiment of the present invention.
3 is a flowchart illustrating a method for diagnosing developmental disorders in infants and young children based on artificial intelligence according to an embodiment of the present invention.
4 is a diagram showing an example of an observation environment created for diagnosing developmental disorders in infants and young children according to an embodiment of the present invention.
5 is a diagram showing an example of movements of a target infant or toddler according to an observation result for diagnosing developmental disorders in infants and young children according to an embodiment of the present invention.
6 to 8 are diagrams illustrating an example of a user terminal receiving a service for diagnosing developmental disorders in infants and young children according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram showing the network structure of an artificial intelligence-based infant development disorder diagnosis system according to an embodiment of the present invention.

Referring to FIG. 1 , an artificial intelligence-based developmental disorder diagnosis system 10 according to an embodiment of the present invention includes a photographing device 200 and an infant development disorder diagnosis device 100 (hereinafter, referred to as a 'diagnostic device'). And it may be configured to include a user terminal (300).

The photographing device 200 is disposed in an observation space created to diagnose developmental disorders in infants and young children, and includes interaction between the examiner and target infants and eye tracking in the space, and body movements and eye tracking of target infants according to task requests. Acquire video.

In this case, the photographing device 200 may include at least one camera, whereby at the time of diagnosis, images taken in various directions are simultaneously acquired through respective cameras disposed at different positions, and at least one image is obtained. can For example, the at least one or more cameras may include a 2D camera, a 3D camera, a Time of Flight (ToF) camera, a light field camera, a stereo camera, an event camera, an infrared camera, a lidar sensor, and an array ( array) may be at least one of cameras. In addition, the photographing device 200 may further include an eye tracker. At this time, the eye tracker may be provided in the form of at least one of a stationary eye tracker and a wearable eye tracker, and may be integrated into the camera. It may be provided and provided. That is, at least one camera and/or eye tracker may be installed in various types and forms, but is not limited thereto.

The diagnosis apparatus 100 acquires at least one image from the photographing apparatus 200 and analyzes the obtained image to determine whether response to joint attention (RJA) or initiative of joint attention (IJA) is obtained. Depending on whether or not there is a diagnosis result for the target infant or toddler is output and provided.

In this case, the diagnosis device 100 and the photographing device 200 may be connected in at least one of wireless communication and wired communication, but are not limited to either method.

Meanwhile, the diagnosis device 100 may provide the diagnosis result to a preset user terminal 300, and at this time, the user terminal 300 may be registered as at least one terminal of each of the examiner and/or the guardian of the target infant or toddler. can

2 is a diagram showing the configuration of an artificial intelligence-based diagnosis apparatus for developmental disorders in infants and young children according to an embodiment of the present invention.

Referring to FIG. 2 , the diagnosis device 100 according to an embodiment of the present invention may include a communication module 110 , a storage module 130 and a control module 150 .

The communication module 110 transmits and receives at least one piece of information or data to and from the photographing device 200 and the user terminal 300 in order to diagnose developmental disorders in infants and young children. At this time, the photographing device 200 and the user terminal 300 may be at least one. In addition, the communication module 101 may communicate with the photographing device 200, the user terminal 300, and other devices, and transmits and receives wireless signals in a communication network based on wireless Internet technologies. .

Wireless Internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc., diagnostic device 100 transmits and receives data according to at least one wireless Internet technology within a range including Internet technologies not listed above.

As for short range communication, Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi -Local communication may be supported using at least one of wireless-fidelity (Fi), Wi-Fi Direct, and wireless universal serial bus (USB) technologies. These wireless area networks may support wireless communication between the diagnosis device 100 and the photographing device 200 and between the diagnosis device 100 and the user terminal 300 . At this time, the local area wireless communication network may be a local area wireless personal area network (Wireless Personal Area Networks).

However, this is only one embodiment, and the diagnosis device 100 and the photographing device 200 may be configured to transmit and receive data based on wired communication, but are not limited thereto.

The storage module 130 stores at least one data (information) and at least one process necessary for diagnosing developmental disorders in infants and young children, as well as various other data (information). For example, a program for diagnosing developmental disorders in infants and young children, information on at least one user (a target infant, a guardian of the target infant or young child, an examiner, a test center, etc.) may be stored as data. In addition, the storage module 130 stores various commands, algorithms, etc. for executing a method for diagnosing developmental disorders in infants and young children. In particular, the storage module 130 of the diagnosis device 100 according to an embodiment of the present invention may store a pre-learned machine learning model to output (derive) a diagnosis result for a target infant or toddler based on artificial intelligence. .

To this end, the storage module 130 may include a memory, and the memory may be of a flash memory type, a hard disk type, a multimedia card micro type, or a card type. Memory (eg SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), It may include at least one type of storage medium among a programmable read-only memory (PROM) magnetic memory, a magnetic disk, and an optical disk. In addition, the memory may store information temporarily, permanently or semi-permanently, and may be provided in a built-in or removable type.

The control module 150 acquires at least one image including the activity of the target infant and young child, and analyzes the movement of the target infant and young child including eye tracking based on the acquired at least one image. Then, the control module 150 determines whether passive joint attention (Response to joint attention, RJA) or voluntary joint attention (IJA) is performed according to the analysis result, and the target based on the artificial intelligence machine learning model. It outputs (derives) and provides diagnosis results for infants and toddlers.

In this case, the machine learning model may be pre-learned using learning data obtained by combining image data including eye tracking data and diagnostic data.

In addition, the control module 150 determines a face landmark that changes over time when determining whether passive joint attention (RJA) or voluntary joint attention (IJA) is performed according to the analysis result. Automatically measure the angle, determine whether a head-turn has occurred based on the measured angle, and when it is determined that an angle change of a preset angle or more has occurred more than a preset number of times from the point of view of the target infant, shift Determine that staring has occurred.

For example, voluntary joint attention can be judged by the examiner placing a toy on a table in a position where the subject infant cannot reach and grab it right away, and observes the child's behavior. In addition, passive joint attention is judged by whether the target infant turns his or her gaze in the direction indicated when the examiner calls the target infant's name and then points to a toy (close object) located on the table with his or her finger. After calling the name of the infant, it can be judged by whether or not the target infant turns his or her gaze in the direction pointed at when the finger is pointed at a picture (an object placed far away) located at the top of the back of the child.

In other words, passive joint attention (Response to joint attention, RJA) or voluntary joint attention (Initiation of joint attention, IJA) can be determined through movements including eye tracking of the target infant as shown in Table 1 below.

Initiation of joint attention Response to joint attention Eye contact + alternating gaze (object - examiner - object
or examiner - thing - inspector) When the examiner points a finger at a nearby object, he or she gazes at the pointing point. (object) pointing + eye contact + showing When the examiner points a finger at a picture located far away, gaze at the pointing point.

However, this also corresponds to an embodiment, and may be determined by further reflecting more subdivided or other motion elements, and is not limited thereto.

However, an eye tracker may be used to determine passive joint attention. Accordingly, it is possible to objectively evaluate whether or not the target infant or toddler is looking in the direction indicated by the examiner's finger.

3 is a flowchart illustrating a method for diagnosing developmental disorders in infants and young children based on artificial intelligence according to an embodiment of the present invention.

Referring to FIG. 3 , the diagnosis apparatus 100 includes at least one interaction including an interaction between an examiner and a target infant in a specific space from the photographing device 200 and body movement and eye tracking of the target infant according to a task request. When the image is acquired (S310), motion analysis including eye tracking of the target infant is performed (S330).

Based on the analysis result, it is determined whether the target infant is passive or voluntary joint attention (S350), and the diagnosis result for the target infant is output based on the artificial intelligence machine learning model using the judgment (S370). ).

4 is a diagram showing an example of an observation environment created for diagnosing developmental disorders in infants and young children according to an embodiment of the present invention.

Referring to FIG. 4 , an observation environment created to diagnose developmental disorders in infants and young children according to an embodiment of the present invention is performed in a certain space (A), and a table (B) disposed in the certain space (A) As a standard, the target infant 10 and the examiner 30 face each other and perform the diagnosis. At this time, the guardian 20 of the target infant 10 may stay together within the predetermined space A according to the situation (necessity).

At this time, at least one toy may be placed on the table B as a close object 41, and at least one image (picture) 42a, 42b as a distant object 42 may be placed on the top of the rear wall of the target infant or toddler. can be placed.

The inspector 30 may determine voluntary joint attention by placing the toy on the table at a position where the target infant 10 cannot reach out and grab it right away and observe the child's behavior. In addition, when the inspector 30 calls the name of the target infant 10 and then points to a toy located on the table with a finger, whether the target infant turns his gaze in the direction indicated, or the inspector 30 determines whether the target infant 10 After calling the name of the child, when pointing to the picture (42a or 42b) located at the top of the back of the child with a finger, passive joint attention can be judged by whether the target infant turns his/her gaze in the direction indicated.

At this time, this diagnosis process may be captured by the photographing device 200 installed in a certain space (A), and at least one captured image is transmitted to the diagnosis device 100 . However, although a single camera is shown as the photographing device 200 in FIG. 4 , this is for convenience of description, and a plurality of cameras may be disposed at different positions in a certain space A to capture images, respectively.

5 is a diagram showing an example of movements of a target infant or toddler according to an observation result for diagnosing developmental disorders in infants and young children according to an embodiment of the present invention.

Referring to FIG. 5 , as an observation result analyzed through at least one captured image, a face landmark angle that changes over time is automatically measured. A head turn was conducted to induce joint attention of target infants, and the angle change according to the movement can be confirmed.

The method for diagnosing developmental disorders in infants and young children according to one embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a hardware device and stored in a medium.

For example, an examiner and/or a guardian of a target infant or toddler may each install and execute an application through a user terminal used by the examiner to receive a diagnosis service for developmental disorders in infants and young children. In this case, as shown in (a) to (b) of FIG. 6 , the inspector and/or the guardian of the target infant may directly perform photographing through each user terminal. At this time, the action guide and the photographing guide are It is provided so that the examiner and/or the guardian of the target infant can easily take an image suitable for diagnosis. The captured image may be de-identified and uploaded as shown in (c) to (d) of FIG. 6 .

On the other hand, as shown in (a) to (b) of FIG. 7, various missions may be given according to each inspection item, and the inspector and/or the guardian of the target infant takes an image corresponding to the mission. can be performed.

On the other hand, as shown in (a) to (c) of FIG. 8, the captured image is stored, and the progress rate, re-shooting, evaluation result, etc. are displayed on the user terminal of the examiner or the guardian of the target infant. can be easily verified.

On the other hand, each terminal of the examiner and/or the guardian of the target infant or toddler who wants to use the infant development disorder diagnosis service is a computer, UMPC (Ultra Mobile PC), workstation, net-book, PDA (Personal Digital Assistants), portable (portable) computer, web tablet, wireless phone, mobile phone, smart phone, pad, smart watch, wearable It may be a terminal, an e-book, a portable multimedia player (PMP), a portable game machine, a navigation device, a black box, a digital camera, and other mobile communication terminals. However, this is only one example, and a diagnosis service for infant development disorders may be provided by accessing the web page.

The aforementioned program is C, C++, JAVA, machine language, etc. It may include a code coded in a computer language of. These codes may include functional codes related to functions defining necessary functions for executing the methods, and include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include memory reference related codes for which location (address address) of the computer's internal or external memory should be referenced for additional information or media required for the computer's processor to execute the functions. there is. In addition, when the processor of the computer needs to communicate with any other remote computer or server in order to execute the functions, the code uses the computer's communication module to determine how to communicate with any other remote computer or server. It may further include communication-related codes for whether to communicate, what kind of information or media to transmit/receive during communication, and the like.

The storage medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and is readable by a device. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers accessible by the computer or various recording media on the user's computer. In addition, the medium may be distributed to computer systems connected through a network, and computer readable codes may be stored in a distributed manner.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

10: infant development disorder diagnosis system 100: diagnosis device
200: photographing device 300: user terminal
110: communication module 130: storage module
150: control module 10: target infants
20: protector 30: inspector
41: nearby object 42: distant object
42a, 42b: at least one image A: constant space
B: table

Claims (5)

In the artificial intelligence-based infant development disorder diagnosis method performed by the device,
Obtaining at least one image including an activity of the target infant or toddler;
analyzing motion including eye tracking of the target infant based on the obtained at least one image;
Determining whether there is passive joint attention (Response to joint attention, RJA) or voluntary joint attention (IJA) according to the analysis result; and
Outputting a diagnosis result for the target infant based on an artificial intelligence machine learning model; includes,
Characterized in that the machine learning model is pre-learned using learning data combining image data and diagnostic data including eye tracking data,
Methods for diagnosing developmental disorders in infants and young children.
According to claim 1,
The obtained at least one image,
The body of the target infant or toddler according to the interaction between the target infant and the examiner based on at least one first object disposed at a distance and at least one second object disposed at a short distance based on the target infant or young child, and a task request. Characterized in that it includes movement and eye tracking,
Methods for diagnosing developmental disorders in infants and young children.
According to claim 1,
The determining step is
automatically measuring face landmark angles that change over time;
determining whether a head-turn has occurred based on the measured angle; and
When it is determined that an angle change equal to or greater than a preset angle has occurred more than a preset number of times from the viewpoint of the target infant or toddler, determining that alternating gaze has occurred; characterized in that it comprises,
Methods for diagnosing developmental disorders in infants and young children.
According to claim 1,
The at least one image,
Characterized in that it includes images obtained by each of a plurality of photographing devices having different arrangement positions, and the images obtained for each photographing device are divided into a plurality of sets as one set.
Methods for diagnosing developmental disorders in infants and young children.
According to claim 1,
The at least one image may include a 2D camera, a 3D camera, a Time of Flight (ToF) camera, a light field camera, a stereo camera, an event camera, an infrared camera, a lidar sensor, and an array Obtained by at least one photographing device of the camera,
Characterized in that the eye tracking is confirmed by an eye tracker,
Methods for diagnosing developmental disorders in infants and young children.

Images