WO2023008607A1 - Chemistry education device using artificial intelligence and augmented reality and method of providing chemistry education service using the same - Google Patents

Chemistry education device using artificial intelligence and augmented reality and method of providing chemistry education service using the same Download PDF

Info

Publication number
WO2023008607A1
WO2023008607A1 PCT/KR2021/009806 KR2021009806W WO2023008607A1 WO 2023008607 A1 WO2023008607 A1 WO 2023008607A1 KR 2021009806 W KR2021009806 W KR 2021009806W WO 2023008607 A1 WO2023008607 A1 WO 2023008607A1
Authority
WO
WIPO (PCT)
Prior art keywords
chemical
formula
equation
output unit
error
Prior art date
Application number
PCT/KR2021/009806
Other languages
French (fr)
Inventor
Nourin Haque RIDI
Yasir Hasan TAQI
Md Rukshar ALAM
Original Assignee
Ridi Nourin Haque
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ridi Nourin Haque filed Critical Ridi Nourin Haque
Publication of WO2023008607A1 publication Critical patent/WO2023008607A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/54—Browsing; Visualisation therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/5846—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using extracted text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
    • G06Q50/10—Services
    • G06Q50/20—Education
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/22—Character recognition characterised by the type of writing
    • G06V30/224—Character recognition characterised by the type of writing of printed characters having additional code marks or containing code marks
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B23/00—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes
    • G09B23/24—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for chemistry
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B7/00—Electrically-operated teaching apparatus or devices working with questions and answers
    • G09B7/02—Electrically-operated teaching apparatus or devices working with questions and answers of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
    • G09B7/04—Electrically-operated teaching apparatus or devices working with questions and answers of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question, supplying a further explanation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions

Definitions

  • the present invention relates to a chemistry education device and a method of providing a chemistry education service using the same, and more particularly, to a chemistry education device using artificial intelligence and augmented reality and a method of providing a chemistry education service using the same.
  • a problem to be solved by the present invention is to provide a method of providing a chemistry education service that can visualize and display a difficult chemical term in augmented reality for easy understanding and can automatically extract an error in a chemical equation and display a correct answer and a chemical structural formula thereof in augmented reality when the error is present in the chemical equation.
  • Another problem to be solved by the present invention is to provide a chemistry education device that provides the chemistry education service.
  • a method of providing a chemistry education service according to an embodiment of the present invention for achieving the problem is a method of providing a chemistry education service using a chemistry education device including a camera unit, an output unit and a database, and may include a step of, by the chemistry education device, selectively providing an AI tutoring service or an AR implementation service.
  • the step of providing the AI tutoring service may include a step of photographing a targeted object and displaying image information of the targeted object on the output unit, a step of detecting a chemical equation written on the targeted object, a step of recognizing a chemical symbol, a number, and other symbols constituting the chemical equation, a step of extracting a context of the chemical equation using a positional relationship between the chemical symbol, the number, and the other symbols, a step of extracting an error by defining a plurality of reference reaction formulas in the database in advance and comparing the chemical equation with the reference reaction formulas, a step of, when an error is present in the chemical equation, displaying a bounding box around the error on the output unit, and a step of displaying a correct answer for the error on the output unit, and visualizing a chemical structural formula of the correct answer in augmented reality according to a user's selection on the output unit.
  • the step of extracting the context may include a step of classifying one side as a reactant and the other side as a product based on an arrow or an equal sign in the chemical equation, and a step of extracting the context of the chemical reaction by distinguishing between (i) a case in which the number is placed in front of the chemical symbol, (ii) a case in which the number is placed as a subscript after the chemical symbol, and (iii) a case in which the number is placed in a superscript after the chemical symbol in the positional relationship between the chemical symbol and the number.
  • the step of extracting the error may include a step of distinguishing between a reactant consisting of one or more substances and a product consisting of one or more substances from the chemical equation and selecting a reference reaction formula with the highest degree of matching from the database based on the substances included in the chemical equation, a step of extracting a chemical formula error by comparing a chemical formula of the reactant and a chemical formula of the product for the chemical equation with a chemical formula of the reactant and a chemical formula of the product for the reference equation, respectively, and a step of, after correcting the chemical formula error, extracting a coefficient error by comparing a coefficient of the reactant and a coefficient of the product for the chemical equation with a coefficient of the reactant and a coefficient of the product for the reference reaction formula, respectively.
  • the step of providing the AR implementation service may include a step of photographing a targeted object and displaying the targeted object on the output unit, a step of detecting a chemical term written on the targeted object (the chemical term includes a chemical name and chemical formula), a step of displaying a bounding box around the chemical term on the output unit when a plurality of reference terms in the database are defined in advance and the chemical term matches the reference term, a step of displaying at least one option window selected from a group consisting of a 'reading method' option window, a 'view molecular formula' option window, a 'view structural formula' option window, and a 'view additional information' option window for the chemical term on the output unit when a selection event occurs for the bounding box, and a step of displaying education information corresponding to the selected option window on the output unit when a selection event occurs with respect to any one of the option windows.
  • a chemistry education device for achieving the other problem is a chemistry education device that provides a chemistry education service, and may include a camera unit that photographs a targeted object, an output unit that displays image information of the targeted object, an AI tutoring unit that provides AI tutoring service for a chemical equation written on the targeted object, an AR implementation unit that provides an AR implementation service for a chemical term written on the targeted object, and a database that stores a plurality of reference reaction formulas and a plurality of reference terms defined in advance.
  • the AI tutoring unit may detect the chemical equation written on the targeted object from the image information displayed on the output unit, recognize a chemical symbol, a number and other symbols constituting the chemical equation, extract a context of the chemical equation using a positional relationship between the chemical symbol, the number, and the other symbols, extract an error by comparing the chemical equation with the reference reaction formula, display a bounding box around an error on the output unit when the error is present in the chemical equation, and display a correct answer for the error on the output unit and visualizes a chemical structural formula of the correct answer in augmented reality according to a user's selection on the output unit.
  • the AI tutoring unit may classify one side as a reactant and the other side as a product based on an arrow or an equal sign in the chemical equation, and extract the context of the chemical reaction by distinguishing between (i) a case in which the number is placed in front of the chemical symbol, (ii) a case in which the number is placed as a subscript after the chemical symbol, and (iii) a case in which the number is placed in a superscript after the chemical symbol in the positional relationship between the chemical symbol and the number.
  • the AI tutoring unit distinguishes between a reactant consisting of one or more substances and a product consisting of one or more substances from the chemical equation and selects a reference reaction formula with the highest degree of matching from the database based on the substances included in the chemical equation, extract a chemical formula error by comparing a chemical formula of the reactant and a chemical formula of the product for the chemical equation with a chemical formula of the reactant and a chemical formula of the product for the reference equation, respectively, and, after correcting the chemical formula error, extract a coefficient error by comparing a coefficient of the reactant and a coefficient of the product for the chemical equation with a coefficient of the reactant and a coefficient of the product for the reference reaction formula, respectively.
  • the AR implementation unit may detect a chemical term written on the targeted object displayed on the output unit (the chemical term includes a chemical name and chemical formula), display a bounding box around the chemical term on the output unit when the chemical term matches the reference term, display at least one option window selected from a group consisting of a 'reading method' option window, a 'view molecular formula' option window, a 'view structural formula' option window, and a 'view additional information' option window for the chemical term on the output unit when a selection event occurs for the bounding box, and display education information corresponding to the selected option window on the output unit when a selection event occurs for any one of the option windows.
  • the chemical term includes a chemical name and chemical formula
  • the chemistry education device when the chemistry education device photographs a targeted object such as paper, the chemistry education device automatically may detect a chemical term or chemical equation among handwritten or printed text on the targeted object and provide the AI tutoring service or AR implementation service according to the user's selection.
  • a chemical symbol, a number, and other symbols can be individually recognized from the detected chemical equations, and the context of the chemical equation can be extracted using a mutual positional relationship between the chemical symbol, the number, and the other symbols.
  • the chemical equation By comparing the chemical equation with the reference reaction formula defined in the database, it is possible to automatically extract a chemical formula error and a coefficient error sequentially.
  • the user By displaying a bounding box around the error on the output unit and displaying a correct answer for the error and the chemical structural formula of the correct answer in augmented reality around the bounding box, the user can easily identify and understand the error which is present in the chemical equation.
  • one or more chemical terms on the targeted object can be detected and bounding boxes can be displayed around the detected chemical terms on the output unit, and the 'reading method' option window, 'view molecular formula' option window, 'view structural formula' option window, 'view additional information' option window, or a combination thereof for the corresponding chemical term can be displayed on the output unit when the user selects any one of the bounding boxes.
  • the user selects any one of these option windows education information corresponding to the selected option window is displayed on the output unit. For example, when a chemical term consists of a predetermined chemical formula, if the user selects the 'reading method' option window, a chemical name of the chemical formula is displayed on the output unit.
  • the chemical term consists of a predetermined chemical name
  • the molecular formula of the chemical name is displayed on the output unit.
  • the structural formula of the chemical formula or chemical name is displayed in the output unit, and if the user selects the 'view additional information' option, additional information about the chemical formula or chemical name, for example, a website or video can be activated. Therefore, when a user learns chemistry, if a chemical term he or she does not know appears, he or she can easily obtain information useful for learning simply by photographing the targeted object on which the chemical term is written with the chemistry education device.
  • FIG. 1 is a configuration diagram conceptually illustrating a chemistry education device according to an embodiment of the present invention.
  • FIG. 2 is a configuration diagram conceptually illustrating an AI tutoring unit of FIG. 1.
  • FIG. 3 is a flowchart illustrating a method of providing an AI tutoring service according to an embodiment of the present invention.
  • FIG. 4 is a diagram exemplarily illustrating a process of detecting a chemical equation using an artificial intelligence algorithm.
  • FIG. 5 is a configuration diagram conceptually illustrating an AR implementation unit of FIG. 1.
  • FIG. 6 is a flowchart illustrating a method of providing an AR implementation service according to an embodiment of the present invention.
  • FIG. 7 is a diagram exemplarily illustrating a process of detecting a chemical term using the artificial intelligence algorithm.
  • FIG. 8 is a diagram illustrating a process of providing the AI tutoring service using the chemistry education device according to an embodiment of the present invention.
  • FIG. 9 is a diagram illustrating a process of providing the AR implementation service using the chemistry education device according to an embodiment of the present invention.
  • a "chemical formula” used in the present invention may be understood as a concept including a molecular formula, empirical formula, rational formula, and structural formula commonly used in the chemical field.
  • a "chemical term” used in the present invention may be understood as a concept including the chemical formula and a chemical name.
  • FIG. 1 is a configuration diagram conceptually illustrating the chemistry education device according to an embodiment of the present invention.
  • a chemistry education device 100 is a device that provides a chemistry education service, and is a device that selectively provides ⁇ AI tutoring service> for error correction of a chemical equation or ⁇ AR implementation service> that provides educational information about a chemical term using artificial intelligence and augmented reality.
  • the chemistry education device 100 includes a camera unit 10, an input unit 20, an output unit 30, an AI tutoring unit 40, an AR implementation unit 50, and a database 60.
  • the camera unit 10 displays image information generated by photographing a targeted object, such as paper, a book, or a print, on the output unit 30.
  • the input unit 20 includes a software or hardware input device
  • the output unit 30 includes a display.
  • the display may include a user interface as a means for detecting a user's touch input in UI/UX of operating system software and UI/UX of application software.
  • the display may be composed of a touch screen that simultaneously executes a function of an output unit for outputting a screen and a function of an input unit for detecting the user's touch event.
  • the output unit 30 outputs image information of the targeted object to the display screen.
  • the AI tutoring unit 40 detects a chemical equation written on the targeted object from image information of the targeted object, and provides an artificial intelligence (AI) tutoring service for the chemical equation.
  • AI artificial intelligence
  • the AR implementation unit 50 detects a chemical term written on the targeted object from the image information of the targeted object and provides an augmented reality (AR) implementation service for the chemical term.
  • AR augmented reality
  • the database 60 generally provides a place to store computer code and data used by the devices. Specifically, the database 60 defines and stores a plurality of reference reaction formulas for the AI tutoring service and a plurality of reference terms for the AR implementation service in advance. In the database 60, a mobile application and resources necessary for its driving/management, as well as a basic input/output system, an operating system, various programs, applications, or firmware for any device including user interface functions, processor functions, etc. executed in the device may be stored.
  • the chemistry education device 100 may include a separate processor (not illustrated), and the processor, along with the operating system, executes the computer code and executes an operation of generating and using data.
  • the processor can receive and process input and output data between components of the chemistry education device 100 using a series of instructions.
  • the processor serves as a control unit for executing functions of operating system software and various application software installed in the chemistry education device 100.
  • the chemistry education device 100 may include a terminal device having a wireless communication function that a user can move and carry.
  • a terminal may be a mobile communication terminal device, a navigation terminal device, a smart phone, a tablet PC, a wearable smart device, a PDA, a computer, a notebook computer, etc. capable of communication connection through a mobile communication network.
  • FIG. 2 is a configuration diagram conceptually illustrating the AI tutoring unit of FIG. 1.
  • FIG. 3 is a flowchart illustrating a method of providing the AI tutoring service according to an embodiment of the present invention.
  • FIG. 4 is a diagram exemplarily illustrating a process of detecting a chemical equation using an artificial intelligence algorithm.
  • the AI tutoring unit 40 includes a chemical equation detection unit 42, a matching unit 44, an error correction unit 46, and a modeling unit 48, and ⁇ AI tutoring service> proceeds as follows.
  • the chemical equation detection unit 42 detects a chemical equation written on the targeted object from the image information of the targeted object displayed on the output unit 30 (S32).
  • the chemical equation detection unit 42 distinguishes and recognizes the chemical symbol, the number, and other symbols constituting the chemical equation through artificial intelligence and machine learning (S33). Specifically, the chemical equation detection unit 42 recognizes a text from a targeted object image using a machine learning engine, and detects a token probability of a LaTeX code from a softmax layer of the machine learning engine. Subsequently, the chemical equation detection unit 42 fetches a LaTeX markup command from a token with the highest probability, and converts the LaTeX command into a mathematical markup language (MathML) version. Subsequently, the chemical equation detection unit 42 stores MathML in the database 60 and recognizes the chemical symbol, the number, and the other symbols by classifying them from a character string constituting the chemical equation.
  • MathML MathML in the database 60 and recognizes the chemical symbol, the number, and the other symbols by classifying them from a character string constituting the chemical equation.
  • the chemical equation of this embodiment may consist of the chemical symbol, the number, and other symbols, and the chemical equation detection unit 42 extracts a context of the chemical equation by using a positional relationship between the chemical symbol, the number, and the other symbols (S34).
  • a chemical symbol is a symbol expressed with one or two Roman characters to simply indicate a chemical element.
  • hydrogen can be represented by the chemical symbol such as H, oxygen as O, and iron as Fe.
  • the chemical symbol may be understood to be substantially the same as an element symbol on a periodic table of elements.
  • the number may represent a stoichiometric coefficient, the number of elements, or the number of ions.
  • the number When the number is placed in front of a chemical symbol, the number may be understood to have the meaning of a coefficient or a stoichiometric coefficient.
  • the number When the number is placed as a subscript after a chemical symbol, the number may be understood that the number has a meaning of the number of elements.
  • the number When the number is placed after a chemical symbol as a superscript and is written together with a negative sign or a positive sign, the number may be understood to have a meaning of the number of ions.
  • the chemical equation detection unit 42 first classifies one side as the reactant and the other side as the product based on another symbol (e.g., an arrow or equal sign) in the chemical equation and subsequently, extracts the context of the chemical reaction by distinguishing between (i) a case in which the number is placed in front of the chemical symbol, (ii) a case in which the number is placed as a subscript after the chemical symbol, and (iii) a case in which the number is placed in a superscript after the chemical symbol in the positional relationship between the chemical symbol and the number.
  • another symbol e.g., an arrow or equal sign
  • the matching unit 44 extracts an error by comparing the chemical equation with a reference reaction formula defined in the database 60 in advance (S35). Specifically, the matching unit 44 sequentially performs ⁇ selection process of reference reaction formula>, ⁇ extraction process of chemical formula error>, and ⁇ extraction process of coefficient error>.
  • the matching unit 44 distinguishes a reactant and a product from the detected chemical equation.
  • the reactant may consist of one or more substances
  • the product may consist of one or more substances.
  • the matching unit 44 retrieves the reference reaction formula stored in the database 60 based on the substances included in the chemical equation, and selects the reference reaction formula with the highest degree of matching from the database 60.
  • the matching unit 44 extracts a chemical formula of the reactant and a chemical formula of the product in the chemical equation.
  • the chemical formula of each substance is extracted.
  • the matching unit 44 extracts a chemical formula error by comparing the chemical formula of the reactant and the chemical formula of the product for the chemical equation with a chemical formula of the reactant and a chemical formula of the product for the reference equation, respectively;
  • the matching unit 44 corrects the chemical formula error, and then extracts a coefficient error by comparing a coefficient of the reactant and a coefficient of the product for the chemical equation with a coefficient of the reactant and a coefficient of the product for the reference reaction formula, respectively.
  • the error extraction process can proceed as follows.
  • the matching unit 44 distinguishes the reactants (H 2 , O 3 ) and the product (H 2 O) in the chemical equation (2H 2 + O 3 ⁇ H 2 O).
  • the matching unit 44 selects a reference reaction formula (2H 2 + O 2 ⁇ 2H 2 O) with the highest degree of matching with three substances (H 2 , O 3 , H 2 O) included in the chemical equation as the reference.
  • two substances (H 2 , H 2 O) match with each other.
  • the matching unit 44 compares the chemical formulas (H 2 , O 3 ) of the reactant and the chemical formula (H 2 O) of the product for the chemical equation with the chemical formulas (H 2 , O 2 ) of the reactant and the chemical formula (H 2 O) of the product for the reference reaction equation, respectively.
  • an error is found in the chemical formula (O 3 ) of the reactant in the chemical equation.
  • the matching unit 44 corrects the chemical formula error, and then compares the coefficient of the chemical equation with the coefficient of the reference reaction formula to extract the coefficient error.
  • an error is found in the coefficient of the product (H 2 O) in the chemical equation.
  • the error correction unit 46 displays a bounding box around the error on the output unit 30 (S36).
  • the error correction unit 46 displays a correct answer for the error on the output unit 30 (S37).
  • the correct answer it may be in the form of a text such as a chemical formula.
  • the modeling unit 48 visualizes a chemical structural formula of the correct answer in 2D or 3D type augmented reality and displays the chemical structural formula on the output unit 30 (S38).
  • a process of detecting a chemical reaction by the AI tutoring unit according to an embodiment of the present invention will be described with reference to FIG. 4.
  • a convolutional neural network (CNN) unit extracts features from an input image of the chemical equation using a convolution layer, a pooling layer, and an activation layer (S41).
  • CNN convolutional neural network
  • the input image is convolved with a series of kernels or image filters.
  • the pooling layer an image size is reduced and a size of a receptive field is increased.
  • the activation layer non-linearity is added to a neural network.
  • a positional encoding unit extracts an important mathematical semantic or context of chemical equations by identifying the positional relationship between the chemical symbol, the number, and the other symbols in the chemical equation (S42). This positional relationship can be extended in various directions, such as left and right, up and down, subscript, superscript, and overlap.
  • a decoder model is based on bidirectional long-short term memory (Bi-LSTM) cells and captures more complex language semantics (S45).
  • Bi-LSTM bidirectional long-short term memory
  • S45 complex language semantics
  • the bidirectional long and short-term memory can handle long sequences better than a standard recurrent neural network (RNN).
  • RNN recurrent neural network
  • LSTM Long short-term memory
  • a markup of a complex chemical equation can be extended to more than 100 tokens.
  • an initial hidden state vector of the RNN is not sufficient to compress all the information of the encoder.
  • an attention mechanism is introduced to solve this problem (S46).
  • the attention layer uses the encoder output to create a context vector.
  • a softmax function is used to create a probability distribution of the next token for the vocabulary (S47).
  • FIG. 5 is a block diagram conceptually illustrating the AR implementation unit of FIG. 1.
  • FIG. 6 is a flowchart illustrating a method of providing the AR implementation service according to an embodiment of the present invention.
  • FIG. 7 is a diagram exemplarily illustrating a process of detecting a chemical term using an artificial intelligence algorithm.
  • the AR implementation unit 50 includes a chemical term detection unit 52, a matching unit 54, and an option execution unit 56, and the ⁇ AR implementation service> proceeds as follows.
  • the chemical term detection unit 52 detects a chemical term written on the targeted object from the image information of the targeted object displayed on the output unit 30 (S62).
  • the chemical term may include a chemical formula and a chemical name.
  • the chemical term detection unit 52 recognizes the chemical formula and/or the chemical name through artificial intelligence and machine learning (S63). Specifically, the chemical term detection unit 52 recognizes a text from an image of targeted object using a machine learning engine, and detects a token probability of the LaTeX code from the softmax layer of the machine learning engine. Subsequently, the chemical term detection unit 52 fetches a LaTeX markup command from the token with the highest probability, and converts the LaTeX command into a mathematical markup language (MathML) version. Subsequently, the chemical term detection unit 52 stores MathML in the database 60 and recognizes the chemical formula or chemical name by classifying the chemical formula or chemical name.
  • MathML mathematical markup language
  • the matching unit 54 compares the chemical term with a reference term defined in the database 60 in advance, and displays a bounding box around the chemical term on the output unit 30 when the chemical term matches the reference term (S64). If a reference term matching the plurality of chemical terms exists in the database 60, a bounding box may be displayed around each of the plurality of chemical terms.
  • the option execution unit 56 displays at least one option window selected from a group consisting of a 'reading method' option window, a 'view molecular formula' option window, a 'view structural formula' option window and a 'view additional information' option window for the chemical term on the output unit (S65).
  • the option execution unit 56 displays education information corresponding to the selected option window on the output unit 30 (S66).
  • the option execution unit 56 displays a method of reading the chemical formula or the chemical name thereof within the selected bounding box on the output unit 30.
  • the selected option execution unit 56 displays the molecular formula corresponding to the chemical name within the bounding box on the output unit 30.
  • the option execution unit 56 displays the structural formula corresponding to the chemical formula or chemical name within the selected bounding box on the output unit 30.
  • additional information corresponding to the chemical formula or chemical name within the selected bounding box for example, an encyclopedia website or a moving picture is activated on the output unit 30.
  • the CNN layer extracts a relevant feature from the image (S71). Specifically, the convolution operation extracts the feature from an image by applying filter kernels of various sizes. In addition, a nonlinear rectified linear unit (ReLU) function is applied.
  • the pooling layer summarizes an image area and outputs a reduced version of the input.
  • a feature sequence provided from the CNN layer is supplied to the RNN cell carrying relevant information through this sequence (S72).
  • a long short-term memory (LSTM) implementation of the RNN is used. It can propagate information over long distances and provide stronger training characteristics than vanilla RNNs.
  • An RNN output sequence is mapped to a matrix encoding temporal information.
  • the RNN output matrix and a ground truth text are provided to connectionist temporal classification (CTC), and the CTC calculates a loss value (S73).
  • CTC connectionist temporal classification
  • S73 a loss value
  • S74 a final text
  • the CTC calculates a loss value for training a neural network (NN), decodes the matrix to fetch the text included in the input image, and infers a handwritten text label.
  • FIG. 8 is a diagram illustrating a process of providing the AI tutoring service using the chemistry education device according to an embodiment of the present invention.
  • FIG. 9 is a diagram illustrating a process of providing the AR implementation service using the chemistry education device according to an embodiment of the present invention.
  • FIGS. 8 and 9 illustrate a case of using a smart phone as an example of the chemistry education device.

Abstract

A chemistry education device using artificial intelligence and augmented reality and a method of providing a chemistry education service using the same are provided. This chemistry education service includes an AI tutoring service and an AR implementation service. In particular, the AI tutoring service recognizes a chemical symbol, a number, and other symbols that constitute a chemical equation, extracts a context of the chemical equation using a positional relationship between the chemical symbol, the number, and the other symbols, and extracts an error by comparing the chemical equation with a reference reaction formula, and visualizes a corrected answer in augmented reality and provides the corrected answer.

Description

CHEMISTRY EDUCATION DEVICE USING ARTIFICIAL INTELLIGENCE AND AUGMENTED REALITY AND METHOD OF PROVIDING CHEMISTRY EDUCATION SERVICE USING THE SAME
The present invention relates to a chemistry education device and a method of providing a chemistry education service using the same, and more particularly, to a chemistry education device using artificial intelligence and augmented reality and a method of providing a chemistry education service using the same.
As a result of a survey conducted on Bangladesh school students, it was found that the students had the most difficulty in understanding and visualizing a chemical formula, a chemical name, and a chemical equation in school education. These difficulties cause many mistakes in school examinations and act as a cause of lowering the desire for academic achievement. Accordingly, the inventors of the present application have completed the present invention so that a difficult chemical term or chemical equation can be visualized and easily understood by using new technologies of augmented reality and artificial intelligence in order to solve the difficulties of students.
A problem to be solved by the present invention is to provide a method of providing a chemistry education service that can visualize and display a difficult chemical term in augmented reality for easy understanding and can automatically extract an error in a chemical equation and display a correct answer and a chemical structural formula thereof in augmented reality when the error is present in the chemical equation.
Another problem to be solved by the present invention is to provide a chemistry education device that provides the chemistry education service.
The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention pertains from the following description.
A method of providing a chemistry education service according to an embodiment of the present invention for achieving the problem is a method of providing a chemistry education service using a chemistry education device including a camera unit, an output unit and a database, and may include a step of, by the chemistry education device, selectively providing an AI tutoring service or an AR implementation service. Here, the step of providing the AI tutoring service may include a step of photographing a targeted object and displaying image information of the targeted object on the output unit, a step of detecting a chemical equation written on the targeted object, a step of recognizing a chemical symbol, a number, and other symbols constituting the chemical equation, a step of extracting a context of the chemical equation using a positional relationship between the chemical symbol, the number, and the other symbols, a step of extracting an error by defining a plurality of reference reaction formulas in the database in advance and comparing the chemical equation with the reference reaction formulas, a step of, when an error is present in the chemical equation, displaying a bounding box around the error on the output unit, and a step of displaying a correct answer for the error on the output unit, and visualizing a chemical structural formula of the correct answer in augmented reality according to a user's selection on the output unit.
The step of extracting the context may include a step of classifying one side as a reactant and the other side as a product based on an arrow or an equal sign in the chemical equation, and a step of extracting the context of the chemical reaction by distinguishing between (i) a case in which the number is placed in front of the chemical symbol, (ii) a case in which the number is placed as a subscript after the chemical symbol, and (iii) a case in which the number is placed in a superscript after the chemical symbol in the positional relationship between the chemical symbol and the number.
The step of extracting the error may include a step of distinguishing between a reactant consisting of one or more substances and a product consisting of one or more substances from the chemical equation and selecting a reference reaction formula with the highest degree of matching from the database based on the substances included in the chemical equation, a step of extracting a chemical formula error by comparing a chemical formula of the reactant and a chemical formula of the product for the chemical equation with a chemical formula of the reactant and a chemical formula of the product for the reference equation, respectively, and a step of, after correcting the chemical formula error, extracting a coefficient error by comparing a coefficient of the reactant and a coefficient of the product for the chemical equation with a coefficient of the reactant and a coefficient of the product for the reference reaction formula, respectively.
The step of providing the AR implementation service may include a step of photographing a targeted object and displaying the targeted object on the output unit, a step of detecting a chemical term written on the targeted object (the chemical term includes a chemical name and chemical formula), a step of displaying a bounding box around the chemical term on the output unit when a plurality of reference terms in the database are defined in advance and the chemical term matches the reference term, a step of displaying at least one option window selected from a group consisting of a 'reading method' option window, a 'view molecular formula' option window, a 'view structural formula' option window, and a 'view additional information' option window for the chemical term on the output unit when a selection event occurs for the bounding box, and a step of displaying education information corresponding to the selected option window on the output unit when a selection event occurs with respect to any one of the option windows.
A chemistry education device according to an embodiment of the present invention for achieving the other problem is a chemistry education device that provides a chemistry education service, and may include a camera unit that photographs a targeted object, an output unit that displays image information of the targeted object, an AI tutoring unit that provides AI tutoring service for a chemical equation written on the targeted object, an AR implementation unit that provides an AR implementation service for a chemical term written on the targeted object, and a database that stores a plurality of reference reaction formulas and a plurality of reference terms defined in advance. Here, the AI tutoring unit may detect the chemical equation written on the targeted object from the image information displayed on the output unit, recognize a chemical symbol, a number and other symbols constituting the chemical equation, extract a context of the chemical equation using a positional relationship between the chemical symbol, the number, and the other symbols, extract an error by comparing the chemical equation with the reference reaction formula, display a bounding box around an error on the output unit when the error is present in the chemical equation, and display a correct answer for the error on the output unit and visualizes a chemical structural formula of the correct answer in augmented reality according to a user's selection on the output unit.
The AI tutoring unit may classify one side as a reactant and the other side as a product based on an arrow or an equal sign in the chemical equation, and extract the context of the chemical reaction by distinguishing between (i) a case in which the number is placed in front of the chemical symbol, (ii) a case in which the number is placed as a subscript after the chemical symbol, and (iii) a case in which the number is placed in a superscript after the chemical symbol in the positional relationship between the chemical symbol and the number.
The AI tutoring unit distinguishes between a reactant consisting of one or more substances and a product consisting of one or more substances from the chemical equation and selects a reference reaction formula with the highest degree of matching from the database based on the substances included in the chemical equation, extract a chemical formula error by comparing a chemical formula of the reactant and a chemical formula of the product for the chemical equation with a chemical formula of the reactant and a chemical formula of the product for the reference equation, respectively, and, after correcting the chemical formula error, extract a coefficient error by comparing a coefficient of the reactant and a coefficient of the product for the chemical equation with a coefficient of the reactant and a coefficient of the product for the reference reaction formula, respectively.
The AR implementation unit may detect a chemical term written on the targeted object displayed on the output unit (the chemical term includes a chemical name and chemical formula), display a bounding box around the chemical term on the output unit when the chemical term matches the reference term, display at least one option window selected from a group consisting of a 'reading method' option window, a 'view molecular formula' option window, a 'view structural formula' option window, and a 'view additional information' option window for the chemical term on the output unit when a selection event occurs for the bounding box, and display education information corresponding to the selected option window on the output unit when a selection event occurs for any one of the option windows.
Specific details of other embodiments are included in the specific contents and drawings.
According to the chemistry education device and method of providing the chemistry education service using the same according to the present invention as described above, when the chemistry education device photographs a targeted object such as paper, the chemistry education device automatically may detect a chemical term or chemical equation among handwritten or printed text on the targeted object and provide the AI tutoring service or AR implementation service according to the user's selection.
In the case of the AI tutoring service, a chemical symbol, a number, and other symbols can be individually recognized from the detected chemical equations, and the context of the chemical equation can be extracted using a mutual positional relationship between the chemical symbol, the number, and the other symbols. By comparing the chemical equation with the reference reaction formula defined in the database, it is possible to automatically extract a chemical formula error and a coefficient error sequentially. By displaying a bounding box around the error on the output unit and displaying a correct answer for the error and the chemical structural formula of the correct answer in augmented reality around the bounding box, the user can easily identify and understand the error which is present in the chemical equation.
In the case of the AR implementation service, one or more chemical terms on the targeted object can be detected and bounding boxes can be displayed around the detected chemical terms on the output unit, and the 'reading method' option window, 'view molecular formula' option window, 'view structural formula' option window, 'view additional information' option window, or a combination thereof for the corresponding chemical term can be displayed on the output unit when the user selects any one of the bounding boxes. When the user selects any one of these option windows, education information corresponding to the selected option window is displayed on the output unit. For example, when a chemical term consists of a predetermined chemical formula, if the user selects the 'reading method' option window, a chemical name of the chemical formula is displayed on the output unit. When the chemical term consists of a predetermined chemical name, if the user selects the 'view molecular formula' option window, the molecular formula of the chemical name is displayed on the output unit. In addition, if the user selects the 'view structural formula' option window, the structural formula of the chemical formula or chemical name is displayed in the output unit, and if the user selects the 'view additional information' option, additional information about the chemical formula or chemical name, for example, a website or video can be activated. Therefore, when a user learns chemistry, if a chemical term he or she does not know appears, he or she can easily obtain information useful for learning simply by photographing the targeted object on which the chemical term is written with the chemistry education device.
FIG. 1 is a configuration diagram conceptually illustrating a chemistry education device according to an embodiment of the present invention.
FIG. 2 is a configuration diagram conceptually illustrating an AI tutoring unit of FIG. 1.
FIG. 3 is a flowchart illustrating a method of providing an AI tutoring service according to an embodiment of the present invention.
FIG. 4 is a diagram exemplarily illustrating a process of detecting a chemical equation using an artificial intelligence algorithm.
FIG. 5 is a configuration diagram conceptually illustrating an AR implementation unit of FIG. 1.
FIG. 6 is a flowchart illustrating a method of providing an AR implementation service according to an embodiment of the present invention.
FIG. 7 is a diagram exemplarily illustrating a process of detecting a chemical term using the artificial intelligence algorithm.
FIG. 8 is a diagram illustrating a process of providing the AI tutoring service using the chemistry education device according to an embodiment of the present invention.
FIG. 9 is a diagram illustrating a process of providing the AR implementation service using the chemistry education device according to an embodiment of the present invention.
Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention will not be limited to the embodiments disclosed below, but will be implemented in various different forms. Only the present embodiments are provided to complete the disclosure of the present invention, and to fully inform those of ordinary skill in the art to which the present invention pertains of the scope of the invention, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.
A "chemical formula" used in the present invention may be understood as a concept including a molecular formula, empirical formula, rational formula, and structural formula commonly used in the chemical field. In addition, a "chemical term" used in the present invention may be understood as a concept including the chemical formula and a chemical name.
Hereinafter, a chemistry education device according to an embodiment of the present invention and a method of providing a chemistry education service using the same will be described in detail with reference to the accompanying drawings. FIG. 1 is a configuration diagram conceptually illustrating the chemistry education device according to an embodiment of the present invention.
Referring to FIG. 1, a chemistry education device 100 is a device that provides a chemistry education service, and is a device that selectively provides <AI tutoring service> for error correction of a chemical equation or <AR implementation service> that provides educational information about a chemical term using artificial intelligence and augmented reality. Specifically, the chemistry education device 100 includes a camera unit 10, an input unit 20, an output unit 30, an AI tutoring unit 40, an AR implementation unit 50, and a database 60.
The camera unit 10 displays image information generated by photographing a targeted object, such as paper, a book, or a print, on the output unit 30.
The input unit 20 includes a software or hardware input device, and the output unit 30 includes a display. The display may include a user interface as a means for detecting a user's touch input in UI/UX of operating system software and UI/UX of application software. The display may be composed of a touch screen that simultaneously executes a function of an output unit for outputting a screen and a function of an input unit for detecting the user's touch event. The output unit 30 outputs image information of the targeted object to the display screen.
The AI tutoring unit 40 detects a chemical equation written on the targeted object from image information of the targeted object, and provides an artificial intelligence (AI) tutoring service for the chemical equation.
The AR implementation unit 50 detects a chemical term written on the targeted object from the image information of the targeted object and provides an augmented reality (AR) implementation service for the chemical term.
The database 60 generally provides a place to store computer code and data used by the devices. Specifically, the database 60 defines and stores a plurality of reference reaction formulas for the AI tutoring service and a plurality of reference terms for the AR implementation service in advance. In the database 60, a mobile application and resources necessary for its driving/management, as well as a basic input/output system, an operating system, various programs, applications, or firmware for any device including user interface functions, processor functions, etc. executed in the device may be stored.
The chemistry education device 100 may include a separate processor (not illustrated), and the processor, along with the operating system, executes the computer code and executes an operation of generating and using data. In addition, the processor can receive and process input and output data between components of the chemistry education device 100 using a series of instructions. In addition, the processor serves as a control unit for executing functions of operating system software and various application software installed in the chemistry education device 100.
The chemistry education device 100 according to an embodiment of the present invention may include a terminal device having a wireless communication function that a user can move and carry. For example, such a terminal may be a mobile communication terminal device, a navigation terminal device, a smart phone, a tablet PC, a wearable smart device, a PDA, a computer, a notebook computer, etc. capable of communication connection through a mobile communication network.
Hereinafter, an AI tutoring unit according to an embodiment of the present invention and a method of providing an AI tutoring service using the same will be described in detail with reference to FIGS. 2 to 4. FIG. 2 is a configuration diagram conceptually illustrating the AI tutoring unit of FIG. 1. FIG. 3 is a flowchart illustrating a method of providing the AI tutoring service according to an embodiment of the present invention. FIG. 4 is a diagram exemplarily illustrating a process of detecting a chemical equation using an artificial intelligence algorithm.
Referring to FIGS. 2 and 3, the AI tutoring unit 40 includes a chemical equation detection unit 42, a matching unit 44, an error correction unit 46, and a modeling unit 48, and <AI tutoring service> proceeds as follows.
First, when the camera unit 10 photographs a targeted object to generate image information and displays the image information on the output unit 30 (S31), the chemical equation detection unit 42 detects a chemical equation written on the targeted object from the image information of the targeted object displayed on the output unit 30 (S32).
The chemical equation detection unit 42 distinguishes and recognizes the chemical symbol, the number, and other symbols constituting the chemical equation through artificial intelligence and machine learning (S33). Specifically, the chemical equation detection unit 42 recognizes a text from a targeted object image using a machine learning engine, and detects a token probability of a LaTeX code from a softmax layer of the machine learning engine. Subsequently, the chemical equation detection unit 42 fetches a LaTeX markup command from a token with the highest probability, and converts the LaTeX command into a mathematical markup language (MathML) version. Subsequently, the chemical equation detection unit 42 stores MathML in the database 60 and recognizes the chemical symbol, the number, and the other symbols by classifying them from a character string constituting the chemical equation.
The chemical equation of this embodiment may consist of the chemical symbol, the number, and other symbols, and the chemical equation detection unit 42 extracts a context of the chemical equation by using a positional relationship between the chemical symbol, the number, and the other symbols (S34). Here, a chemical symbol is a symbol expressed with one or two Roman characters to simply indicate a chemical element. For example, hydrogen can be represented by the chemical symbol such as H, oxygen as O, and iron as Fe. In addition, the chemical symbol may be understood to be substantially the same as an element symbol on a periodic table of elements.
The number may represent a stoichiometric coefficient, the number of elements, or the number of ions. When the number is placed in front of a chemical symbol, the number may be understood to have the meaning of a coefficient or a stoichiometric coefficient. When the number is placed as a subscript after a chemical symbol, the number may be understood that the number has a meaning of the number of elements. When the number is placed after a chemical symbol as a superscript and is written together with a negative sign or a positive sign, the number may be understood to have a meaning of the number of ions.
Other symbols may include an arrow, an equal sign (=), a positive sign (+), a negative sign (-), etc. In a character string constituting the chemical equation, one side may be classified as a reactant and the other side as a product based on an arrow or an equal sign.
As such, the chemical equation detection unit 42 first classifies one side as the reactant and the other side as the product based on another symbol (e.g., an arrow or equal sign) in the chemical equation and subsequently, extracts the context of the chemical reaction by distinguishing between (i) a case in which the number is placed in front of the chemical symbol, (ii) a case in which the number is placed as a subscript after the chemical symbol, and (iii) a case in which the number is placed in a superscript after the chemical symbol in the positional relationship between the chemical symbol and the number.
Subsequently, the matching unit 44 extracts an error by comparing the chemical equation with a reference reaction formula defined in the database 60 in advance (S35). Specifically, the matching unit 44 sequentially performs <selection process of reference reaction formula>, <extraction process of chemical formula error>, and <extraction process of coefficient error>.
In the case of the <selection process of the reference reaction equation>, the matching unit 44 distinguishes a reactant and a product from the detected chemical equation. Here, the reactant may consist of one or more substances, and the product may consist of one or more substances. The matching unit 44 retrieves the reference reaction formula stored in the database 60 based on the substances included in the chemical equation, and selects the reference reaction formula with the highest degree of matching from the database 60.
In the case of the <extraction process of formula error>, the matching unit 44 extracts a chemical formula of the reactant and a chemical formula of the product in the chemical equation. When the reactant or product consists of a plurality of substances, the chemical formula of each substance is extracted. The matching unit 44 extracts a chemical formula error by comparing the chemical formula of the reactant and the chemical formula of the product for the chemical equation with a chemical formula of the reactant and a chemical formula of the product for the reference equation, respectively;
In the case of the <extraction process of coefficient error>, the matching unit 44 corrects the chemical formula error, and then extracts a coefficient error by comparing a coefficient of the reactant and a coefficient of the product for the chemical equation with a coefficient of the reactant and a coefficient of the product for the reference reaction formula, respectively.
Referring to the example below, the error extraction process can proceed as follows.
[Chemical equation] 2H2 + O3 → H2O
[Reference reaction formula] 2H2 + O2 → 2H2O
[Extraction of chemical formula error] 2H2 + O2 → H2O
[Extract coefficient error] 2H2 + O2 → 2H2O
First, the matching unit 44 distinguishes the reactants (H2, O3) and the product (H2O) in the chemical equation (2H2 + O3 → H2O). The matching unit 44 selects a reference reaction formula (2H2 + O2 → 2H2O) with the highest degree of matching with three substances (H2, O3, H2O) included in the chemical equation as the reference. In the reference reaction formula and the chemical equation selected in this example, two substances (H2, H2O) match with each other. Subsequently, the matching unit 44 compares the chemical formulas (H2, O3) of the reactant and the chemical formula (H2O) of the product for the chemical equation with the chemical formulas (H2, O2) of the reactant and the chemical formula (H2O) of the product for the reference reaction equation, respectively. In this example, an error is found in the chemical formula (O3) of the reactant in the chemical equation. Subsequently, the matching unit 44 corrects the chemical formula error, and then compares the coefficient of the chemical equation with the coefficient of the reference reaction formula to extract the coefficient error. In this example, an error is found in the coefficient of the product (H2O) in the chemical equation.
Referring back to FIGS. 2 and 3, when an error is present in the chemical equation, the error correction unit 46 displays a bounding box around the error on the output unit 30 (S36).
The error correction unit 46 displays a correct answer for the error on the output unit 30 (S37). Here, in the case of the correct answer, it may be in the form of a text such as a chemical formula.
According to the user's selection, the modeling unit 48 visualizes a chemical structural formula of the correct answer in 2D or 3D type augmented reality and displays the chemical structural formula on the output unit 30 (S38).
A process of detecting a chemical reaction by the AI tutoring unit according to an embodiment of the present invention will be described with reference to FIG. 4.
A convolutional neural network (CNN) unit extracts features from an input image of the chemical equation using a convolution layer, a pooling layer, and an activation layer (S41). In each convolutional layer, the input image is convolved with a series of kernels or image filters. In the pooling layer, an image size is reduced and a size of a receptive field is increased. In the activation layer, non-linearity is added to a neural network.
A positional encoding unit extracts an important mathematical semantic or context of chemical equations by identifying the positional relationship between the chemical symbol, the number, and the other symbols in the chemical equation (S42). This positional relationship can be extended in various directions, such as left and right, up and down, subscript, superscript, and overlap.
The context of the positional encoding unit and a feature map of the CNN unit are combined together and unfolded into a one-dimensional array to constitute an encoder output (S43).
Since chemical equation tokens are related to each other, a word embedding layer, which is generally used in natural language processing (NLP), is added, and the token is projected as a high-dimensional vector (S44). These embeddings capture the interrelationships between different tokens.
A decoder model is based on bidirectional long-short term memory (Bi-LSTM) cells and captures more complex language semantics (S45). Here, the bidirectional long and short-term memory can handle long sequences better than a standard recurrent neural network (RNN). The use of bidirectional cells in each instrumentation helps to capture context in both forward and backward directions.
Long short-term memory (LSTM) can be extended to capture long-term memory, but it is not uncommon for a markup of a complex chemical equation to extend to more than 100 tokens. In this case, an initial hidden state vector of the RNN is not sufficient to compress all the information of the encoder. In addition, since the CNN encoder does not have a memory function, an attention mechanism is introduced to solve this problem (S46). The attention layer uses the encoder output to create a context vector.
A softmax function is used to create a probability distribution of the next token for the vocabulary (S47).
Hereinafter, the AR implementation unit according to an embodiment of the present invention and the method of providing the AR implementation service using the same will be described in detail with reference to FIGS. 5 to 7. FIG. 5 is a block diagram conceptually illustrating the AR implementation unit of FIG. 1. FIG. 6 is a flowchart illustrating a method of providing the AR implementation service according to an embodiment of the present invention. FIG. 7 is a diagram exemplarily illustrating a process of detecting a chemical term using an artificial intelligence algorithm.
Referring to FIGS. 5 and 6, the AR implementation unit 50 includes a chemical term detection unit 52, a matching unit 54, and an option execution unit 56, and the <AR implementation service> proceeds as follows.
First, when the camera unit 10 captures a targeted object to generate image information and displays the image information on the output unit 30 (S61), the chemical term detection unit 52 detects a chemical term written on the targeted object from the image information of the targeted object displayed on the output unit 30 (S62). Here, the chemical term may include a chemical formula and a chemical name.
The chemical term detection unit 52 recognizes the chemical formula and/or the chemical name through artificial intelligence and machine learning (S63). Specifically, the chemical term detection unit 52 recognizes a text from an image of targeted object using a machine learning engine, and detects a token probability of the LaTeX code from the softmax layer of the machine learning engine. Subsequently, the chemical term detection unit 52 fetches a LaTeX markup command from the token with the highest probability, and converts the LaTeX command into a mathematical markup language (MathML) version. Subsequently, the chemical term detection unit 52 stores MathML in the database 60 and recognizes the chemical formula or chemical name by classifying the chemical formula or chemical name.
The matching unit 54 compares the chemical term with a reference term defined in the database 60 in advance, and displays a bounding box around the chemical term on the output unit 30 when the chemical term matches the reference term (S64). If a reference term matching the plurality of chemical terms exists in the database 60, a bounding box may be displayed around each of the plurality of chemical terms.
When a selection event by the user occurs for any one of the bounding boxes displayed on the output unit 30, the option execution unit 56 displays at least one option window selected from a group consisting of a 'reading method' option window, a 'view molecular formula' option window, a 'view structural formula' option window and a 'view additional information' option window for the chemical term on the output unit (S65).
When a selection event occurs for any one of the option windows by the user, the option execution unit 56 displays education information corresponding to the selected option window on the output unit 30 (S66). When the user selects the 'reading method' option window, the option execution unit 56 displays a method of reading the chemical formula or the chemical name thereof within the selected bounding box on the output unit 30. When the user selects the 'view molecular formula' option window, the selected option execution unit 56 displays the molecular formula corresponding to the chemical name within the bounding box on the output unit 30. When the user selects the 'view structural formula' option window, the option execution unit 56 displays the structural formula corresponding to the chemical formula or chemical name within the selected bounding box on the output unit 30. When the user selects the 'view additional information' option window, additional information corresponding to the chemical formula or chemical name within the selected bounding box, for example, an encyclopedia website or a moving picture is activated on the output unit 30.
A process of detecting a chemical term by the AR implementation unit according to an embodiment of the present invention will be described with reference to FIG. 7.
When an input image of the targeted object is supplied to the CNN layer, the CNN layer extracts a relevant feature from the image (S71). Specifically, the convolution operation extracts the feature from an image by applying filter kernels of various sizes. In addition, a nonlinear rectified linear unit (ReLU) function is applied. The pooling layer summarizes an image area and outputs a reduced version of the input.
Subsequently, a feature sequence provided from the CNN layer is supplied to the RNN cell carrying relevant information through this sequence (S72). A long short-term memory (LSTM) implementation of the RNN is used. It can propagate information over long distances and provide stronger training characteristics than vanilla RNNs. An RNN output sequence is mapped to a matrix encoding temporal information.
The RNN output matrix and a ground truth text are provided to connectionist temporal classification (CTC), and the CTC calculates a loss value (S73). During inference, only a matrix is provided to the CTC and the CTC decodes the matrix into a final text (S74). Therefore, in the above process, the CTC calculates a loss value for training a neural network (NN), decodes the matrix to fetch the text included in the input image, and infers a handwritten text label.
FIG. 8 is a diagram illustrating a process of providing the AI tutoring service using the chemistry education device according to an embodiment of the present invention. FIG. 9 is a diagram illustrating a process of providing the AR implementation service using the chemistry education device according to an embodiment of the present invention. FIGS. 8 and 9 illustrate a case of using a smart phone as an example of the chemistry education device.
Although the embodiments of the present invention have been described with reference to the accompanying drawings, those of ordinary skill in the art to which the present invention pertains will understand that the present invention may be embodied in other specific forms without changing the technical spirit or essential features thereof. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

Claims (8)

  1. A method of providing a chemistry education service using a chemistry education device including a camera unit, an output unit and a database, the method comprising:
    a step of, by the chemistry education device, selectively providing an AI tutoring service or an AR implementation service,
    wherein the step of providing the AI tutoring service includes
    a step of photographing a targeted object and displaying image information of the targeted object on the output unit,
    a step of detecting a chemical equation written on the targeted object,
    a step of recognizing a chemical symbol, a number, and other symbols constituting the chemical equation,
    a step of extracting a context of the chemical equation using a positional relationship between the chemical symbol, the number, and the other symbols,
    a step of extracting an error by defining a plurality of reference reaction formulas in the database in advance and comparing the chemical equation with the reference reaction formulas,
    a step of, when an error is present in the chemical equation, displaying a bounding box around the error on the output unit, and
    a step of displaying a correct answer for the error on the output unit, and visualizing a chemical structural formula of the correct answer in augmented reality according to a user's selection on the output unit.
  2. The method of providing the chemistry education service according to claim 1,
    wherein the step of extracting the context includes
    a step of classifying one side as a reactant and the other side as a product based on an arrow or an equal sign in the chemical equation, and
    a step of extracting the context of the chemical reaction by distinguishing between (i) a case in which the number is placed in front of the chemical symbol, (ii) a case in which the number is placed as a subscript after the chemical symbol, and (iii) a case in which the number is placed in a superscript after the chemical symbol in the positional relationship between the chemical symbol and the number.
  3. The method of providing the chemistry education service according to claim 1,
    wherein the step of extracting the error includes
    a step of distinguishing between a reactant consisting of one or more substances and a product consisting of one or more substances from the chemical equation and selecting a reference reaction formula with the highest degree of matching from the database based on the substances included in the chemical equation,
    a step of extracting a chemical formula error by comparing a chemical formula of the reactant and a chemical formula of the product for the chemical equation with a chemical formula of the reactant and a chemical formula of the product for the reference equation, respectively, and
    a step of, after correcting the chemical formula error, extracting a coefficient error by comparing a coefficient of the reactant and a coefficient of the product for the chemical equation with a coefficient of the reactant and a coefficient of the product for the reference reaction formula, respectively.
  4. The method of providing the chemistry education service according to claim 1,
    wherein the step of providing the AR implementation service includes
    a step of photographing a targeted object and displaying the targeted object on the output unit,
    a step of detecting a chemical term written on the targeted object (the chemical term includes a chemical name and chemical formula),
    a step displaying a bounding box around the chemical term on the output unit when a plurality of reference terms in the database are defined in advance and the chemical term matches the reference term,
    a step of displaying at least one option window selected from a group consisting of a 'reading method' option window, a 'view molecular formula' option window, a 'view structural formula' option window and a 'view additional information' option window for the chemical term on the output unit when a selection event occurs with respect to the bounding box, and
    a step of displaying education information corresponding to the selected option window on the output unit when a selection event occurs for any one of the option windows.
  5. A chemistry education device that provides a chemistry education service, comprising:
    a camera unit that photographs a targeted object;
    an output unit that displays image information of the targeted object;
    an AI tutoring unit that provides AI tutoring service for a chemical equation written on the targeted object;
    an AR implementation unit that provides an AR implementation service for a chemical term written on the targeted object; and
    a database that stores a plurality of reference reaction formulas and a plurality of reference terms defined in advance,
    wherein the AI tutoring unit
    detects the chemical equation written on the targeted object from the image information displayed on the output unit,
    recognizes a chemical symbol, a number and other symbols constituting the chemical equation,
    extracts a context of the chemical equation using a positional relationship between the chemical symbol, the number, and the other symbols,
    extracts an error by comparing the chemical equation with the reference reaction formula,
    displays a bounding box around the error on the output unit when the error is present in the chemical equation, and
    displays a correct answer for the error on the output unit and visualizes a chemical structural formula of the correct answer in augmented reality according to a user's selection on the output unit.
  6. The chemistry education device according to claim 5,
    wherein the AI tutoring unit
    classifies one side as a reactant and the other side as a product based on an arrow or an equal sign in the chemical equation; and
    extracts the context of the chemical reaction by distinguishing between (i) a case in which the number is placed in front of the chemical symbol, (ii) a case in which the number is placed as a subscript after the chemical symbol, and (iii) a case in which the number is placed in a superscript after the chemical symbol in the positional relationship between the chemical symbol and the number.
  7. The chemistry education device according to claim 5,
    wherein the AI tutoring unit
    distinguishes between a reactant consisting of one or more substances and a product consisting of one or more substances from the chemical equation and selects a reference reaction formula with the highest degree of matching from the database based on the substances included in the chemical equation,
    extracts a chemical formula error by comparing a chemical formula of the reactant and a chemical formula of the product for the chemical equation with a chemical formula of the reactant and a chemical formula of the product for the reference equation, respectively, and
    after correcting the chemical formula error, extracts a coefficient error by comparing a coefficient of the reactant and a coefficient of the product for the chemical equation with a coefficient of the reactant and a coefficient of the product for the reference reaction formula, respectively.
  8. The chemistry education device according to claim 5,
    wherein the AR implementation unit
    detects a chemical term written on the targeted object displayed on the output unit (the chemical term includes a chemical name and chemical formula),
    displays a bounding box around the chemical term on the output unit when the chemical term matches the reference term,
    displays at least one option window selected from a group consisting of a 'reading method' option window, a 'view molecular formula' option window, a 'view structural formula' option window and a 'view additional information' option window for the chemical term on the output unit when a selection event occurs with respect to the bounding box, and
    displays education information corresponding to the selected option window on the output unit when a selection event occurs for any one of the option windows.
PCT/KR2021/009806 2021-07-28 2021-07-28 Chemistry education device using artificial intelligence and augmented reality and method of providing chemistry education service using the same WO2023008607A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR10-2021-0098886 2021-07-28
KR1020210098886A KR20230017935A (en) 2021-07-28 2021-07-28 Chemistry education device using artificial intelligence and augmented reality and method of providing chemistry education service using the same

Publications (1)

Publication Number Publication Date
WO2023008607A1 true WO2023008607A1 (en) 2023-02-02

Family

ID=85087391

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2021/009806 WO2023008607A1 (en) 2021-07-28 2021-07-28 Chemistry education device using artificial intelligence and augmented reality and method of providing chemistry education service using the same

Country Status (2)

Country Link
KR (1) KR20230017935A (en)
WO (1) WO2023008607A1 (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150112604A1 (en) * 2013-03-13 2015-04-23 Cambridgesoft Corporation Visually augmenting a graphical rendering of a chemical structure representation or biological sequence representation with multi-dimensional information
KR20190132169A (en) * 2018-05-18 2019-11-27 삼성전자주식회사 Apparatus and method for generating molecular structure
US20200184223A1 (en) * 2017-10-06 2020-06-11 Steve Rad Augmented and virtual reality system and kit
KR20200104814A (en) * 2019-02-27 2020-09-04 한지훈 Vr/ar based education platform providing system using method of loci and thereof method
US20210217501A1 (en) * 2020-01-10 2021-07-15 Fujitsu Limited Learning device and learning method

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150112604A1 (en) * 2013-03-13 2015-04-23 Cambridgesoft Corporation Visually augmenting a graphical rendering of a chemical structure representation or biological sequence representation with multi-dimensional information
US20200184223A1 (en) * 2017-10-06 2020-06-11 Steve Rad Augmented and virtual reality system and kit
KR20190132169A (en) * 2018-05-18 2019-11-27 삼성전자주식회사 Apparatus and method for generating molecular structure
KR20200104814A (en) * 2019-02-27 2020-09-04 한지훈 Vr/ar based education platform providing system using method of loci and thereof method
US20210217501A1 (en) * 2020-01-10 2021-07-15 Fujitsu Limited Learning device and learning method

Also Published As

Publication number Publication date
KR20230017935A (en) 2023-02-07

Similar Documents

Publication Publication Date Title
WO2017039086A1 (en) Deep learning modularization system on basis of web plug-in and image recognition method using same
WO2012053867A1 (en) Method and apparatus for recognizing an emotion of an individual based on facial action units
WO2018169115A1 (en) Method and system for supporting learning, and non-transitory computer-readable recording medium
US9075835B2 (en) Learning support device, learning support method and storage medium in which learning support program is stored
CN109753968A (en) Generation method, device, equipment and the medium of character recognition model
WO2020111505A1 (en) Method and system for generating object gt information for image machine learning
CN107016022A (en) Teaching difficulty big data analysis system
WO2018143486A1 (en) Method for providing content using modularizing system for deep learning analysis
CN103680219A (en) Kanji stroke order learning device, and kanji stroke order learning support method
US11138896B2 (en) Information display apparatus, information display method, and computer-readable recording medium
WO2020262800A1 (en) System and method for automating natural language understanding (nlu) in skill development
WO2012060645A2 (en) Method and device for inputting an equation&lt;0}
WO2014003520A1 (en) Led display for outdoor advertising, and interaction method
CN110059636B (en) Method and system for checking and correcting learning homework of students
WO2023008607A1 (en) Chemistry education device using artificial intelligence and augmented reality and method of providing chemistry education service using the same
WO2024005413A1 (en) Artificial intelligence-based method and device for extracting information from electronic document
WO2017126770A1 (en) Method for adjusting display area of electronic book contents
WO2022270840A1 (en) Deep learning-based word recommendation system for predicting and improving foreign language learner&#39;s vocabulary ability
WO2022177372A1 (en) System for providing tutoring service by using artificial intelligence and method therefor
CN112201116B (en) Logic board identification method and device and terminal equipment
WO2024029946A1 (en) Method and device for providing customized mathematical concept through problem-solving image analysis
WO2017090856A1 (en) Method for providing wrong answer feedback note using smart pen and mobile device for processing same
WO2014027782A1 (en) System for providing captcha based on drawing mode image, and method for providing captcha
WO2023095991A1 (en) System for automatically extracting question area and type within content for learning included in electronic document and method therefor
CN112201118B (en) Logic board identification method and device and terminal equipment

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21951977

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE