WO2023075434A1 - Machine learning-based digital reference book provision system using bounding boxes - Google Patents

Machine learning-based digital reference book provision system using bounding boxes Download PDF

Info

Publication number
WO2023075434A1
WO2023075434A1 PCT/KR2022/016531 KR2022016531W WO2023075434A1 WO 2023075434 A1 WO2023075434 A1 WO 2023075434A1 KR 2022016531 W KR2022016531 W KR 2022016531W WO 2023075434 A1 WO2023075434 A1 WO 2023075434A1
Authority
WO
WIPO (PCT)
Prior art keywords
area
reference book
answer
digital reference
detection unit
Prior art date
Application number
PCT/KR2022/016531
Other languages
French (fr)
Korean (ko)
Inventor
최현욱
Original Assignee
주식회사 테스트뱅크
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 주식회사 테스트뱅크 filed Critical 주식회사 테스트뱅크
Publication of WO2023075434A1 publication Critical patent/WO2023075434A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10—Services
    • G06Q50/20—Education
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/40—Document-oriented image-based pattern recognition
    • G06V30/41—Analysis of document content
    • G06V30/413—Classification of content, e.g. text, photographs or tables
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/40—Document-oriented image-based pattern recognition
    • G06V30/41—Analysis of document content
    • G06V30/414—Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Definitions

  • the present invention relates to a system and method for providing a digital reference using a machine learning-based bounding box, and more particularly, to a system and method for generating a digital reference enabling interaction with a user.
  • OCR technology is used to recognize characters in images and extract them as text strings.
  • a problem occurs in which the text string is extracted in an inaccurate form.
  • a license agreement for using the font is required.
  • there is a problem that the accuracy of the conversion result from image to text string is not 100% guaranteed.
  • syntax containing mathematical formulas (1) additional development of OCR technology that recognizes formulas, (2) technology development such as latex that converts the format of recognized formulas is required.
  • An object of the present invention to solve the above problems is to apply semantic grouping to elements included in a digital reference book using machine learning, thereby enabling smooth interaction with a digital reference book.
  • To provide a reference book providing system and method.
  • the digital reference book providing system performs Semantic Gourping using machine learning. Although the number and order of specific procedures may vary, a digital reference book that enables smooth interaction with users can be created through the process below.
  • the digital reference book providing system does not use resources for restoring styling, discovers only the areas that require interaction, and implements multi-layer interaction on top of it.
  • the following procedure is performed do. (1) Detect problem areas within the page, (2) Discover areas that require answer records in each problem area
  • the present invention provides a system and method for providing a digital reference book that enables smooth interaction with users by applying semantic grouping to elements included in a digital reference book using machine learning.
  • FIG. 1 is a diagram showing the configuration of a digital reference book providing system according to an embodiment of the present invention.
  • FIG. 2 is a diagram showing a sequence of a digital reference book providing method according to an embodiment of the present invention.
  • 3 to 18 show specific examples for explaining the present invention.
  • first, second, A, B, (a), and (b) may be used to describe components of an embodiment of the present invention. These terms are only used to distinguish the component from other components, and the nature, order, or order of the corresponding component is not limited by the term.
  • the x coordinate means a horizontal coordinate value within a page
  • the y coordinate means a vertical coordinate value within a page
  • FIG. 1 is a diagram showing the configuration of a digital reference book providing system (hereinafter referred to as “the present digital reference book providing system”) using a machine learning-based bounding box according to an embodiment of the present invention.
  • this digital reference book providing system 1000 includes an external server 10 and/or a digital conversion server 100 .
  • the external server 10 may indicate a server that stores PDF files requiring conversion, and the digital conversion server 100 may receive PDF files requiring conversion from the external server 10 .
  • the digital conversion server 100 separates, extracts, stores and manages the elements included in the PDF (Portable Document Format) file, semantically groups the extracted elements to create a digital reference book that can be interacted with, This can be provided to the user terminal.
  • PDF Portable Document Format
  • Each component of the digital conversion server 100 may be driven by a known library.
  • fitz a PDF library used for converting PDFs into images and extracting text page by page
  • opencv Open Source Computer Vision
  • a computer vision library used for image transformation and border extraction
  • Libraries such as MathPix used as object detection pytorch and tensorflow, MathML and LaTex grammar OCR APIs can be used, and they can be run on Python, but are not limited thereto.
  • libraries such as numpy, sklearn, re, and glob can be used and run on Python, and text can be detected on PDF using the Font-forge library, matched with a suitable font, and extracted in a renderable form.
  • Font format may be calculated in the form of TTF, TTF2, OTF, woff, etc., but is not limited thereto.
  • the digital conversion server 100 may include a file generator 110, a region detector 120, and/or an element extractor 130.
  • the file generator 110 extracts a plurality of pages from the PDF file of the digital reference book and creates a page file. That is, the file generator 110 creates a page file by dividing a plurality of pages of the PDF file into single pages.
  • the page file may be a bitmap format file of BMP, GIF, JPEG, and PNG using pixels, respectively.
  • the PDF file from which the file creation unit 110 extracts pages may be a PDF file that has been created and stored in the database unit 150 .
  • the PDF file from which the file generator 110 extracts pages may be a PDF file generated by scanning a reference book by a user terminal such as a smartphone, scanner, or multifunction device.
  • the PDF file from which the file generator 110 extracts pages may be received from another external server 10 connected through a communication network.
  • a PDF file created by scanning a reference book contains multiple pages.
  • the file generator 110 extracts a plurality of pages into a single page and creates a page file.
  • the PDF file from which the file creation unit 110 extracts pages may be a structured PDF file or a scanned PDF file depending on how it is created.
  • Structured PDF is a PDF file created electronically using computer software
  • Scanned PDF is a PDF file created in the form of an image using a specific book in the real world.
  • the big difference between Structured PDF and Scanned PDF is that in Structured PDF, referring to Figure 3 below, each element such as text, image or table included in each page can be identified and extracted according to selection. The blue highlights in FIG. 3 show the selected elements.
  • Scanned PDF referring to FIG. 4 , it is impossible to select and extract each element.
  • the file generator 110 restores the structure through the OCR process. do.
  • the OCR process may be performed by the optical character recognition unit 140 to be described later.
  • the file creation unit 110 extracts the question area, answer area, and commentary area by the area detection unit 120 to be described later without going through the OCR process, and extracts the extracted area. Images of may be stored in the database unit 150 and managed.
  • the digital reference book according to the present invention includes a question paper area and a commentary area.
  • the problem area is usually composed of two columns regardless of subject, grade, book type, etc.
  • the number of questions observed within one page is distributed within a minimum of 4 to 8 questions.
  • Most of the questions are in the form of multiple choice, and depending on the case, there may be short answer and subjective questions, but they are not frequent.
  • the most important element to find in the problem sheet area is the problem area.
  • the problem area consists of a problem number, problem title, and options, and has a similar form regardless of the type of subject.
  • the commentary area does not have a shared geometric feature regardless of subject or book type like the problem area of the problem area, but all pages and elements within the commentary have the same layout design and element shape. There is no overall trend found in the commentary area, each book has many individual characteristics, there is no commonly found pattern like the two column structure, there is no pattern found in the form of multiple choice, and each book (especially book series) has its own unique font, styling, Layout is used frequently. In the case of commentary areas with these characteristics, it is very likely that Machine Learning for Semantic Binding for the purpose of implementing Interaction in questions will not work well.
  • the area detection unit 120 in the page file generated by the file creation unit 110, the entire image of the page file, text included in the page file, HTML, CSS, Font, Background vector Extract elements such as images. After that, the area detection unit 120 converts each page file into an HTML page.
  • Structured PDF is converted to HTML page, each element can be detected (clickable) after conversion, and for Scanned PDF, the entire page is extracted as a single image while each element is undetectable. Therefore, as described above, the scanned PDF needs to be converted into a form in which characters/numbers in the image can be detected through OCR processing.
  • the region detection unit 120 converts each page file into an HTML page
  • an id value is assigned to an identified element within each page.
  • the selectable element includes each element such as text, image, or table, and may include, for example, a problem area, a question area and an answer area within the problem area, and a number option within the answer area.
  • the area detection unit 120 converts the structured PDF file produced during the book production process at the publishing house into HTML, which is a form usable for the service.
  • the PDF file used at this time has the characteristics that text can be copied, font information is accurately expressed, and object files can be selected individually.
  • the area detection unit 120 After extracting all the elements included in the PDF file, the area detection unit 120 combines the images into one large image to make a background image, and the text is placed at the same position as in the original PDF by creating a new font. do. Specifically, it is performed in the following procedure (see FIG. 5).
  • the region detector 120 may use an image-based PDF file calculated through a scan procedure when a typesetting PDF file is not secured.
  • Structured PDF files can select objects, highlighting by designating a text block, selecting an image or table, and declaring a selector tag by accessing an object.
  • a scanned PDF file the entire page is implemented as an image, so it is impossible to highlight by designating a text block, and it is impossible to select an image or table, and a tag must be declared using Machine Learning Selector Detection technology.
  • the area detection unit 120 When converting scanned PDF into HTML, the area detection unit 120 discovers a text area, designates a text block, and implements a text block area so that a highlight can be designated. Specifically, the following procedure is performed. (1) Determine whether a specific character is text based on the OCR model, (2) Text box detection to designate and select the text box and implement interaction, (3) Text box overlay at the same position for each text character
  • the area detection unit 120 sets a location-based tag point so that an event can be implemented in a selector within a question, and is specifically performed in the following procedure. (1) Discrimination of questions within a page, (2) Discrimination of choice area within a question, (3) Determination of the position of a circle number within a choice, (4) Assignment of a tag capable of setting a javascript event for each circle number area
  • HTML that is not semantically implemented because a meaningful tag is not assigned cannot be used for interaction.
  • tags based on location values have the same screen configuration that users see on the web page, but are not semantically grouped, making it difficult to connect in semantic units. Therefore, scoring and answer connection, etc. It is difficult to implement the event of
  • tags based on semantic values have the same screen configuration that users see on the webpage compared to cases based on location values, but are semantically grouped and connected in semantic units. It is easy, so it is easy to implement events such as scoring and answer connection.
  • the class id value (Tag) given to the selectable element is generated based on location information within each page in the process of converting PDF to HTML page, and the id value has no meaning. Therefore, in order to group these id values into semantic units, such as pages, problems, and options, the following machine learning process is performed on the image of the page file. In this case, grouping may mean, for example, grouping problems, options, and multiple-choice questions attached to one passage.
  • the region detection unit 120 groups classes for each problem region and assigns an id. Specifically, the corresponding work is performed in the following procedure. (1) Detection of problem areas within the page, (2) Detection of areas requiring interaction application within the problem area, (3) Detection of elements present in each area
  • the area detection unit 120 first detects a problem area in each page file, assigns an ID, separates the problem area into a question area and an answer area requiring interaction, and detects it.
  • An ID (Sub-id) is also assigned to the detected question area and/or answer area.
  • the area requiring interaction within the problem area may differ for each subject in the reference book. For example, in the case of Korean or English, there is a bundle of texts, and these bundles of texts are combined with the text for the convenience of learners. It is desirable to be able to view them on one screen at the same time.
  • the area detection unit 120 may detect an answer area within the problem area and classify an area other than the answer area as a question area.
  • the area detection unit 120 detects a problem area (A in FIG. 9 ) in the page file extracted by the file creation unit 110 .
  • the area detection unit 120 may extract object information in a JSON (JavaScript Object Notation) format using the fitz library.
  • the fitz library can extract text from a page as shown in FIG. 8 .
  • the area detection unit 120 detects text from the page through the fitz library.
  • the area detection unit 120 may detect each problem area as shown in FIG. 9 based on a boundary between text spaces or vertical lengths longer than other text spacings or vertical lengths.
  • the area detection unit 120 may generate a bounding box surrounding each problem area using machine learning.
  • the region detection unit 120 may add a unified background color to the problem region within the bounding box generated by machine learning.
  • the region detection unit 120 determines whether other problem regions exist by expanding adjacent pixels in the remaining region except for the bounding box through opencv.
  • a bounding box may be generated based on a machine learning model based on Fast RCNN, which will be described later.
  • the area detection unit 120 detects and stores information about the position and size of the created bounding box.
  • the region detection unit 120 detects a problem region in each page file
  • the x-coordinate (x1) of the leftmost point and the x-coordinate (x1) of the rightmost point of the bounding box are higher than the x-coordinate (x half) of the midpoint in the horizontal direction of the page.
  • the value of x2) is larger, it can be determined that the layout of the page is two columns.
  • the midpoint coordinates in the horizontal direction of the page may be coordinate values corresponding to the center point of the entire page or the center of a line segment connecting the left and right sides of the page.
  • the area detection unit 120 separates the detected problem area into a question area and an answer area (B, FIG. 10) as shown in FIG. 10 .
  • the area detector 120 first selects and extracts the answer area, and selects the remaining problem areas excluding the selected answer area as the question area.
  • options may be arranged horizontally or vertically. If the answer area is aligned vertically, the choice of answer area is as follows.
  • the area detection unit 120 may detect a row including a circle number.
  • the area detection unit 120 has a problem. Select an area containing a sentence containing a circle number detected in the area as an answer area.
  • the area detection unit 120 may detect lines included so that the original number continuously increases when detecting the answer areas in the page file.
  • the area detection unit 120 selects an area containing a sentence including the circle numbers detected in the problem area as an answer area.
  • the element extraction unit 130 extracts elements (C to F, FIG. 11) from the question area and the answer area separated by the area detection unit 120 and stores the extracted elements in the database unit 150. Save.
  • the elements include a problem index (C), text (D), image (E), text box (not shown), and table (not shown).
  • the element extraction unit 130 selects and extracts the index located at the leftmost side of the question area.
  • the element extraction unit 130 may select and extract the image E from the problem area.
  • the problem index extracted by the element extractor 130 can be used for scoring automation by being connected to an answer area of the commentary area, which will be described later.
  • the rest may be text (D).
  • each text may be selected and extracted based on a region having a different bounding box according to each position of the text.
  • the element extraction unit 130 may select and extract a series of options (F) from text in the selected question area.
  • a series of options (F) are 'a. you. c.', '(a) (b) (c)', 'a. me. all.'
  • the sentences or the area can be selected and extracted as a series of options (F). That is, the element extractor 130 may select and extract sentences within a region in which consonants are sequentially detected in sequence or consonants and vowels are sequentially detected in sequence in a combined state as a series of options (F). there is.
  • the element extraction unit 130 stores the selected and extracted elements in the database unit 150.
  • Each element selected and extracted by the element extraction unit 130 has a bounding box, and the area within the bounding box is stored in the database unit 150 in the form of an image.
  • the element extraction unit 130 determines that a sentence including a circle number is an option in an area selected as an answer area because the above-described area detection unit 120 has a sentence including a circle number in the problem area.
  • the element extraction unit 130 extracts the aforementioned elements from the answer area including the options, and stores the extracted elements in the database unit 150.
  • Bundled fingerprints refer to fingerprints in which a plurality of questions are included in one fingerprint.
  • the area detection unit 120 may detect a bundle fingerprint number (G, FIG. 12 ) at the top of the problem area. When a bundled fingerprint number is detected, the area detection unit 120 performs a document merging operation to include a problem area corresponding to the bundled fingerprint number (refer to the description area to be described later), and the area including the bundled fingerprint number. Create a bounding box in At this time, the region detection unit 120 may create a bounding box to include problem regions according to bundled fingerprint numbers. For example, as shown in Figure 12, when '35 to 37' is the bundled fingerprint number, the area detection unit 120 sets the bounding box so that all problem areas having problem indexes 35 to 37 (C) are included together with the bundled fingerprint.
  • the element extraction unit 130 extracts elements from the question area and the answer area and stores them in the database unit 150, it also extracts elements from the aforementioned bundled fingerprints and stores them in the database unit 150.
  • the question area and Elements extracted from the answer area and elements extracted from the bundled fingerprints may be stored in the database unit 150 so as to be connected to each other.
  • the detection of bundled fingerprints is not limited only to cases where the subject of the reference book is Korean or English, and whenever the region detection unit 120 detects a number array in the form of 'numbers to numbers' such as the aforementioned bundled fingerprint numbers, bundled fingerprint elements can be extracted and stored.
  • the element extraction unit 130 extracts the extracted elements, that is, the problem index (C), text (D), image (E), text box (not shown), table (not shown), a series of options (F), and page information. etc. are grouped into one problem and stored in the database unit 150.
  • the grouped elements may be provided to the user terminal by the problem providing unit 160 to be described later.
  • problems corresponding to one bundled fingerprint are simultaneously provided on one screen. Therefore, the element extraction unit 130 determines whether there is a bundled fingerprint in the page file, how many questions are the questions corresponding to the bundled fingerprint, that is, which problem index it has, and if the bundled fingerprint is turned over, the page information at that time. are extracted and stored in the database unit 150.
  • the element extraction unit 130 may assign a specific class id value to the elements when grouping the extracted elements.
  • the optical character recognition unit 140 may extract text from elements extracted and stored in the database unit 150 through Optical Character Recognition (OCR) processing or the Fitz library, and store them in the database unit 150 again. .
  • OCR Optical Character Recognition
  • the optical character recognition unit 140 may perform optical character recognition processing on an element using a known optical character recognition algorithm.
  • the problem providing unit 160 provides the elements grouped by the element extraction unit 130 to the screen of the user terminal.
  • provided elements may be provided as an HTML layer in a state of being arranged at positions suitable for respective position values.
  • the problem providing unit 160 may provide a multi-layer structure to the user terminal. That is, a writing layer for writing may be further provided on the HTML layer in which the grouped problems are provided to the user terminal. The user terminal may proceed with writing and problem solving in the writing layer provided on the HTML layer.
  • the question providing unit 160 may recognize an answer check through a known library capable of recognizing handwriting written in the handwriting layer. At this time, if the answer is checked in the handwriting method in the handwriting layer at the same position as the position value where a certain number included in the answer area of the HTML layer is located, the problem providing unit 160 is the problem index of the answer area to be described later. Based on the answer and the correct answer, it is possible to automate the scoring of the problem by comparing the answer check of the user terminal and the correct answer.
  • the area detection unit 120 decomposes the commentary area for each item through the following procedure. (1) Determine the center line of the page, (2) Create a page by converting the two-column structure into a one-column structure based on the determined center line, (3) Detect the separator within the page, (4) Detect the area of the separator, (5) ) extract the area to be mapped, (6) map the area to the problem
  • the area detection unit 120 extracts the entire image file from the page file.
  • the area detection unit 120 extracts a commentary area from the extracted image file.
  • the extraction of the commentary area may be performed by the same process as the detection of the problem area described above.
  • the region detection unit 120 extracts the commentary region, in the entire image file extracted from each page file, first in the n-column configuration, referring to FIG. 14, the dividing line (midline, H, figure 14) can be detected.
  • the area detection unit 120 may also detect an outer line (I, FIG. 14 ), which is the innermost dividing line constituting content within each page.
  • the area detector 120 may cut images containing internal contents based on the center line and the outer line, and merge them into one image by cutting them based on the center line. An example of merging into one image is shown in FIG. 15 . Such merged images may be obtained by merging images corresponding to all pages of the commentary area into a single image in one column.
  • the element extraction unit 130 divides the images merged into one column into chapter areas.
  • a first delimiter for dividing the chapters is selected, and referring to FIG. 17, feature points of the selected first delimiter are extracted, , Referring to FIG. 18, each chapter may be distinguished by detecting a second delimiter based on the extracted first delimiter.
  • the element extraction unit 130 separates the merged commentary area images by chapter and section through the above-mentioned feature points (detection of the area of the separator), and extracts each page information and problem index for each chapter (extracts the area to be mapped ).
  • the element extraction unit 130 extracts the correct answers and explanations for each problem index and stores them in the database unit 150.
  • the element extraction unit 130 connects the problem index in the problem area in the aforementioned problem area with the problem index in the explanation area (mapping the area with the problem). Therefore, as described above, the problem providing unit 160 compares the location information of the answer check of the user terminal with the location information of the answer area of the question to automatically score the problem.
  • the area detection unit 120 detects a problem area within the page, detects an area requiring interaction within the problem area (eg, an answer area), and uses a pre-learned machine learning model to detect elements within the problem area.
  • the machine learning model used at this time was based on a Fast RCNN-based machine learning model, and through this, it was possible to compensate for the slow speed with a structure that combined classification after CNN and Bounding Box regression with RPN (Region Proposal Network).
  • the digital conversion server 100 provides information on a reference book for which the system according to the present invention has a copyright or is licensed for copyright, and a reference book that does not have a copyright and is not licensed for copyright. It is possible to access the external server 10 including.
  • the digital reference book providing system 1000 may further include a copyright determination unit (not shown).
  • the copyright judging unit may determine whether or not the present system has the copyright of the reference file (PDF file) uploaded to this system by the user terminal or the reference file to be provided to the user terminal.
  • PDF file the copyright of the reference file
  • the database unit 150 stores all stored reference PDF files by dividing them into PDF files with copyright and PDF files without copyright in this system.
  • the copyright determination unit determines whether or not the system has the copyright for the reference file uploaded from the user terminal or the reference file to be provided to the user terminal, based on the copyright existence information for each reference file stored in the database unit 150. can judge
  • the digital reference book is provided to the user terminal through the process of the above-described area detection unit 120 and element extraction unit 130.
  • the system can generate only an additional layer irrelevant to copyright, such as automatic scoring, and provide it to the user terminal. That is, the present system can provide only the HTML layer and/or the handwriting layer to the user terminal without extracting elements from the problem area and the commentary area. If necessary, only the writing layer may be provided without region detection.
  • the problem providing unit 160 provides HTML for providing problems of the reference book to the user terminal. layer and/or handwriting layer. After the user checks the answer through the handwriting layer of the user terminal, that is, the handwriting layer at the same position as the position value of the original number in the HTML layer answer area as described above and automatically scores, if the correct answer to the question is inconsistent, problem solving The study 160 displays the commentary area of the commentary corresponding to the question on the screen of the user terminal.
  • the present digital reference book providing system 1000 may further include a purchase authentication unit 170 that verifies that the user has legally purchased a book corresponding to the digital reference book.
  • the purchase authentication unit 170 transmits a specific random number generated together with a specific page to the user terminal.
  • the user terminal writes the received specific random number on the same page as the specific page received from the purchase authentication unit 170 and transmits an image of the page with the random number to the purchase authentication unit 170 .
  • the purchase authentication unit 170 mutually compares elements included in the page on which the user terminal writes a specific random number with elements included in the same page of the reference book stored in the database unit 150, and when the same element is detected, It can be judged that the user has legally purchased the book. Therefore, the elements extracted and grouped as described above are combined into an HTML layer and provided to the user terminal.
  • This digital reference book providing system 1000 concludes a copyright licensing agreement with reference book publishers and/or their distributors for distributing specific reference books, receives books for distribution, or is provided by reference book publishers and/or their distributors.
  • a PDF file of the specific reference can be received from an external server 10 .
  • the digital conversion server 100 builds an application that can operate a website and/or application-based sales channel and a digitally converted reference book, and provides the built application to a user terminal.
  • a provision unit (not shown) may be further included.
  • the application built by the application providing unit is installed in the user terminal.
  • the user terminal purchases a reference book such as a book by generating an ID in a sales channel established by the application providing unit.
  • the user terminal can use the digitally converted reference book by authenticating the ID on the application.
  • the user terminal acquires the right to read and use the digitally converted reference book with the same contents as the purchased reference book.
  • the application built by the application providing unit according to the present invention may have the following safety devices implemented.
  • a procedure for verifying whether the purchaser of the reference book and the holder of the logged-in user terminal are the same may be implemented through the login and channel authentication procedures.
  • a procedure for requesting an irrevocable expression of intent to confirm purchase from the user terminal that purchased the reference book and providing access to the digital reference book to the user terminal only when there is an expression of intent to confirm purchase may be implemented. This is to prevent users who have not purchased reference books from using digital reference books, which are an additional function of purchasing reference books.
  • an application provides a digital reference to a user terminal, it can be provided as fragmented pages in a CDN/security-enhanced protocol, not in the form of a PDF file.
  • FIG. 2 is a diagram showing a sequence of a digital reference book providing method according to an embodiment of the present invention.
  • the method for providing a digital reference book includes generating a page file by extracting a plurality of pages from a PDF file (S10), and detecting a problem area in the page file (S20). , Separating the problem area into a question area and an answer area (S30), extracting and storing elements from the question area and answer area (S40), and / or OCR processing the extracted elements and storing them in the database unit Steps (not shown) may be included.
  • the file generation unit, the area detection unit, the element extraction unit, the optical character recognition unit, the database unit, the problem providing unit, and/or the purchase authentication unit may be processors that execute successive processes stored in memory. Or, it can operate as software modules driven and controlled by a processor. Further, a processor may be a hardware device.
  • the digital reference book providing method may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium.
  • the computer readable medium may include program instructions, data files, data structures, etc. alone or in combination.
  • Program instructions recorded on the medium may be those specially designed and configured for the present invention or those known and usable to those skilled in computer software.
  • Examples of computer readable media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROMs, RAMs,
  • a hardware device specially configured to store and execute program instructions, such as flash memory, may be included.
  • Examples of program instructions include not only machine code generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like.
  • the hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Business, Economics & Management (AREA)
  • Tourism & Hospitality (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Economics (AREA)
  • Data Mining & Analysis (AREA)
  • Educational Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Resources & Organizations (AREA)
  • Marketing (AREA)
  • Primary Health Care (AREA)
  • Strategic Management (AREA)
  • General Business, Economics & Management (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Educational Administration (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Character Input (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The present invention relates to a machine learning-based digital reference book provision system using bounding boxes. The machine learning-based digital reference book provision system using bounding boxes according to the present invention comprises a digital conversion server. The digital conversion server comprises: an area detection unit that detects problem areas in a portable document format (PDF) file of a digital reference book, and divides the problem area into a question area and an answer area by detecting the answer area in the problem area; and an element extraction unit that extracts elements from the question area and the answer area, and saves the extracted elements to a database unit, wherein the area detection unit detects, as the answer area, an area in the question area in which a circled number is detected.

Description

머신러닝 기반 바운딩 박스를 이용한 디지털 참고서 제공 시스템 및 그 방법Digital reference book providing system and method using machine learning based bounding box
본 발명은 머신러닝 기반 바운딩 박스를 이용한 디지털 참고서 제공 시스템 및 그 방법에 관한 것으로서, 보다 상세하게는 이용자와의 상호작용(Interaction)이 가능하도록 하는 디지털 참고서를 생성하는 시스템 및 그 방법에 관한 것이다.The present invention relates to a system and method for providing a digital reference using a machine learning-based bounding box, and more particularly, to a system and method for generating a digital reference enabling interaction with a user.
종래 서책의 생산을 위해서는 “기획, 원고, 삽화, 조판”의 형태로 절차가 진행되는데 각 절차가 완료된 후 출력 직전에 산출되는 것이 서책에 대한 PDF 파일이다. 이러한 PDF 파일은 출력을 목적으로 고안된 파일로 이를 활용하여 디지털 콘텐츠에서 필요한 저작권 보호, 풀이 데이터 기록, 다양한 Interaction을 구현하는 것은 어렵다.For the production of conventional books, procedures are carried out in the form of “planning, manuscript, illustration, typesetting”, and after each procedure is completed, the PDF file for the book is calculated immediately before output. These PDF files are designed for output, and it is difficult to implement copyright protection, solution data recording, and various interactions required for digital content by utilizing them.
이용자가 원하는 다양한 기능을 구현하기 위해서는 다음의 절차가 필요하다. (1) 각 서책 PDF에 포함된 데이터를 체계적으로 추출하여 저장, (2) 저장된 데이터를 원래의 Font, Styling, Layout 형태로 조립하여 복원, (3) 이러한 저장 및 추출된 데이터가 적절한 형태로 작동하도록 서비스 기획, (4) 서비스 기획에 맞는 어플리케이션 개발In order to implement various functions desired by users, the following procedures are required. (1) systematically extract and save the data included in each book PDF, (2) restore the saved data by assembling it in the form of original font, styling, and layout, (3) operate the saved and extracted data in an appropriate form service planning, (4) application development suitable for service planning
이러한 복잡한 절차에서 발생하는 시간과 비용으로 인하여, 각 서책은 출판을 위해서 PDF 파일은 생성되나, Application 형태로 변환하여 유통되지 못하며 서책중심의 유통구조가 유지되고 있다. 이러한 복잡한 종래 절차가 디지털 참고서와 같은 디지털 콘텐츠의 공급 부족을 야기하고 있다.Due to the time and cost incurred in these complicated procedures, PDF files are created for publication of each book, but it is not converted into an application form and distributed, and a book-centered distribution structure is maintained. Such complex conventional procedures are causing a lack of supply of digital contents such as digital reference books.
서책 PDF로부터 데이터를 추출하는 단계에서는, OCR 기술을 이용해 이미지 내 Character를 인식하고 이를 Text string으로 추출한다. 이때, OCR 기술에 학습되지 않은 Font를 사용할 경우 부정확한 형태로 Text string이 추출되는 문제가 발생한다. 나아가, OCR 기술을 이용한 데이터 추출 시 특정 Font를 사용할 경우 Font 사용을 위한 license 계약이 필요하다는 문제가 발생한다. 또한, Image로부터 Text string으로의 변환 결과에 대한 정확성이 100% 담보되지 않는다는 문제가 발생한다. 나아가, 수학공식이 포함된 구문의 경우 해당 구문에서 (1) 수식을 인식하는 OCR 기술의 추가 개발, (2) 인식된 수식 format을 변환하는 Latex 등과 같은 기술 개발이 필요하다.In the step of extracting data from book PDF, OCR technology is used to recognize characters in images and extract them as text strings. At this time, when using a font that has not been learned in the OCR technology, a problem occurs in which the text string is extracted in an inaccurate form. Furthermore, when using a specific font when extracting data using OCR technology, a license agreement for using the font is required. In addition, there is a problem that the accuracy of the conversion result from image to text string is not 100% guaranteed. Furthermore, in the case of syntax containing mathematical formulas, (1) additional development of OCR technology that recognizes formulas, (2) technology development such as latex that converts the format of recognized formulas is required.
추출된 데이터를 저장하는 단계에서, 학습 문항은 단순 Text로 이루어진 형태는 드물고 지문 내 이미지, 문항 내 이미지, 선택 원숫자 등으로 구현되어 있어 이를 체계적으로 분류하여 저장하는 것이 요구된다. 이를 단순 OCR로 처리하여 Text string을 추출할 경우, 문항 내 모든 Text들이 문제번호, 질문영역, 원숫자, 선택지, 삽화 등 세부 분류에 포함여부와 무관하게 1개의 Text로 병합되어 추출돼 버린다. 즉, 맥락에 맞게 분류되지 않고 Text string으로 추출된 데이터는 다시 그 맥락에 맞도록 분리 작업이 필요하나, 이러한 작업이 매우 복잡하고 시간과 자원의 소요가 많다. 각 문항 내에서 추출된 데이터가 맥락에 맞도록 조립되고 복원되기 위해서는 문항 내에서 문항 Description, 원숫자, 삽화 등과 같이 소분류에 대한 Tagging을 진행하고 OCR로 분해하여 Grouping을 용이하게 해야하나 단순 OCR은 이를 지원하지 않는다.In the step of storing the extracted data, learning questions are rarely composed of simple text and are implemented with images in fingerprints, images in questions, and selected numbers, so it is required to classify and store them systematically. When text strings are extracted by processing this with simple OCR, all texts in the question are merged into one text and extracted regardless of whether they are included in detailed classifications such as question numbers, question areas, original numbers, options, and illustrations. That is, the data extracted as text strings without being classified according to the context needs to be separated to fit the context again, but this task is very complicated and requires a lot of time and resources. In order for the data extracted from each item to be assembled and restored in accordance with the context, tagging of subclasses such as item description, original number, and illustration within the item must be performed, and grouping must be facilitated by decomposition by OCR. do not support
저장된 데이터를 복원하는 단계에서, 체계적으로 저장된 데이터를 원데이터에서 사용된 Font를 활용하여 복원하고 Styling(Font size, 자간, Layout 등)을 고려하여 재조립 해야한다. 이러한 조립 절차 시 기존에 사용된 Font를 활용하여야 하며, 이때 Font license를 확보해야 하는 이슈가 발생할 수 있다. 한편, OCR을 이용하여 추출된 Text 데이터를 기반으로 원래의 Styling을 조립하는 것은 매우 어려운 일이며 추출 시 OCR을 이용한 Text extraction뿐만 아니라 각 Styling에 대한 체계적인 추출이 요구되나 이는 매우 어려운 일이다.In the stage of restoring the saved data, it is necessary to systematically restore the saved data using the font used in the original data and reassemble it by considering the styling (font size, spacing, layout, etc.). During this assembly process, previously used fonts must be used, and at this time, an issue of securing a font license may occur. On the other hand, it is very difficult to assemble the original styling based on the text data extracted using OCR, and systematic extraction of each styling as well as text extraction using OCR is required during extraction, but this is very difficult.
상술한 문제점을 해결하기 위한 본 발명의 목적은, 머신러닝을 활용하여 디지털 참고서에 포함된 요소(Element)들에 의미론적 그룹핑(Semantic Grouping)을 적용함으로써 이용자와의 원활한 상호작용이 가능하도록 하는 디지털 참고서 제공 시스템 및 그 방법을 제공하는 것이다.An object of the present invention to solve the above problems is to apply semantic grouping to elements included in a digital reference book using machine learning, thereby enabling smooth interaction with a digital reference book. To provide a reference book providing system and method.
상기 문제점을 해결하기 위하여, 본 발명에 따른 디지털 참고서 제공 시스템은, 머신러닝(Machine Learning)을 이용한 Semantic Gourping을 수행한다. 구체적인 절차의 수, 순서 등은 다를 수 있으나, 아래의 과정을 통해 이용자와의 원활한 상호작용이 가능한 디지털 참고서를 생성할 수 있다.In order to solve the above problem, the digital reference book providing system according to the present invention performs Semantic Gourping using machine learning. Although the number and order of specific procedures may vary, a digital reference book that enables smooth interaction with users can be created through the process below.
각 문항을 체계적으로 데이터베이스화하기 위해 OCR 처리 이전에 머신러닝을 이용해 Column 분류작업을 수행하는데 다음과 같은 절차로 수행한다. (1) 페이지 내에서 문제영역을 검출, (2) 각 문제영역에서 원숫자 영역을 검출, (3) 각 문제영역에서 문제번호, 질문 Text, 이미지, Sub 질문영역, Sub 영역 및 원숫자를 각 숫자 및 Text 영역으로 분해, (4) 각 문항을 Semantic Binding box 산출Column classification is performed using machine learning prior to OCR processing in order to systematically database each question, which is performed in the following procedure. (1) Detect problem area within the page, (2) Detect circle number area in each problem area, (3) In each problem area, problem number, question text, image, sub question area, sub area and circle number Decomposition into numbers and text areas, (4) Calculation of Semantic Binding box for each question
Styling의 유실이 없는 방법을 구현하기 위하여, 본 발명에 따른 디지털 참고서 제공 시스템은 Styling의 복원에 자원을 쓰지 않고 Interaction이 필요한 영역만 발굴하여 그 위에 Multi-layer Interaction을 구현하는데 다음과 같은 절차로 수행한다. (1) 페이지 내에서 문제영역을 검출, (2) 각 문제영역에서 답변기록이 필요한 영역을 발굴In order to implement a method without loss of styling, the digital reference book providing system according to the present invention does not use resources for restoring styling, discovers only the areas that require interaction, and implements multi-layer interaction on top of it. The following procedure is performed do. (1) Detect problem areas within the page, (2) Discover areas that require answer records in each problem area
본 발명은, 머신러닝을 활용하여 디지털 참고서에 포함된 요소(Element)들에 의미론적 그룹핑(Semantic Grouping)을 적용함으로써 이용자와의 원활한 상호작용이 가능하도록 하는 디지털 참고서 제공 시스템 및 그 방법을 제공할 수 있다.The present invention provides a system and method for providing a digital reference book that enables smooth interaction with users by applying semantic grouping to elements included in a digital reference book using machine learning. can
도 1은 본 발명의 일 실시예에 따른 디지털 참고서 제공 시스템의 구성을 나타낸 도면이다.1 is a diagram showing the configuration of a digital reference book providing system according to an embodiment of the present invention.
[규칙 제91조에 의한 정정 29.12.2022]　
도 2는 본 발명의 일 실시예에 따른 디지털 참고서 제공 방법의 순서를 나타낸 도면이다. 도 3 내지 18은 본 발명을 설명하기 위한 구체적인 예시를 나타낸다.
[Correction under Rule 91 29.12.2022]
2 is a diagram showing a sequence of a digital reference book providing method according to an embodiment of the present invention. 3 to 18 show specific examples for explaining the present invention.
이하, 본 발명의 일부 실시예들을 예시적인 도면을 통해 상세하게 설명한다. 각 도면의 구성 요소들에 참조 부호를 부가함에 있어서, 동일한 구성 요소들에 대해서는 비록 다른 도면상에 표시되더라도 가능한 한 동일한 부호를 가지도록 하고 있음에 유의해야 한다.Hereinafter, some embodiments of the present invention will be described in detail through exemplary drawings. In adding reference numerals to components of each drawing, it should be noted that the same components have the same numerals as much as possible, even if they are displayed on different drawings.
그리고 본 발명의 실시예를 설명함에 있어, 관련된 공지 구성 또는 기능에 대한 구체적인 설명이 본 발명의 실시예에 대한 이해를 방해한다고 판단되는 경우에는 그 상세한 설명은 생략한다.And, in describing the embodiments of the present invention, if it is determined that a detailed description of a related known configuration or function hinders understanding of the embodiments of the present invention, the detailed description will be omitted.
또한, 본 발명의 실시예의 구성 요소를 설명하는 데 있어서, 제1, 제2, A, B, (a), (b) 등의 용어를 사용할 수 있다. 이러한 용어는 그 구성 요소를 다른 구성 요소와 구별하기 위한 것일 뿐, 그 용어에 의해 해당 구성 요소의 본질이나 차례 또는 순서 등이 한정되지 않는다.Also, terms such as first, second, A, B, (a), and (b) may be used to describe components of an embodiment of the present invention. These terms are only used to distinguish the component from other components, and the nature, order, or order of the corresponding component is not limited by the term.
본 명세서에서 x 좌표는 페이지 내 가로방향 좌표 값을 의미하고, y 좌표는 페이지 내 세로방향 좌표 값을 의미한다.In this specification, the x coordinate means a horizontal coordinate value within a page, and the y coordinate means a vertical coordinate value within a page.
도 1을 참조하여, 본 발명의 일 실시예에 따른 머신러닝 기반 바운딩 박스를 이용한 디지털 참고서 제공 시스템의 구성 및 동작에 대하여, 이하 설명한다.Referring to FIG. 1, the configuration and operation of a digital reference book providing system using a machine learning-based bounding box according to an embodiment of the present invention will be described below.
도 1은 본 발명의 일 실시예에 따른 머신러닝 기반 바운딩 박스를 이용한 디지털 참고서 제공 시스템(이하, “본 디지털 참고서 제공 시스템”이라 칭함)의 구성을 나타낸 도면이다.1 is a diagram showing the configuration of a digital reference book providing system (hereinafter referred to as “the present digital reference book providing system”) using a machine learning-based bounding box according to an embodiment of the present invention.
도 1을 참조하면, 본 디지털 참고서 제공 시스템(1000)은 외부서버(10) 및/또는 디지털 변환 서버(100)를 포함한다.Referring to FIG. 1 , this digital reference book providing system 1000 includes an external server 10 and/or a digital conversion server 100 .
외부서버(10)는 변환이 필요한 PDF 파일을 저장하고 있는 서버를 나타낼 수 있고, 디지털 변환 서버(100)는 외부서버(10)로부터 변환이 필요한 PDF 파일을 수신할 수 있다.The external server 10 may indicate a server that stores PDF files requiring conversion, and the digital conversion server 100 may receive PDF files requiring conversion from the external server 10 .
디지털 변환 서버(100)는 PDF(Portable Document Format) 파일에 포함된 요소들을 분리 추출하여 저장 및 관리하며, 추출된 요소들을 의미론적(Semantic) 그룹핑하여 상호작용이 가능한 디지털 참고서를 생성할 수 있고, 이를 이용자단말기에 제공할 수 있다.The digital conversion server 100 separates, extracts, stores and manages the elements included in the PDF (Portable Document Format) file, semantically groups the extracted elements to create a digital reference book that can be interacted with, This can be provided to the user terminal.
디지털 변환 서버(100)의 각 구성은, 공지된 라이브러리에 의해 구동될 수 있다. 예를 들어, PDF를 페이지별로 이미지 변환 및 텍스트 추출 시 사용되는 PDF 라이브러리 fitz, 이미지 변형과 경계선 추출에서 사용되는 컴퓨터 비전 라이브러리인 opencv(Open Source Computer Vision), Object Detection 모델 학습, 예측 및 도구로써 사용되는 객체감지 pytorch와 tensorflow, MathML 및 LaTex 문법 OCR API로 사용되는 MathPix 등 라이브러리가 이용될 수 있고 이들은 파이썬 상에서 구동될 수 있으나, 이에 한정되지 않는다. 또한, numpy, sklearn, re, glob 등 라이브러리가 이용될 수 있고 파이썬 상에서 구동될 수 있으며, Font-forge 라이브러리를 사용하여 PDF 상에서 텍스트를 검출하고 이를 적합한 폰트와 매칭하여 rendering 가능한 형태로 추출할 수 있으며, Font fomat은 TTF, TTF2, OTF, woff 등의 형태로 산출될 수 있으나, 이에 한정되지 않는다.Each component of the digital conversion server 100 may be driven by a known library. For example, fitz, a PDF library used for converting PDFs into images and extracting text page by page; opencv (Open Source Computer Vision), a computer vision library used for image transformation and border extraction; object detection model training, prediction, and use as a tool Libraries such as MathPix used as object detection pytorch and tensorflow, MathML and LaTex grammar OCR APIs can be used, and they can be run on Python, but are not limited thereto. In addition, libraries such as numpy, sklearn, re, and glob can be used and run on Python, and text can be detected on PDF using the Font-forge library, matched with a suitable font, and extracted in a renderable form. , Font format may be calculated in the form of TTF, TTF2, OTF, woff, etc., but is not limited thereto.
디지털 변환 서버(100)는 파일생성부(110), 영역검출부(120) 및/또는 요소추출부(130)를 포함할 수 있다.The digital conversion server 100 may include a file generator 110, a region detector 120, and/or an element extractor 130.
파일생성부(110)는 디지털 참고서의 PDF 파일에서 복수 개의 페이지를 추출하여 페이지 파일을 생성한다. 즉, 파일생성부(110)는 PDF 파일이 가지고 있는 복수 개의 페이지를 각각 단일의 페이지로 분리한 페이지 파일을 생성한다. 이때, 페이지 파일은 각각, 픽셀을 사용하는 BMP, GIF, JPEG, PNG의 비트맵 형식의 파일일 수 있다.The file generator 110 extracts a plurality of pages from the PDF file of the digital reference book and creates a page file. That is, the file generator 110 creates a page file by dividing a plurality of pages of the PDF file into single pages. In this case, the page file may be a bitmap format file of BMP, GIF, JPEG, and PNG using pixels, respectively.
파일생성부(110)가 페이지를 추출하는 PDF 파일은 기 생성되어 데이터베이스부(150)에 저장된 PDF 파일일 수 있다. 파일생성부(110)가 페이지를 추출하는 PDF 파일은 스마트폰, 스캐너, 복합기 등 이용자단말기에 의해 참고서가 스캔되어 생성된 PDF 파일일 수도 있다. 또는, 파일생성부(110)가 페이지를 추출하는 PDF 파일은 통신망을 통해 연결된 다른 외부서버(10)로부터 수신될 수도 있다. 참고서가 스캔되어 생성된 PDF 파일은 복수 개의 페이지를 포함한다. 파일생성부(110)는 복수 개의 페이지를 각각 단일의 페이지로 추출하여 페이지 파일을 생성한다.The PDF file from which the file creation unit 110 extracts pages may be a PDF file that has been created and stored in the database unit 150 . The PDF file from which the file generator 110 extracts pages may be a PDF file generated by scanning a reference book by a user terminal such as a smartphone, scanner, or multifunction device. Alternatively, the PDF file from which the file generator 110 extracts pages may be received from another external server 10 connected through a communication network. A PDF file created by scanning a reference book contains multiple pages. The file generator 110 extracts a plurality of pages into a single page and creates a page file.
[규칙 제91조에 의한 정정 29.12.2022]　
한편, 파일생성부(110)가 페이지를 추출하는 PDF 파일은 그 생성된 방법에 따라 Structured PDF 파일 또는 Scanned PDF 파일일 수 있다. Structured PDF는 컴퓨터 소프트웨어를 이용하여 전자적으로 생성된 PDF 파일이고, Scanned PDF은 실물의 특정 서책을 스캐너 등을 이용하여 이미지 형태로 생성된 PDF 파일이다. Structured PDF과 Scanned PDF의 큰 차이점으로 Structured PDF는, 아래 도 3을 참조하면, 각 페이지가 포함하는 텍스트, 이미지 또는 표 등 각 요소에 대해서 선택(identify) 및 선택에 따른 추출이 가능하다는 점이다. 도 3의 파란색 하이라이트는 요소들이 선택된 모습을 보여준다. Scanned PDF는 Structured PDF와 달리, 도 4를 참조하면, 각 요소에 대한 선택 및 추출이 불가능하며, 따라서 파일생성부(110)는, 준비된 PDF 파일이 Scanned PDF 일 경우, OCR process를 통해서 Structure를 복원한다. 이때 OCR process는 후술하게 될 광학문자인식부(140)에 의해 진행될 수 있다. 또는, 파일생성부(110)는 준비된 PDF 파일이 Scanned PDF 일 경우, OCR process를 통하지 않고, 후술하게 될 영역검출부(120)에 의해 질문영역, 답변영역 및 해설영역 등이 추출되고, 추출된 영역의 이미지가 데이터베이스부(150)에 저장되어 관리될 수도 있다.[Correction under Rule 91 29.12.2022]
Meanwhile, the PDF file from which the file creation unit 110 extracts pages may be a structured PDF file or a scanned PDF file depending on how it is created. Structured PDF is a PDF file created electronically using computer software, and Scanned PDF is a PDF file created in the form of an image using a specific book in the real world. The big difference between Structured PDF and Scanned PDF is that in Structured PDF, referring to Figure 3 below, each element such as text, image or table included in each page can be identified and extracted according to selection. The blue highlights in FIG. 3 show the selected elements. Unlike Structured PDF, in Scanned PDF, referring to FIG. 4 , it is impossible to select and extract each element. Therefore, if the prepared PDF file is Scanned PDF, the file generator 110 restores the structure through the OCR process. do. At this time, the OCR process may be performed by the optical character recognition unit 140 to be described later. Alternatively, when the prepared PDF file is a scanned PDF, the file creation unit 110 extracts the question area, answer area, and commentary area by the area detection unit 120 to be described later without going through the OCR process, and extracts the extracted area. Images of may be stored in the database unit 150 and managed.
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 3: Structured PDF 예시>[Correction under Rule 91 29.12.2022]
<Figure 3: Structured PDF example>
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 4: Scanned PDF 예시>[Correction under Rule 91 29.12.2022]
<Figure 4: Scanned PDF example>
본 발명에 따른 디지털 참고서는 문제지 영역 및 해설지 영역을 포함한다. 이 중 문제지 영역은 과목, 학년, 책 종류 등에 상관없이 보통 Two column(2열) 형식으로 구성된다. 또한, 한 페이지 내에서 관측되는 문항의 수는 최소 4 내지 8문제 내에서 분포한다. 대부분의 문항은 객관식 형태로 이루어져 있고, 경우에 따라 단답형, 주관식 문항이 있을 수 있으나 그 빈도가 많지 않다. 문제지 영역에서 찾아야 할 가장 중요한 요소는 문제영역으로, 문제영역은 문제번호, 문제타이틀 및 선택지로 구성되며 과목 종류에 상관없이 유사한 형태를 가지고 있다. The digital reference book according to the present invention includes a question paper area and a commentary area. Among them, the problem area is usually composed of two columns regardless of subject, grade, book type, etc. In addition, the number of questions observed within one page is distributed within a minimum of 4 to 8 questions. Most of the questions are in the form of multiple choice, and depending on the case, there may be short answer and subjective questions, but they are not frequent. The most important element to find in the problem sheet area is the problem area. The problem area consists of a problem number, problem title, and options, and has a similar form regardless of the type of subject.
해설지 영역은 문제지 영역의 문제영역처럼 과목, 책 종류 등에 상관없이 공유되는 기하학적 특징이 없지만, 해설지 내부에서 모든 페이지와 요소들이 같은 레이아웃 디자인과 요소형태를 가지고 있다. 해설지 영역에는 전체적인 추세가 발견되지 않고 책마다 개별적인 특성이 많고, Two column 구조처럼 공통적으로 발견되는 패턴이 없으며, 객관식 형태로 발견되는 Pattern이 없고, 책(특히 서책 시리즈)별로 특유의 Font, Styling, Layout의 사용빈도가 높다. 이러한 특성을 갖는 해설지 영역의 경우, 문항에서 Interaction 구현을 목적으로 하기 위한 Semantic Binding을 위한 Machine Learning이 잘 작동하지 않을 가능성이 매우 높다.The commentary area does not have a shared geometric feature regardless of subject or book type like the problem area of the problem area, but all pages and elements within the commentary have the same layout design and element shape. There is no overall trend found in the commentary area, each book has many individual characteristics, there is no commonly found pattern like the two column structure, there is no pattern found in the form of multiple choice, and each book (especially book series) has its own unique font, styling, Layout is used frequently. In the case of commentary areas with these characteristics, it is very likely that Machine Learning for Semantic Binding for the purpose of implementing Interaction in questions will not work well.
문제지 영역의 경우, 본 발명에 따른 영역검출부(120)는, 파일생성부(110)가 생성한 페이지 파일에서, 페이지 파일의 전체 이미지, 페이지 파일 내 포함된 텍스트, HTML, CSS, Font, Background vector image 등 요소를 추출한다. 이후, 영역검출부(120)는 페이지 파일 각각을 HTML page로 변환한다. Structured PDF가 HTML page로 변환된 경우, 각 요소들은 변환 후 검출(Clickable) 가능하며, Scanned PDF에 대해서는 각 요소들의 검출이 불가능한 상태에서 전체 페이지가 한 개의 이미지로 추출된다. 따라서 전술한 바와 같이 Scanned PDF는 OCR processing을 통해서 이미지 내 character/number의 검출이 가능한 형태로 변환될 필요가 있다. 한편, 영역검출부(120)가 페이지 파일 각각을 HTML page로 변환한 후, 각 페이지 내에서 선택 가능한 요소(identified element)에 대해 id 값을 부여한다. 이때, 선택 가능한 요소란, 텍스트, 이미지 또는 표 등 각 요소를 포함하며, 예를 들어, 문제영역, 문제영역 내 질문영역 및 답변영역, 답변영역 내 원숫자 선택지 등을 포함할 수 있다.In the case of the problem paper area, the area detection unit 120 according to the present invention, in the page file generated by the file creation unit 110, the entire image of the page file, text included in the page file, HTML, CSS, Font, Background vector Extract elements such as images. After that, the area detection unit 120 converts each page file into an HTML page. When Structured PDF is converted to HTML page, each element can be detected (clickable) after conversion, and for Scanned PDF, the entire page is extracted as a single image while each element is undetectable. Therefore, as described above, the scanned PDF needs to be converted into a form in which characters/numbers in the image can be detected through OCR processing. Meanwhile, after the region detection unit 120 converts each page file into an HTML page, an id value is assigned to an identified element within each page. At this time, the selectable element includes each element such as text, image, or table, and may include, for example, a problem area, a question area and an answer area within the problem area, and a number option within the answer area.
보다 상세하게, 영역검출부(120)는 출판사에서 서책을 생산하는 프로세스 중 산출되는 Structured PDF 파일을 서비스에 사용할 수 있는 형태인 HTML로 변환한다. 이때 사용되는 PDF 파일은 Text 복사가 가능하고, Font 정보가 정확히 표현돼 있으며, Object 파일의 선택이 개별적으로 가능하다는 특징을 갖고 있다.In more detail, the area detection unit 120 converts the structured PDF file produced during the book production process at the publishing house into HTML, which is a form usable for the service. The PDF file used at this time has the characteristics that text can be copied, font information is accurately expressed, and object files can be selected individually.
[규칙 제91조에 의한 정정 29.12.2022]　
영역검출부(120)는 PDF 파일 안에 포함된 모든 요소(Element)를 추출한 뒤, 이미지들은 하나의 큰 이미지로 합쳐 배경이이미지로 만들고, 텍스트는 폰트를 새롭게 만들어 원본 PDF에서의 위치와 동일한 위치에 배치한다. 구체적으로, 다음과 같은 절차로 수행한다(도 5 참조). (1) PDF 렌더링 라이브러리(PDF rendering Library)를 사용하여 PDF 내부에 있는 모든 객체(Object, Element)들을 위치값, 크기값 등과 함께 추출, (2) 추출된 객체 중 백그라운드(배경) 이미지와 객체 이미지들을 하나의 이미지로 병합(객체 이미지의 위치값, 크기값 등을 고려하여, 백그라운드 이미지 내 객체 이미지들을 해당 위치에 배치하여 병합), (3) 추출된 객체 중, 텍스트에 대해서는 추출된 텍스트의 형태에 맞는 폰트를 새롭게 생성, (4) 텍스트들과 생성된 폰트를 사용하여 텍스트들이, 페이지 크기의 행열에 원본 형태와 동일한 크기로 동일한 위치에 위치하도록 배치하고 스타일링을 추가, (5) 완성된 이미지와 텍스트 행열을 Tag 형식으로 HTML에 내재화[Correction under Rule 91 29.12.2022]
After extracting all the elements included in the PDF file, the area detection unit 120 combines the images into one large image to make a background image, and the text is placed at the same position as in the original PDF by creating a new font. do. Specifically, it is performed in the following procedure (see FIG. 5). (1) Using the PDF rendering library, extract all objects (objects and elements) inside the PDF along with their position and size values, (2) among the extracted objects, the background (background) image and object image Merge them into one image (by arranging and merging object images in the background image at the corresponding location, considering the position value and size value of the object image), (3) Among the extracted objects, the form of the extracted text for text (4) Using the texts and the generated fonts, place the texts in the page-sized matrix in the same size and position as the original form and add styling, (5) Finished image and text strings are internalized in HTML in Tag format
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 5: Structured PDF를 HTML로 변환하는 방법>[Correction under Rule 91 29.12.2022]
<Figure 5: How to convert Structured PDF to HTML>
한편, 영역검출부(120)는 조판을 거친 PDF 파일을 확보하지 못했을 경우 Scan 절차를 통해 산출된 이미지 기반의 PDF 파일을 이용할 수 있다.Meanwhile, the region detector 120 may use an image-based PDF file calculated through a scan procedure when a typesetting PDF file is not secured.
Structrured PDF 파일은 객체들을 선택할 수 있어서, Text block을 지정하여 Highlight 등이 가능하고, 이미지나 표 선택이 가능하며, 객체에 접근하여 Selector Tag 선언이 가능하다. 반면, Scanned PDF 파일은 페이지 전체가 이미지로 구현돼 있어서, Text block을 지정하여 Highlight 등이 불가하고, 이미지나 표 선택이 불가하며, Machine Learning Selector(선택지) Detection 기술을 이용해 Tag를 선언해야 한다.Structured PDF files can select objects, highlighting by designating a text block, selecting an image or table, and declaring a selector tag by accessing an object. On the other hand, in a scanned PDF file, the entire page is implemented as an image, so it is impossible to highlight by designating a text block, and it is impossible to select an image or table, and a tag must be declared using Machine Learning Selector Detection technology.
Structured PDF 파일은 각 페이지에서 모든 Element들이 인식가능한 형태로 구현되어 있고 각각 혹은 집합적으로 Class가 부여되어 있어서 특정 class javascript event 구현이 가능하다. 다만, Scanned PDF 파일의 경우에는 전체 이미지가 하나의 Class로 conversion되기 때문에 각 이미지에서 interaction이 필요한 부분을 발굴하여 overlay로 javascript event 구현이 필요한 point를 구성하는 작업이 선행되어야 한다.In a structured PDF file, all elements in each page are implemented in a recognizable form, and classes are assigned individually or collectively, so it is possible to implement a specific class javascript event. However, in the case of a scanned PDF file, since the entire image is converted into a single class, it is necessary to discover the parts that require interaction in each image and configure the points that require javascript event implementation with overlay.
영역검출부(120)는, Scanned PDF를 HTML로 변환하는 경우, Text 영역을 발굴하여 Text block을 지정하고 Highlight를 지정할 수 있도록 Text Block 영역을 구현하는데, 구체적으로 다음의 절차로 수행된다. (1) OCR model을 기반으로 특정 글자가 Text인지 판별, (2) 해당 Text의 box를 지정하여 선택하고 Interaction이 구현될 수 있도록 Text Box detection, (3) Text 글자별로 동일위치에 Text box overlayWhen converting scanned PDF into HTML, the area detection unit 120 discovers a text area, designates a text block, and implements a text block area so that a highlight can be designated. Specifically, the following procedure is performed. (1) Determine whether a specific character is text based on the OCR model, (2) Text box detection to designate and select the text box and implement interaction, (3) Text box overlay at the same position for each text character
한편, 영역검출부(120)는 문항 내 선택지(Selector)에 Event를 구현할 수 있도록 위치 기반 Tag point 설정하는데, 구체적으로 다음의 절차로 수행된다. (1) 페이지 내 문항 판별, (2) 문항 내 선택지 영역 판별, (3) 선택지 내 원숫자 위치 판별, (4) 각 원숫자 영역에 대해서 javascript event 설정이 가능한 tag 부여On the other hand, the area detection unit 120 sets a location-based tag point so that an event can be implemented in a selector within a question, and is specifically performed in the following procedure. (1) Discrimination of questions within a page, (2) Discrimination of choice area within a question, (3) Determination of the position of a circle number within a choice, (4) Assignment of a tag capable of setting a javascript event for each circle number area
다만, 상기와 같이 Structured PDF 파일 또는 Scanned PDF 파일이 효과적으로 HTML로 변환되더라도 유의미한 Tag가 부여되지 않아 Semantic하게 구현되지 않은 HTML은 Interaction에 사용될 수 없다.However, even if a Structured PDF file or a Scanned PDF file is effectively converted to HTML as described above, HTML that is not semantically implemented because a meaningful tag is not assigned cannot be used for interaction.
[규칙 제91조에 의한 정정 29.12.2022]　
도 6을 참조하면, 위치값을 기반으로 한 Tag는, 이용자가 웹페이지에서 보는 화면 구성은 동일하지만, 의미론적(Semantic)으로 그룹핑되어 있지 않아 의미론 단위로 연결이 어려우며, 따라서 채점 및 해답연결 등의 이벤트 구현이 어렵다.[Correction under Rule 91 29.12.2022]
Referring to FIG. 6, tags based on location values have the same screen configuration that users see on the web page, but are not semantically grouped, making it difficult to connect in semantic units. Therefore, scoring and answer connection, etc. It is difficult to implement the event of
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 6: 위치값을 기반으로 한 Tag>[Correction under Rule 91 29.12.2022]
<Figure 6: Tag based on location value>
[규칙 제91조에 의한 정정 29.12.2022]　
반면, 도 7을 참조하면, 의미값을 기반으로 한 Tag는, 위치값을 기반으로 한 경우 대비 이용자가 웹페이지에서 보는 화면 구성은 동일하지만, 의미론적(Semantic)으로 그룹핑되어 의미론 단위로 연결이 쉬우며, 따라서 채점 및 해답연결 등의 이벤트 구현이 쉽다.[Correction under Rule 91 29.12.2022]
On the other hand, referring to FIG. 7, tags based on semantic values have the same screen configuration that users see on the webpage compared to cases based on location values, but are semantically grouped and connected in semantic units. It is easy, so it is easy to implement events such as scoring and answer connection.
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 7: 의미값을 기반으로 한 Tag>[Correction under Rule 91 29.12.2022]
<Figure 7: Tag based on semantic value>
즉, 선택 가능한 요소에 부여된 class id 값(Tag)은 PDF가 HTML 페이지로의 변환과정에서 각 페이지 내 위치정보를 기반으로 생성된 것으로 그 id 값에 의미는 없다. 따라서, 이러한 id 값을 의미단위, 즉 페이지, 문제, 선택지 등의 단위로 그룹화하기 위해 페이지 파일의 이미지를 아래와 같은 머신러닝 과정을 진행한다. 이때 그룹화는, 예를 들어, 하나의 지문에 딸린 문제, 선택지, 객관식 문항 등을 그룹화하는 것을 의미할 수 있다.That is, the class id value (Tag) given to the selectable element is generated based on location information within each page in the process of converting PDF to HTML page, and the id value has no meaning. Therefore, in order to group these id values into semantic units, such as pages, problems, and options, the following machine learning process is performed on the image of the page file. In this case, grouping may mean, for example, grouping problems, options, and multiple-choice questions attached to one passage.
즉, 영역검출부(120)는 각 문제영역별로 class를 grouping하고 id를 부여하는 작업을 수행한다. 구체적으로, 다음의 절차로 해당작업을 수행한다. (1) 페이지 내에서 문제영역을 detection, (2) 문제영역 내 Interaction 적용이 필요한 영역을 detection, (3) 각 영역에 존재하는 요소 detectionThat is, the region detection unit 120 groups classes for each problem region and assigns an id. Specifically, the corresponding work is performed in the following procedure. (1) Detection of problem areas within the page, (2) Detection of areas requiring interaction application within the problem area, (3) Detection of elements present in each area
영역검출부(120)는 먼저, 각 페이지 파일에서 문제영역을 검출하여 ID를 부여하고, 그 문제영역 내 상호작용(interaction)이 필요한 질문영역 및 답변영역으로 분리하여 검출한다. 검출된 질문영역 및/또는 답변영역에도 ID(Sub-id)가 부여된다. 또한, 문제영역 내 상호작용이 필요한 영역은, 참고서의 과목마다 차이가 있을 수 있으며, 예를 들어, 국어 또는 영어의 경우 묶음지문이 존재하고, 이러한 묶음지문은 학습자의 편의를 위해서는 지문과 문제를 동시에 한 화면에서 볼 수 있도록 하는 것이 바람직하다. 일 실시예에 따르면, 영역검출부(120)는 문제영역 내 답변영역을 검출하고, 답변영역 외의 영역을 질문영역으로 구분할 수 있다.The area detection unit 120 first detects a problem area in each page file, assigns an ID, separates the problem area into a question area and an answer area requiring interaction, and detects it. An ID (Sub-id) is also assigned to the detected question area and/or answer area. In addition, the area requiring interaction within the problem area may differ for each subject in the reference book. For example, in the case of Korean or English, there is a bundle of texts, and these bundles of texts are combined with the text for the convenience of learners. It is desirable to be able to view them on one screen at the same time. According to an embodiment, the area detection unit 120 may detect an answer area within the problem area and classify an area other than the answer area as a question area.
[규칙 제91조에 의한 정정 29.12.2022]　
본 발명에 따른 영역검출부(120)는, 파일생성부(110)가 추출한 페이지 파일에서 문제영역(도 9, A)을 검출한다. 영역검출부(120)는 fitz 라이브러리를 사용하여 JSON(JavaScript Object Notation) format의 Object 정보를 추출할 수 있다. fitz 라이브러리는 도 8과 같이 페이지에서 텍스트를 추출할 수 있다.[Correction under Rule 91 29.12.2022]
The area detection unit 120 according to the present invention detects a problem area (A in FIG. 9 ) in the page file extracted by the file creation unit 110 . The area detection unit 120 may extract object information in a JSON (JavaScript Object Notation) format using the fitz library. The fitz library can extract text from a page as shown in FIG. 8 .
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 8: fitz 라이브러리를 사용한 object 추출 예시>[Correction under Rule 91 29.12.2022]
<Figure 8: Example of object extraction using fitz library>
[규칙 제91조에 의한 정정 29.12.2022]　
영역검출부(120)는 fitz 라이브러리를 통해 페이지에서 텍스트를 검출한다. 영역검출부(120)는 텍스트간 자간 또는 상하방향 길이가 다른 텍스트간 자간 또는 상하방향 길이보다 더 긴 부분을 경계로 하여 도 9와 같이 각각의 문제영역을 검출할 수 있다.[Correction under Rule 91 29.12.2022]
The area detection unit 120 detects text from the page through the fitz library. The area detection unit 120 may detect each problem area as shown in FIG. 9 based on a boundary between text spaces or vertical lengths longer than other text spacings or vertical lengths.
영역검출부(120)는 각 페이지 파일에서 문제영역을 검출할 때, 머신러닝을 이용하여 각 문제영역을 둘러싼 바운딩박스(bounding box)를 생성할 수 있다. 영역검출부(120)는 머신러닝으로 생성한 바운딩박스 내 문제영역에 통일된 배경색을 추가할 수 있다. 영역검출부(120)는 바운딩박스를 제외한 나머지 영역에 대해 opencv를 통해 인접한 픽셀끼리 팽창화를 통해 다른 문제영역이 존재하는 지 판단한다. 한편, 본 명세서에서는 Fast RCNN 기반의 머신러닝 모델을 기초로 바운딩박스를 생성할 수 있으며, 이는 후술하도록 한다.When detecting problem areas in each page file, the area detection unit 120 may generate a bounding box surrounding each problem area using machine learning. The region detection unit 120 may add a unified background color to the problem region within the bounding box generated by machine learning. The region detection unit 120 determines whether other problem regions exist by expanding adjacent pixels in the remaining region except for the bounding box through opencv. Meanwhile, in the present specification, a bounding box may be generated based on a machine learning model based on Fast RCNN, which will be described later.
영역검출부(120)가 생성한 바운딩박스 내의 정보만이 작업영역으로 할당되어 전체 페이지를 처리하는 것보다 본 발명에 따른 처리속도가 더욱 빨라질 수 있다. 한편, 영역검출부(120)는 생성한 바운딩박스의 위치 및 사이즈에 대한 정보를 검출하여 저장한다.Only the information within the bounding box generated by the region detection unit 120 is allocated to the work region, and the processing speed according to the present invention can be faster than processing the entire page. Meanwhile, the area detection unit 120 detects and stores information about the position and size of the created bounding box.
한편, 영역검출부(120)는, 각 페이지 파일에서 문제영역을 검출할 때, 페이지 가로방향 중간점 x 좌표(x half)보다, 바운딩박스의 최좌측점 x 좌표(x1) 및 최우측점 x 좌표(x2)의 값이 더 클 때 페이지의 레이아웃이 2열인 것으로 판단할 수 있다. 이때, 페이지 가로방향의 중간점 좌표란, 전체 페이지의 중심점 또는 페이지의 좌측과 우측을 잇는 선분의 중심에 해당하는 좌표 값일 수 있다.On the other hand, when the region detection unit 120 detects a problem region in each page file, the x-coordinate (x1) of the leftmost point and the x-coordinate (x1) of the rightmost point of the bounding box are higher than the x-coordinate (x half) of the midpoint in the horizontal direction of the page. When the value of x2) is larger, it can be determined that the layout of the page is two columns. In this case, the midpoint coordinates in the horizontal direction of the page may be coordinate values corresponding to the center point of the entire page or the center of a line segment connecting the left and right sides of the page.
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 9: 바운딩박스를 이용한 문제영역 추출 예시>[Correction under Rule 91 29.12.2022]
<Figure 9: Example of problem area extraction using bounding box>
[규칙 제91조에 의한 정정 29.12.2022]　
영역검출부(120)는 검출한 문제영역을, 도 10과 같이 질문영역 및 답변영역(B, 도 10)으로 분리한다. 영역검출부(120)는 문제영역을 질문영역 및 답변영역으로 분리할 때 먼저 답변영역을 먼저 선택하여 추출하고, 선택한 답변영역을 제외한 나머지 문제영역을 질문영역으로 선택한다.[Correction under Rule 91 29.12.2022]
The area detection unit 120 separates the detected problem area into a question area and an answer area (B, FIG. 10) as shown in FIG. 10 . When dividing the problem area into a question area and an answer area, the area detector 120 first selects and extracts the answer area, and selects the remaining problem areas excluding the selected answer area as the question area.
답변영역은 선택지가 가로로 정렬될 수도 있고, 세로로 정렬될 수도 있다. 답변영역이 세로로 정렬될 경우 답변영역의 선택은 아래와 같다.In the answer area, options may be arranged horizontally or vertically. If the answer area is aligned vertically, the choice of answer area is as follows.
영역검출부(120)는 페이지 파일에서 답변영역을 검출할 때, 원숫자가 포함된 행을 검출할 수 있다. 영역검출부(120)에 의해 일련의 연속된 원숫자가 행의 최좌측에서 검출되고, 그 원숫자의 열의 아랫방향으로 일련의 증가되도록 연속된 원숫자가 검출된 경우, 영역검출부(120)는 문제영역에서 검출된 원숫자가 포함된 문장이 있는 영역을 답변영역으로 선택한다.When detecting an answer area in a page file, the area detection unit 120 may detect a row including a circle number. When a series of consecutive circular numbers is detected at the leftmost side of a row by the area detection unit 120 and a series of consecutive circular numbers are detected in a downward direction in a column of the circular numbers, the area detection unit 120 has a problem. Select an area containing a sentence containing a circle number detected in the area as an answer area.
또한, 답변영역이 가로로 정렬될 경우, 영역검출부(120)는 페이지 파일에서 답변영역을 검출할 때, 원숫자가 연속적으로 증가하도록 포함된 행을 검출할 수 있다. 영역검출부(120)에 의해 일련의 연속된 원숫자가 행을 따라 검출된 경우, 영역검출부(120)는 문제영역에서 검출된 원숫자가 포함된 문장이 있는 영역을 답변영역으로 선택한다.In addition, when the answer areas are aligned horizontally, the area detection unit 120 may detect lines included so that the original number continuously increases when detecting the answer areas in the page file. When a series of consecutive circular numbers is detected along a row by the area detection unit 120, the area detection unit 120 selects an area containing a sentence including the circle numbers detected in the problem area as an answer area.
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 10: 문제영역 내 답변영역 추출 예시>[Correction under Rule 91 29.12.2022]
<Figure 10: Example of extracting answer areas within problem areas>
[규칙 제91조에 의한 정정 29.12.2022]　
요소추출부(130)는, 도 11과 같이, 영역검출부(120)에 의해 분리된 질문영역 및 답변영역에서 요소(C 내지 F, 도 11)를 추출하고 추출된 요소를 데이터베이스부(150)에 저장한다. 이때 요소는 문제인덱스(C), 텍스트(D), 이미지(E), 글박스(미도시) 및 표(미도시)를 포함한다. 요소추출부(130)는 문제인덱스(C)를 추출할 때 질문영역의 최좌측에 위치하는 인덱스를 선택하여 추출한다. 요소추출부(130)는 문제영역에서 이미지(E)를 선택하여 추출할 수 있다. 요소추출부(130)가 추출한 문제인덱스는, 후술하게 될, 해설지 영역의 답변영역과 서로 연결되어 채점 자동화에 이용될 수 있다.[Correction under Rule 91 29.12.2022]
As shown in FIG. 11, the element extraction unit 130 extracts elements (C to F, FIG. 11) from the question area and the answer area separated by the area detection unit 120 and stores the extracted elements in the database unit 150. Save. At this time, the elements include a problem index (C), text (D), image (E), text box (not shown), and table (not shown). When extracting the problem index C, the element extraction unit 130 selects and extracts the index located at the leftmost side of the question area. The element extraction unit 130 may select and extract the image E from the problem area. The problem index extracted by the element extractor 130 can be used for scoring automation by being connected to an answer area of the commentary area, which will be described later.
요소추출부(130)가 질문영역에서 추출한 문제인덱스(C) 및 이미지(E)를 제외한 나머지는 텍스트(D)일 수 있다. 이때의 텍스트의 각 위치에 따라 서로 다른 바운딩박스를 가진 영역을 기초로 각 텍스트는 선택 및 추출될 수 있다. 한편, 요소추출부(130)는, 선택한 질문영역에서의 텍스트 중 일련의 선택지(F)를 선택 및 추출할 수 있다. 일련의 선택지(F)는, 각 문장에 각각 'ㄱ. ㄴ. ㄷ.', '(ㄱ) (ㄴ) (ㄷ)', '가. 나. 다.' 또는 '(가) (나) (다)'가 일련으로 검출된 경우, 그 문장들 또는 그 영역을 일련의 선택지(F)로 선택 및 추출할 수 있다. 즉, 요소추출부(130)는 자음이 순서에 따라 일련으로 검출되거나, 자음과 모음이 결합된 상태로 순서에 따라 일련으로 검출된 영역 내 문장들을 일련의 선택지(F)로 선택 및 추출할 수 있다.Except for the problem index (C) and the image (E) extracted from the question area by the element extraction unit 130, the rest may be text (D). At this time, each text may be selected and extracted based on a region having a different bounding box according to each position of the text. Meanwhile, the element extraction unit 130 may select and extract a series of options (F) from text in the selected question area. A series of options (F) are 'a. you. c.', '(a) (b) (c)', 'a. me. all.' Alternatively, when '(a) (b) (c)' is detected in series, the sentences or the area can be selected and extracted as a series of options (F). That is, the element extractor 130 may select and extract sentences within a region in which consonants are sequentially detected in sequence or consonants and vowels are sequentially detected in sequence in a combined state as a series of options (F). there is.
요소추출부(130)는, 선택 및 추출한 각 요소들을 데이터베이스부(150)에 저장한다. 요소추출부(130)가 선택 및 추출한 각 요소들은 각각 바운딩박스를 가지고, 그 바운딩박스 내의 영역을 이미지의 형태로 데이터베이스부(150)에 저장한다.The element extraction unit 130 stores the selected and extracted elements in the database unit 150. Each element selected and extracted by the element extraction unit 130 has a bounding box, and the area within the bounding box is stored in the database unit 150 in the form of an image.
또한 요소추출부(130)는, 전술한 영역검출부(120)가 문제영역에서 원숫자가 포함된 문장이 있어 답변영역으로 선택한 영역에서, 원숫자가 포함된 문장을 선택지로 판단한다. 요소추출부(130)는 선택지가 포함된 답변영역에서 전술한 요소들을 추출하고, 추출된 요소를 데이터베이스부(150)에 저장한다.In addition, the element extraction unit 130 determines that a sentence including a circle number is an option in an area selected as an answer area because the above-described area detection unit 120 has a sentence including a circle number in the problem area. The element extraction unit 130 extracts the aforementioned elements from the answer area including the options, and stores the extracted elements in the database unit 150.
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제]
[Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 11: 문제영역 내 요소추출 예시>[Correction under Rule 91 29.12.2022]
<Figure 11: Example of element extraction in problem area>
한편, 참고서의 과목 종류에 따라 문제의 형식에 차이가 있는 경우가 있다. 예를 들어 참고서의 과목이 수학인 경우, 문제영역 내에 수식이 존재할 수 있다. 요소추출부(130)가 추출하는 요소에는 수식이 더 포함될 수 있으며, 요소추출부(130)는 문제영역 내 수식 요소를 추출할 때, MatchOCR(예를 들어, Mathpix)를 사용하여 수식에 대하여 MathML 혹은 LaTex 형태로 수식을 추출할 수 있다. 한편, 참고서의 과목이 수학인 경우에만 문제영역에서 수식 요소가 추출되는 것에 한정되는 것은 아니고, 문제영역 내 수식이 검출될 때에는 언제나 수식 요소가 추출될 수 있다.On the other hand, depending on the subject type of the reference book, there may be a difference in the format of the problem. For example, if the subject of the reference book is mathematics, formulas may exist in the problem area. Elements extracted by the element extractor 130 may further include formulas, and when the element extractor 130 extracts formula elements in the problem area, MathML for formulas using MatchOCR (eg, Mathpix). Alternatively, you can extract formulas in LaTex format. On the other hand, it is not limited to extracting formula elements from the problem area only when the subject of the reference book is mathematics, and formula elements can be extracted anytime when formulas in the problem area are detected.
[규칙 제91조에 의한 정정 08.03.2023]　
또한 참고서의 과목이 국어 또는 영어인 경우에는, 도 12와 같이 묶음지문이 포함될 수 있다. 묶음지문이란, 하나의 지문에 복수 개의 문제가 포함된 지문을 의미한다.[Correction by Rule 91 08.03.2023]
In addition, when the subject of the reference book is Korean or English, a bundled fingerprint may be included as shown in FIG. 12. Bundled fingerprints refer to fingerprints in which a plurality of questions are included in one fingerprint.
[규칙 제91조에 의한 정정 08.03.2023]
영역검출부(120)는, 문제영역 최상단에서 묶음지문 번호(G, 도 12)를 검출할 수 있다. 영역검출부(120)는, 묶음지문 번호가 검출되면, 묶음지문 번호가 해당하는 문제 영역을 포함시킬 수 있도록 Document 병합 작업을 실시하고(후술하게 될 해설영역 참조), 그 묶음지문 번호를 포함하는 영역에 바운딩박스를 생성한다. 이때 영역검출부(120)는 묶음지문 번호에 따른 문제영역을 함께 포함하도록 바운딩박스를 생성할 수 있다. 예를 들어 그림 12과 같이 '35~37' 이 묶음지문 번호인 경우, 영역검출부(120)는, 묶음지문과 함께 35 내지 37번 문제인덱스(C)를 가진 문제영역이 모두 포함되도록 바운딩박스를 생성한다. 그리고 요소추출부(130)는, 질문영역 및 답변영역에서 요소를 추출하여 데이터베이스부(150)에 저장할 때 전술한 묶음지문에서도 요소를 함께 추출하여 데이터베이스부(150)에 저장하며, 이때 질문영역 및 답변영역에서 추출한 요소와 묶음지문에서 추출한 요소가 서로 연결되도록 데이터베이스부(150)에 저장할 수 있다.[Correction by Rule 91 08.03.2023]
The area detection unit 120 may detect a bundle fingerprint number (G, FIG. 12 ) at the top of the problem area. When a bundled fingerprint number is detected, the area detection unit 120 performs a document merging operation to include a problem area corresponding to the bundled fingerprint number (refer to the description area to be described later), and the area including the bundled fingerprint number. Create a bounding box in At this time, the region detection unit 120 may create a bounding box to include problem regions according to bundled fingerprint numbers. For example, as shown in Figure 12, when '35 to 37' is the bundled fingerprint number, the area detection unit 120 sets the bounding box so that all problem areas having problem indexes 35 to 37 (C) are included together with the bundled fingerprint. create And when the element extraction unit 130 extracts elements from the question area and the answer area and stores them in the database unit 150, it also extracts elements from the aforementioned bundled fingerprints and stores them in the database unit 150. At this time, the question area and Elements extracted from the answer area and elements extracted from the bundled fingerprints may be stored in the database unit 150 so as to be connected to each other.
묶음지문의 검출은 참고서의 과목이 국어 또는 영어인 경우에만 한정되는 것은 아니고, 영역검출부(120)가 전술한 묶음지문 번호와 같은 '숫자~숫자' 형식의 숫자배열이 검출될 때에는 언제나 묶음지문 요소가 추출되고 저장될 수 있다.The detection of bundled fingerprints is not limited only to cases where the subject of the reference book is Korean or English, and whenever the region detection unit 120 detects a number array in the form of 'numbers to numbers' such as the aforementioned bundled fingerprint numbers, bundled fingerprint elements can be extracted and stored.
[규칙 제91조에 의한 정정 29.12.2022]　
한편, 특정영역의 묶음 지문은 도 12와는 달리 묶음 번호가 부여되지 않는 경우도 존재하며, 이 경우 각 문제영역을 검출한 후 남은 영역을 지문영역으로 규정하고 각 페이지별 검수절차를 통해서 남은 지문을 관계있는 문제영역과 후 보정으로 매핑 프로세스를 진행할 수 있다.[Correction under Rule 91 29.12.2022]
On the other hand, unlike FIG. 12, there is a case where a bundle number is not assigned to a bundled fingerprint of a specific area. In this case, the remaining area after detecting each problem area is defined as a fingerprint area, and the remaining fingerprints are checked through the inspection procedure for each page. You can proceed with the mapping process with relevant problem areas and post-correction.
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 12: 묶음 지문에서 묶음지문 번호를 나타낸 그림>[Correction under Rule 91 29.12.2022]
<Figure 12: Figure showing bundled fingerprint numbers in bundled fingerprints>
요소추출부(130)는, 추출한 요소들 즉 문제인덱스(C), 텍스트(D), 이미지(E), 글박스(미도시), 표(미도시), 일련의 선택지(F), 페이지 정보 등을 하나의 문제로 그룹화하여 데이터베이스부(150)에 저장한다. 그룹화된 요소들은, 후술하게 될 문제제공부(160)에 의해 이용자단말기로 제공될 수 있다. 묶음지문이 있는 경우, 하나의 묶음지문과 대응되는 문제들이 동시에 한 화면에 제공되는 것이 바람직하다. 따라서 요소추출부(130)는, 페이지 파일에 묶음지문이 존재하는지, 묶음 지문이 대응되는 문항들이 몇 번 문제들인지, 즉 어느 문제인덱스를 가지고 있는지, 묶음 지문이 페이지가 넘어간다면, 그때의 페이지 정보들을 추출하여 데이터베이스부(150)에 저장한다. 요소추출부(130)는, 추출한 요소들을 그룹화할 때 해당 요소들에 특정 class id 값을 부여할 수 있다.The element extraction unit 130 extracts the extracted elements, that is, the problem index (C), text (D), image (E), text box (not shown), table (not shown), a series of options (F), and page information. etc. are grouped into one problem and stored in the database unit 150. The grouped elements may be provided to the user terminal by the problem providing unit 160 to be described later. When there is a bundled fingerprint, it is preferable that problems corresponding to one bundled fingerprint are simultaneously provided on one screen. Therefore, the element extraction unit 130 determines whether there is a bundled fingerprint in the page file, how many questions are the questions corresponding to the bundled fingerprint, that is, which problem index it has, and if the bundled fingerprint is turned over, the page information at that time. are extracted and stored in the database unit 150. The element extraction unit 130 may assign a specific class id value to the elements when grouping the extracted elements.
광학문자인식부(140)는, 추출되어 데이터베이스부(150)에 저장된 요소를 광학문자인식(OCR, Optical Character Recognition) 처리 또는 Fitz library를 통해 텍스트를 추출하여 다시 데이터베이스부(150)에 저장할 수 있다. 광학문자인식부(140)는 공지된 광학문자인식 알고리즘을 이용하여 요소를 광학문자인식 처리할 수 있다.The optical character recognition unit 140 may extract text from elements extracted and stored in the database unit 150 through Optical Character Recognition (OCR) processing or the Fitz library, and store them in the database unit 150 again. . The optical character recognition unit 140 may perform optical character recognition processing on an element using a known optical character recognition algorithm.
문제제공부(160)는, 요소추출부(130)가 그룹화한 요소들을 이용자단말기의 화면에 제공한다. 이때, 제공되는 요소들은 각각 위치값에 맞는 위치에 배열된 상태에서 HTML layer로 제공될 수 있다.The problem providing unit 160 provides the elements grouped by the element extraction unit 130 to the screen of the user terminal. At this time, provided elements may be provided as an HTML layer in a state of being arranged at positions suitable for respective position values.
한편, 문제제공부(160)는 Multi-layer structure를 이용자단말기로 제공할 수 있다. 즉, 그룹화된 문제들이 이용자단말기로 제공되는 HTML layer 상에 필기를 위한 필기 layer 가 더 제공될 수 있다. 이용자단말기는 HTML layer 상에 제공된 필기 layer에 필기와 문제풀기를 진행할 수 있다. 문제제공부(160)는 필기 layer에 기재된 손글씨를 인식할 수 있는 공지된 라이브러리를 통해 답 체크를 인식할 수 있다. 이때의 답 체크는 HTML layer의 답변영역에 포함된 어느 원숫자가 위치한 위치값과 동일한 위치의 필기 layer에 손글씨의 방식으로 체크된 경우, 문제제공부(160)는 후술하게 될 답변영역의 문제인덱스와 정답을 기반으로 이용자단말기의 답 체크와 정답을 비교하여 문제의 채점을 자동화할 수 있다.On the other hand, the problem providing unit 160 may provide a multi-layer structure to the user terminal. That is, a writing layer for writing may be further provided on the HTML layer in which the grouped problems are provided to the user terminal. The user terminal may proceed with writing and problem solving in the writing layer provided on the HTML layer. The question providing unit 160 may recognize an answer check through a known library capable of recognizing handwriting written in the handwriting layer. At this time, if the answer is checked in the handwriting method in the handwriting layer at the same position as the position value where a certain number included in the answer area of the HTML layer is located, the problem providing unit 160 is the problem index of the answer area to be described later. Based on the answer and the correct answer, it is possible to automate the scoring of the problem by comparing the answer check of the user terminal and the correct answer.
[규칙 제91조에 의한 정정 29.12.2022]　
변환의 대상이, 문제지 영역에 따른 문제의 정답과 해설이 포함된 해설지 영역의 경우, 서책 내부에서는 레이아웃, Font, Styling이 반복적으로 사용되어 내부의 규칙을 이용한 데이터 분류가 가능하다. 구체적으로, 영역검출부(120)는, 도 13을 참조하면, 다음의 절차를 통해 해설지 영역을 문항별로 분해한다. (1) 페이지의 중앙선을 판별, (2) 판별된 중앙선을 기준으로 2열 구조를 1열 구조로 변환하여 페이지 생성, (3) 페이지 내에서 구분자 탐지, (4) 구분자의 영역 탐지, (5) 매핑할 영역을 추출, (6) 영역을 문제와 매핑[Correction under Rule 91 29.12.2022]
In the case where the target of conversion is the explanation sheet area that includes the answers and explanations of the problems according to the problem sheet area, the layout, font, and styling are repeatedly used inside the book, enabling data classification using internal rules. Specifically, referring to FIG. 13 , the area detection unit 120 decomposes the commentary area for each item through the following procedure. (1) Determine the center line of the page, (2) Create a page by converting the two-column structure into a one-column structure based on the determined center line, (3) Detect the separator within the page, (4) Detect the area of the separator, (5) ) extract the area to be mapped, (6) map the area to the problem
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 13: 해설지 영역의 문항 분해 절차>[Correction under Rule 91 29.12.2022]
<Figure 13: Item decomposition procedure in the commentary area>
보다 구체적으로, 영역검출부(120)는, 페이지 파일에서 전체 이미지 파일을 추출한다. 영역검출부(120)는 추출한 이미지 파일에서 해설영역을 추출한다. 이때 해설영역의 추출은, 전술한 문제영역의 검출과 동일한 process에 의해 진행될 수 있다.More specifically, the area detection unit 120 extracts the entire image file from the page file. The area detection unit 120 extracts a commentary area from the extracted image file. At this time, the extraction of the commentary area may be performed by the same process as the detection of the problem area described above.
[규칙 제91조에 의한 정정 08.03.2023]
한편, 영역검출부(120)는 해설영역을 추출할 때, 각 페이지 파일에서 추출된 전체 이미지 파일에서, 먼저 n열 방식의 구성에서, 도 14를 참조하면, 구분선인 중앙라인(midline, H, 도 14)을 검출할 수 있다. 또한 영역검출부(120)는 각 페이지 내에서 콘텐츠를 구성하는 최내측 구분선인 바깥라인(outline, I, 도 14)도 검출할 수 있다. 영역검출부(120)는, 중앙라인과 바깥라인을 기준으로 내부 콘텐츠가 담긴 이미지를, 중앙라인을 기준으로 잘라 하나의 이미지로 병합할 수 있다. 하나의 이미지로 병합된 예시는 도 15와 같다. 이와 같은 병합된 이미지는 해설영역의 모든 페이지에 대응하는 이미지가 1열의 단일의 이미지로 병합된 것일 수 있다.[Correction under Rule 91 08.03.2023]
On the other hand, when the region detection unit 120 extracts the commentary region, in the entire image file extracted from each page file, first in the n-column configuration, referring to FIG. 14, the dividing line (midline, H, figure 14) can be detected. In addition, the area detection unit 120 may also detect an outer line (I, FIG. 14 ), which is the innermost dividing line constituting content within each page. The area detector 120 may cut images containing internal contents based on the center line and the outer line, and merge them into one image by cutting them based on the center line. An example of merging into one image is shown in FIG. 15 . Such merged images may be obtained by merging images corresponding to all pages of the commentary area into a single image in one column.
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 14: 페이지 내 중앙라인, 바깥라인의 검출 예시>[Correction under Rule 91 29.12.2022]
<Figure 14: Example of detecting center line and outer line in a page>
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 15: 1열의 단일 이미지 예시>[Correction under Rule 91 29.12.2022]
<Figure 15: Example of a single image in column 1>
[규칙 제91조에 의한 정정 29.12.2022]　
요소추출부(130)는, 1열로 병합된 이미지를 각 챕터 영역으로 구분한다. 요소추출부(130)는 병합된 이미지를 각 챕터 영역으로 구분할 때, 도 16을 참조하면, 챕터를 구분하는 제1 구분자를 선택하고, 도 17을 참조하면, 선택한 제1 구분자의 특징점을 추출하고, 도 18을 참조하면, 추출한 제1 구분자를 기준으로 제2 구분자를 검출하여 각 챕터를 구분할 수 있다.[Correction under Rule 91 29.12.2022]
The element extraction unit 130 divides the images merged into one column into chapter areas. When the element extractor 130 divides the merged image into chapter areas, referring to FIG. 16, a first delimiter for dividing the chapters is selected, and referring to FIG. 17, feature points of the selected first delimiter are extracted, , Referring to FIG. 18, each chapter may be distinguished by detecting a second delimiter based on the extracted first delimiter.
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 16: 챕터를 구분하는 제1 구분자 선택 예시>[Correction under Rule 91 29.12.2022]
<Figure 16: Example of selecting a first delimiter for dividing chapters>
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 17: 제1 구분자의 특징점 추출 예시>[Correction under Rule 91 29.12.2022]
<Figure 17: Example of feature point extraction of the first delimiter>
[규칙 제91조에 의한 정정 29.12.2022]　
[삭제][Correction under Rule 91 29.12.2022]
[delete]
[규칙 제91조에 의한 정정 29.12.2022]　
<도 18: 제1 구분자 기준 제2 구분자(타 챕터) 검출 예시>[Correction under Rule 91 29.12.2022]
<Figure 18: Example of detecting the second separator (other chapters) based on the first separator>
요소추출부(130)는, 전술한 특징점을 통해 병합된 해설지 영역 이미지를 챕터별, 섹션별 분리하고(구분자의 영역 탐지), 각 페이지 정보와 챕터별 문제인덱스를 추출한다(매핑할 영역을 추출). 요소추출부(130)는 문제인덱스별 정답과 해설을 추출하여 데이터베이스부(150)에 저장한다. 요소추출부(130)는, 전술한 문제지 영역 내 문제영역에서의 문제인덱스와 해설영역에서의 문제인덱스를 서로 연결한다(영역을 문제와 매핑). 따라서 문제제공부(160)는, 전술한 바와 같이 이용자단말기의 답 체크의 위치정보와 문제의 답변영역의 위치정보를 비교하여 문제의 자동채점이 가능하다.The element extraction unit 130 separates the merged commentary area images by chapter and section through the above-mentioned feature points (detection of the area of the separator), and extracts each page information and problem index for each chapter (extracts the area to be mapped ). The element extraction unit 130 extracts the correct answers and explanations for each problem index and stores them in the database unit 150. The element extraction unit 130 connects the problem index in the problem area in the aforementioned problem area with the problem index in the explanation area (mapping the area with the problem). Therefore, as described above, the problem providing unit 160 compares the location information of the answer check of the user terminal with the location information of the answer area of the question to automatically score the problem.
영역검출부(120)가 페이지 내 문제영역을 탐지하고, 문제영역 내 상호작용이 필요한 영역(예를 들어, 답변영역)을 탐지하며, 문제영역 내 요소들을 탐지하는 작업에는 기 학습된 머신러닝 모델을 이용한다. 이때 이용되는 머신러닝 모델은, Fast RCNN 기반의 머신러닝 모델을 기초로 하였고 이를 통해 CNN 후 분류와 Bounding Box regression에 RPN(Region Proposal Network)을 합친 구조로서 느린 속도를 보완할 수 있었다. 서책 형태의 데이터를 디지털로 변환한 프로젝트(A unified toolkit for Deep Learning Based Document Image Analysis)를 활용하여 해당 데이터를 Transfer learning을 활용하기 위하여 동일한 model인 Faster-RCNN 초기 모델을 구현했으며, iBM PubLayNet 데이터셋 35만장을 가지고 Pre Training을 통해 초기 모델을 설정하였으며, 단일 클래스 Object Detection이며, 서책 이미지 데이터에 적합하도록 하이퍼파라미터 튜닝을 진행했다. 이후, 시중 참고서, 문제집 이미지 Grayscale 및 Image Augmentation 기법을 사용하여 2만장을 Pre Training하여 성능 고도화하였다.The area detection unit 120 detects a problem area within the page, detects an area requiring interaction within the problem area (eg, an answer area), and uses a pre-learned machine learning model to detect elements within the problem area. use The machine learning model used at this time was based on a Fast RCNN-based machine learning model, and through this, it was possible to compensate for the slow speed with a structure that combined classification after CNN and Bounding Box regression with RPN (Region Proposal Network). Using a project that digitally converted book-type data (A unified toolkit for Deep Learning Based Document Image Analysis), the same model, Faster-RCNN initial model, was implemented to utilize transfer learning for the data, and the iBM PubLayNet dataset An initial model was set up through pre-training with 350,000 sheets, single-class object detection, and hyperparameter tuning was performed to suit the book image data. After that, 20,000 sheets were pre-trained using commercial reference books, workbook image grayscale, and image augmentation techniques to enhance performance.
본 발명에 따른 데이터베이스부(150)에는, 본 발명에 따른 시스템이 저작권을 갖고 있거나 저작권에 대한 이용허락을 받은 참고서와, 저작권을 갖고 있지 않고 저작권에 대한 이용허락도 받지 못한 참고서에 대한 정보가 모두 저장될 수 있다. 또는, 본 발명에 따른 디지털 변환 서버(100)는 본 발명에 따른 시스템이 저작권을 갖고 있거나 저작권에 대한 이용허락을 받은 참고서와, 저작권을 갖고 있지 않고 저작권에 대한 이용허락도 받지 못한 참고서에 대한 정보를 포함하는 외부서버(10)에 접근하는 것이 가능하다.In the database unit 150 according to the present invention, information on reference books for which the system according to the present invention has copyrights or is licensed for copyrights and reference books for which copyrights are not held and for which licenses for copyrights have not been obtained are all stored. can be stored Alternatively, the digital conversion server 100 according to the present invention provides information on a reference book for which the system according to the present invention has a copyright or is licensed for copyright, and a reference book that does not have a copyright and is not licensed for copyright. It is possible to access the external server 10 including.
한편, 본 디지털 참고서 제공 시스템(1000)은 저작권판단부(미도시)를 더 포함할 수 있다. 저작권판단부는 이용자단말기에 의해 본 시스템에 업로드된 참고서 파일(PDF 파일) 또는 이용자단말기에 제공해야 할 참고서 파일에 관한 저작권을 본 시스템이 갖고 있는지 여부를 판단할 수 있다. 데이터베이스부(150)는 저장된 모든 참고서 PDF 파일에 대해 본 시스템이 저작권을 갖고 있는 PDF 파일과 저작권을 갖고 있지 않는 PDF 파일을 구분하여 저장한다. 그리고, 저작권판단부는 데이터베이이스부(150)에 저장된 참고서 파일별 저작권 존부 정보를 기초로, 이용자단말기로부터 업로드된 참고서 파일 또는 이용자단말기에 제공할 참고서 파일에 관한 저작권을 본 시스템이 갖고 있는지 여부를 판단할 수 있다.Meanwhile, the digital reference book providing system 1000 may further include a copyright determination unit (not shown). The copyright judging unit may determine whether or not the present system has the copyright of the reference file (PDF file) uploaded to this system by the user terminal or the reference file to be provided to the user terminal. The database unit 150 stores all stored reference PDF files by dividing them into PDF files with copyright and PDF files without copyright in this system. In addition, the copyright determination unit determines whether or not the system has the copyright for the reference file uploaded from the user terminal or the reference file to be provided to the user terminal, based on the copyright existence information for each reference file stored in the database unit 150. can judge
본 시스템이 저작권을 갖고 있는 참고서에 대한 디지털 참고서를 생성하고 제공하는 경우에는 상술한 영역검출부(120), 요소추출부(130)에 의한 과정을 통해 디지털 참고서를 이용자단말기에 제공한다. 반면, 본 시스템이 저작권을 갖고 있지 않은 참고서에 대한 디지털 참고서를 제공해야될 경우에는, 본 시스템은 자동채점만 될 수 있는 등의 저작권에 무관한 추가 레이어만 생성하여 이용자단말기에 제공할 수 있다. 즉, 본 시스템은 문제지 영역과 해설지 영역에서 요소 추출을 진행하지 않고, 이용자단말기로 HTML layer 및/또는 필기 layer만을 제공할 수 있다. 필요에 따라, 영역 검출도 이뤄지지 않고 필기 layer만이 제공될 수도 있다. 즉, 본 시스템에 저작권을 갖고 있지 않고 저작권에 대한 이용허락도 받지 못한 참고서의 PDF 파일을 사용자가 이용자단말기에 저장하여 읽은 경우, 문제제공부(160)는 이용자단말기로 참고서의 문제를 제공하는 HTML layer 및/또는 필기 layer를 제공한다. 사용자가 이용자단말기의 필기 layer를 통해, 즉, 전술한 바와 같이 HTML layer 답변영역 원숫자의 위치값과 동일한 위치의 필기 layer에 답 체크 및 자동 채점 후, 해당 문항에 대한 정답이 불일치한 경우 문제제공부(160)는 그 문항에 해당하는 해설집의 해설영역을 이용자단말기의 화면에 표출한다.When the present system creates and provides a digital reference book for a copyrighted reference book, the digital reference book is provided to the user terminal through the process of the above-described area detection unit 120 and element extraction unit 130. On the other hand, if the present system needs to provide a digital reference book for a reference book that does not have copyright, the system can generate only an additional layer irrelevant to copyright, such as automatic scoring, and provide it to the user terminal. That is, the present system can provide only the HTML layer and/or the handwriting layer to the user terminal without extracting elements from the problem area and the commentary area. If necessary, only the writing layer may be provided without region detection. That is, when a user stores and reads a PDF file of a reference book that does not have copyright in this system and is not licensed for copyright, the problem providing unit 160 provides HTML for providing problems of the reference book to the user terminal. layer and/or handwriting layer. After the user checks the answer through the handwriting layer of the user terminal, that is, the handwriting layer at the same position as the position value of the original number in the HTML layer answer area as described above and automatically scores, if the correct answer to the question is inconsistent, problem solving The study 160 displays the commentary area of the commentary corresponding to the question on the screen of the user terminal.
본 디지털 참고서 제공 시스템(1000)은, 이용자가 디지털 참고서에 대응하는 서책을 적법하게 구매한 것이 맞는지 확인하는 구매인증부(170)를 더 포함할 수 있다. 구매인증부(170)는 이용자단말기로 특정 페이지와 함께 생성한 특정 난수를 송신한다. The present digital reference book providing system 1000 may further include a purchase authentication unit 170 that verifies that the user has legally purchased a book corresponding to the digital reference book. The purchase authentication unit 170 transmits a specific random number generated together with a specific page to the user terminal.
이용자단말기는 구매인증부(170)로부터 수신한 특정 페이지와 동일한 페이지에, 수신한 특정 난수를 기재하고 난수가 기재된 페이지를 촬영한 이미지를 구매인증부(170)로 전송한다. 구매인증부(170)는 이용자단말기가 특정 난수를 기재한 그 페이지에 포함된 요소와, 데이터베이스부(150)에 저장된 참고서의 동일한 페이지에 포함된 요소를 상호비교하여, 동일한 요소가 검출된 경우, 이용자가 서책을 적법하게 구매한 것으로 판단할 수 있다. 따라서, 전술한 바와 같이 추출하여 그룹화된 요소들을 HTML layer로 조합하여 이용자단말기로 제공한다.The user terminal writes the received specific random number on the same page as the specific page received from the purchase authentication unit 170 and transmits an image of the page with the random number to the purchase authentication unit 170 . The purchase authentication unit 170 mutually compares elements included in the page on which the user terminal writes a specific random number with elements included in the same page of the reference book stored in the database unit 150, and when the same element is detected, It can be judged that the user has legally purchased the book. Therefore, the elements extracted and grouped as described above are combined into an HTML layer and provided to the user terminal.
본 디지털 참고서 제공 시스템(1000)은, 참고서 출판사 및/또는 그들의 총판과 특정 참고서를 유통할 수 있는 저작권 이용허락 약정을 체결하고, 유통을 위한 서책을 납품받거나, 참고서 출판사 및/또는 그들의 총판이 구비한 외부서버(10)로부터 그 특정 참고서의 PDF 파일을 수신할 수 있다. This digital reference book providing system 1000 concludes a copyright licensing agreement with reference book publishers and/or their distributors for distributing specific reference books, receives books for distribution, or is provided by reference book publishers and/or their distributors. A PDF file of the specific reference can be received from an external server 10 .
본 발명에 따른 디지털 변환 서버(100)는, 웹사이트 및/또는 어플리케이션 기반의 판매채널과 디지털 변환된 참고서가 작동할 수 있는 어플리케이션(Application)을 구축하고, 구축한 어플리케이션을 이용자단말기로 제공하는 어플리케이션제공부(미도시)를 더 포함할 수 있다. 어플리케이션제공부가 구축한 어플리케이션은 이용자단말기에 설치된다.The digital conversion server 100 according to the present invention builds an application that can operate a website and/or application-based sales channel and a digitally converted reference book, and provides the built application to a user terminal. A provision unit (not shown) may be further included. The application built by the application providing unit is installed in the user terminal.
이용자단말기는, 어플리케이션제공부가 구축한 판매 채널에서 ID를 생성하여 서책 등 참고서를 구매한다. 이용자단말기는 어플리케이션 상에서 ID를 인증하여 디지털 변환된 참고서를 이용할 수 있다. The user terminal purchases a reference book such as a book by generating an ID in a sales channel established by the application providing unit. The user terminal can use the digitally converted reference book by authenticating the ID on the application.
이용자단말기는, 구매한 참고서에 대해서 수령 후 구매확정 절차 완료 시, 구매한 참고서와 동일한 내용의 디지털 변환된 참고서를 열람 및 사용할 수 있는 권한을 획득한다.When the purchase confirmation process is completed after receiving the purchased reference book, the user terminal acquires the right to read and use the digitally converted reference book with the same contents as the purchased reference book.
한편, 본 발명에 따른 어플리케이션제공부가 구축한 어플리케이션은, 다음과 같은 안전장치가 구현되어 있을 수 있다. 먼저 어플리케이션에는, 로그인 및 채널 인증절차를 통해서 참고서의 구매자와 로그인 한 이용자단말기의 소지자가 동일한 지 확인하는 절차가 구현될 수 있다. 또한 어플리케이션에는, 참고서를 구매한 이용자단말기로 취소 불가한 구매확정의 의사표시를 요구하고 구매확정의 의사표시가 있을 시에만 이용자단말기로 디지털 참고서의 접근 권한을 제공하는 절차가 구현될 수 있다. 이는 참고서를 구매하지 않은 이용자에게는 참고서 구매의 부가적인 기능인 디지털 참고서의 이용을 방지하기 위한 것이다. 한편, 어플리케이션은 이용자단말기로 디지털 참고서를 제공할 때, PDF 파일 형태가 아닌 CDN/보안기능을 강화한 프로토콜에서 파편화된 페이지로 제공할 수 있다.On the other hand, the application built by the application providing unit according to the present invention may have the following safety devices implemented. First, in the application, a procedure for verifying whether the purchaser of the reference book and the holder of the logged-in user terminal are the same may be implemented through the login and channel authentication procedures. In addition, in the application, a procedure for requesting an irrevocable expression of intent to confirm purchase from the user terminal that purchased the reference book and providing access to the digital reference book to the user terminal only when there is an expression of intent to confirm purchase may be implemented. This is to prevent users who have not purchased reference books from using digital reference books, which are an additional function of purchasing reference books. On the other hand, when an application provides a digital reference to a user terminal, it can be provided as fragmented pages in a CDN/security-enhanced protocol, not in the form of a PDF file.
도 2는 본 발명의 일 실시예에 따른 디지털 참고서 제공 방법의 순서를 나타낸 도면이다.2 is a diagram showing a sequence of a digital reference book providing method according to an embodiment of the present invention.
도 2를 참조하면, 본 발명의 일 실시예에 따른 디지털 참고서 제공 방법은 PDF 파일에서 복수 개의 페이지를 추출하여 페이지 파일을 생성하는 단계(S10), 페이지 파일에서 문제영역을 검출하는 단계(S20), 문제영역을, 질문영역과 답변영역으로 분리하는 단계(S30), 질문영역 및 답변영역에서 요소를 추출하고 저장하는 단계(S40) 및/또는 추출된 요소를 OCR 처리하여 상기 데이터베이스부에 저장하는 단계(미도시)를 포함할 수 있다.Referring to FIG. 2 , the method for providing a digital reference book according to an embodiment of the present invention includes generating a page file by extracting a plurality of pages from a PDF file (S10), and detecting a problem area in the page file (S20). , Separating the problem area into a question area and an answer area (S30), extracting and storing elements from the question area and answer area (S40), and / or OCR processing the extracted elements and storing them in the database unit Steps (not shown) may be included.
상술한 방법의 각 단계에 대한 상세한 설명은 전술한 디지털 참고서 제공 시스템의 동작에 대한 설명으로 대체한다.A detailed description of each step of the above-described method is replaced with a description of the operation of the above-described digital reference book providing system.
본 명세서에 기재된 실시예들은 서로 동일한 기술 분야에 속하는 것으로서 일 실시예를 구성하는 구성 요소들은 다른 일 실시예를 구성하는 구성 요소들과 결합되어 새로운 실시예를 구성할 수 있다.The embodiments described in this specification belong to the same technical field, and components constituting one embodiment may be combined with components constituting another embodiment to form a new embodiment.
본 명세서에서 파일생성부, 영역검출부, 요소추출부, 광학문자인식부, 데이터베이스부, 문제제공부 및/또는 구매인증부는 메모리에 저장된 연속된 수행과정들을 실행하는 프로세서들일 수 있다. 또는, 프로세서에 의해 구동되고 제어되는 소프트웨어 모듈들로서 동작할 수 있다. 나아가, 프로세서는 하드웨어 장치일 수 있다.In the present specification, the file generation unit, the area detection unit, the element extraction unit, the optical character recognition unit, the database unit, the problem providing unit, and/or the purchase authentication unit may be processors that execute successive processes stored in memory. Or, it can operate as software modules driven and controlled by a processor. Further, a processor may be a hardware device.
참고로, 본 발명의 일 실시예에 따른 디지털 참고서 제공 방법은 다양한 컴퓨터 수단을 통하여 수행될 수 있는 프로그램 명령 형태로 구현되어 컴퓨터 판독가능매체에 기록될 수 있다. 상기 컴퓨터 판독가능매체는 프로그램 명령, 데이터 파일, 데이터 구조 등을 단독으로 또는 조합하여 포함할 수 있다. 상기 매체에 기록되는 프로그램 명령은 본 발명을 위하여 특별히 설계되고 구성된 것들이거나 컴퓨터 소프트웨어 당업자에게 공지되어 사용가능한 것일 수도 있다. 컴퓨터 판독가능매체의 예에는 하드 디스크, 플로피 디스크 및 자기 테이프와 같은 자기 매체, CD-ROM, DVD와 같은 광기록 매체, 플롭티컬 디스크(floptical disk)와 같은 자기-광 매체, 및 ROM, RAM, 플래시 메모리 등과 같은 프로그램 명령을 저장하고 수행하도록 특별히 구성된 하드웨어 장치가 포함될 수 있다. 프로그램 명령의 예에는 컴파일러에 의해 만들어지는 것과 같은 기계어코드뿐만 아니라 인터프리터 등을 사용해서 컴퓨터에 의해서 실행될 수 있는 고급언어코드를 포함한다. 상술한 하드웨어 장치는 본 발명의 동작을 수행하기 위해 하나 이상의 소프트웨어 모듈로서 작동하도록 구성될 수 있으며, 그 역도 마찬가지이다.For reference, the digital reference book providing method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the medium may be those specially designed and configured for the present invention or those known and usable to those skilled in computer software. Examples of computer readable media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROMs, RAMs, A hardware device specially configured to store and execute program instructions, such as flash memory, may be included. Examples of program instructions include not only machine code generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.
본 발명의 보호범위가 이상에서 명시적으로 설명한 실시예의 기재와 표현에 제한되는 것은 아니다. 또한, 본 발명이 속하는 기술분야에서 자명한 변경이나 치환으로 말미암아 본 발명이 보호범위가 제한될 수도 없음을 다시 한 번 첨언한다.The protection scope of the present invention is not limited to the description and expression of the embodiments explicitly described above. In addition, it is added once again that the scope of protection of the present invention cannot be limited due to obvious changes or substitutions in the technical field to which the present invention belongs.

Claims (6)

  1. 디지털 변환 서버를 포함하는, 머신러닝 기반 바운딩 박스를 이용한 디지털 참고서 제공 시스템에 있어서,In a digital reference book providing system using a machine learning-based bounding box, including a digital conversion server,
    상기 디지털 변환 서버는,The digital conversion server,
    디지털 참고서의 PDF(Portable Document Format) 파일에서 문제영역을 검출하는 영역검출부로서, 상기 영역검출부는 상기 문제영역에서 답변영역을 검출하여 상기 문제영역을 질문영역과 답변영역으로 분리하고; 및An area detection unit for detecting a problem area in a PDF (Portable Document Format) file of a digital reference, wherein the area detection unit detects an answer area in the problem area and separates the problem area into a question area and an answer area; and
    상기 질문영역 및 상기 답변영역에서 요소를 추출하고, 상기 추출된 요소를 데이터베이스부에 저장하는 요소추출부를 포함하되,An element extraction unit extracting elements from the question area and the answer area and storing the extracted elements in a database unit,
    상기 영역검출부는,The area detection unit,
    상기 문제영역 내 원숫자가 검출된 영역을 상기 답변영역으로 검출하는, 머신러닝 기반 바운딩 박스를 이용한 디지털 참고서 제공 시스템.A digital reference book providing system using a machine learning-based bounding box that detects an area where a circle number is detected in the problem area as the answer area.
  2. 청구항 1에 있어서,The method of claim 1,
    상기 영역검출부는, 상기 문제영역을 검출할 때,When the region detection unit detects the problem region,
    머신러닝을 이용하여 상기 문제영역을 둘러싼 바운딩박스를 생성하며,Creating a bounding box surrounding the problem area using machine learning,
    상기 영역검출부 및 상기 요소추출부는,The area detection unit and the element extraction unit,
    상기 바운딩박스 내 콘텐츠만으로 데이터를 처리하여, 데이터의 처리 속도를 향상시키고,Processing data only with the content in the bounding box to improve data processing speed;
    상기 영역검출부는, 상기 문제영역을 검출할 때, 페이지 가로방향의 중간점 좌표보다 상기 바운딩박스의 최좌측점 좌표 및 최우측점 좌표의 값이 더 클 때 상기 페이지의 레이아웃이 2열인 것으로 판단하는 것을 특징으로 하는, 머신러닝 기반 바운딩 박스를 이용한 디지털 참고서 제공 시스템.When detecting the problem area, the area detection unit determines that the layout of the page is two columns when values of the coordinates of the leftmost point and the rightmost point of the bounding box are greater than the coordinates of the midpoint in the horizontal direction of the page. Characterized by, a digital reference book providing system using a machine learning-based bounding box.
  3. 청구항 2에 있어서,The method of claim 2,
    상기 디지털 참고서가 수학 과목에 대한 참고서인 경우, 상기 요소추출부는,If the digital reference book is a reference book for mathematics subjects, the element extraction unit,
    상기 요소를 추출할 때 MathOCR을 통해 수식을 추출하고, When extracting the elements, extract the formula through MathOCR,
    상기 디지털 참고서가 국어 또는 영어 과목에 대한 참고서인 경우, 상기 영역검출부는, When the digital reference book is a reference book for Korean or English subjects, the area detection unit,
    상기 문제영역 최상단에서 묶음지문 번호가 검출되면, 상기 묶음지문 및 상기 묶음지문 번호에 따른 문제영역을 포함하도록 상기 바운딩박스를 생성하는 것을 특징으로 하는, 머신러닝 기반 바운딩 박스를 이용한 디지털 참고서 제공 시스템.When a bundled fingerprint number is detected at the top of the problem area, the bounding box is generated to include the problem area according to the bundled fingerprint and the bundled fingerprint number.
  4. 청구항 1에 있어서,The method of claim 1,
    상기 요소추출부는, 상기 원숫자가 포함된 문장을 선택지로 판단하며, 상기 선택지에서 상기 요소를 추출하고, 상기 선택지에서 추출된 요소를 상기 데이터베이스부에 저장하며,The element extraction unit determines a sentence including the original number as an option, extracts the element from the option, and stores the element extracted from the option in the database unit;
    상기 요소추출부는, 상기 질문영역에서 일련의 선택지를 추출하고, 상기 일련의 선택지에서 상기 요소를 추출하며, 상기 일련의 선택지에서 추출된 요소를 상기 데이터베이스부에 저장하는 것을 특징으로 하는, 머신러닝 기반 바운딩 박스를 이용한 디지털 참고서 제공 시스템.The element extraction unit extracts a series of options from the question area, extracts the elements from the series of options, and stores the elements extracted from the series of options in the database unit, based on machine learning. Digital reference book providing system using bounding box.
  5. 청구항 4에 있어서,The method of claim 4,
    상기 요소추출부는, The element extraction unit,
    상기 질문영역에서 일련의 선택지를 추출할 때, When extracting a series of options from the question area,
    상기 질문영역 내에서 'ㄱ. ㄴ. ㄷ.', '(ㄱ) (ㄴ) (ㄷ)', '가. 나. 다.' 또는 '(가) (나) (다)' 가 일련으로 검출된 영역 내 문장들을 상기 일련의 선택지로 추출하는 것을 특징으로 하는, 머신러닝 기반 바운딩 박스를 이용한 디지털 참고서 제공 시스템.Within the above question area, 'a. you. c.', '(a) (b) (c)', 'a. me. all.' Or, a digital reference book providing system using a machine learning-based bounding box, characterized in that sentences within a region in which '(a) (b) (c)' are detected in series are extracted as the series of options.
  6. 디지털 참고서에서 문제영역을 검출하는 단계;Detecting problem areas in digital reference books;
    상기 문제영역에서 답변영역을 검출하여 상기 문제영역을 질문영역과 답변영역으로 분리하는 단계로서,A step of detecting an answer area in the problem area and separating the problem area into a question area and an answer area,
    상기 문제영역 내 원숫자가 검출된 영역을 상기 답변영역으로 검출하고; 및detecting an area in which the original number is detected in the problem area as the answer area; and
    상기 질문영역 및 상기 답변영역에서 요소를 추출하고, 상기 추출된 요소를 데이터베이스부에 저장하는 단계를 포함하는, 머신러닝 기반 바운딩 박스를 이용한 디지털 참고서 제공 방법.A method for providing a digital reference book using a machine learning-based bounding box, comprising extracting elements from the question area and the answer area and storing the extracted elements in a database unit.
PCT/KR2022/016531 2021-10-28 2022-10-27 Machine learning-based digital reference book provision system using bounding boxes WO2023075434A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR10-2021-0145534 2021-10-28
KR1020210145534A KR102473059B1 (en) 2021-10-28 2021-10-28 Digitized reference book element extraction system and method the same

Publications (1)

Publication Number Publication Date
WO2023075434A1 true WO2023075434A1 (en) 2023-05-04

Family

ID=84413185

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2022/016531 WO2023075434A1 (en) 2021-10-28 2022-10-27 Machine learning-based digital reference book provision system using bounding boxes

Country Status (2)

Country Link
KR (1) KR102473059B1 (en)
WO (1) WO2023075434A1 (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102591757B1 (en) * 2023-02-14 2023-10-20 주식회사 테스트뱅크 Methods and devices for converting pdf files to semantic html format for producing digital reference books
KR102610681B1 (en) * 2023-03-31 2023-12-06 주식회사 테스트뱅크 Methods and devices for generating metadata for the content of questions contained in the reference book

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20060041465A (en) * 2004-11-09 2006-05-12 김대희 On-line grading system by mobile camera phone
KR20160027862A (en) * 2014-09-02 2016-03-10 삼성전자주식회사 Method for processing image data and electronic device supporting thereof
KR101985612B1 (en) * 2018-01-16 2019-06-03 김학선 Method for manufacturing digital articles of paper-articles
KR20210001412A (en) * 2019-06-28 2021-01-06 한양대학교 에리카산학협력단 System and method for providing learning service
KR102256667B1 (en) * 2020-03-23 2021-05-26 주식회사 신한디에스 Method for recognizing documents and apparatus therefor

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20060041465A (en) * 2004-11-09 2006-05-12 김대희 On-line grading system by mobile camera phone
KR20160027862A (en) * 2014-09-02 2016-03-10 삼성전자주식회사 Method for processing image data and electronic device supporting thereof
KR101985612B1 (en) * 2018-01-16 2019-06-03 김학선 Method for manufacturing digital articles of paper-articles
KR20210001412A (en) * 2019-06-28 2021-01-06 한양대학교 에리카산학협력단 System and method for providing learning service
KR102256667B1 (en) * 2020-03-23 2021-05-26 주식회사 신한디에스 Method for recognizing documents and apparatus therefor

Also Published As

Publication number Publication date
KR102473059B1 (en) 2022-12-02

Similar Documents

Publication Publication Date Title
WO2023075434A1 (en) Machine learning-based digital reference book provision system using bounding boxes
US5960448A (en) System and method for displaying a graphically enhanced view of a region of a document image in which the enhanced view is correlated with text derived from the document image
Bansal et al. Segmentation of touching and fused Devanagari characters
CN107656922A (en) Translation method, translation device, translation terminal and storage medium
JP5661663B2 (en) Information extraction device
JP2014102669A (en) Information processor, information processing method and program
CN101661465A (en) Image processing apparatus, image processing method and image processing program
CN107479791A (en) Associated annotation information determination method and device, intelligent teaching equipment and storage medium
Harit et al. Table detection in document images using header and trailer patterns
CN111753120B (en) Question searching method and device, electronic equipment and storage medium
WO2020060019A1 (en) Character detection device, method, and system
CN111738252B (en) Text line detection method, device and computer system in image
US11494588B2 (en) Ground truth generation for image segmentation
JP2021043478A (en) Information processing device, control method thereof and program
KR102542174B1 (en) Digital reference book provision system
KR20210077251A (en) Database building device that can build a knowledge database from a table-inserted image and operating method thereof
KR101638511B1 (en) Computer readable medium recording program for authoring online learning contents and d method of authoring online learning contents
WO2023149618A1 (en) System and method for providing digital reference book according to copyright ownership
WO2023149617A1 (en) System for providing digital reference book based on page edition and delimiter detection, and method thereof
KR102542170B1 (en) Digital reference book provision system by authentication of purchase of real reference books
JP4807618B2 (en) Image processing apparatus and image processing program
JP3171626B2 (en) Character recognition processing area / processing condition specification method
Li et al. A platform for creating Smartphone apps to enhance Chinese learning using augmented reality
JP7107138B2 (en) Information processing device, information processing method and program
Lawrie et al. Building OCR/NER test collections

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22887630

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE