KR20220084679A - Method for providing drawing reading and search function and apparatus and system thereof - Google Patents

Abstract

A method for providing a drawing reading and retrieval function is provided. The method for providing a drawing reading and searching function includes the steps of detecting a position of a reference number in a drawing; and extracting reference numbers in text format from reference numbers in which the position is sensed.

Description

Drawing comprehension and search function providing method and device and system

The present invention relates to a method for providing a drawing reading and retrieval function, and an apparatus and system therefor, and more specifically, by detecting the position of the reference number in the drawing in the patent specification, extracting and recognizing the reference number in text format, and extracting the recognized reference number and A method of providing a drawing reading and search function that enables a user to easily grasp the invention by linking and displaying the descriptions of reference numerals in the detailed description of the invention, and enables the drawing to be searched using technical keywords and literature information, etc. and an invention related to the device and system.

In general, when a user intends to grasp an invention through a patent specification and a utility model specification, the drawings are grasped together while reading the detailed description of the invention included in the specification. Reference numerals displayed in the drawings of the specification are indicated as numbers, and since only reference numbers indicated by numbers exist in the drawings, there is inevitably a limit to the user's ability to grasp the invention only with the drawings.

In addition, when an invention is searched for through a technical keyword to be found on a site where a patent can be searched, the contents of the invention containing the keyword are displayed, but there is no method of searching for drawings including reference numerals corresponding to the keyword. to be.

In addition, even if an invention is searched for through the keyword of the invention to be found and an invention related to a specification including a drawing containing a reference number corresponding to the keyword is searched, the part corresponding to the keyword that I want to find in the drawings of the searched invention specification is not found. There is a difficulty in figuring out the detailed description of the invention, such as the description of the code, where it is located.

In addition, even if you want to search for another invention that includes a part having a function or technology similar to the part indicated by the reference number in the drawing in which the reference number is indicated, after confirming the name of the reference number through the detailed description of the invention, the name The hassle of having to search again arises.

In addition, the user can easily edit text, figure, color, etc. as needed in a drawing in which only reference signs are displayed or drawings that are displayed together with reference signs and descriptions of reference signs, and save the edited images separately. There is no device or system that can be used.

In addition, there is a problem in that even if descriptions corresponding to the reference numerals in the drawings are displayed in the related art, the reference numerals are not detected or are incorrectly detected, so that the reference numerals and the corresponding description are not accurately displayed.

Therefore, in order to solve this problem, a description of the corresponding reference number can be displayed in the reference number of the drawing, a drawing containing the reference number of the technical keyword to be found can be searched for, and only the drawing in which the reference number is displayed can correspond Users can easily search for other inventions that include parts having functions or technologies similar to those indicated by reference numerals, and users can view drawings in which only reference numbers are displayed or drawings that are displayed together with reference numbers and descriptions of the reference numbers. can easily edit text, figure, color, etc. as needed, save edited images, etc. separately for use The need for a method for providing a function and an apparatus and a system therefor is gradually increasing.

The present invention has been devised to solve the above problems, and the present invention detects the position of a reference in a drawing, extracts a reference number in text form from the detected reference number, and explains the reference number to the extracted reference number An object of the present invention is to provide a method for providing a drawing reading comprehension and search function capable of displaying

In addition, according to the present invention, by searching for a technical keyword to be found, it is possible to easily search for drawings including reference numbers related to the corresponding technology keywords, and only the drawings in which the reference numbers are displayed have functions similar to the parts indicated by the reference numbers or An object of the present invention is to provide a method for providing a drawing comprehension and retrieval function, and an apparatus and system therefor, which can easily search for other inventions including parts having the technology.

In addition, the present invention provides a drawing in which only reference signs are displayed, drawings in which descriptions of reference signs are displayed, or drawings in which reference signs and descriptions of reference signs are displayed together with text, figures, colors, etc. according to the user's convenience. An object of the present invention is to provide a method for providing a drawing reading and retrieval function that can be edited and the edited image can be separately stored and used, and an apparatus and system therefor.

In addition, the present invention extracts reference numbers in text form from reference numbers, and maps the extracted reference numbers and reference numbers described in the claims, so that the user can easily understand the contents of the invention and a drawing reading and search function An object of the present invention is to provide a method for providing and an apparatus and system for the same.

In addition, the present invention provides a method for providing a drawing comprehension and search function capable of displaying or interlocking a description corresponding to a reference number in a drawing by accurately detecting a reference number in a drawing and extracting a reference number in text form, and a method thereof It aims to provide devices and systems.

In addition, the present invention can generate a reference number list by processing the reference mark in which the description corresponding to the reference number is displayed or interlocked, and drawing reading that can generate a search expression through the reference number or the generated reference number list An object of the present invention is to provide a method for providing a search function and an apparatus and system therefor.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

A method for providing a drawing reading and searching function according to an embodiment of the present invention for solving the above technical problem includes the steps of: detecting a position of a reference in a drawing; and extracting reference numbers in text format from reference numbers in which the position is sensed.

Also, the method for providing a drawing reading comprehension and search function may further include pre-processing an image of a drawing.

In addition, the method of providing a drawing reading comprehension and search function may further include the step of associating the reference numerals in the extracted text format with the description corresponding to the reference numerals.

In addition, the method of providing a drawing comprehension and search function comprises the steps of: generating a reference number list by processing the reference number in the text format in which the description corresponding to the reference number is linked; And it may further include the step of generating a search expression through the reference number in the text format or the generated reference number list linked with the description corresponding to the reference number.

In a computer-readable recording medium according to another embodiment of the present invention for solving the above technical problem, a computer program for performing the above methods may be stored.

An apparatus for providing a drawing reading and searching function according to an additional embodiment of the present invention for solving the above technical problem includes: a reference number position detecting unit configured to detect the position of a reference number in the drawing; and a text recognition unit configured to extract a reference number in text format from the reference number for which the position is sensed.

In addition, the apparatus for providing a drawing reading comprehension and search function may further include an image pre-processing unit configured to pre-process an image of a drawing.

In addition, the apparatus for providing a drawing comprehension and search function may further include a reference reference linkage unit configured to link the extracted and recognized text format reference numbers with the description corresponding to the reference numbers.

In addition, the drawing comprehension and search function providing apparatus may include: a processing unit configured to generate a reference number list by processing the text-type reference numerals linked to the description corresponding to the reference numerals; And it may further include a patent search unit configured to generate a search expression through the reference number or the generated reference number list in a text format in which the description corresponding to the reference number is linked.

According to the method for providing a drawing reading and searching function according to an embodiment of the present invention, and an apparatus and system therefor, by detecting the position of a reference number in a drawing, extracting a textual reference number from the detected reference number, and adding the extracted reference number to the reference number. A description of the corresponding reference number may be displayed.

In addition, according to the method for providing a drawing reading and search function according to an embodiment of the present invention, and an apparatus and system therefor, it is possible to easily search for a drawing including reference numerals related to the technology keyword by searching for a technology keyword to be found. In addition, it is possible to easily search for other inventions including parts having functions or technologies similar to the parts indicated by the corresponding reference numbers only in the drawings in which the reference numerals are indicated.

In addition, according to the method for providing a drawing reading and search function according to an embodiment of the present invention, and an apparatus and system therefor, a drawing in which only reference numerals are displayed, a drawing in which a description of a reference number is displayed, or a reference to a reference number and a reference number The user can easily edit the text, shape, color, etc. according to the need of the drawing, which is displayed together with the description, and the edited image can be saved and used separately.

In addition, according to the method for providing a drawing reading and searching function according to an embodiment of the present invention, and an apparatus and system therefor, a reference number in text form is extracted from a reference number, and the extracted reference number and reference number described in the claims By mapping the , it is possible for the user to easily grasp the contents of the invention.

In addition, according to the method for providing a drawing reading and retrieval function according to an embodiment of the present invention, and an apparatus and system therefor, by accurately detecting a reference number in a drawing and extracting a reference number in text form, the reference number is added to the reference number of the drawing It is possible to display or link a description corresponding to .

In addition, according to the method for providing a drawing reading and searching function according to an embodiment of the present invention, and an apparatus and system therefor, it is possible to generate a reference number list by processing reference numbers in which descriptions corresponding to reference numbers are displayed or interlocked, A search expression can be created through a reference number or a generated reference number list.

In order to more fully understand the drawings cited in the Detailed Description, a brief description of each drawing is provided.
1 is a schematic block diagram of a system 1000 for providing a drawing reading and searching function according to an embodiment of the present invention.
Figure 2 (a) is a flowchart for explaining a process of detecting a reference number in a drawing and extracting a reference number in text format by the method and the apparatus for providing a drawing reading and searching function according to an embodiment of the present invention; 2(b) is a schematic diagram for explaining a method for providing a drawing comprehension and search function according to an embodiment of the present invention, and a process by which the apparatus detects reference numbers in a drawing and extracts reference numbers in text format.
Figure 3 (a) is a block diagram for explaining the reference numeral position detection unit 240 and the image pre-processing unit 241 according to an embodiment of the present invention, Figure 3 (b) is an embodiment of the present invention It is a flowchart for explaining an image pre-processing step according to an example, and FIG. 3 (c) is a schematic diagram for explaining a detailed process of the image pre-processing step according to an embodiment of the present invention.
Figure 4 (a) is an exemplary view for explaining an example in which the reference number is sensed by the reference number position detection unit 240 according to an embodiment of the present invention, Figure 4 (b) is an example of the present invention It is an exemplary diagram for explaining an example in which the reference number is not detected by the reference number position detecting unit 240 according to the embodiment.
5 is an algorithm and algorithm tuning used for the apparatus 200 for providing a drawing reading comprehension and search function according to an embodiment of the present invention to detect the position of a reference number and to extract a reference number in text form from the detected reference number. It is the experimental result data to explain the process.
6 is an experimental result data according to the algorithm tuning version of the reference numeral position detecting unit 240 according to an embodiment of the present invention.
7 is a schematic block diagram for explaining the processing unit 270, the patent search unit 280, and the mapping unit 290 of the drawing reading and searching function providing system 1000 according to an embodiment of the present invention.
8 is a schematic block flow diagram illustrating each step performed by a method for providing a drawing reading and searching function and an apparatus according to an embodiment of the present invention.
9 is an exemplary view for explaining a drawing information providing screen generated by the processing unit 270 according to an embodiment of the present invention.
10 is an exemplary diagram for explaining a method for providing a drawing reading and searching function and an exemplary drawing search method performed by the apparatus according to an embodiment of the present invention.
11 is an exemplary diagram for explaining mapping between reference numerals in a drawing and reference numerals described in the claims by the method and apparatus for providing a drawing reading and searching function according to an embodiment of the present invention.
12 is a method for providing a drawing comprehension and search function according to an embodiment of the present invention and a word cloud in which the degree of association between the technical keywords of the reference numerals described in the claims and the technical keywords of the entire specification is provided by the device. It is an example diagram for explanation.

Hereinafter, embodiments according to the present invention will be described with reference to the accompanying drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the embodiment of the present invention, if it is determined that a detailed description of a related known configuration or function interferes with the understanding of the embodiment of the present invention, the detailed description thereof will be omitted. In addition, the embodiments of the present invention will be described below, but the technical spirit of the present invention is not limited thereto and may be variously implemented by those skilled in the art.

Throughout the specification, when a part is "connected" with another part, this includes not only the case where it is "directly connected" but also the case where it is "indirectly connected" with another element interposed therebetween. . Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, in describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms.

1 is a schematic block diagram of a system 1000 for providing a drawing reading and searching function according to an embodiment of the present invention. As shown in FIG. 1 , the drawing reading and searching function providing system 1000 according to an embodiment of the present invention includes a patent information server 100 , a drawing reading and searching function providing device 200 , and a user terminal 300 . and so on.

The patent information server 100 according to an embodiment of the present invention may be configured to transmit/receive reference number information to and from the drawing reading and searching function providing apparatus 200 . In addition, the patent information server 100 according to an embodiment of the present invention may store or store a vast amount of data related to patent documents, utility model documents, invention data, patent specifications, utility model specifications, drawings, and the like.

In addition, the user terminal 300 according to an embodiment of the present invention may be configured to receive information about reference numbers in a text format in which descriptions corresponding to reference numbers are linked from the drawing reading comprehension and search function providing apparatus 200 . have.

For reference, the user terminal 300 includes a subscriber unit, a subscriber station, a mobile station, a mobile terminal, a remote station, a remote terminal, a mobile device, an access terminal, a terminal, a wireless communication device, a user agent, a user device, a user equipment (UE), It may also be referred to as a PC, computer, personal computer, or personal PC. An access terminal is a cellular phone, cordless phone, Session Initiation Protocol (SIP) phone, wireless local loop (WLL) station, personal digital assistant (PDA), handheld device with wireless connectivity capability, computing device or wireless modem connected to It may be another processing device.

In addition, as shown in FIG. 1 , the apparatus 200 for providing a drawing reading and searching function according to an embodiment of the present invention includes a control unit 210 , a communication unit 220 , a storage unit 230 , and a reference position detection unit. 240 , a text recognition unit 250 , a reference reference linkage unit 260 , and the like.

In addition, the drawing comprehension and search function providing apparatus 200 according to an embodiment of the present invention corresponds to a device based on its own computer program produced and operated by an individual or company, and according to various expression methods in the related art field It may be referred to as a term such as a device, equipment, server, and platform.

For reference, the components 210 , 220 , 230 , 240 , 250 and 260 of the apparatus 200 for providing a drawing reading and searching function shown in FIG. 1 provide a drawing reading and searching function according to an embodiment of the present invention. These are merely exemplary components for describing the operation, function, and the like of the device 200 . That is, other components (eg, a power supply unit) other than the components ( 210 , 220 , 230 , 240 , 250 and 260 ) shown in the drawing reading and searching function providing apparatus 200 according to an embodiment of the present invention etc.) may be further included.

The control unit 210 according to an embodiment of the present invention includes a communication unit 220 , a storage unit 230 , a reference position detection unit 240 , a text recognition unit 250 , a reference reference linkage unit 260 , and processing It may be configured to control the configuration of the drawing reading and search function providing apparatus 200 such as the unit 270 (refer to FIG. 7), the patent search unit 280 (refer to FIG. 7), and the mapping unit 290 (refer to FIG. 7). .

For reference, the controller 210 may be implemented as a processor, a micro-processor, a controller, a micro-controller, or the like.

In addition, according to an embodiment of the present invention, the communication unit 220 transmits all information sensed, extracted, processed, linked or generated by the drawing comprehension and search function providing apparatus 200 to the patent information server 100 or the user terminal. It may be configured to transmit and receive with 300 .

For reference, the communication unit 220 is provided for direct connection with the outside or connection through a network, and may be a wired and/or wireless communication unit 220 . Specifically, the communication unit 220 includes a control unit 210 , a storage unit 230 , a reference position detection unit 240 , a text recognition unit 250 , a reference description linkage unit 260 , a processing unit 270 , The control unit 210, the storage unit 230, the reference position detection unit ( 240 ), the text recognition unit 250 , the reference description linkage unit 260 , the processing unit 270 , the patent search unit 280 , or the mapping unit 290 , or may be stored in the storage unit 230 . The data may include content such as text, images, and moving images.

Communication unit 220 is a LAN, WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), WiBro (Wireless Broadband Internet), RF (Radio Frequency) communication, wireless LAN (Wireless LAN), Wi-Fi (Wireless) Fidelity), NFC (Near Field Communication), Bluetooth, infrared communication, etc. However, this is an example, and according to an embodiment to which the present invention is applied, various wired and wireless communication technologies applicable in the art may be used.

The storage unit 230 according to an embodiment of the present invention may store all information sensed, extracted, processed, linked or generated by the drawing reading comprehension and search function providing apparatus 200 . Illustratively, the storage unit 230 according to an embodiment of the present invention includes patent specification data, drawing data, reference number data, reference number position data, reference number position coordinate data, reference number data in text format, and reference number. Corresponding explanatory data, reference code in text format and interlocking data of description corresponding to the reference number, reference number list data generated by processing the reference number in text format in which the description corresponding to the reference number is linked, and the reference number corresponding to the reference number Search-type data generated through a reference number or reference number list in text format linked with description, image pre-processing data of drawings, image pre-processing method data of drawings, mapping data of reference numbers and reference numbers described in claims, reference information Information such as drawing data edited on the provision screen, drawing editing method data, drawing search method data through technical keywords or document numbers, etc. may be stored.

In addition, information stored in the storage unit 230 according to an embodiment of the present invention may be shared with the patent information server 100 or the user terminal 300 by the communication unit 220 . Similarly, information stored in the patent information server 100 or the user terminal 300 may also be shared with the storage unit 230 .

For reference, the storage unit 230 is, as known to those skilled in the art, a hard disk drive (HDD), read only memory (ROM), random access memory (RAM), electrically erasable and programmable read only memory (EEPROM), flash Various types of storage devices that allow input and output of information, such as flash memory, CF (Compact Flash) card, SD (Secure Digital) card, SM (Smart Media) card, MMC (Multimedia) card, or Memory Stick may be implemented as shown in FIG. 1 , and may be provided inside the drawing comprehension and search function providing apparatus 200 or may be provided in a separate external device.

According to an embodiment of the present invention, the apparatus 200 for providing a drawing comprehension and search function may include a reference number position detecting unit 240 configured to detect the positions of reference symbols in the drawings. Reference number position detection unit 240 according to an embodiment of the present invention may detect the position coordinates of the reference number in the drawing based on a deep learning or machine learning algorithm. In addition, the reference number position detecting unit 240 according to an embodiment of the present invention may detect reference signs displayed in the drawing, and detect the location coordinates and the location area.

For example, the reference number position detecting unit 240 may detect an object such as a reference number and use the RetinaNet algorithm to detect the position of the reference number. In general, dense object detection models based on deep learning can be broadly classified into two types. One is a one-stage network object detection represented by YOLO, SSD, etc., and the other is object detection in a secondary network (R-CNN series) in which region proposal and object classification are separated. In general, primary network object detection is faster and performance is lower than secondary network object detection.

The cause of this phenomenon is the extreme class imbalance problem. First-order network object detection samples candidate positions from an image or an entire image, in which most images or images have a meaningful object occupying only a portion of the image or image, and most of the rest is a background independent of the object. Therefore, learning is inefficient and non-general as it learns most of these easily classified negative samples for most of the time in the learning process. On the other hand, the secondary network is less affected by this problem because it can filter excessive samples in the region proposal stage.

로 구성될 수 있다. 기존의 크로스 엔트로피 로스 함수와 비교했을 때 초점 손실 함수는 변조 인수

가 추가되어 이러한 변조 인수가 쉬운 부정적인 샘플의 중요도를 낮추며 어려운 샘플들을 학습하는데 집중하게 만들 수 있다.

은 조절 가능한 초점 인수이며 0보다 큰 값이어야 한다.

값이 클수록 쉬운 부정적인 샘플이 모델에 미치는 영향이 줄어들 수 있다. 초점 손실 함수는 잘 분류된 예제들에 대해서는 작은 가중치를 부여하는 반면 분류하기 어려운 일부 예제들에게는 큰 가중치를 부여하여 학습을 어려운 예제에 집중시킬 수 있다.RetinaNet can propose a focal loss function instead of the commonly used cross entropy loss function to solve the extreme class imbalance problem. The focus loss function is

can be composed of Compared with the conventional cross-entropy loss function, the focus loss function is a modulation factor

can be added that these modulation factors reduce the importance of easy negative samples and focus on learning difficult samples.

is an adjustable focus factor and must be greater than zero.

Larger values can reduce the influence of easy negative samples on the model. The focus loss function can focus the learning on the difficult examples by giving small weights to well-classified examples while giving large weights to some examples that are difficult to classify.

Therefore, it is possible to solve the problem that learning is overwhelmed by most negative samples that are easily classified. In addition, by using the focus loss function, even the first-order network can achieve better performance than the second-order network. RetinaNet is a simple first-order network object detection using an anchor box based on FPN (Feature Pyramid Network). It is as fast as first-order network object detection, but it can show accuracy that surpasses all existing best performing object detection algorithms. . Reference number position detection unit 240 according to an embodiment of the present invention may detect the reference number position by tuning the RetinaNet algorithm. The configuration and method for the reference position detection unit 240 to tune the RetinaNet algorithm will be described in more detail in FIG. 5 .

Also, the reference number position detecting unit 240 according to an embodiment of the present invention may generate a bounding box in the reference number of which the position is sensed. The bounding box generated by the reference number position detecting unit 240 may have a rectangular border shape formed around the reference number where the position is sensed. However, the bounding box may not be visually exposed. For example, the bounding box may be configured with a transparent color.

In addition, the reference symbol position detection unit 240 according to an embodiment of the present invention, unlike the general image data processing object recognition technology, object recognition technology using RetinaNet specialized for drawings of the patent specification consisting of only black and white figures and numbers. By using it, it is possible to obtain the coordinate information of the reference number by figuring out where the reference number is located in the drawing. Thereafter, the text recognition unit 250 may extract a reference number located at the corresponding coordinate as a reference number in a text format using OCR (Optical Character Recognition) technology specialized for drawings in the patent specification.

Meanwhile, according to an embodiment of the present invention, the text recognition unit 250 may extract a reference number in text format based on the bounding box generated in the reference number by the reference symbol position detecting unit 240 . According to an embodiment of the present invention, the apparatus 200 for providing a drawing comprehension and search function may include a text recognition unit 250 configured to extract reference numbers in text format from reference numbers for which a position is detected.

The text recognition unit 250 according to an embodiment of the present invention may extract a reference number in text format from an area within a bounding box generated in the reference number by the reference number position detecting unit 240 . In addition, the text recognition unit 250 according to another embodiment of the present invention is a text-type drawing in the entire area of the image of the drawing regardless of the area within the bounding box generated in the reference number by the reference number position detection unit 240 . code can be extracted. In addition, the text recognition unit 250 according to another embodiment of the present invention may be configured to generate a bounding box in an area corresponding to a reference number in a text format in an image format drawing.

Also, according to an embodiment of the present invention, the text recognition unit 250 may extract a reference number in text format from reference numbers whose positions are detected based on a deep learning algorithm or a machine learning algorithm. Illustratively, the text recognition unit 250 according to an embodiment of the present invention may use an OCR algorithm called ASTER to extract textual reference numbers from reference numbers.

For reference, ASTER is an OCR algorithm. A tricky aspect when recognizing text from an image is recognizing text that has a distorted or irregular layout. In particular, text with perspective and curved text are common in images and are difficult to recognize.

ASTER is an end-to-end neural network model consisting of a rectification network and a recognition network. ASTER's rectification network can modify the text by transforming the input image into a new one, and can be driven by flexible TPS (Thin-Plate Spline) transformations that handle various text irregularities and learn without human annotation. Recognition networks are attentional sequence-to-sequence models that can predict character sequences directly from modified images. This overall ASTER model is trained end-to-end and may only need images and the text on which they are based.

The text recognition unit 250 according to an embodiment of the present invention may extract reference numbers in text format from reference numbers whose positions are detected by tuning the ASTER algorithm. The configuration and method of the text recognition unit 250 tuning the ASTER algorithm will be described in more detail with reference to FIG. 5 .

According to an embodiment of the present invention, the drawing reading comprehension and search function providing apparatus 200 further includes a reference number description interlocking unit 260 configured to link the extracted text format reference numbers with the description corresponding to the reference numbers. may include

The reference number description linkage unit 260 according to an embodiment of the present invention may be configured to inquire a description corresponding to the reference number in text format extracted from the description DB of the reference number. The code description DB may be stored in the patent information server 100 or the storage unit 230 . The reference number description linkage unit 260 according to an embodiment of the present invention inquires for a description corresponding to the reference symbol in the text format extracted from the description DB of the symbol, and provides a description of the reference number that matches the reference number in the extracted text format. It can be configured to link. For reference, the code description DB may be stored in the patent information server 100 or the storage unit 230 .

Exemplarily, if the reference number of the extracted text format is '10' and there is '10: fixed unit' in the description DB of the symbol, the reference description linkage unit 260 inquires the description DB of the symbol, and the extracted text It is possible to link '10: fixed unit' that matches the reference numeral '10' of the format. A detailed configuration in which the reference numeral description interlocking unit 260 according to an embodiment of the present invention interlocks the extracted textual reference numerals with the description corresponding to the reference numerals will be described in more detail with reference to FIG. 9 to be described later.

In addition, the reference reference linkage linker 260 according to an embodiment of the present invention may be configured to inquire in the detailed description of the invention in the specification. In addition, the reference reference interlocking unit 260 may be configured to link the reference number in the text format extracted by creating a pattern in the detailed description of the invention and the description of the corresponding reference number.

In addition, the reference number description linkage unit 260 according to an embodiment of the present invention provides a detailed description of the invention when reference numbers matching the reference numbers in the extracted text format are not queried in the description DB of the reference numbers or the detailed description of the invention. It may be configured to link the extracted textual reference numerals with the description corresponding to the reference numerals by using the surrounding sentences of the positions of the parts matching the extracted textual reference numerals in the description.

Illustratively, the reference reference interlocking unit 260 interlocks the extracted textual reference number with the part predicted to be a description corresponding to the textual reference number extracted from the detailed description of the invention or the portion determined to be most appropriate. It can be configured to

Or, the reference reference linkage unit 260 is a description corresponding to the reference number by combining several sentences in the vicinity of the location of the part coincident with the reference number in text format extracted from the detailed description of the invention can be created, and it can be configured to link the description corresponding to the generated reference number with the reference number in the extracted text format.

Alternatively, the reference number description linkage unit 260 interlocks the extracted textual reference number with a part predicted to be a description corresponding to the textual reference number extracted from the detailed description of the invention or a plurality of portions determined to be appropriate. It can be configured to

Unlike the existing technology of displaying reference numerals and descriptions corresponding to the reference numbers, the reference reference linkage unit 260 according to an embodiment of the present invention provides not only words such as names, but also adjectives, adverbs, etc. Including, it can be configured to extract/generate a description corresponding to a reference number in a syntax unit and link it with a reference number in text format.

Figure 2 (a) is a flowchart for explaining a process of detecting a reference number in a drawing and extracting a reference number in text format by the method and the apparatus for providing a drawing reading and searching function according to an embodiment of the present invention; 2(b) is a schematic diagram for explaining a method for providing a drawing comprehension and search function according to an embodiment of the present invention, and a process by which the apparatus detects reference numbers in a drawing and extracts reference numbers in text format.

The method for providing a drawing comprehension and search function according to an embodiment of the present invention can detect the position of a reference number in the drawing (S10), and extract the reference number in text form from the reference number where the position is detected (S20). have. Subsequently, the extracted textual reference numerals may be linked with the description corresponding to the reference numerals ( S30 ).

In addition, as described above, the step of detecting the position of the reference number in the drawing performed by the reference position detection unit 240 according to an embodiment of the present invention may be performed through a tuned RetinaNet algorithm. As a detection model, the tuned RetinaNet algorithm can detect the position coordinates of the reference numbers predicted from the position coordinates of the drawing image source and the target reference number.

In addition, as described above, the step of extracting the reference number in text format from the reference number in which the position is detected performed by the text recognition unit 250 according to an embodiment of the present invention may be performed by the tuned ASTER algorithm. have. As an OCR model, the tuned ASTER algorithm can extract the description of the reference number predicted from the drawing image source, the source of the position coordinates of the reference number, and the description of the target reference number.

In addition, the step of detecting the position of the reference number in the drawing performed by the drawing comprehension and search function providing apparatus 200 according to an embodiment of the present invention and extracting the reference number in text form from the detected reference number The steps may be made sequentially.

More specifically, the step of detecting the position of a reference number in the drawing performed by the apparatus 200 for providing a drawing comprehension and search function according to an embodiment of the present invention and a reference number in text form from the reference number detected in the position The extracting step may be configured as a pipeline. For reference, a pipeline is a piece of hardware designed to parallelize several special tasks while executing commands to increase system efficiency. is a technique

Figure 3 (a) is a block diagram for explaining the reference numeral position detection unit 240 and the image pre-processing unit 241 according to an embodiment of the present invention, Figure 3 (b) is an embodiment of the present invention It is a flowchart for explaining an image pre-processing step according to an example, and FIG. 3 (c) is a schematic diagram for explaining a detailed process of the image pre-processing step according to an embodiment of the present invention.

According to an embodiment of the present invention, the apparatus 200 for providing a drawing reading and searching function may further include an image pre-processing unit 241 configured to pre-process an image of a drawing. The image pre-processing unit 241 is shown in FIG. 3 As shown in (a) of (a), it may be provided in the reference position detection unit 240, or may be separately provided in the drawing reading and search function providing device 200 separately from the reference number position detection unit 240 have.

In addition, in the method for providing a drawing reading and searching function according to an embodiment of the present invention, the step of detecting the position of the reference number in the drawing (S10) is pre-processing the image of the drawing (S11) and then detecting the position of the reference number (S12) ) can be done by Alternatively, according to another embodiment of the present invention, the method for providing a drawing comprehension and search function detects the position of a reference number in the drawing after pre-processing the image of the drawing, and extracts the reference number in text form from the detected reference number. can

The image preprocessor 241 according to an embodiment of the present invention may preprocess the image of the drawing by adjusting color jitter of the image of the drawing. Illustratively, the image pre-processing unit 241 according to an embodiment of the present invention uses hue jitter of the image of the drawing as a default, and calculates brightness jitter and saturation jitter. You can preprocess the image by adjusting the contrast jitter by 2 to default and 0, respectively.

According to an embodiment of the present invention, when the image pre-processing unit 241 adjusts the color jitter of the image in the drawing to the basic and zero, respectively, the brightness jitter and the saturation jitter to the basic and zero, and the contrast jitter is doubled, the reference mark position is detected The unit 240 may most accurately detect the position of the reference number in the drawing, and the text recognition unit 250 may also extract the reference number in text form from the reference number where the position is most accurately detected.

Figure 4 (a) is an exemplary view for explaining an example in which the reference number is sensed by the reference number position detection unit 240 according to an embodiment of the present invention, Figure 4 (b) is an example of the present invention It is an exemplary diagram for explaining an example in which the reference number is not detected by the reference number position detecting unit 240 according to the embodiment.

As described above, according to an embodiment of the present invention, the image pre-processing unit 241 may be configured to pre-process the image of the drawing. The reference number position detecting unit 240 detects the position of the reference number in the drawing, and before the text recognition unit 250 extracts the reference number in text form from the reference number where the position is detected, according to an embodiment of the present invention The image pre-processing unit 241 pre-processes the image-type drawing so that the position of the reference numerals is optimally detected.

As shown in (a) of FIG. 4 , the reference number position detecting unit 240 may generate a bounding box around the detected reference number by detecting the positions of most of the reference numbers indicated by numbers in the drawing. On the other hand, as shown in (b) of FIG. 4 , the reference number position detecting unit 240 does not detect the positions of reference numbers indicated by numbers in the drawing, so it may not be able to generate a bounding box for the reference numbers. .

In the case of (a) of FIG. 4 , before the position of a reference in the drawing is detected by the reference position detecting unit 240 , the image pre-processing unit 241 according to an embodiment of the present invention pre-processes the image in the form of a drawing. By way of example, the image preprocessing unit 241 preprocesses the color jitter of the image of the drawing by adjusting the brightness jitter and saturation jitter to default and 0, respectively, and the contrast jitter by doubling the position of the reference numerals. It is possible to create an optimal state to be sensed, and thus the reference number position detecting unit 240 can detect the positions of almost all reference numbers.

In the case of (b) of FIG. 4 , since the image of the drawing by the image pre-processing unit 241 was not pre-processed before the position of the reference in the drawing was detected by the reference position detecting unit 240, the reference position detection The unit 240 may not detect the positions of the reference numerals well. Alternatively, even if the image of the drawing has been preprocessed by the image preprocessing unit 241, the color jitter of the image of the drawing is improperly adjusted, so that the position of the reference number may not be detected well. If the image of the drawing by the image preprocessing unit 241 is not preprocessed or if the color jitter of the image of the drawing is preprocessed to be improperly adjusted, the reference numerals in some drawings may have blurred colors or unclear lines forming the reference numerals The position of the reference numerals may not be detected well because it is too thin.

However, not all drawings may necessarily require an image pre-processing step, and even if the image pre-processing step is not performed, the position of the reference number may be well detected because the reference number is clearly displayed. According to a further embodiment of the present invention, the image pre-processing unit 241 may be configured to pre-process the image of the drawing by selecting only the image of the drawing requiring the image pre-processing step. Alternatively, the image pre-processing unit 241 according to another embodiment of the present invention may pre-process the images of all drawings to make the image in an optimal state for the reference position detection unit 240 to detect the reference number position.

5 is an algorithm and algorithm tuning used for the apparatus 200 for providing a drawing reading comprehension and search function according to an embodiment of the present invention to detect the position of a reference number and to extract a reference number in text form from the detected reference number. It is the experimental result data to explain the process.

As an object detection technique for finding the position of a reference number in the drawings of the specification, the reference number position detecting unit 240 according to an embodiment of the present invention may be configured to detect the position of the reference number using the RetinaNet algorithm. . In addition, it is possible to detect the position of the reference number in the drawing by tuning the RetinaNet algorithm so that the reference number position detecting unit 240 is specialized in detecting reference numbers displayed in numbers in a drawing composed of black and white.

Meanwhile, a study was conducted to find the most appropriate object detection technique for finding the position of reference numerals in the drawing. The details of the research plan for this study will be described later. For reference, the experimental result data is a drawing image in which reference numbers are displayed, and for about 10,000 cases, the bounding box Intersection over Union (IoU) for the true label was measured to be 0.5 or more.

To explain the direction of the study, first of all, some of the drawing data that were not prepared using a computer, the resolution of the image of the drawing is too low, or the drawing data that is not suitable for processing, such as a drawing with severe noise, was excluded from the test data set. We aimed to find the area corresponding to the reference number in the image of the drawing and extract the reference number in text format. We aimed to have an average IoU of 0.5 or more for newly worked labels. In addition, according to the size and characteristics of the data and the required computational cost, an appropriate technique is selected and used among machine learning methodologies including deep learning.

To explain the detailed research method, the area of the reference symbol in the drawing image is found using object detection recognition, the description of the reference symbol is recognized through the object recognition model, and the result is returned in xml format. In the object detection step, a reference mark area of the drawing image is created as a bounding box, and an area in which the IoU value of the true label and the detected bounding box is greater than or equal to a certain threshold is found. In the object recognition step, text is extracted from the characters in the reference area, and in this step, the text in the reference area is extracted using OCR.

When explaining the evaluation method, the performance evaluation scale, which is the final inspection standard, detects the position of the reference mark, extracts the reference number in text format, and returns it as data. to be made The performance of the final system is evaluated by obtaining an F1 score. The signs in the given data contain some data containing English characters or symbols in addition to numbers, and the proportion is small, so it is judged that there is a possibility of being treated as noise in the learning process. .

The precision used to evaluate the F1 score is measured by the number of correctly recognized reference numbers/total number of predicted reference numbers, and recall can be measured by the number of correctly recognized reference numbers/total number of actual reference numbers. The F1 score can be measured as a harmonic average of precision and recall.

As an inspection standard, 5% (about 500 pieces) randomly selected from the provided data set (about 10,000 pieces) is judged as the test data for accuracy. As a definition for the performance indicator of object detection, precision is measured by the number of correctly recognized reference numbers/total number of predicted reference numbers, and recall can be measured as the number of correctly recognized reference numbers/total number of actual reference numbers . The F1 score can be measured as a harmonic average of precision and recall.

In addition, as a definition of the performance index of OCR, accuracy can be measured by the number of correctly recognized reference numbers/total number of test samples. The final system can be composed of pipelines of two systems, object detection and OCR, and the performance evaluation of the final system aims to achieve an F1 score of 70%.

A process of tuning the RetinaNet algorithm so that the reference number position detecting unit 240 according to an embodiment of the present invention as shown in FIG. 5 accurately detects the position of the reference number in the drawing will be described. As a result of using the untuned RetinaNet algorithm on 1,000 Korean patent specifications on June 27, 2019, the F1 score was measured to be 90%. On July 22, 2019, for 1000 US patent specifications, the depth of the RetinaNet algorithm was tuned to 152 and the inference time to 346 (ms). As a result, the F1 score was measured to be 91%. .

In addition, as a result of using only the depth of the RetinaNet algorithm by tuning to 152 for all about 12,000 patent specifications including Korea and the United States on September 9, 2019, the F1 score was measured to be 75%. As of September 27, 2019, the depth of RetinaNet algorithm was 152 for all patent specifications of about 12,000 cases, and the image of the drawing was preprocessed and color jitter was adjusted and tuned. As a result, the F1 score was 94%, showing the best performance. Illustratively, the tonal jitter of the image of the drawing may be adjusted by default, the brightness jitter and saturation jitter may be adjusted to the default and zero, respectively, and the contrast jitter may be adjusted by doubling.

Next, the process of selecting and tuning an appropriate algorithm so that the text recognition unit 250 according to an embodiment of the present invention, as shown in FIG. 5, extracts reference numbers in text format from reference numbers whose positions are accurately detected Let's take a look. June 27, 2019 Tuning the tesseract 4.1 0.639 algorithm for 1000 domestic patent specifications to resize (x2) 0.739, grayscale to 0.740, sharpness, and contrast to 0.770 As a result, the accuracy was measured to be 77%. For reference, the tesseract algorithm is an OCR algorithm for various operating systems that is distributed free of charge.

On July 22, 2019, for 1000 US patent specifications, the default tesseract 4.1 algorithm was tuned to resize (x2), grayscale, sharpness, and contrast. As a result, the accuracy was measured to be 75%. As a result of tuning and using the ASTER model algorithm with pretrained parameters, RGB scale, and resizing (x5) for all about 12,000 patent specifications on September 9, 2019, the accuracy was measured at 93%, showing the best performance. showed

In addition, the drawing reading comprehension and search function providing apparatus 200 according to an embodiment of the present invention includes a reference mark position detection unit 240 and a text recognition unit 250, so that object detection and OCR are pipelined. can As an experimental result in which both object detection and OCR are performed, Detection v0.3, which tuned the depth of the RetinaNet algorithm to 152 for all about 12,000 patent specifications on September 25, 2019, and the ASTER model algorithm as pretrained parameters, As a result of using OCR v0.3 tuned to RGB scale and resize (x5), the F1 score was measured to be 63%.

On October 11, 2019, the depth of RetinaNet algorithm was set to 152 for all patent specifications of about 12,000 cases, and Detection v0.4 and ASTER model algorithm tuned by pre-processing the image of the drawing and adjusting color jitter were pretrained parameters, As a result of using OCR v0.3 tuned to RGB scale and resize (x5), the F1 score was measured to be 82%.

In summary, the reference position detection unit 240 according to an embodiment of the present invention sets the depth of the RetinaNet algorithm to 152, and uses Detection v0.4 tuned by preprocessing the image of the drawing and adjusting color jitter in the drawing. It may be configured to detect the position of the reference number, and when the reference number position detecting unit 240 detects the position of the reference number in the drawing using Detection v0.4, it is possible to most accurately detect the position of the reference number.

In addition, the text recognition unit 250 according to an embodiment of the present invention uses OCR v0.3 in which the ASTER model algorithm is tuned with pretrained parameters, RGB scale, and resize (x5). It can be configured to extract reference numbers in text format, and when the text recognition unit 250 extracts reference numbers in text format from reference numbers using OCR v0.3, it is possible to most accurately extract reference numbers in text format. have.

In addition, the drawing reading comprehension and search function providing apparatus 200 according to an embodiment of the present invention uses Detection v0.4 tuned by preprocessing the image of the drawing and adjusting the color jitter with the depth of the RetinaNet algorithm set to 152. A text recognition unit 250 using OCR v0.3 tuned to the pretrained parameter, RGB scale, and resizing (x5) of the symbol position detection unit 240 and the ASTER model algorithm is provided to determine the position of the reference mark in the drawing. It can be configured to detect and extract reference numbers in text form from the location-detected reference numbers.

In addition, the drawing comprehension and search function providing device 200 sets the depth of the RetinaNet algorithm to 152, the image of the drawing is pre-processed, and the reference mark position detection unit 240 using Detection v0.4 tuned by adjusting color jitter and Equipped with text recognition unit 250 using OCR v0.3 tuned with pretrained parameters, RGB scale, and resizing (x5) of the ASTER model algorithm, that is, Detect_OCR v0 using Detection v0.4 and OCR v0.3 When using .2 to detect the position of a reference in a drawing and extracting a reference in text form from a position-detected reference, it most accurately detects the position of a reference in the drawing and extracts the reference number in text form from the position-detected reference. It is possible to extract the reference numerals of

6 is an experimental result data according to the algorithm tuning version of the reference numeral position detecting unit 240 according to an embodiment of the present invention.

As a result of detecting the position of the reference symbol in the drawing, the reference symbol position detecting unit 240 according to an embodiment of the present invention detects the position of the reference symbol in the drawing using Detection v0.3, in which the depth of the RetinaNet algorithm is tuned to 152. The position may not be detected and the bounding box may not be created.

On the other hand, the reference mark position detection unit 240 according to an embodiment of the present invention sets the depth of the RetinaNet algorithm to 152, and uses Detection v0.4 tuned by preprocessing the image of the drawing and adjusting the color jitter in the drawing. As a result of detecting the positions of the reference signs, the positions of all reference signs are detected, and a bounding box may be generated for all reference signs.

That is, it is much more accurate to detect the position of a reference in the drawing using Detection v0.4 than the reference position detection unit 240 detects the position of the reference in the drawing by using Detection v0.3. A bounding box can be created by detecting the position of the reference numeral.

7 is a schematic block diagram for explaining the processing unit 270, the patent search unit 280, and the mapping unit 290 of the drawing reading and searching function providing system 1000 according to an embodiment of the present invention.

As shown in FIG. 7 , an apparatus 200 for providing a drawing reading and searching function according to an embodiment of the present invention includes a control unit 210 , a communication unit 220 , a storage unit 230 , and a reference position detection unit 240 . ), a text recognition unit 250 , a reference description linkage unit 260 , a processing unit 270 , a patent search unit 280 , a mapping unit 290 , and the like.

For reference, the components 210 , 220 , 230 , 240 , 250 , 260 , 270 , 280 and 290 of the drawing comprehension and search function providing apparatus 200 shown in FIG. 7 are These are merely exemplary components for explaining the operation and function of the drawing reading and search function providing apparatus 200 . That is, components other than the illustrated components 210 , 220 , 230 , 240 , 250 , 260 , 270 , 280 and 290 of the drawing comprehension and search function providing apparatus 200 according to an embodiment of the present invention (For example, it is clear that it may further include a power supply unit, etc.).

According to an embodiment of the present invention, the apparatus 200 for providing a drawing comprehension and search function is a processing unit 270 configured to generate a reference number list by processing reference numerals in a text format in which descriptions corresponding to reference numerals are linked. may further include. A detailed configuration in which the processing unit 270 according to an embodiment of the present invention generates a reference number list will be described in more detail with reference to FIG. 9 to be described later.

In addition, the processing unit 270 according to an embodiment of the present invention may generate a drawing information providing screen by processing a reference number in a text format in which a description corresponding to the reference number is linked, a drawing image, and a reference number list. The drawing information providing screen generated by the processing unit 270 according to an embodiment of the present invention contains information about a drawing list, a reference number in a text format in which a description corresponding to the reference number is linked, a reference number list, and information on the description of the reference number. As a displayed screen, the user may be provided with information on drawings and reference numerals through the drawing information providing screen, and may easily grasp the invention using the provided information.

In addition, according to an embodiment of the present invention, the processing unit 270 converts the reference number position coordinates detected by the reference number position detection unit 240 and the reference number in text format extracted by the text recognition unit 250 . By using the reference numerals can be displayed together with the descriptions of the symbols on the information providing screen.

Additionally, according to an embodiment of the present invention, the processing unit 270 may generate a drawing information providing screen including a search field. The search field of the drawing information providing screen generated by the processing unit 270 according to an embodiment of the present invention may be configured so that a user can search for an invention related to drawing information directly on the drawing information providing screen.

In addition, the processing unit 270 according to an embodiment of the present invention may translate and display in real time a user's desired language when the description corresponding to the reference numerals displayed on the drawing information providing screen is written in another language. Illustratively, since the US patent specification is written in English, the description corresponding to the reference numerals displayed on the drawing information providing screen must also be written in English, so the processing unit 270 according to an embodiment of the present invention is Descriptions corresponding to the reference numerals written in English displayed on the drawing information providing screen can be translated and displayed in real time into Korean.

In addition, according to an embodiment of the present invention, the processing unit 270 may be configured so that the user can edit the generated drawing information providing screen. The processing unit 270 according to an embodiment of the present invention may be configured to include a separate button that can be switched to the edit mode on the drawing information providing screen, and the user selects a separate button that can be switched to the edit mode Click to use the drawing information provision screen editing mode.

The processing unit 270 according to an embodiment of the present invention may be configured so that a user can edit a reference number on the drawing information providing screen or a text of a reference number list, a selection display figure, a color, and the like. Meanwhile, the control unit 210 according to an embodiment of the present invention may be configured to store an edited drawing information providing screen image or an unedited drawing information providing screen image in the storage unit 230 .

In addition, a separate button for storing an edited drawing information providing screen image or an unedited drawing information providing screen image may be provided on the drawing information providing screen. Accordingly, the user clicks a separate button capable of storing the drawing information providing screen image, etc. in the storage unit 230 to store the edited drawing information providing screen image or the unedited drawing information providing screen image, etc. in the storage unit 230 . In addition, the communication unit 220 may be configured to transmit such stored information to the user terminal (300).

On the other hand, the drawing reading comprehension and search function providing apparatus 200 according to an embodiment of the present invention is configured to generate a search expression through a text-type reference number or a generated reference number list in which a description corresponding to the reference number is linked. A patent search unit 280 may be further included. The patent search unit 280 according to an embodiment of the present invention may generate a search expression in a search field of the drawing information providing screen generated by the processing unit 270 .

More specifically, when the user selects a reference number in a text format in which a description corresponding to a reference number displayed in a drawing is linked or a reference number in the generated reference number list, the patent search unit ( 280) may create a search expression in the search field of the drawing information providing screen.

Illustratively, a reference number in a text format (eg, '4') or a reference number in the generated reference number list (eg, a description corresponding to the reference number displayed in the drawing on the drawing information providing screen by the user is linked) For example, if '4: discharge valve') is clicked, a search expression (eg, 'discharge valve') may be generated in the search field generated by the processing unit 270 . In addition, a reference number in a text format (eg, '10') or a reference number in the generated reference number list (for example, If '10: conveyor' is additionally clicked, an additional search expression (eg, 'discharge valve or conveyor') may be generated in the search field generated by the processing unit 270 . However, the search expression for 'discharge valve or conveyor' given as an example is simply expressed only with the description of the reference numerals, and the actual search expression is '((discharge valve) or (conveyor)).DN,DD,CLA.' A search expression may be generated in the form of, or the like.

In addition, the patent search unit 280 according to an embodiment of the present invention ?v the search expression generated in the search field by the user's selection to include a portion having a function or technology similar to the portion indicated by the reference numeral. It can be configured to easily search for other inventions.

According to an embodiment of the present invention, the apparatus 200 for providing a drawing comprehension and search function is a mapping unit configured to map reference numerals in text format extracted by the text recognition unit 250 and reference numerals described in the claims. (290). The mapping unit 290 according to an embodiment of the present invention may inquire the claims and map the reference numbers described in the claims that match the reference numbers in the extracted text format.

More specifically, when the user selects (for example, clicks, etc.) a reference number in text format or a reference number to be found from the generated reference number list, the mapping unit 290 according to an embodiment of the present invention provides a claim It can be configured to give different colored highlighting to reference numbers that match user-selected reference numbers in the range. Accordingly, the user can easily grasp the contents of the invention through the drawings, and can see the reference signs that are highlighted in the claims, so that the user can easily grasp the claims or the scope of the rights in which the reference signs are described.

In addition, according to an embodiment of the present invention, the mapping unit 290 may be configured to display the reference numerals described in the claims and the reference numerals mapped on the drawing information providing screen. That is, the user can grasp the reference numerals described in the claims on the drawing information providing screen, so that the user can easily grasp the scope of the patent.

8 is a schematic block flow diagram illustrating each step performed by a method for providing a drawing reading and searching function and an apparatus according to an embodiment of the present invention.

As described above, the method for providing a drawing reading and searching function according to an embodiment of the present invention can detect the position of a reference number in the drawing (S10), and extract a reference number in text form from the reference number where the position is detected. (S20) can be done. Subsequently, the extracted textual reference numerals may be linked with a description corresponding to the reference numerals ( S30 ).

Subsequently, a reference number list can be generated (S40) by processing the reference number in the text format in which the description corresponding to the reference number is linked, and the reference number or the generated reference number in the text format in which the description corresponding to the reference symbol is linked A search expression can be created through the list (S50).

In addition, the extracted text-type reference number and the description corresponding to the reference number are linked (S30), and the text-type reference number in which the description corresponding to the reference number is linked with the reference number described in the claims is mapped (S60) can

In addition, after interlocking the extracted textual reference number with the description corresponding to the reference number (S30) and the drawing information providing screen is generated, the user can edit the drawing information providing screen (S70).

In addition, according to an additional embodiment of the present invention, the extracted text-type reference number and the description corresponding to the reference number are interlocked (S30), and the text-type reference number and the claim in which the description corresponding to the reference number is linked Alternatively, the reference numerals described in the detailed description may be mapped.

In addition, according to a further embodiment of the present invention, a text format reference number in which a description corresponding to a reference number is linked with a reference number described in a claim or detailed description of the invention is mapped, and a description corresponding to the reference number is linked It is possible to provide a word cloud in which the degree of association between the technical keywords of the reference numerals described in the claims or detailed description of the invention mapped to the textual reference numerals and the technical keywords in the entire specification is expressed.

9 is an exemplary view for explaining a drawing information providing screen generated by the processing unit 270 according to an embodiment of the present invention.

According to an embodiment of the present invention, the reference reference interlocking unit 260 may be configured to associate the extracted textual reference number with a description corresponding to the reference number. As shown in the center of FIG. 9 , an image of a drawing and a reference number in the extracted text format may be displayed in the center of the drawing information providing screen, and the reference number position detecting unit 240 includes the reference number in the extracted text format. and a bounding box may be displayed by the text recognition unit 250 .

The reference reference linkage unit 260 according to an embodiment of the present invention displays a bounding box in the center of the drawing information providing screen and provides a description corresponding to the reference number of the reference number in the text format in which the corresponding description is linked to the user. can be configured to provide.

More specifically, the reference number description linkage unit 260 according to an embodiment of the present invention displays a bounding box in the center of the drawing information providing screen and selects a reference number in a text format in which a corresponding description is linked, A description corresponding to the selected reference number may be configured to pop up in an area surrounding the selected reference number. More specifically, the reference reference linkage unit 260 may be configured such that a description corresponding to the selected reference number is popped up when the user clicks on a reference number to be selected or places a cursor on the reference number to be selected.

In addition, the reference numeral description linkage unit 260 according to an embodiment of the present invention is a drawing of a text format extracted by using the surrounding sentences of the positions of the parts matching the reference numbers in the text format extracted within the detailed description of the present invention. It may be configured to link the descriptions corresponding to the reference numerals and the reference numerals.

As shown in FIG. 9, in the list of reference numerals on the right side of the drawing information provision screen, '4: discharge valve (or rotary valve) at the bottom bin of the adsorption tower', '5: roll feeder at the bottom of the activated carbon bed layer ( roll feeder) (or rotary valve) as indicated, the reference description linkage 260 uses the surrounding text of the location of the reference number within the detailed description to link the description corresponding to the reference number. can be configured.

On the other hand, according to an embodiment of the present invention, the processing unit 270 may be configured to generate a reference number list by processing the reference number in a text format in which the description corresponding to the reference number is linked. More specifically, the processing unit 270 according to an embodiment of the present invention combines a reference number in a text format in which a bounding box is displayed in the center of the screen and a corresponding description is linked and a description corresponding thereto, and a plurality of combinations A reference number list can be created by arranging reference numbers in text format and corresponding descriptions in parallel. According to an embodiment of the present invention, the processing unit 270 may generate a list of reference numerals on the right side of the screen information providing screen.

Also, the processing unit 270 according to an embodiment of the present invention may generate a drawing information providing screen including a search field. More specifically, the processing unit 270 according to an embodiment of the present invention may create a search field at the lower side of the screen information providing screen.

Subsequently, the patent search unit 280 according to an embodiment of the present invention may be configured to generate a search expression through a reference number in a text format or a generated reference number list in which a description corresponding to the reference number is linked. The patent search unit 280 according to an embodiment of the present invention may generate a search expression in a search field of the drawing information providing screen generated by the processing unit 270 .

More specifically, at least one of the reference numbers in the text format in which the user has linked a description corresponding to a plurality of reference numbers displayed in the drawing in the center of the drawing information providing screen or at least one drawing of the generated reference number list When a symbol is selected, the patent search unit 280 according to an embodiment of the present invention may generate a search expression in the search field of the drawing information providing screen.

In addition, according to an additional embodiment of the present invention, the reference reference linkage unit 260 moves to the part where the selected reference number is written in the detailed description of the invention when the user selects at least one reference number from the created reference number list. It can be configured to

In addition, according to an embodiment of the present invention, the processing unit 270 may be configured to list a plurality of drawings on the left side of the drawing information providing screen, and the user can view the plurality of drawings listed on the left side of the drawing information providing screen. You can select a drawing you want to understand more specifically through.

10 is an exemplary diagram for explaining a method for providing a drawing reading and searching function and an exemplary drawing search method performed by the apparatus according to an embodiment of the present invention.

The apparatus 200 for providing a drawing comprehension and search function according to an embodiment of the present invention may provide a search field in which a technology keyword can be input, and the technology keyword input by the user when the user inputs the technology keyword in the search field may be configured to search for drawings included as reference numerals. Accordingly, the user can access the drawing search field through the technical keyword search field. Alternatively, by inputting a reference number in the drawing search field, the user can access the technology keyword search field having the same and similar function or technology as the reference number.

In addition, according to an embodiment of the present invention, the apparatus 200 for providing a drawing reading and searching function may provide a search field for inputting a document number, and when the user inputs a specific document number into the search field, a specific document It may be configured to search for drawings of a numbered specification. Also, the drawing searched for by the drawing reading and searching function providing apparatus 200 may be provided to the user as the drawing information providing screen as described above.

According to a further embodiment of the present invention, the apparatus 200 for providing a drawing reading and retrieval function, despite the fact that reference numerals in text format, such as reference numbers in text format extracted from drawings, are described in the detailed description of the invention, If the description corresponding to the code (the description of the code) is different, it may be determined as a typo and configured to provide typographic information to the user.

In addition, in the drawing reading and searching function providing apparatus 200 according to an additional embodiment of the present invention, although corresponding descriptions such as reference numerals and corresponding descriptions displayed on the drawing information providing screen are described in the detailed description of the invention, , when the reference numerals linked to the corresponding description are different, it may be determined as a typo and configured to provide typographic information to the user.

11 is an exemplary diagram for explaining mapping between reference numerals in a drawing and reference numerals described in the claims by the method and apparatus for providing a drawing reading and searching function according to an embodiment of the present invention.

As described above, the drawing reading comprehension and search function providing apparatus 200 includes a mapping unit 290 configured to map reference numerals in text format extracted by the text recognition unit 250 and reference numerals described in the claims. may include The mapping unit 290 according to an embodiment of the present invention may search the claims and map the reference numerals described in the claims that match the extracted text format reference numbers.

More specifically, when the user selects (for example, clicks, etc.) a reference number in the text format of a drawing or a reference number to be found from the generated reference number list, the mapping unit 290 according to an embodiment of the present invention is In the claims, reference numbers that match the reference numbers selected by the user may be configured with a different color highlighting. Accordingly, the user can easily grasp the contents of the invention through the drawings, and can see the reference signs that are highlighted in the claims, so that the user can easily grasp the claims or the scope of the rights in which the reference signs are described.

According to a further embodiment of the present invention, the drawing comprehension and search function providing apparatus 200 maps reference numerals in a text format in which descriptions corresponding to reference numerals are linked with reference numbers described in the claims or detailed description of the invention. It may further include a configured mapping unit 290 .

In addition, according to an additional embodiment of the present invention, the apparatus 200 for providing a drawing reading and searching function displays a screen on which a reference number in which a description is linked with a screen on which a claim or detailed description of the invention is displayed and a corresponding description is displayed. can be provided together.

As shown in FIG. 11 , the drawing comprehension and search function providing apparatus 200 allows the user to press the claims tab and the description of the invention tab arranged in one area of the screen so that the user can view the detailed description of the claims or inventions. It can be configured to In addition, the drawing comprehension and search function providing apparatus 200 allows the user to display a screen on which a detailed description of a claim or an invention disposed on both sides of the screen is displayed, and a screen on which a reference number in which a corresponding description is linked is displayed. It can be configured to be visible (dual mode can be provided).

In addition, the mapping unit 290 according to an embodiment of the present invention selects (for example, clicking, etc. ), it may be configured to display a description corresponding to the reference number in a position adjacent to the reference number.

In addition, the mapping unit 290 is the corresponding description displayed at a position adjacent to the selected reference number when the user selects a reference number associated with the corresponding description on the screen of the drawing on which the reference symbol linked to the corresponding description is displayed. It may be configured to highlight (highlight) the description and at the same time highlight the description corresponding to the reference number that matches the reference number selected by the user on the screen on which the accompanying claims or detailed description of the invention are displayed.

Conversely, when the user selects a reference number on the screen on which the claims or detailed description of the invention are displayed, the mapping unit 290 highlights and displays the description corresponding to the reference number selected by the user, and at the same time provides a corresponding description. On the screen of the drawing on which the linked reference numbers are displayed, the description corresponding to the reference numbers matching the reference numbers selected by the user may be configured to be highlighted.

12 is a method for providing a drawing comprehension and search function according to an embodiment of the present invention and a word cloud in which the degree of association between the technical keywords of the reference numerals described in the claims and the technical keywords of the entire specification is provided by the device. It is an example diagram for explanation.

According to an embodiment of the present invention, the apparatus 200 for providing a drawing reading and retrieval function provides a description of a reference number in a claim or a detailed description of the invention in which a description corresponding to the reference number is mapped to a reference number in a text format linked thereto. The apparatus may further include an apparatus for providing a word cloud in which a degree of association between the technical keyword and the technical keyword of the entire specification is expressed.

12 , when the user selects a technical keyword of a reference number that is included in the description of a reference or mapped to a reference number in the text form of a drawing, described in a claim or detailed description of the invention, drawing reading and retrieval The function providing apparatus 200 may provide a word cloud in which a degree of association with a plurality of technical keywords throughout the specification is expressed.

A specification including a plurality of technical keywords for which the degree of relevance is measured with the technical keyword searched by the user is one case including the reference number searched by the user or a plurality of specifications including the technical keyword of the reference number searched by the user it may be

For reference, the drawing comprehension and search function providing apparatus 200 may display a larger size of the technical keywords in the word cloud as the degree of relevance between the plurality of technical keywords in the entire specification with the technical keywords of reference numerals selected by the user increases.

In addition, the degree of association between the technical keyword selected by the user and the technical keyword in the entire specification is determined by the adjacency in the text of the claims or the detailed description of the invention, the frequency of being shown together in the drawings, the proximity of the locations in the drawings, the proximity of the positions to be searched together. It can be measured by various criteria, such as frequency, the frequency described together with the corresponding technical keyword.

As described above, according to the method for providing a drawing reading and retrieval function according to an embodiment of the present invention and an apparatus and system therefor, by detecting the position of a reference number in a drawing and extracting a reference number in text form from the detected reference number, A description of the corresponding reference number may be displayed on the extracted reference number.

In addition, according to the method for providing a drawing reading and search function according to an embodiment of the present invention, and an apparatus and system therefor, it is possible to easily search for a drawing including reference numerals related to the technology keyword by searching for a technology keyword to be found. In addition, it is possible to easily search for other inventions including parts having functions or technologies similar to the parts indicated by the corresponding reference numbers only in the drawings in which the reference numerals are indicated.

In addition, according to the method for providing a drawing reading and search function according to an embodiment of the present invention, and an apparatus and system therefor, a drawing in which only reference numerals are displayed, a drawing in which a description of a reference number is displayed, or a reference to a reference number and a reference number The user can easily edit the text, shape, color, etc. according to the need of the drawing, which is displayed together with the description, and the edited image can be saved and used separately.

In addition, according to the method for providing a drawing reading and searching function according to an embodiment of the present invention, and an apparatus and system therefor, a reference number in text form is extracted from a reference number, and the extracted reference number and reference number described in the claims By mapping the , it is possible to enable the user to easily grasp the contents of the invention.

In addition, according to the method for providing a drawing reading and retrieval function according to an embodiment of the present invention, and an apparatus and system therefor, by accurately detecting a reference number in a drawing and extracting a reference number in text form, the reference number is added to the reference number of the drawing It is possible to display or link a description corresponding to .

In addition, according to the method for providing a drawing reading and searching function according to an embodiment of the present invention, and an apparatus and system therefor, it is possible to generate a reference number list by processing reference numbers in which descriptions corresponding to reference numbers are displayed or interlocked, A search expression can be created through a reference number or a generated reference number list.

Meanwhile, various embodiments described herein may be implemented by hardware, middleware, microcode, software, and/or a combination thereof. For example, various embodiments may include one or more application specific semiconductors (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs). ), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the functions presented herein, or a combination thereof.

Also, for example, the various embodiments may be embodied in or encoded on a computer-readable medium comprising instructions. The instructions embodied in or encoded on a computer-readable medium may cause a programmable processor or other processor to perform a method, for example, when the instructions are executed. Computer-readable media includes computer storage media, which may be any available media that can be accessed by a computer. For example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage media, magnetic disk storage media, or other magnetic storage devices.

Such hardware, software, firmware, etc. may be implemented in the same device or in separate devices to support the various operations and functions described herein. Additionally, components, units, modules, components, etc. described as “parts” in the present invention may be implemented together or individually as separate but interoperable logic devices. Depictions of different features of modules, units, etc. are intended to emphasize different functional embodiments, and do not necessarily imply that they must be realized by separate hardware or software components. Rather, functionality associated with one or more modules or units may be performed by separate hardware or software components or integrated within common or separate hardware or software components.

Although acts are shown in the drawings in a particular order, it should not be understood that these acts need to be performed in the specific order, or sequential order, shown, or that all shown acts need to be performed to achieve a desired result. . In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the division of various components in the above-described embodiments should not be construed as requiring such division in all embodiments, and that the described components will generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there can be

As described above, an optimal embodiment has been disclosed in the drawings and the specification. Although specific terms are used herein, they are used only for the purpose of describing the present invention, and are not used to limit the meaning or scope of the present invention described in the claims. Therefore, it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: patent information server
200: drawing comprehension and search function providing device
210: control unit
220: communication department
230: storage
240: reference position detection unit
241: image preprocessor
250: text recognition unit
260: reference number description linkage
270: processing unit
280: patent search unit
290: mapping unit
300: user terminal
1000: drawing comprehension and search function providing system

Claims (13)

A method for providing drawing reading and retrieval functions, the method comprising:
detecting the positions of reference numerals in the drawing; and
Including the step of extracting a reference number in text form from the reference number for which the position is sensed,
How to provide drawing comprehension and search function. The method of claim 1,
Further comprising the step of preprocessing the image of the drawing,
How to provide drawing comprehension and search function. The method of claim 1,
Further comprising the step of associating a reference number in the extracted text format with a description corresponding to the reference number,
How to provide drawing comprehension and search function. 4. The method of claim 3,
generating a reference number list by processing the reference number in the text format in which a description corresponding to the reference number is linked; and
Further comprising the step of generating a search expression through the reference number in the text format or the generated reference number list in which the description corresponding to the reference number is linked,
How to provide drawing comprehension and search function. 4. The method of claim 3,
Further comprising the step of mapping the reference number in the text format in which the description corresponding to the reference number is linked with the reference number described in the claims or detailed description of the invention,
How to provide drawing comprehension and search function. 6. The method of claim 5,
A word expressing the degree of association between the technical keywords of the reference numbers described in the claims or the detailed description of the invention in which the description corresponding to the reference signs is mapped with the reference signs in the text format linked to the description and the technical keywords of the entire specification Further comprising the step of providing a cloud,
How to provide drawing comprehension and search function. A computer program for performing the method of any one of claims 1 to 6 is stored,
computer readable recording medium. As a drawing reading and searching function providing apparatus 200,
Reference number position detection unit 240 configured to detect the position of the reference number in the drawing; and
and a text recognition unit 250 configured to extract reference numbers in text form from reference numbers for which the position is sensed,
Drawing comprehension and search function providing apparatus 200. 9. The method of claim 8,
Further comprising an image preprocessor 241 configured to preprocess the image of the drawing,
Drawing comprehension and search function providing apparatus 200. 9. The method of claim 8,
Further comprising a reference number description interlocking unit 260 configured to link the extracted textual reference number with a description corresponding to the reference number,
Drawing comprehension and search function providing apparatus 200. 11. The method of claim 10,
a processing unit 270 configured to generate a reference number list by processing the reference number in the text format in which the description corresponding to the reference number is linked; and
Further comprising a patent search unit 280 configured to generate a search expression through the reference number in text format or the generated reference number list in which the description corresponding to the reference number is linked,
Drawing comprehension and search function providing apparatus 200. 11. The method of claim 10,
Further comprising a mapping unit 290 configured to map the reference numerals in the text format in which the description corresponding to the reference numerals are linked and the reference numerals described in the claims or detailed description of the invention,
Drawing comprehension and search function providing apparatus 200. 13. The method of claim 12,
A word expressing the degree of association between the technical keywords of the reference numbers described in the claims or the detailed description of the invention in which the description corresponding to the reference signs is mapped with the reference signs in the text format linked to the description and the technical keywords of the entire specification Further comprising a device for providing a cloud,
Drawing comprehension and search function providing apparatus 200.

Images