KR102524124B1 - Metadata generation apparatus and method for verifying transformation and pragiarism of the image object in documents - Google Patents

Abstract

Provided is a method for generating metadata, which includes the steps of: receiving document input; converting the input document into an image file; extracting bibliographic information of the input document; extracting format information from the input document; extracting an image area within the input document and at least one image object within the image area; extracting structural feature points of at least one image object; extracting contextual feature points of at least one image object; and generating integrated metadata based on the bibliographic information, format information, structural feature points, and contextual feature points.

Description

Apparatus and Method for Generating Metadata for Transformation and Plagiarism Verification of Image Objects in Documents

The present disclosure relates to a metadata generating apparatus and method. More specifically, the present disclosure relates to an apparatus and method for generating metadata for deformation and plagiarism verification of an image object in a document.

In the field of computer vision, similarity determination between images may be performed in units of pixels. Specifically, similarity determination may be performed based on the degree of matching of entire pixel arrangements between the original image and the comparison target image. However, in the pixel-unit similarity determination, when the comparison target image is an image obtained by adding transformations to some pixels, scales, angles, brightness, saturation, etc. in the original image, it is difficult to verify such transformations. In addition, the pixel-unit similarity determination has a problem in that it is difficult to verify the image to be compared when it is a part of the original image. In order to compensate for the above-mentioned disadvantages, various techniques for determining similarity by extracting feature points in images have been devised.

A comparison target image of the present disclosure may be an image object included in a document (eg, a thesis or a contribution). In this case, the types of image objects (eg, photos, drawings, illustrations, tables, charts) and types of transformations that can be applied to the image objects may vary. Therefore, it may be difficult to verify deformation and plagiarism of image objects in a document only by determining similarity based on existing pixels and feature points.

KR 10-2019-0064288 KR 10-2020-0046182 KR 10-2021-0086836

As described above, in order to verify deformation and plagiarism of an image area in a document, the metadata generating device disclosed in the present disclosure processes structural information and contextual information of an image object into integrated metadata to build a database. there is.

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

A metadata generation method according to an aspect of the present disclosure for achieving the above technical problem includes receiving a document, converting the input document into an image file, extracting bibliographic information of the input document, and inputting the input document. Extracting format information of a document, extracting an image area within an input document and at least one image object within the image area, extracting structural feature points of at least one image object, It may be characterized by including the step of extracting feature points, generating integrated metadata based on bibliographic information, format information, structural feature points, and contextual feature points.

In addition, the bibliography information includes the title, type, journal name, author, publisher, publication site, keyword, DOI (Digital Object Identifier), original text link, information link, search link, registration date, status code, volume, number, and at least one of page information.

In addition, the step of extracting the format information of the input document includes the step of extracting the metadata of the input document, the step of classifying the format of the input document, the step of extracting byte codes from the input document, and the structure of the input document. The step of analyzing the structure of the input document includes the step of analyzing, and the step of generating final form information, and the step of analyzing the structure of the input document converts at least one image object into a picture, figure, illustration, or chart. ), and a step of classifying at least one of tables.

In addition, the step of extracting structural feature points of at least one image object may include at least one using at least one of energy threshold, contrast enhancement, scale-invariant feature transform (SIFT), and features from accelerated segment test (FAST) techniques. It may include extracting a robust key point of the image object.

In addition, the step of extracting contextual feature points of at least one image object may include identifying an image object including text among the at least one image object based on OCR (Optical Character Recognition) technology, and the identified text information and extracting detailed context information based on the method.

In addition, a metadata generating apparatus according to another aspect of the present disclosure for achieving the above technical problem includes a memory; a document input unit for receiving documents; and a control unit for controlling the operation of the metadata generation unit, wherein the metadata generation unit includes: a document conversion unit that converts the input document into an image file; a bibliographic information extraction unit extracting bibliographic information of the input document; a format information extraction unit extracting format information of the input document; an image area extraction unit extracting an image area within the input document and at least one image object within the image area; a structural feature extraction unit extracting structural feature points of the at least one image object; a contextual feature extraction unit extracting contextual feature points of the at least one image object; and an integrated metadata generation unit generating integrated metadata based on the bibliography information, the format information, the structural feature points, and the contextual feature points.

In addition to this, a computer program stored in a computer readable recording medium for execution to implement the present disclosure may be further provided.

In addition to this, a computer readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

According to the above-mentioned problem solving means of the present disclosure, meta data specialized for an image object in a document is generated to provide an effect of easily verifying or detecting image modification and plagiarism by a third party.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a block diagram of a metadata generating device of the present disclosure.
2 is a block diagram illustrating a metadata generator according to an embodiment of the present disclosure.
3 is a flowchart illustrating an operation of an integrated metadata generating device according to an embodiment of the present disclosure.
4 is a flowchart illustrating an operation of extracting form information according to an embodiment of the present disclosure.
5 illustrates a result of extracting an image area for an input document according to an embodiment of the present disclosure.

Like reference numbers designate like elements throughout this disclosure. The present disclosure does not describe all elements of the embodiments, and general content or overlapping content between the embodiments in the technical field to which the present disclosure belongs is omitted. The term 'unit, module, member, or block' used in the specification may be implemented as software or hardware, and according to embodiments, a plurality of 'units, modules, members, or blocks' may be implemented as one component, It is also possible that one 'part, module, member, block' includes a plurality of components.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case of being directly connected but also the case of being indirectly connected, and indirect connection includes being connected through a wireless communication network. do.

In addition, when a certain component is said to "include", this means that it may further include other components without excluding other components unless otherwise stated.

Throughout the specification, when a member is said to be located “on” another member, this includes not only a case where a member is in contact with another member, but also a case where another member exists between the two members.

Terms such as first and second are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

Expressions in the singular number include plural expressions unless the context clearly dictates otherwise.

In each step, the identification code is used for convenience of description, and the identification code does not explain the order of each step, and each step may be performed in a different order from the specified order unless a specific order is clearly described in context. there is.

Hereinafter, the working principle and embodiments of the present disclosure will be described with reference to the accompanying drawings.

In this specification, the 'apparatus according to the present disclosure' includes all various devices capable of providing results to users by performing calculation processing. For example, a device according to the present disclosure may include a computer, a server device, and a portable terminal, or may be in any one form.

Here, the computer may include, for example, a laptop computer, a desktop computer, a laptop computer, a tablet PC, a slate PC, and the like equipped with a web browser.

The server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

The portable terminal is, for example, a wireless communication device that ensures portability and mobility, and includes a Personal Communication System (PCS), a Global System for Mobile communications (GSM), a Personal Digital Cellular (PDC), a Personal Handyphone System (PHS), and a PDA. (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, smart phone ) and wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted-devices (HMDs). can include

1 is a block diagram of a metadata generating device of the present disclosure.

Referring to FIG. 1 , the metadata generating device 100 may include a controller 110, a memory 120, a document input unit 122, and/or a metadata generator 130.

In one embodiment, the control unit 110 uses a memory 120 that stores data for an algorithm or a program that reproduces the algorithm for controlling the operation of the components in the device, and the data stored in the memory 120. It may be implemented with at least one processor (not shown) that performs the above-described operation. In this case, the memory 120 and the processor may be implemented as separate chips. Alternatively, the memory 120 and the processor may be implemented as a single chip.

In addition, the control unit may control any one or a combination of the components described above in order to implement various embodiments according to the present disclosure described in FIGS. 2 to 5 below on the present device.

In one embodiment, the memory 120 may store data supporting various functions of the device and a program for the operation of the control unit 110, and input/output data (eg, music files, still image, video, etc.), and can store a plurality of application programs (application programs or applications) running in the device, data for operation of the device, and commands. At least some of these application programs may be downloaded from an external server through wireless communication.

The memory 120 may be a flash memory type, a hard disk type, a solid state disk type, a silicon disk drive type, or a multimedia card micro type. micro type), card type memory (eg SD or XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable EEPROM (EEPROM) It may include a storage medium of at least one type of a programmable read-only memory (PROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. In addition, the memory 120 is separated from the apparatus, but may be a database connected by wire or wirelessly.

In one embodiment, the document input unit 122 may receive a document (eg, a thesis, a journal) from a user. For example, a document may include at least one text area and/or image area.

In one embodiment, the metadata generator 130 may extract bibliographic information and format information of an input document. In an embodiment, the metadata generating unit 130 may extract at least one image object from an input document and extract structural and contextual features of the extracted image object. In an embodiment, the metadata generator 130 may generate integrated metadata for each image object based on document information (eg, bibliographic information and format information), structural characteristics, and contextual characteristics.

2 is a block diagram illustrating a metadata generator according to an embodiment of the present disclosure.

Referring to FIG. 2 , the metadata generation unit 130 includes a document conversion unit 200, a bibliographic information extraction unit 210, a format information extraction unit 220, an image area extraction unit 230, a structural feature extraction unit ( 240), a contextual feature extractor 250, and/or an integrated metadata generator 260. Hereinafter, it may be understood that the operation of the metadata generator 130 and each component of the metadata generator 130 is performed by the control unit 110 of the metadata generator 100 .

In one embodiment, the document conversion unit 200 may receive a document input from the document input unit 122 . For example, the controller 110 may receive a document input from a user through the document input unit 122 . The document conversion unit 200 may convert an input document into an image file.

In one embodiment, the bibliographic information extraction unit 210 may extract bibliographic information of an input document.

In one embodiment, the format information extractor 220 may extract format information of an input document.

In an embodiment, the image area extractor 230 may extract an image area within an input document. The image area may include at least one image object. The controller 110 may identify at least one image object included in the image area.

In an embodiment, the structural feature extractor 240 may extract structural feature points of at least one image object.

In an embodiment, the contextual feature extractor 250 may identify text information included in at least one image object, and extract detailed contextual information based on the identified text information.

In one embodiment, the integrated metadata generator 260 includes document information (eg, bibliographic information and format information of an input document) extracted by the above components, structural characteristics (eg, feature point information of an image object, pixel area) related information) and contextual characteristics (eg, contextual information included in an image object) may be merged into one meta data (eg, integrated metadata). When there are a plurality of at least one image object, the integrated metadata generating unit 260 may generate integrated metadata for each image object.

At least one component may be added or deleted corresponding to the performance of the components shown in FIGS. 1 and 2 . In addition, it will be easily understood by those skilled in the art that the mutual positions of the components may be changed corresponding to the performance or structure of the system.

Meanwhile, each component shown in FIGS. 1 and 2 means software and/or hardware components such as a Field Programmable Gate Array (FPGA) and Application Specific Integrated Circuit (ASIC).

3 is a flowchart illustrating an operation of an integrated metadata generating device according to an embodiment of the present disclosure.

In operation 300, the control unit 110 of the metadata generating device 100 may receive a document through the document input unit 122. In one embodiment, a document may include a text area and/or an image area.

In operation 305, the controller 110 may convert the document input through the document conversion unit 200 into an image file. For example, conversion of an input document into an image file may be performed in units of pages.

In operation 310, the control unit 110 may extract bibliographic information of the document input through the bibliographic information extracting unit 210. For example, the bibliography information includes the title, type, journal name, author, publisher, publication site, keyword, DOI (Digital Object Identifier), original text link, information link, search link, registration date, status code, volume, call, and/or page information.

In operation 320, the controller 110 may extract format information of the document input through the format information extractor 220.

Referring to FIG. 4 , operation 320 may include operations 400 to 440 . In operation 400, the format information extractor 220 may extract metadata recorded in the input document. For example, the metadata extracted in operation 400 may include document type, document structure, document length, document author, and/or creation date. In operation 410, the format information extractor 220 may classify the format of the input document. For example, the format of a document may mean an extension such as doc, docx, hwp, or pdf. In operation 420, the format information extraction unit 220 may extract byte codes from the input document. In operation 430, the format information extractor 220 may analyze the document structure. In one embodiment, the form information extractor 220 may classify the input document into a text area and/or an image area. The format information extractor 220 analyzes the classified image area and converts at least one image object included in the image area into pictures, figures, illustrations, charts, and tables. ) can be subdivided into at least one of them. In operation 440, the format information extractor 220 may generate final format information based on the results of operations 400 to 430.

Referring back to FIG. 3 , in operation 330, the controller 110 may extract an image area within the document input through the image area extractor 230. In one embodiment, the image area may include at least one image object. The controller 110 may identify at least one image object within the image area.

Referring to FIG. 5 , the image area extraction unit 230 may identify a text area and/or an image area within the input document 500 . In one embodiment, the text area and/or image area may be identified as a bounding box. For example, image regions may be identified by bounding boxes 510 and 540 . For example, text areas may be identified as bounding boxes 520 and 530 . Each bounding box can be identified based on identification coordinates (eg text0, text1, figure0, figure1). Identification coordinates may be assigned based on characteristics (eg, text, image) of the identified area.

In one embodiment, the image area 510 may include an image object. For example, the image object 512 of the image area 510 may be understood as an illustration. The controller 110 may identify an image object from the image area 510 .

In one embodiment, image area 540 may include an image object. For example, an image object in the image area 540 may be understood as a chart. The controller 110 may identify an image object from the image area 540 .

Referring back to FIG. 3 , after extracting the image area in operation 330 , the controller 110 may proceed to operation 340 and/or operation 350 . Operations 340 and 350 may be performed sequentially or concurrently. The order in which operations 340 and 350 are performed may be arbitrarily determined.

In operation 340, the controller 110 may extract structural feature points from at least one image object through the structural feature extractor 240. For example, the structural feature extraction unit 240 uses at least one of energy threshold, contrast enhancement, Scale-Invariant Feature Transform (SIFT), and Features from Accelerated Segment Test (FAST) techniques to robustly extract at least one image object. A robust key point can be extracted. For example, the robust feature points may include corner points of at least one image object. The controller 110 may extract robust feature points as structural feature points of at least one image object. In an embodiment, the structural feature extractor 240 may convert and store the extracted structural feature points into relative coordinates instead of absolute coordinates in order to maintain robust characteristics of the feature points.

In operation 350, the controller 110 may extract contextual information of an image object including text from among at least one image object through the contextual feature extractor 250. In an embodiment, the contextual feature extractor 250 may identify text information from an image object including text among at least one extracted image object (eg, the image objects 510 and 540 of FIG. 5 ). . The contextual feature extractor 250 may extract detailed contextual information based on the identified text information. For example, the detailed context information may refer to text information included inside a chart, table, image, etc., which is not created inside a document program but is externally worked and attached in the form of an image. In one embodiment, the contextual feature extractor 250 may use Optical Character Recognition (OCR) technology to identify text information.

In operation 360, the controller 110 may generate integrated metadata for at least one image object through the integrated metadata generator 260. In one embodiment, when there are a plurality of at least one image object, the integrated metadata generating unit 260 may generate metadata for each image object.

In one embodiment, the integrated metadata generation unit 260 converts the document information extracted by the above configurations, the extracted structural feature points and contextual feature points of at least one image object into one piece of metadata (eg, integrated metadata). can be merged with For example, document information may include bibliography information and format information of an input document. For example, structural feature points may include feature point information and pixel area information of an image object. For example, the contextual feature point may include detailed contextual information of text included in the image object.

In one embodiment, the metadata generated by the integrated metadata generator 260 may be stored in the memory 120 in the form of a database. Metadata for each image object stored in the memory 120 may be used to detect deformation and plagiarism of the image object.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing instructions executable by a computer. Instructions may be stored in the form of program codes, and when executed by a processor, create program modules to perform operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable recording media include all types of recording media in which instructions that can be decoded by a computer are stored. For example, there may be read only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage device, and the like.

As above, the disclosed embodiments have been described with reference to the accompanying drawings. Those skilled in the art to which the present disclosure pertains will understand that the present disclosure may be implemented in a form different from the disclosed embodiments without changing the technical spirit or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims (10)

In the method performed by the metadata generating device,
Receiving a document input;
converting the input document into an image file;
extracting an image area within the input document and at least one image object within the image area;
extracting bibliography information of the input document;
extracting format information of the input document;
extracting structural feature points of the at least one image object;
extracting contextual feature points of the at least one image object;
Generating integrated metadata based on the bibliographic information, the format information, the structural feature, and the contextual feature;
The bibliographic information includes a link to the original text of the input document, an information link, and a search link,
The step of extracting the format information of the input document,
extracting metadata of the input document;
classifying the type of the input document;
extracting byte codes from the input document;
analyzing the structure of the input document, and classifying the at least one image object into a picture, a figure, an illustration, a chart, and a table; and
Generating final format information based on the extracted metadata, the classified format, the extracted byte code, and the detailed classified result;
The step of extracting structural feature points of the at least one image object,
Using at least one of energy threshold, contrast enhancement, scale-invariant feature transform (SIFT), and features from accelerated segment test (FAST) techniques, robust feature points (robust extracting a key point); and
Converting the extracted robust feature points into relative coordinates and storing them;
The step of extracting contextual feature points of the at least one image object,
identifying text information for an image object including text among the at least one image object based on OCR (Optical Character Recognition) technology; and
Extracting detailed context information based on the identified text information; includes,
The detailed context information is text information included in charts, tables, and images that are not created inside the document program but are worked outside and attached in the form of images,
The step of generating the integrated metadata,
When a plurality of image objects are included in the image area, generating integrated metadata for each of the plurality of image objects;
The integrated metadata generated for each of the plurality of image objects is used to determine deformation and plagiarism for each image object,
How to create metadata. delete delete delete delete A program stored in a computer readable recording medium in order to execute the method of generating metadata according to claim 1 in combination with a computer. Memory;
a document input unit for receiving documents; and
Including; a controller for controlling the operation of the metadata generator;
The metadata generator,
a document conversion unit that converts the input document into an image file;
an image area extraction unit extracting an image area within the input document and at least one image object within the image area;
a bibliographic information extraction unit extracting bibliographic information of the input document;
a format information extraction unit extracting format information of the input document;
a structural feature extraction unit extracting structural feature points of the at least one image object;
a contextual feature extraction unit extracting contextual feature points of the at least one image object;
An integrated metadata generator for generating integrated metadata based on the bibliography information, the format information, the structural feature points, and the contextual feature points;
The bibliographic information includes a link to the original text of the input document, an information link, and a search link,
The control unit, through the form information extraction unit,
Extract format information of the input document, extract metadata of the input document, classify the format of the input document, extract byte codes from the input document, and extract the structure of the input document. While analyzing, the at least one image object is classified in detail into a picture, a figure, an illustration, a chart, and a table, and the extracted metadata and the classified format , Based on the extracted byte code and the detailed classified result, final format information is generated,
The control unit, through the structural feature extraction unit,
Using at least one of energy threshold, contrast enhancement, scale-invariant feature transform (SIFT), and features from accelerated segment test (FAST) techniques, robust feature points (robust key point) is extracted, and the extracted robust feature points are converted into relative coordinates and stored,
The control unit, through the contextual feature extraction unit,
Identifying text information based on OCR (Optical Character Recognition) technology for an image object including text among the at least one image object, extracting detailed context information based on the identified text information,
The detailed context information is text information included in charts, tables, and images that are not created inside the document program but are worked outside and attached in the form of images,
The control unit, through the integrated metadata generation unit,
When a plurality of image objects are included in the image area, generating integrated metadata for each of the plurality of image objects;
The integrated metadata generated for each of the plurality of image objects is used to determine deformation and plagiarism for each image object,
Metadata generator. delete delete delete

Images