KR102620260B1 - Method and device for recogniting object based on graph - Google Patents

Abstract

A graph-based object recognition method and device that can recognize objects using different types of text data and image data for objects are provided. The object recognition method receives and groups a plurality of text data and a plurality of image data about the object from the outside, converts the text data into a text graph through a neural network model, converts the image data into an image graph, and generates a text graph and Recognize objects by connecting image graphs.

Description

Graph-based object recognition method and device {METHOD AND DEVICE FOR RECOGNITING OBJECT BASED ON GRAPH}

The present invention relates to a graph-based object recognition method and device.

Object recognition technology is a technology that identifies objects such as people or objects from images such as images or videos collected by sensors.

This object recognition technology is applied in various fields. For example, it can be used to identify battlefield situations or detect objects such as targets from images provided by SAR (synthetic aperture radar).

Recently, with the development of sensor technology, multiple images of various types are being collected from sensors. Accordingly, methods for accurately recognizing objects from multiple images are being studied.

However, conventional object recognition technology performs object recognition using only collected images, which has the problem of deteriorating object recognition performance for images collected in low-light environments or limited visibility environments due to fog or the like.

Korean Patent Publication No. 10-2022-0148053 (2022.11.04.)

The purpose of the present invention is to provide a graph-based object recognition method and device that can recognize objects using different types of text data and image data for objects.

An object recognition method according to an embodiment of the present invention includes receiving a plurality of text data and a plurality of image data for an object from the outside and grouping them; Converting the text data of each group into a text graph using a previously learned text graph conversion unit; Converting the image data of each group into an image graph using a previously learned image graph conversion unit; and recognizing the object by connecting the text graph and the image graph using a previously learned object recognition unit.

The grouping may include determining spatiotemporal identity between the plurality of text data and the plurality of image data based on one or more spatiotemporal context information; and grouping one or more text data and image data having the same time and space into one group among the plurality of text data and the plurality of image data based on the determination result.

Here, the one or more spatio-temporal context information includes location coordinates and timestamps where each of the plurality of text data and the plurality of image data occurred.

The converting into a text graph includes extracting a plurality of keywords from the text data of each group; Extracting one or more key keywords by determining the importance of each of the plurality of keywords; generating an initial text graph using the one or more key keywords as nodes; determining a common frequency of appearance of the one or more key keywords in the text data; and generating the text graph by applying edge weights to the initial text graph based on the determination result.

The text graph conversion unit includes a keyword extraction unit, and the keyword extraction unit is trained to extract and output the one or more key keywords from the text data when the extracted correct answer is input as label data together with the text data.

The converting into an image graph includes extracting a plurality of objects and a plurality of tags corresponding to each of the plurality of objects from the image data of each group; generating an initial image graph using each of the plurality of tags as a node; determining the importance of each node of the initial image graph; and generating the image graph through node pruning of the initial image graph based on the determination result.

The image graph conversion unit includes an object extraction unit, and the object extraction unit is trained to extract and output the plurality of objects and the plurality of tags from the image data when receiving an extraction answer as label data together with the image data. .

Recognizing the object may include determining whether entities of the text graph and the image graph are identical; connecting the text graph and the image graph based on a determination result; and outputting a recognition result for the object based on the connected graph.

The object recognition unit includes an identity determination unit, and the identity determination unit is trained to determine whether the text graph and the image graph are identical when receiving the correct answer as label data together with the text graph and the image graph and output the same. .

An object recognition device according to an embodiment of the present invention includes a memory storing an object recognition program; and executing the object recognition program, wherein the object recognition unit includes an identity determination unit, receives a plurality of text data and a plurality of image data for the object from the outside, groups them, and uses a previously learned text graph conversion unit. Convert the text data of each group into a text graph, convert the image data of each group into an image graph using a previously learned image graph conversion unit, and convert the text graph and the image graph using a previously learned object recognition unit. It includes a processor that connects and recognizes the object.

The present invention can convert different types of text data and image data about objects into graphs, determine whether the converted graphs are identical, and recognize and output the objects.

Accordingly, the present invention can increase the accuracy of object recognition at a specific place or time by recognizing the object using the corresponding text data even if image data is collected in a low-light environment or a limited visibility environment.

In addition, the present invention determines whether the converted graphs of image data and text data are identical and recognizes objects using the connected graphs, thereby resolving the complexity of the neural network model for object recognition compared to the prior art. This can improve the speed of object recognition.

1 is a diagram showing a graph-based object recognition device according to an embodiment of the present invention.
FIG. 2 is a diagram conceptually showing the function of the object recognition program of FIG. 1.
FIG. 3 is a diagram showing the text graph conversion unit of FIG. 2.
Figure 4 is a diagram showing the image graph conversion unit of Figure 2.
FIG. 5 is a diagram showing the object recognition unit of FIG. 2.
Figure 6 is a diagram showing a graph-based object recognition method according to an embodiment of the present invention.
7 to 9 are diagrams specifically showing the graph-based object recognition method of the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to be understood by those skilled in the art in the technical field to which the present invention pertains. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

In describing embodiments of the present invention, if a detailed description of a known function or configuration is judged to unnecessarily obscure the gist of the present invention, the detailed description will be omitted. The terms described below are terms defined in consideration of functions in the embodiments of the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

1 is a diagram showing a graph-based object recognition device according to an embodiment of the present invention.

Referring to FIG. 1, the object recognition device 100 of this embodiment may include an input/output unit 110, a processor 120, and a memory 130.

The input/output unit 110 may receive various types of data about objects detected by an external collection device, for example, a plurality of sensors. Additionally, the input/output unit 110 may output an object recognition result corresponding to object data generated by the processor 120, which will be described later, to an external device, such as a server.

Here, the object data may include a plurality of text data and a plurality of image data collected from each of one or more objects at a specific location or a specific point in time. Such object data may include information such as the type, name, shape, quantity, and location of the object, information such as the location, terrain, and environment of the place where the object is located, and information on the time when the object data was generated.

The processor 120 receives object data, that is, a plurality of text data and a plurality of image data through the input/output unit 110, and uses the object recognition program 140 stored in the memory 130 to recognize the object data at a specific location or at a specific point in time. Recognition results for one or more objects can be generated and output.

The memory 130 may store the object recognition program 140 and information necessary for its execution. The object recognition program 140 may be software that includes instructions that can recognize one or more objects from a plurality of text data and a plurality of image data provided through the input/output unit 110 and generate a result.

Accordingly, when a plurality of text data and image data for an object are received through the input/output unit 110, the processor 120 executes the object recognition program 140 of the memory 130 to identify the previously received plurality of texts and image data. Object recognition results for one or more objects can be generated and output from image data.

FIG. 2 is a diagram conceptually showing the function of the object recognition program of FIG. 1.

Referring to FIG. 2, the object recognition program 140 of this embodiment includes a data grouping unit 141, a text graph conversion unit 143, an image graph conversion unit 145, and an object recognition unit 147. can do.

The data grouping unit 141, text graph conversion unit 143, image graph conversion unit 145, and object recognition unit 147 shown in FIG. 2 are used to easily explain the functions of the object recognition program 140 of this embodiment. As conceptually divided for the sake of clarity, the present invention is not limited thereto.

For example, according to an embodiment of the present invention, the functions of the data grouping unit 141, text graph conversion unit 143, image graph conversion unit 145, and object recognition unit 147 of the object recognition program 140 are merged. It can be separated or implemented as a series of instructions included in one program.

The data grouping unit 141 may group a plurality of text data and a plurality of image data provided through the input/output unit 110 into a plurality of groups.

For example, the data grouping unit 141 may determine spatiotemporal identity between a plurality of text data and a plurality of image data based on one or more spatiotemporal context information extracted from each of the plurality of text data and the plurality of image data. .

Additionally, the data grouping unit 141 may group one or more text data and image data having the same time and space into one group among a plurality of text data and a plurality of image data according to the identity determination result.

Accordingly, the grouped data group may include one or more text data and image data collected for one or more objects at a specific place and time.

Here, the spatiotemporal context information may include the location where each of the object data, that is, the text data and the image data, was generated, that is, the coordinates for the collected location, and a timestamp for the point of collection.

Meanwhile, the data grouping unit 141 of this embodiment further expands the spatiotemporal context information collected from each of the plurality of text data and the plurality of image data temporally or spatially, thereby improving the grouping performance of the plurality of text data and the plurality of image data. You can also increase it.

The text graph conversion unit 143 may generate a text graph by converting one or more text data from each of the plurality of groups grouped by the data grouping unit 141 into a graph form.

This text graph conversion unit 143 may include a previously learned neural network model for generating a text graph.

FIG. 3 is a diagram showing the text graph conversion unit of FIG. 2.

Referring to FIG. 3, the text graph conversion unit 143 of this embodiment may include a keyword extraction unit 151 and a text graph creation unit 153.

When receiving text data, the keyword extraction unit 151 learns to extract a plurality of keywords, such as nouns or verbs, from the text data, determine the importance of each of the plurality of keywords, and extract and output one or more key keywords. It can be.

Additionally, the keyword extraction unit 151 may generate a loss value using the output core keywords and repeatedly perform learning to extract one or more core keywords from the above-described text data so that the loss value is minimized.

To this end, the keyword extraction unit 151 may receive the extracted correct answer, for example, the actual key keyword for the text data, as label data along with the text data. Accordingly, the keyword extraction unit 151 may compare the extracted correct answer with the actual output extraction result, that is, one or more key keywords, and generate a loss value according to the comparison result.

The text graph generator 153 may generate a text graph corresponding to text data based on one or more key keywords output from the keyword extractor 151.

For example, the text graph generator 153 may generate an initial text graph using one or more key keywords as nodes. Here, each node of the initial text graph may be connected by an edge.

The text graph generator 153 may determine the frequency of appearance of one or more key keywords in the text data, for example, the common frequency of appearance for one or more key keywords. The text graph generator 153 may assign edge weights to each node of the initial text graph based on the determination result, and generate and output a text graph including the weighted edges.

Referring again to FIG. 2, the image graph conversion unit 145 may generate an image graph by converting one or more image data from each of a plurality of groups grouped by the data grouping unit 141 into a graph form.

This image graph conversion unit 145 may include a previously learned neural network model for generating an image graph.

Figure 4 is a diagram showing the image graph conversion unit of Figure 2.

Referring to FIG. 4, the image graph conversion unit 145 of this embodiment may include an object extraction unit 161 and an image graph creation unit 163.

When receiving image data, the object extractor 161 may be trained to extract and output one or more objects included in the image data and tags corresponding to them.

Here, the object can be extracted in the form of an image, and the tag can be extracted in the form of text for the name of the object.

Additionally, the object extractor 161 may generate a loss value using the output object and tag, and repeatedly perform learning to extract the object and tag from the image data to minimize the loss value.

To this end, the object extraction unit 161 may receive the image data as well as the extraction answer, for example, the actual object and tag of the image data as label data. Accordingly, the object extraction unit 161 may compare the extracted correct answer with the actual output result, that is, one or more objects and tags, and generate a loss value according to the comparison result.

The image graph generator 163 may generate an image graph corresponding to image data based on the objects and tags extracted from the object extractor 161.

For example, the image graph generator 163 may generate an initial image graph using one or more tags as nodes. Here, each node in the initial image graph may be connected to an edge.

The image graph generator 163 may calculate information entropy for each node of the initial image graph and determine the importance of each node based on the calculated information entropy.

The image graph generator 163 may prune each node of the initial image graph based on the determined importance, and generate and output an image graph accordingly.

Here, the image graph generator 163 performs pruning to delete at least one node of the initial image graph based on the entropy change calculated in the path between the lowest node of the initial image graph and the upper nodes connected to it through edges. can do.

Referring again to FIG. 2, the object recognition unit 147 connects the text graph and the image graph to create a connected graph, recognizes one or more objects in each of the plurality of groups based on the created connected graph, and outputs the result. can do.

This object recognition unit 147 may include a previously learned neural network model for generating object recognition results.

FIG. 5 is a diagram showing the object recognition unit of FIG. 2.

Referring to FIG. 5 , the object recognition unit 147 may include an identity determination unit 171 and a graph connection unit 173.

The identity determination unit 171 may be trained to determine whether the entities of the text graph and the entities of the image graph are identical when receiving text graphs and image graphs respectively, and output the same.

Additionally, the identity determination unit 171 may generate a loss value based on the output identity determination result and repeat learning of the identity determination to minimize the loss value.

To this end, the identity determination unit 171 may receive the text graph and the image graph as well as the correct answer to determine, for example, whether the entities of the two graphs are the same as label data. Accordingly, the identity determination unit 171 may compare the determination correct answer with the actual output result, that is, the identity determination result, and generate a loss value according to the comparison result.

The graph connection unit 173 may create a connected graph by connecting the text graph and the image graph based on the determination result of the identity determination unit 171.

Accordingly, the object recognition unit 147 can recognize the object from the connected graph and output the object recognition result.

In this way, the object recognition device 100 of this embodiment converts different types of text data and image data provided from the outside into graphs, determines whether the converted graphs are identical, and recognizes and outputs the object. .

Accordingly, the present invention can increase the accuracy of object recognition at a specific place or time by recognizing the object using the corresponding text data even if image data is collected in a low-light environment or a limited visibility environment.

In addition, the present invention converts each of the image data and text data into a graph, determines whether the converted graphs are identical, and recognizes the object using the connected graph, thereby reducing the complexity of the neural network model for object recognition compared to the prior art. This can be resolved, thereby improving the speed of object recognition.

Figure 6 is a diagram showing a graph-based object recognition method according to an embodiment of the present invention, and Figures 7 to 9 are diagrams specifically showing the graph-based object recognition method of the present invention.

Referring to FIG. 6, the object recognition device 100 of this embodiment may receive object data including a plurality of text data and a plurality of image data for one or more objects from a plurality of external sensors.

Accordingly, the processor 120 executes the object recognition program 140 stored in the memory 130, recognizes one or more objects at a specific location or at a specific time from the received plurality of text data and plurality of image data, and results in can be output.

For example, the processor 120 may group a plurality of text data and a plurality of image data into a plurality of groups through the data grouping unit 141 (S10).

At this time, the data grouping unit 141 extracts one or more context information from each of the plurality of text data and the plurality of image data, and determines spatiotemporal identity between the plurality of text data and the plurality of image data based on the extracted context information. You can.

Next, the data grouping unit 141 may group one or more text data and image data having the same time and space among the plurality of text data and the plurality of image data into the same group based on the determination result.

Next, the processor 120 may generate a text graph by converting the text data of each of the plurality of groups into a graph form using the previously learned text graph conversion unit 143 (S20).

Referring to FIG. 7, the keyword extraction unit 151 of the text graph conversion unit 143 can extract a plurality of keywords from text data. Next, the keyword extraction unit 151 determines the importance of each of the plurality of extracted keywords, and extracts at least one core keyword from the plurality of keywords according to the judgment result (S110).

Next, the text graph generation unit 153 of the text graph conversion unit 143 may generate an initial text graph using the extracted key keywords as nodes (S120). At this time, each node in the initial text graph may be connected through an edge.

Continuing, the text graph generator 153 may determine the common frequency of appearance of key keywords within the text data (S130).

Next, the text graph generator 153 may calculate a weight for the edges between nodes of the initial text graph according to the determination result, and apply the calculated edge weight to generate a text graph corresponding to the text data (S140). .

Referring again to FIG. 6, the processor 120 may generate an image graph by converting the image data of each of the plurality of groups into a graph form using the previously learned image graph conversion unit 145 (S30).

Referring to FIG. 8, the object extraction unit 161 of the image graph conversion unit 145 may extract one or more objects and tags corresponding to them from image data (S210).

Next, the image graph generation unit 163 of the image graph conversion unit 145 may generate an initial image graph using one or more extracted tags as nodes (S220). At this time, each node in the initial image graph may be connected through an edge.

Subsequently, the image graph generator 163 may calculate the information entropy for each node of the initial image graph and determine the importance of each node based on the calculated information entropy (S230).

Next, the image graph generator 163 may prune each node of the initial image graph based on the determined importance, and generate and output an image graph accordingly (S240).

Here, the image graph generator 163 performs pruning to delete at least one node of the initial image graph based on the entropy change calculated in the path between the lowest node of the initial image graph and the upper nodes connected to it through edges. can do.

Referring again to FIG. 6, the processor 120 connects the text graph and the image graph of each of the plurality of groups using the previously learned object recognition unit 147, and based on the connected graph, one or more Objects can be recognized and the results can be output (S40).

Referring to FIG. 9, the identity determination unit 171 of the object recognition unit 147 may extract one or more entities from each of the text graph and the image graph. Next, the identity determination unit 171 may analyze the equivalence relationship between the entities of the text graph and the entities of the image graph, and determine whether the text graph and the image graph are identical accordingly (S310).

If it is determined that the text graph and the image graph are the same, the graph connection unit 173 may connect the text graph and the image graph based on the determination result (S320).

Accordingly, the object recognition unit 147 can recognize the object from the connected graph and output the object recognition result (S330).

On the other hand, if it is determined that the text graph and the image graph are not identical, the identity determination unit 171 may determine whether other text graphs and image graphs are identical.

In this way, the present invention can convert different types of text data and image data provided from the outside into graphs, determine whether the converted graphs are identical, and recognize and output objects.

Accordingly, the present invention can increase the accuracy of object recognition at a specific place or time by recognizing the object using the corresponding text data even if image data is collected in a low-light environment or a limited visibility environment.

Combinations of each block of the block diagram of the present invention and each step of the flowchart described above may be performed by computer program instructions. Since these computer program instructions can be mounted on the encoding processor of a general-purpose computer, special-purpose computer, or other programmable data processing equipment, the instructions performed through the encoding processor of the computer or other programmable data processing equipment are included in each block or block of the block diagram. Each step of the flowchart creates a means to perform the functions described. These computer program instructions may also be stored in computer-usable or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement a function in a particular way, so that the computer-usable or computer-readable memory The instructions stored in can also produce manufactured items containing instruction means that perform the functions described in each block of the block diagram or each step of the flowchart. Computer program instructions can also be mounted on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable data processing equipment to create a process that is executed by the computer, thereby generating a process that is executed by the computer or other programmable data processing equipment. Instructions that perform processing equipment may also provide steps for executing functions described in each block of the block diagram and each step of the flowchart.

Additionally, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s). Additionally, it should be noted that in some alternative embodiments it is possible for the functions mentioned in the blocks or steps to occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially simultaneously, or the blocks or steps may sometimes be performed in reverse order depending on the corresponding function.

The above description is merely an illustrative explanation of the technical idea of the present invention, and those skilled in the art will be able to make various modifications and variations without departing from the essential quality of the present invention. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but are for illustrative purposes, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention shall be interpreted in accordance with the claims below, and all technical ideas within the scope equivalent thereto shall be construed as being included in the scope of rights of the present invention.

100: Object recognition device
110: input/output unit
120: processor
130: memory
140: Object recognition program
141: Data grouping unit
143: Text graph conversion unit
145: Image graph conversion unit
147: Object recognition unit

Claims (20)

Receiving a plurality of text data and a plurality of image data for an object from the outside and grouping them;
Converting the text data of each group into a text graph using a previously learned text graph conversion unit;
Extracting a plurality of objects and a plurality of tags corresponding to each of the plurality of objects from the image data of each group using a pre-learned image graph conversion unit, and generating an initial image graph with each of the plurality of tags as a node, converting the image data of each group into an image graph by pruning nodes of the initial image graph based on a result of determining the importance of each node of the initial image graph; and
A graph-based object recognition method comprising recognizing the object by connecting the text graph and the image graph using a previously learned object recognition unit.
According to paragraph 1,
The grouping step is,
determining spatiotemporal identity between the plurality of text data and the plurality of image data based on one or more spatiotemporal context information; and
A graph-based object recognition method comprising grouping one or more text data and image data having the same time and space among the plurality of text data and the plurality of image data into one group based on a determination result.
According to paragraph 2,
The one or more spatio-temporal context information is,
A graph-based object recognition method including location coordinates and timestamps where each of the plurality of text data and the plurality of image data occurred.
According to paragraph 1,
The step of converting to a text graph is,
extracting a plurality of keywords from the text data of each group;
Extracting one or more key keywords by determining the importance of each of the plurality of keywords;
generating an initial text graph using the one or more key keywords as nodes;
determining a common frequency of appearance of the one or more key keywords in the text data; and
A graph-based object recognition method comprising generating the text graph by applying an edge weight to the initial text graph based on a determination result.
According to paragraph 1,
The text graph conversion unit includes a keyword extraction unit,
The keyword extraction unit,
A graph-based object recognition method that is learned to extract and output the one or more core keywords from the text data when the extracted answer is input as label data together with the text data.
delete According to paragraph 1,
The image graph conversion unit includes an object extraction unit,
The object extraction unit,
A graph-based object recognition method that is learned to extract and output the plurality of objects and the plurality of tags from the image data when an extraction answer is input as label data together with the image data.
According to paragraph 1,
The step of recognizing the object is,
determining whether entities of the text graph and the image graph are identical;
connecting the text graph and the image graph based on a determination result; and
A graph-based object recognition method comprising outputting a recognition result for the object based on a connected graph.
According to clause 8,
The object recognition unit includes an identity determination unit,
The identity determination unit,
A graph-based object recognition method that is learned to determine whether the text graph and the image graph are identical and output them when the correct answer is input as label data along with the text graph and the image graph.
Memory in which an object recognition program is stored; and
Execute the object recognition program, receive a plurality of text data and a plurality of image data for the object from the outside, group them, and convert the text data of each group into a text graph using a previously learned text graph conversion unit. Then, a plurality of objects and a plurality of tags corresponding to each of the plurality of objects are extracted from the image data of each group using a pre-learned image graph conversion unit, and an initial image graph is generated with each of the plurality of tags as a node. And, based on the result of determining the importance of each node of the initial image graph, the image data of each group is converted into an image graph through node pruning of the initial image graph, and the image data of each group is converted into an image graph using a pre-learned object recognition unit. A graph-based object recognition device comprising a processor that recognizes the object by connecting a text graph and the image graph.
According to clause 10,
The processor,
Determine spatiotemporal identity between the plurality of text data and the plurality of image data based on one or more spatiotemporal context information, and determine the spatiotemporal identity between the plurality of text data and the plurality of image data based on the determination result. One or more texts having the same space and time among the plurality of text data and the plurality of image data A graph-based object recognition device that groups data and image data into one group.
According to clause 11,
The one or more spatio-temporal context information is,
A graph-based object recognition device including location coordinates and timestamps where each of the plurality of text data and the plurality of image data occurred.
According to clause 10,
The processor,
Extracting a plurality of keywords from the text data of each group, determining the importance of each of the plurality of keywords to extract one or more core keywords, generating an initial text graph with the one or more core keywords as nodes, and generating the text A graph-based object recognition device that determines the common frequency of appearance of the one or more key keywords in data and generates the text graph by applying edge weights to the initial text graph based on the determination result.
According to clause 10,
The text graph conversion unit includes a keyword extraction unit,
The keyword extraction unit,
A graph-based object recognition device that is trained to extract and output the one or more key keywords from the text data when the extracted answer is input as label data together with the text data.
delete According to clause 10,
The image graph conversion unit includes an object extraction unit,
The object extraction unit,
A graph-based object recognition device that is trained to extract and output the plurality of objects and the plurality of tags from the image data when an extraction answer is input as label data together with the image data.
According to clause 10,
The processor,
Determine whether the entities of the text graph and the image graph are identical, connect the text graph and the image graph based on the determination result, and output recognition results for the object based on the connected graph. A graph-based object recognition device.
According to clause 10,
The object recognition unit includes an identity determination unit,
The identity determination unit,
A graph-based object recognition device that is trained to determine whether the text graph and the image graph are identical when receiving a correct answer as label data along with the text graph and the image graph and output the result.
A computer-readable recording medium storing a computer program,
The computer program is,
Receiving a plurality of text data and a plurality of image data for an object from the outside and grouping them;
Converting the text data of each group into a text graph using a previously learned text graph conversion unit;
Extracting a plurality of objects and a plurality of tags corresponding to each of the plurality of objects from the image data of each group using a pre-learned image graph conversion unit, and generating an initial image graph with each of the plurality of tags as a node, converting the image data of each group into an image graph by pruning nodes of the initial image graph based on a result of determining the importance of each node of the initial image graph; and
A computer-readable recording medium comprising instructions for a processor to perform a graph-based object recognition method comprising recognizing the object by connecting the text graph and the image graph using a previously learned object recognition unit.
A computer program stored on a computer-readable recording medium,
The computer program is,
Receiving a plurality of text data and a plurality of image data for an object from the outside and grouping them;
Converting the text data of each group into a text graph using a previously learned text graph conversion unit;
Extracting a plurality of objects and a plurality of tags corresponding to each of the plurality of objects from the image data of each group using a pre-learned image graph conversion unit, and generating an initial image graph with each of the plurality of tags as a node, converting the image data of each group into an image graph by pruning nodes of the initial image graph based on a result of determining the importance of each node of the initial image graph; and
A computer program stored in a recording medium including instructions for a processor to perform a graph-based object recognition method including recognizing the object by connecting the text graph and the image graph using a previously learned object recognition unit.