JP7395915B2 - Information processing device, computer program and information processing method - Google Patents

Description

The present invention relates to an information processing device, a computer program, and an information processing method.

Various methods have been proposed to support layout creation for magazines, books, newspapers, etc. Patent Document 1 discloses a method for extracting a plurality of contents from a document, determining the position of each content on the document based on the degree of semantic relationship between the extracted contents, and arranging the content at the determined position. Disclosed is an information processing device that generates a new document based on a document.

JP2009-169536A

However, in the information processing apparatus disclosed in Patent Document 1, when performing a required operation (for example, copying, moving, etc.) on content within a document, it is necessary to perform the operation individually for each content. In particular, for related content, there is a high possibility that the same operation will be repeated, making it complicated.

The present invention has been made in view of the above circumstances, and it is an object of the present invention to provide an information processing device, a computer program, and an information processing method that can improve the operability of content within a document.

An information processing device according to an embodiment of the present invention includes a specifying unit that identifies whether or not an input set of content is related, using a trained model that has learned a set of related content as training data. , and an extraction unit that associates the related content sets identified by the identification unit and extracts them as cluster content.

A computer program according to an embodiment of the present invention provides a computer with a process of determining whether or not a set of input contents is related, using a trained model that has learned a set of related contents as teacher data. and a process of correlating the identified set of related contents and extracting them as cluster contents.

An information processing method according to an embodiment of the present invention uses a trained model that has learned a set of related contents as training data to identify whether or not there is a relationship between a set of input contents. A set of related content is associated and extracted as cluster content.

According to the present invention, the operability of content within a document is improved.

FIG. 1 is a block diagram showing an example of the configuration of an information processing device according to the present embodiment. FIG. 3 is a schematic diagram showing an example of layout data. It is a schematic diagram which shows an example of a related graph. FIG. 2 is a schematic diagram illustrating an example of a method for calculating feature amounts of an image. FIG. 2 is a schematic diagram illustrating an example of a method for calculating a feature amount of a caption. FIG. 2 is a schematic diagram illustrating a first example of determining the relevance between an image and a caption. FIG. 7 is a schematic diagram showing a second example of determining the relevance between an image and a caption. FIG. 2 is a schematic diagram showing a first example of a neural network learning method. FIG. 2 is a schematic diagram showing a second example of a neural network learning method. FIG. 3 is a schematic diagram showing a first example of operations on cluster content. FIG. 7 is a schematic diagram showing a second example of operations on cluster content. 3 is a flowchart illustrating an example of cluster content extraction processing of the information processing device.

Embodiments of the present disclosure will be described below based on the drawings. FIG. 1 is a block diagram showing an example of the configuration of an information processing device 50 according to the present embodiment. The information processing device 50 can be connected to the server 10 via the communication network 1. Further, a scanner 20 can be connected to the information processing device 50. The server 10 can be a data server that stores layout data, but is not limited thereto.

The information processing device 50 includes a control section 51 that controls the entire device, a communication section 52, a storage section 53, a layout data estimation section 54, a specification section 55, an extraction section 56, a display panel 57, a display section 58, and an operation section 59. .

The control unit 51 can be configured with a CPU, ROM, RAM, and the like.

The communication unit 52 has a function of communicating with the server 10 via the communication network 1, and can send and receive required information. More specifically, the communication unit 52 can acquire layout data from the server 10.

FIG. 2 is a schematic diagram showing an example of layout data. The layout data includes, for example, information representing a state in which a plurality of contents are arranged in a layout frame excluding a blank space in an area corresponding to one page of a document. The content includes, for example, a title, a main body (text), an image (figure), a caption (an explanatory text of the image), and the like. The layout data is data necessary for arranging the contents, and includes, for example, the size of each content, the coordinates of the contents, the relative coordinates between the contents, and the like. In the example of FIG. 2, a title, texts A, B, images A, B, C, and captions A, B, C are arranged within one page of the document. Note that the layout data is not limited to the example shown in FIG. 2.

The communication unit 52 also has an interface function with the scanner 20 and can obtain, for example, a PDF file from the scanner 20.

The layout data estimation unit 54 includes an object detection neural network, and can image the PDF file obtained from the scanner 20 and estimate layout data.

The storage unit 53 can be configured with a hard disk or a flash memory, and can store the layout data acquired via the communication unit 52 and the layout data estimated by the layout data estimation unit 54.

The specifying unit 55 uses a learned model that has learned a set of related content as teacher data to specify whether or not the input set of content is related. The trained models are neural networks 551, 552, and 553, which will be described later.

The specifying unit 55 specifies a set of related contents based on the layout data. "Related content" means, for example, that there is a high possibility that the user will repeat similar operations (for example, copying, moving, deleting, scaling, etc.) on the content in the document displayed on the display panel 57. The content can be the same. Contents that are related to each other can be schematically represented, for example, by a relation graph.

FIG. 3 is a schematic diagram showing an example of a related graph. As shown in FIG. 3, it is assumed that one layout includes images G1, G2, G3 and captions C1, C2, C3 as a plurality of contents. It is assumed that image G1 and captions C1 and C2 are related to each other, and caption C3 and images G2 and G3 are related to each other. In this case, the related graph can be represented by a graph in which the image G1 and the caption C1 are connected, and the image G1 and the caption C2 are connected. Further, the related graph can be represented by a graph in which the caption C3 and the image G2 are connected, and the caption C3 and the image G3 are connected.

Next, a method for determining the relevance between contents will be explained. In the following, as shown in FIG. 3, images and captions will be used as examples of content, but the same applies to other content. To determine the relevance, image feature amounts, caption feature amounts, and relative position information (relative coordinates) between contents (image and caption) are used. First, a method for calculating the feature amount of an image will be explained.

FIG. 4 is a schematic diagram showing an example of a method for calculating the feature amount of an image. The identification unit 55 includes a neural network 551. The neural network 551 is, for example, a convolutional neural network, and includes an input layer 551a, a convolution layer 551b, a pooling layer 551c, a convolution layer 551d, a pooling layer 551e, and a fully connected layer 551f connected in this order. Note that the numbers of convolution layers, pooling layers, and fully connected layers are for convenience and are not limited to the numbers shown in FIG. 4. Furthermore, for convenience, the activation function layer and output layer are omitted. The image G1 is input to the input layer 551a. Since the fully connected layer 551f is a combination of the features of the input image G1, the feature amount g1 (vector) can be calculated from the fully connected layer 551f. The feature amounts g2 and g3 of the other images G2 and G3 can be calculated in the same manner. Note that the calculation of image features is not limited to a method using a neural network, and general image processing such as edge detection, line detection, region segmentation, and texture analysis may be used.

Next, a method for calculating the feature amount of a caption will be explained.

FIG. 5 is a schematic diagram showing an example of a method for calculating the feature amount of a caption. The identification unit 55 includes a neural network 552. The neural network 552 is, for example, word2vec, and has an input layer 552a, a hidden layer 552b, and an output layer 552c connected in this order. The input layer 552a and the hidden layer 552b and the hidden layer 552b and the output layer 552c are fully connected with weights W and W', respectively. A word (or a string of words) extracted from the caption C1 is input to the input layer 552a by a language processing unit (for example, morphological analysis processing) included in the specifying unit 55. Specifically, the caption C1 is divided into a plurality of words by morphological analysis, and the divided words are converted into vectors and input into the input layer 552a. In this case, a vector obtained by averaging the vectors of each word may be input to the input layer 552a. When a vector is input to the input layer 552a, the meaning of the caption C1 is expressed as a vector, and the feature quantity c1 (vector) can be calculated. The feature amounts c2 and c3 of the other captions C2 and C3 can be calculated in the same manner. Note that the calculation of the feature amount of the caption is not limited to the method using a neural network, and general language processing may be used. For example, dictionary data may be used to extract words, which are the smallest meaningful units, from the caption, and the extracted words may be converted into a vector with a required number of dimensions. Note that the feature amount of the title and the feature amount of the text can also be calculated using the neural network 552.

Next, a method for determining the relevance between contents (images and captions) will be explained.

FIG. 6 is a schematic diagram showing a first example of determining the relevance between images and captions. The identification unit 55 includes a neural network 553. Input data is input to the neural network 553. The input data is a vector whose components are an image feature amount, a caption feature amount, and relative position information between the image and the caption. The relative position information includes the relative coordinates between the coordinates of the image G1 on the layout and the coordinates of the caption C1, the size of the image G1 on the layout, the size of the caption C1 on the layout, and the like. In the example of FIG. 6, the feature amount g1 (vector) of the image G1, the feature amount c1 (vector) of the caption C1, and the relative position information (vector) between the image G1 and the caption C1 are set as one vector.

Since the score output by the neural network 553 is equal to or greater than the threshold value, it can be determined that the image G1 and the caption C1 are related, as shown in the related graph shown in FIG. 3. Note that the neural network 553 may use other machine learning methods such as SVM (Support Vector Machine) and Bayesian network.

FIG. 7 is a schematic diagram showing a second example of determining the relevance between images and captions. In the example of FIG. 7, the feature amount g2 (vector) of the image G2, the feature amount c1 (vector) of the caption C1, and the relative position information (vector) between the image G2 and the caption C1 are set as one vector, and the neural network 553 is I am typing.

Since the score output by the neural network 553 is less than the threshold, it can be determined that the image G2 and the caption C1 are not related, as shown in the relation graph shown in FIG. 3.

As described above, each feature amount and relative position information are input to the neural network 553 for any set of images and captions on the layout, and the presence or absence of relevance is determined for each set.

Next, a learning method of the neural network 553 will be explained.

FIG. 8 is a schematic diagram showing a first example of a learning method of the neural network 553. Vectors as learning input data are input to the input layer of the neural network 553. The learning vector is a vector whose components are the feature amount of the image, the feature amount of the caption, and the relative position information between the image and the caption. In the example of FIG. 8, the feature amount g1 (vector) of the image G1, the feature amount c2 (vector) of the caption C2, and the relative position information (vector) between the image G1 and the caption C2 are set as one vector. As shown in FIG. 3, when there is a relationship between the image G1 and the caption C2, a teacher label "1" is given to the output layer, and the neural network 553 is trained. The teacher label "1" is a label indicating that this is a correct answer set.

FIG. 9 is a schematic diagram showing a second example of the learning method of the neural network 553. Vectors as learning input data are input to the input layer of the neural network 553. The learning vector is a vector whose components are the feature amount of the image, the feature amount of the caption, and the relative position information between the image and the caption. In the example of FIG. 9, the feature amount g2 (vector) of the image G2, the feature amount c1 (vector) of the caption C1, and the relative position information (vector) between the image G2 and the caption C1 are set as one vector. As shown in FIG. 3, when there is no relationship between the image G2 and the caption C1, a teacher label "0" is given to the output layer, and the neural network 553 is trained. The teacher label "0" is a label indicating that the set is an incorrect answer.

The neural network 553 can be trained using a large number of learning input data and teacher labels as shown in FIGS. 8 and 9.

As described above, any two contents are selected from multiple contents in a document, a score (index) indicating the relationship between the selected contents is calculated, and if the calculated score is equal to or higher than a predetermined threshold, , it is determined that the content is relevant, and if the calculated score is less than the threshold value, it is determined that the content is not relevant, thereby making it possible to specify a set of relevant content.

Further, by simply reading a paper document using the scanner 20 or the like, it is possible to identify a set of related contents within the document.

The extraction unit 56 associates the related content sets identified by the identification unit and extracts them as cluster content. Cluster content is a collection of related contents, and the related contents can be treated as one piece of content.

With the above configuration, when a required operation is performed on cluster content, the operation is treated as having been performed on related contents, thereby improving the operability of content within a document.

Next, operations on cluster content will be explained.

The display unit 58 can display a document in which content is arranged on the display panel 57. The display panel 57 can be configured with a liquid crystal panel, an organic EL (Electro Luminescence) display, or the like. Note that the display panel 57 may be replaced with a configuration including a display device separate from the information processing device 50.

The operation unit 59 is configured with, for example, a hardware keyboard, a mouse, etc., and is capable of operating icons displayed on the display panel 57, inputting characters, etc. Note that the operation unit 59 may be configured with a touch panel.

FIG. 10 is a schematic diagram showing a first example of operations on cluster content. As shown in FIG. 10, a document in which a plurality of contents are arranged (for example, one page or two double-page pages) is displayed on the display panel 57. In the example of FIG. 10, the title, text A, text B, image A, caption A, and caption B are displayed as the contents. Further, it is assumed that image A and captions A and B are related to each other.

As shown in the left figure of FIG. 10, when the icon 100 is brought close to image A (or the vicinity of image A, or caption A or B) and a touch operation and a drag operation are performed, the image A is displayed as shown in the right figure. Captions A and B can be moved in the same way. Image A, caption A, and caption B constitute one cluster content 101.

In this way, when the display unit 58 receives an operation to select the cluster content displayed on the display panel 57, the display unit 58 displays each of the contents associated by the cluster content in the selected display mode. For example, if you perform an operation to select one content or the vicinity of the content in the cluster content displayed on the display panel 57 and move (drag) on the display panel 57, all the content in the cluster content is displayed as selected. The entire cluster content can be moved (dragged). This eliminates the need to repeat similar operations for related content, improving the operability of content within a document.

FIG. 11 is a schematic diagram showing a second example of operations on cluster content. As shown in FIG. 11, a document in which a plurality of contents are arranged (for example, one page or two double-page pages) is displayed on the display panel 57. In the example of FIG. 11, text A, text B, text C, image A, and caption A are displayed as contents. Further, it is assumed that text B and text C are related to each other.

As shown in the left diagram of FIG. 11, when the icon 100 is brought close to text B (or the vicinity of text B, or even text C) and touch and drag operations are performed, the text C can be moved in the same way. Main text B and main text C constitute one cluster content 102. This eliminates the need to repeat similar operations for related content, improving the operability of content within a document.

The control unit 51 has a function as a grouping processing unit, and groups cluster contents as one content. Specifically, when performing a predetermined process on one content of the cluster content, the control unit 51 can perform the same process as the predetermined process on other contents of the cluster content. For example, if the predetermined process is to edit content within a document (e.g., copy, move, scale, delete, etc.), if the editing process is performed on one content, other content related to the one content will be edited. Similar editing processing can be performed on the contents of the document, and the operability of the contents within the document can be improved.

Further, when performing a predetermined process on cluster content, the control unit 51 can perform the same process as the predetermined process on each content in the cluster content. For example, in the case of searching for content within a document as the predetermined process, the search can be performed in units of cluster content, thereby improving the operability of the content within the document.

FIG. 12 is a flowchart illustrating an example of cluster content extraction processing by the information processing device 50. Hereinafter, for convenience, the main body of processing will be described as the control unit 51. The control unit 51 acquires layout data (S11), and acquires structural information (for example, content size and coordinates) of the layout data (S12).

The control unit 51 selects a content set (S13) and extracts the feature amount of the selected content (S14). The control unit 51 determines the relevance of the selected content based on the extracted feature amount and the relative position information of the selected content (S15).

The control unit 51 determines whether there is any unprocessed content (S16), and if there is any unprocessed content (YES at S16), it continues the processing from step S13 onward. If there is no unprocessed content (NO in S16), the control unit 51 extracts related content together as cluster content (S17), and ends the process.

The information processing device 50 can also be realized using, for example, a computer including a CPU (for example, a multi-processor equipped with a plurality of processor cores), a GPU (Graphics Processing Unit), a RAM, and the like. Information processing is performed on the computer by loading a computer program (recordable on a recording medium) that defines the processing procedure shown in FIG. Apparatus 50 can be implemented.

The information processing device according to the present embodiment includes a specifying unit that uses a learned model that has learned a set of related contents as teacher data to specify whether or not a set of input contents is related; and an extracting section that associates a set of related contents identified by the section and extracts them as cluster contents.

The computer program of this embodiment includes a process of specifying whether or not a set of input contents is related, using a trained model that has been trained with a set of related contents as teacher data, and A process of associating a set of related contents and extracting them as cluster contents is executed.

The information processing method of this embodiment uses a learned model that has learned a set of related contents as training data to identify whether or not there is a relationship between the input set of content, and A certain set of contents are associated and extracted as cluster contents.

The identification unit identifies whether or not the input content set is related, using a trained model that has learned a set of related content as teacher data.
The content includes, for example, a title, a main body (text), an image (figure), a caption (an explanatory text of the image), and the like. The layout data is data necessary for arranging the contents, and includes, for example, the size of each content, the coordinates of the contents, the relative coordinates between the contents, and the like. "Related content" can be, for example, content that is likely to be repeatedly performed by a user in a similar manner (for example, copying, moving, etc.).

Select any two contents from a plurality of contents in a document, calculate an index indicating the relationship between the selected contents, and if the calculated index is equal to or higher than a predetermined threshold, determine that there is a relationship, If the calculated index is less than the threshold value, it is determined that there is no relevance, thereby making it possible to identify a set of relevant content.

The extraction unit associates the related content sets identified by the identification unit and extracts them as cluster content. Cluster content is a collection of related contents, and the related contents can be treated as one piece of content.

With the above configuration, when a required operation is performed on cluster content, the operation is treated as having been performed on related contents, thereby improving the operability of content within a document.

In the information processing apparatus of this embodiment, the input content set is a set of image data and document data.

This allows images and documents to be extracted as cluster content.

The information processing apparatus of this embodiment includes a grouping processing unit that groups the cluster contents as one content.

When performing a predetermined process on the cluster content extracted by the extraction unit, the grouping processing unit can perform the same process as the predetermined process on each content associated by the cluster content. For example, in the case of searching for content within a document as the predetermined process, the search can be performed in units of cluster content, thereby improving the operability of the content within the document.

Further, when performing a predetermined process on one content of the cluster content, the grouping processing unit can perform the same process as the predetermined process on other contents of the cluster content. For example, if the predetermined process is to edit content within a document (e.g., copy, move, scale, delete, etc.), if the editing process is performed on one content, other content related to the one content will be edited. Similar editing processing can be performed on the contents of the document, and the operability of the contents within the document can be improved.

The information processing device according to the present embodiment includes a display unit that displays cluster content extracted by the extraction unit on a display screen, and a reception unit that receives an operation for selecting the cluster content displayed on the display screen, When the reception unit receives the operation, the unit displays each of the contents associated by the cluster content in the selected display mode.

The display section displays the cluster content extracted by the extraction section on the display screen. When the reception unit receives an operation to select cluster content displayed on the display screen, the display unit displays each of the contents associated by the cluster content in the selected display mode. For example, if you perform an operation to select one content or the area around the content in the cluster content displayed on the display screen, and move (drag) the content on the display screen, all the content in the cluster content will be displayed as selected. The entire cluster content can be moved (dragged). This eliminates the need to repeat similar operations for related content, improving the operability of content within a document.

1 communication network 10 server 20 scanner 50 information processing device 51 control unit 52 communication unit 53 storage unit 54 layout data estimation unit 55 identification unit 551, 552, 553 neural network 56 extraction unit 57 display panel 58 display unit 59 operation unit

Claims (4)

If the image feature amount of an image included in a related content set, the document feature amount of a document included in the content set, and the relative position information between the image and the document are input, the relationship of the content set is input. an identification unit that identifies whether or not a set of contents is related from input layout data having a plurality of contents using a trained model that has been trained to output the presence or absence of a relationship;
an extraction unit that associates a set of related content identified by the identification unit and extracts it as cluster content;
a display unit that displays layout data including the cluster content extracted by the extraction unit on a display screen;
a reception unit that receives an operation for selecting cluster content displayed on the display screen;
The input content set is a set of an image and a document ,
The display section is
An information processing device that displays each of the contents associated by the cluster content in a selected display mode when the operation is accepted by the reception unit. The information processing apparatus according to claim 1, further comprising a grouping processing unit that groups the cluster contents as one content. to the computer,
If the image feature amount of an image included in a related content set, the document feature amount of a document included in the content set, and the relative position information between the image and the document are input, the relationship of the content set is input. A process of identifying the presence or absence of a relationship between a set of contents from input layout data having a plurality of contents using a trained model that has been trained to output the presence or absence of a relationship;
a process of associating the identified set of related content and extracting it as cluster content;
a process of displaying layout data including the extracted cluster content on a display screen;
executing a process of accepting an operation for selecting cluster content displayed on the display screen;
The input content set is a set of an image and a document ,
Furthermore, when the operation is accepted, the computer program causes the computer program to execute a process of displaying each of the contents associated by the cluster content in a selected display mode. If the image feature amount of an image included in a related content set, the document feature amount of a document included in the content set, and the relative position information between the image and the document are input, the relationship of the content set is input. Using a trained model that has been trained to output the presence or absence of gender , the system identifies the presence or absence of a relationship between a set of contents from input layout data containing multiple contents,
Correlate the identified sets of related content and extract them as cluster content,
Display the layout data including the extracted cluster content on the display screen,
Accepting an operation to select cluster content displayed on the display screen,
The input content set is a set of an image and a document ,
Furthermore, when the operation is accepted, the information processing method displays each content associated by the cluster content in a selected display mode.

Images