JP2021056871A - Information processor, computer program, and information processing method - Google Patents

Abstract

To provide an information processor, a computer program, and an information processing method capable of improving the operability for contents in a document.SOLUTION: An information processor includes: a determination part which uses a learned model obtained by learning a set of related contents as teacher data to determine whether or not there is relation in the set of contents inputted; and an extraction part which extracts in association with each other, as cluster contents, the set of the contents determined to be related to each other by the determination part.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device, a computer program, and an information processing method.

Various methods have been proposed to support the layout creation of magazines, books, newspapers, etc. In Patent Document 1, a plurality of contents are extracted from the document, the position of each content on the document is determined based on the degree of semantic relevance between the extracted plurality of contents, and the content is arranged at the determined position. An information processing device that generates a new document is disclosed.

JP-A-2009-169536

However, in the information processing apparatus of Patent Document 1, when performing a required operation (for example, an operation such as copying or moving) on the content in the document, it is necessary to perform the operation individually for each content. In particular, for related contents, there is a high possibility that the same operation will be repeated, which is complicated.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide an information processing device, a computer program, and an information processing method capable of improving the operability of the contents in a document.

The information processing apparatus according to the embodiment of the present invention uses a trained model in which a set of related contents is learned as teacher data, and a specific unit that specifies whether or not the set of input contents is related. , And an extraction unit that extracts a set of related contents specified by the specific unit as cluster contents in association with each other.

The computer program according to the embodiment of the present invention is a process of identifying whether or not a set of input contents is related by using a trained model in which a set of related contents is learned as teacher data on a computer. And the process of associating the specified set of related contents and extracting them as cluster contents.

In the information processing method according to the embodiment of the present invention, the presence or absence of relevance of the input content set is specified and specified by using a learned model in which the related content set is learned as teacher data. The related sets of contents are associated and extracted as cluster contents.

According to the present invention, the operability of the content in the document is improved.

It is a block diagram which shows an example of the structure of the information processing apparatus of this embodiment. It is a schematic diagram which shows an example of layout data. It is a schematic diagram which shows an example of the relation graph. It is a schematic diagram which shows an example of the calculation method of the feature amount of an image. It is a schematic diagram which shows an example of the calculation method of the feature amount of a caption. It is a schematic diagram which shows the 1st example of the relevance determination of an image and a caption. It is a schematic diagram which shows the 2nd example of the relevance determination of an image and a caption. It is a schematic diagram which shows the 1st example of the learning method of a neural network. It is a schematic diagram which shows the 2nd example of the learning method of a neural network. It is a schematic diagram which shows the 1st example of the operation with respect to a cluster content. It is a schematic diagram which shows the 2nd example of the operation with respect to a cluster content. It is a flowchart which shows an example of the cluster content extraction processing of an information processing apparatus.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of the information processing device 50 of the present embodiment. The information processing device 50 can be connected to the server 10 via the communication network 1. Further, the scanner 20 can be connected to the information processing device 50. The server 10 can be a data server that stores layout data, but is not limited thereto.

The information processing device 50 includes a control unit 51, a communication unit 52, a storage unit 53, a layout data estimation unit 54, a specific unit 55, an extraction unit 56, a display panel 57, a display unit 58, and an operation unit 59 that control the entire device. ..

The control unit 51 can be composed of a CPU, a ROM, a RAM, and the like.

The communication unit 52 has a function of communicating with the server 10 via the communication network 1, and can transmit and receive necessary information. More specifically, the communication unit 52 can acquire layout data from the server 10.

FIG. 2 is a schematic diagram showing an example of layout data. The layout data includes, for example, information indicating a state in which a plurality of contents are arranged in a layout frame excluding margins in an area corresponding to one page of a document. The content includes, for example, a title, a text (text), an image (figure), a caption (description of the image), and the like. The layout data is data necessary for arranging the contents, and includes, for example, the size of each content, the coordinates of the contents, the relative coordinates between the contents, and the like. In the example of FIG. 2, the title, the text A, B, the images A, B, C, and the captions A, B, and C are arranged on one page of the document. The layout data is not limited to the example of FIG.

The communication unit 52 also has an interface function with the scanner 20, and can acquire, for example, a PDF file from the scanner 20.

The layout data estimation unit 54 includes an object detection neural network, can image a PDF file obtained from the scanner 20, and estimate the layout data.

The storage unit 53 can be configured by a hard disk, a flash memory, or the like, and can store layout data acquired via the communication unit 52 and layout data estimated by the layout data estimation unit 54.

The identification unit 55 uses a trained model in which a set of related contents is learned as teacher data to specify whether or not the set of input contents is related. The trained models are neural networks 551, 552, and 553, which will be described later.

The identification unit 55 identifies a set of related contents based on the layout data. “Relevant content” means, for example, that the user is likely to repeat the same operation (for example, copying, moving, deleting, scaling, etc.) on the content in the document displayed on the display panel 57. It can be content to each other. Relevant contents can be schematically represented by a relevance graph, for example.

FIG. 3 is a schematic diagram showing an example of a related graph. As shown in FIG. 3, it is assumed that the images G1, G2, G3, and the captions C1, C2, and C3 are included as a plurality of contents in one layout. It is assumed that the image G1 and the captions C1 and C2 are related to each other, and the caption C3 and the images G2 and G3 are related to each other. In this case, the related graph can be represented by a graph in which the image G1 and the caption C1 are connected and the image G1 and the caption C2 are connected. Further, the related graph can be represented by a graph in which the caption C3 and the image G2 are connected and the caption C3 and the image G3 are connected.

Next, a method of determining the relevance between the contents will be described. Hereinafter, as the content, as shown in FIG. 3, an image and a caption will be described as an example, but the same applies to other contents. Image features, caption features, and relative position information (relative coordinates) between contents (images and captions) are used to determine the relevance. First, a method of calculating the feature amount of the image will be described.

FIG. 4 is a schematic diagram showing an example of a method for calculating the feature amount of an image. The specific unit 55 has a neural network 551. The neural network 551 is, for example, a convolutional neural network, in which an input layer 551a, a convolutional layer 551b, a pooling layer 551c, a convolutional layer 551d, a pooling layer 551e, and a fully connected layer 551f are connected in this order. The number of convolutional layers, pooling layers, and fully connected layers is for convenience, and is not limited to the number shown in FIG. For convenience, the activation function layer and the output layer are omitted. The image G1 is input to the input layer 551a. Since the fully connected layer 551f is a combination of the features of the input image G1, the feature amount g1 (vector) can be calculated from the fully connected layer 551f. The feature amounts g2 and g3 can be calculated in the same manner for the other images G2 and G3. The calculation of the feature amount of the image is not limited to the method using the neural network, and general image processing such as edge detection, line detection, region division, and texture analysis may be used.

Next, a method of calculating the feature amount of the caption will be described.

FIG. 5 is a schematic diagram showing an example of a method for calculating a caption feature amount. The specific unit 55 has a neural network 552. The neural network 552 is, for example, word2vec, and the input layer 552a, the hidden layer 552b, and the output layer 552c are connected in this order. The input layer 552a and the hidden layer 552b, and the hidden layer 552b and the output layer 552c are fully connected by weights W and W', respectively. A word (or a sequence of words) extracted from the caption C1 is input to the input layer 552a by the language processing unit (for example, morpheme analysis processing) possessed by the specific unit 55. Specifically, the caption C1 is divided into a plurality of words by morphological analysis, and the divided words are converted into a vector and input to the input layer 552a. In this case, the vector obtained by averaging the vectors of each word may be input to the input layer 552a. When a vector is input to the input layer 552a, the meaning of the caption C1 is expressed as a vector, and the feature amount c1 (vector) can be calculated. The feature amounts c2 and c3 can be calculated in the same manner for the other captions C2 and C3. The calculation of the feature amount of the caption is not limited to the method using the neural network, and general language processing may be used. For example, dictionary data may be used to extract a word, which is the smallest unit having meaning, from a caption, and the extracted word may be converted into a vector having a required number of dimensions. The feature amount of the title and the feature amount of the text can also be calculated using the neural network 552.

Next, a method of determining the relationship between the contents (images and captions) will be described.

FIG. 6 is a schematic diagram showing a first example of determining the relevance of an image and a caption. The specific unit 55 has a neural network 553. Input data is input to the neural network 553. The input data is a vector whose components are the feature amount of the image, the feature amount of the caption, and the relative position information of the image and the caption. The relative position information includes the relative coordinates between the coordinates of the image G1 on the layout and the coordinates of the caption C1, the size on the layout of the image G1, the size on the layout of the caption C1, and the like. In the example of FIG. 6, the feature amount g1 (vector) of the image G1, the feature amount c1 (vector) of the caption C1, and the relative position information (vector) between the image G1 and the caption C1 are regarded as one vector.

Since the score output by the neural network 553 is equal to or higher than the threshold value, it can be determined that the image G1 and the caption C1 are related as shown in the related graph shown in FIG. The neural network 553 may use other machine learning such as SVM (Support Vector Machine) or Bayesian network.

FIG. 7 is a schematic diagram showing a second example of determining the relevance of an image and a caption. In the example of FIG. 7, the feature amount g2 (vector) of the image G2, the feature amount c1 (vector) of the caption C1, and the relative position information (vector) between the image G2 and the caption C1 are used as one vector in the neural network 553. I'm typing.

Since the score output by the neural network 553 is less than the threshold value, it can be determined that the image G2 and the caption C1 are not related as shown in the related graph shown in FIG.

As described above, each feature amount and relative position information are input to the neural network 553 for any set of images and captions on the layout, and the presence or absence of relevance is determined for each set.

Next, the learning method of the neural network 553 will be described.

FIG. 8 is a schematic diagram showing a first example of a learning method of the neural network 553. A vector as input data for learning is input to the input layer of the neural network 553. The learning vector is a vector whose components are the feature amount of the image, the feature amount of the caption, and the relative position information of the image and the caption. In the example of FIG. 8, the feature amount g1 (vector) of the image G1, the feature amount c2 (vector) of the caption C2, and the relative position information (vector) between the image G1 and the caption C2 are regarded as one vector. As shown in FIG. 3, when there is a relationship between the image G1 and the caption C2, the teacher label “1” is given to the output layer, and the neural network 553 is trained. The teacher label "1" is a label indicating that it is a set of correct answers.

FIG. 9 is a schematic diagram showing a second example of the learning method of the neural network 553. A vector as input data for learning is input to the input layer of the neural network 553. The learning vector is a vector whose components are the feature amount of the image, the feature amount of the caption, and the relative position information of the image and the caption. In the example of FIG. 9, the feature amount g2 (vector) of the image G2, the feature amount c1 (vector) of the caption C1, and the relative position information (vector) between the image G2 and the caption C1 are regarded as one vector. As shown in FIG. 3, when there is no relationship between the image G2 and the caption C1, the teacher label “0” is given to the output layer, and the neural network 553 is trained. The teacher label "0" is a label indicating that the set is an incorrect answer.

The neural network 553 can be trained by using a large number of learning input data and teacher labels as shown in FIGS. 8 and 9.

As described above, if any two contents are selected from a plurality of contents in the document, a score (index) indicating the relationship between the selected contents is calculated, and the calculated score is equal to or higher than a predetermined threshold value. , If it is determined that there is relevance and the calculated score is less than the threshold value, it is possible to identify a set of related contents by determining that there is no relevance.

In addition, a set of related contents in the document can be specified only by reading the document on the paper medium with a scanner 20 or the like.

The extraction unit 56 extracts a set of related contents specified by the specific unit as cluster contents in association with each other. The cluster content is a collection of related contents as one content, and the related contents can be treated as one content.

With the above configuration, when a required operation is performed on the cluster contents, it is treated as if the operations were performed on the related contents, so that the operability for the contents in the document is improved.

Next, the operation for the cluster contents will be described.

The display unit 58 can display a document in which the content is arranged on the display panel 57. The display panel 57 can be composed of a liquid crystal panel, an organic EL (Electro Luminescence) display, or the like. In addition, instead of the display panel 57, a display device separate from the information processing device 50 may be provided.

The operation unit 59 is composed of, for example, a hardware keyboard, a mouse, or the like, and can operate an icon or the like displayed on the display panel 57, input characters or the like, and the like. The operation unit 59 may be composed of a touch panel.

FIG. 10 is a schematic diagram showing a first example of an operation on cluster contents. As shown in FIG. 10, a document in which a plurality of contents are arranged (for example, one page or two pages of double-sided opening) is displayed on the display panel 57. In the example of FIG. 10, the title, the text A, the text B, the image A, the caption A, and the caption B are displayed as the contents. Further, it is assumed that the image A and the captions A and B are related to each other.

As shown in the left figure of FIG. 10, when the icon 100 is brought close to the image A (or the periphery of the image A, the caption A or B may be used) and the touch operation and the drag operation are performed, the image A is as shown in the right figure. Captions A and B can be moved in the same way. The image A, the caption A, and the caption B constitute one cluster content 101.

In this way, when the display unit 58 accepts the operation of selecting the cluster content displayed on the display panel 57, the display unit 58 displays each of the contents associated with the cluster content in the selected display mode. For example, when one content in the cluster content displayed on the display panel 57 or the periphery of the content is selected and moved (drag) on the display panel 57, all the contents in the cluster content are selected and displayed. It is displayed in a mode, and the entire cluster content can be moved (drag). As a result, it is not necessary to repeat the same operation for the related contents, and the operability for the contents in the document is improved.

FIG. 11 is a schematic diagram showing a second example of the operation for the cluster contents. As shown in FIG. 11, a document in which a plurality of contents are arranged (for example, one page or two pages of double-sided opening) is displayed on the display panel 57. In the example of FIG. 11, the text A, the text B, the text C, the image A, and the caption A are displayed as the contents. Further, it is assumed that the text B and the text C are related to each other.

As shown in the left figure of FIG. 11, when the icon 100 is brought close to the text B (or the periphery of the text B, the text C may be used) and the touch operation and the drag operation are performed, the text is combined with the text B as shown in the right figure. C can be moved in the same way. The text B and the text C constitute one cluster content 102. As a result, it is not necessary to repeat the same operation for the related contents, and the operability for the contents in the document is improved.

The control unit 51 has a function as a grouping processing unit, and groups cluster contents as one content. Specifically, when the control unit 51 performs a predetermined process on one content of the cluster content, the control unit 51 can perform the same process as the predetermined process on the other contents of the cluster content. For example, in the case of editing processing for content in a document (for example, copying, moving, scaling, deleting, etc.) as a predetermined process, if the editing process is performed for one content, it is related to the other content. The same editing process can be performed on the contents of the document, and the operability of the contents in the document can be improved.

Further, when the control unit 51 performs a predetermined process on the cluster contents, the control unit 51 can perform the same process as the predetermined process on each of the contents in the cluster contents. For example, in the case of a process of searching for the content in the document as a predetermined process, the search can be performed for each cluster content, and the operability of the content in the document can be improved.

FIG. 12 is a flowchart showing an example of the cluster content extraction process of the information processing apparatus 50. Hereinafter, for convenience, the subject of processing will be described as the control unit 51. The control unit 51 acquires layout data (S11) and acquires structural information (for example, content size and coordinates) of layout data (S12).

The control unit 51 selects a set of contents (S13) and extracts the feature amount of the selected contents (S14). The control unit 51 determines the relevance of the selected content based on the extracted feature amount and the relative position information of the selected content (S15).

The control unit 51 determines whether or not there is unprocessed content (S16), and if there is unprocessed content (YES in S16), the control unit 51 continues the processing after step S13. When there is no unprocessed content (NO in S16), the control unit 51 collectively extracts related contents as cluster content (S17), and ends the process.

The information processing device 50 can also be realized by using, for example, a computer including a CPU (for example, a multiprocessor having a plurality of processor cores mounted on it), a GPU (Graphics Processing Units), a RAM, and the like. Information processing is performed on the computer by loading a computer program (which can be recorded on a recording medium) that defines the processing procedure as shown in FIG. 12 into the RAM provided in the computer and executing the computer program on the CPU (processor). The device 50 can be realized.

The information processing apparatus of the present embodiment uses a trained model in which a set of related contents is learned as teacher data, and uses a specific unit that specifies whether or not the set of input contents is related, and the above-mentioned identification. It is provided with an extraction unit that extracts a set of related contents specified in the unit as cluster contents by associating them with each other.

The computer program of the present embodiment uses a trained model in which a set of related contents is trained as teacher data in a computer, and specifies whether or not the set of input contents is related. The process of associating the related sets of contents and extracting them as cluster contents is executed.

In the information processing method of the present embodiment, the presence or absence of the relevance of the input content set is specified by using the trained model in which the related content set is learned as the teacher data, and the specified relevance is specified. The set of contents with is associated and extracted as cluster contents.

The specific part uses a trained model in which a set of related contents is learned as teacher data to specify whether or not the set of input contents is related.
The content includes, for example, a title, a text (text), an image (figure), a caption (description of the image), and the like. The layout data is data necessary for arranging the contents, and includes, for example, the size of each content, the coordinates of the contents, the relative coordinates between the contents, and the like. The “relevant content” can be, for example, content that the user is likely to repeat the same operation (for example, copying, moving, etc.).

Arbitrary two contents are selected from a plurality of contents in the document, an index showing the relevance between the selected contents is calculated, and if the calculated index is equal to or more than a predetermined threshold value, it is determined to be relevance. If the calculated index is less than the threshold value, it is possible to identify a set of related contents by determining that there is no relevance.

The extraction unit extracts a set of related contents specified in the specific unit as cluster contents in association with each other. The cluster content is a collection of related contents as one content, and the related contents can be treated as one content.

With the above configuration, when a required operation is performed on the cluster contents, it is treated as if the operations were performed on the related contents, so that the operability for the contents in the document is improved.

In the information processing apparatus of the present embodiment, the set of the input contents is a set of image data and document data.

As a result, the image and the document can be extracted as cluster contents.

The information processing device of the present embodiment includes a grouping processing unit that groups the cluster contents as one content.

When the grouping processing unit performs a predetermined processing on the cluster contents extracted by the extraction unit, the grouping processing unit can perform the same processing as the predetermined processing on each of the contents associated with the cluster contents. For example, in the case of a process of searching for the content in the document as a predetermined process, the search can be performed for each cluster content, and the operability of the content in the document can be improved.

Further, when the grouping processing unit performs a predetermined process on one content of the cluster content, the grouping process unit can perform the same process as the predetermined process on the other contents of the cluster content. For example, in the case of editing processing for content in a document (for example, copying, moving, scaling, deleting, etc.) as a predetermined process, if the editing process is performed for one content, it is related to the other content. The same editing process can be performed on the contents of the document, and the operability of the contents in the document can be improved.

The information processing device of the present embodiment includes a display unit that displays the cluster content extracted by the extraction unit on the display screen, and a reception unit that accepts an operation of selecting the cluster content displayed on the display screen. When the reception unit receives the operation, the unit displays each of the contents associated with the cluster contents in a selected display mode.

The display unit displays the cluster content extracted by the extraction unit on the display screen. When the reception unit receives the operation of selecting the cluster content displayed on the display screen, the display unit displays each of the contents associated with the cluster content in the selected display mode. For example, if you perform an operation to select one content in the cluster content displayed on the display screen or the periphery of the content, and move (drag) on the display screen, for example, all the content in the cluster content is selected and displayed. It is displayed in a mode, and the entire cluster content can be moved (drag). As a result, it is not necessary to repeat the same operation for the related contents, and the operability for the contents in the document is improved.

1 Communication network 10 Server 20 Scanner 50 Information processing device 51 Control unit 52 Communication unit 53 Storage unit 54 Layout data estimation unit 55 Specific unit 551, 552, 553 Neural network 56 Extraction unit 57 Display panel 58 Display unit 59 Operation unit

Claims (6)

Using a trained model in which a set of related contents is trained as teacher data, a specific part that identifies whether or not the set of input contents is related, and a specific part,
An information processing device including an extraction unit that extracts a set of related contents specified by the specific unit as cluster contents in association with each other. The information processing device according to claim 1, wherein the input content set is a set of image data and document data. The information processing apparatus according to claim 1 or 2, further comprising a grouping processing unit for grouping the cluster contents as one content. A display unit that displays the cluster content extracted by the extraction unit on the display screen,
It is equipped with a reception unit that accepts operations to select the cluster content displayed on the display screen.
The display unit
The information processing apparatus according to any one of claims 1 to 3, wherein when the reception unit receives the operation, each of the contents associated with the cluster contents is displayed in a selected display mode. On the computer
Using a trained model that trains a set of related contents as teacher data, a process to identify whether or not the set of input contents is related, and
A computer program that executes the process of associating the specified set of related contents and extracting them as cluster contents. Using a trained model in which a set of related contents is trained as teacher data, the presence or absence of relevance of the set of input contents is identified.
An information processing method that extracts a set of identified related contents as cluster contents by associating them with each other.

Images