JP6734323B2 - Program, system, and method for determining similarity of objects - Google Patents

Description

The present invention relates to a program, a system, and a method for determining the similarity of an object, and more particularly, to a program, a system, and a method for determining the similarity of an object using a convolutional neural network (CNN). ..

The neural network is a model that is created by imitating neurons and synapses in the brain, and is configured by a two-step process of learning and discrimination. In the learning stage, the features are learned from a large number of inputs, and a neural network for discrimination processing is constructed. In the identification stage, a neural network is used to identify what the new input is. In recent years, the technology at the learning stage has been greatly developed, and it is becoming possible to construct a multilayer neural network having high expressive power, for example, by deep learning. Especially, in the tests of voice recognition and image recognition, the effectiveness of the multilayer neural network has been confirmed, and the effectiveness of deep learning has been widely recognized.

A method of using a convolutional neural network (CNN) is known as a method of constructing such a multilayer neural network and performing image identification (for example, Non-Patent Document 1). A multilayer neural network using a convolutional neural network (CNN) in Non-Patent Document 1 is referred to as AlexNet, which extends LeNet5 into multiple layers, and further, a linear rectifying unit (ReLU) as an output function of each unit. )) etc. are used.

"ImageNet Classification with Deep Conservative Neural Networks", Alex Krizhevsky, Ilya Suskever, Geoffrey E. Hinton

It has been found that the above-described conventional image identification method can further reduce the error rate when specifying an object included in an image. However, this method has a problem that extraction cannot be performed accurately and efficiently when a search is performed by focusing on a specific element of an object including many elements.

An embodiment of the present invention has an object to appropriately determine the degree of similarity between elements included in an object. Other objects of embodiments of the present invention will become apparent by reference to the entire specification.

A method according to an embodiment of the present invention is a similar image determination method for determining a similarity between a plurality of objects using a convolutional neural network (CNN) including one or a plurality of convolution layers and a fully connected layer. Then, according to being executed on one or more computers, a step of extracting a plurality of feature amounts from each of the plurality of objects on the one or more computers, and each of the plurality of objects. Extracting the output values of all connected layers after one or more convolutional layers of the convolutional neural network (CNN) based on the plurality of feature quantities from It is configured to execute a step of performing a conversion process for setting a range of values within a range to extract a converted output value, and a step of determining a degree of similarity of an object based on the converted output value.

A system according to an embodiment of the present invention is a similarity determination system for determining a similarity between a plurality of objects using a convolutional neural network (CNN) including one or more convolution layers and fully connected layers. Then, according to being executed on one or more computers, a step of extracting a plurality of feature amounts from each of the plurality of objects on the one or more computers, and each of the plurality of objects. Extracting output values of all connected layers after one or more convolution layers of a convolutional neural network (CNN) based on the plurality of feature quantities from The step of performing a conversion process to extract the converted output value and the step of determining the similarity of the object based on the converted output value are configured to be executed.

The program according to one embodiment described above is a program for determining the degree of similarity between a plurality of objects using a convolutional neural network (CNN) including one or a plurality of convolution layers and a fully connected layer. Or, in response to being executed on a plurality of computers, a step of extracting a plurality of feature amounts from each of the plurality of objects in the one or more computers, and a step of extracting the plurality of feature amounts from each of the plurality of object images. A step of extracting output values of all connected layers after one or a plurality of convolution layers of a convolutional neural network (CNN) based on a plurality of feature quantities; and a range of output values of the all connected layers within a predetermined range. And a step of extracting a converted output value, and a step of determining the degree of similarity of the object based on the converted output value.

According to various embodiments of the present invention, by utilizing a multilayer neural network using a convolutional neural network (CNN), it is possible to appropriately perform similarity determination between elements included in an object.

The block diagram which shows roughly the structure of the system 1 which concerns on one Embodiment of this invention. The block diagram which shows roughly the function which the system 1 in one embodiment has. The figure which shows an example of the similar image determination flow in one embodiment. The figure which shows an example of the flow of the category classification of the object of each image using the existing convolution network in one Embodiment. FIG. 6 is a flow chart showing an example of a sigmoid function in one embodiment. The figure which shows the concept of the similarity determination by the distance scale comparison in one embodiment.

FIG. 1 is a configuration diagram schematically showing a configuration of a system 1 according to an embodiment of the present invention. As shown in the figure, a system 1 according to an embodiment includes a server 10 and a plurality of terminal devices 30 connected to the server 10 via a communication network 20 such as the Internet. Provide e-commerce services. Further, the system 1 according to one embodiment provides a service for providing various digital contents other than games such as games using characters, electronic books, moving image contents, and music contents, as well as text chat (minimail), circles, and avatars. , Diary, message board, and various Internet services such as a communication platform (SNS platform) service that realizes communication functions between various users such as greetings can be provided to the user of the terminal device 30.

The server 10 in one embodiment is configured as a general computer, and as illustrated, a CPU (computer processor) 11, a main memory 12, a user I/F 13, a communication I/F 14, and a storage (storage). Device 15), and these respective components are electrically connected to each other via a bus 17. The CPU 11 loads the operating system and various other programs from the storage 15 into the main memory 12, and executes the instructions included in the loaded program. The main memory 12 is used to store a program executed by the CPU 11, and is composed of, for example, a DRAM or the like. It should be noted that the server 10 in one embodiment may be configured using a plurality of computers each having the hardware configuration described above. The CPU (computer processor) 11 described above is an example, and it goes without saying that a GPU (graphics processing unit) may be used instead. How to select the CPU and/or GPU can be appropriately determined in consideration of desired cost or efficiency. Hereinafter, the CPU 11 will be described as an example.

The user I/F 13 includes, for example, an information input device such as a keyboard or a mouse that receives an operator's input, and an information output device such as a liquid crystal display that outputs a calculation result of the CPU 11. The communication I/F 14 is implemented as hardware, firmware, communication software such as a TCP/IP driver or a PPP driver, or a combination thereof, and is configured to be communicable with the terminal device 30 via the communication network 20.

The storage 15 is composed of, for example, a magnetic disk drive, and stores various programs such as a control program for providing various services. Further, the storage 15 may also store various data for providing various services. The various data that can be stored in the storage 15 may be stored in a database server or the like that is physically separate from the server 10 that is communicatively connected to the server 10.

In one embodiment, the server 10 also functions as a web server that manages a website composed of a plurality of hierarchically structured web pages, and can provide various services to the user of the terminal device 30 via such website. .. The storage 15 may also store HTML data corresponding to this web page. Various image data are associated with the HTML data, and various programs described in a script language such as JavaScript (registered trademark) may be embedded.

In addition, in one embodiment, the server 10 may provide various services via an application (program) executed on an execution environment other than a web browser in the terminal device 30. Such applications may also be stored in the storage 15. This application is created using a programming language such as Objective-C or Java (registered trademark), for example. The application stored in the storage 15 is delivered to the terminal device 30 in response to the delivery request. The terminal device 30 can also download such an application from a server other than the server 10 (a server that provides an application market) or the like.

In this way, the server 10 can manage a website for providing various services and deliver a web page (HTML data) constituting the website in response to a request from the terminal device 30. Further, as described above, the server 10 is provided with an application executed in the terminal device 30 as an alternative to or in addition to the provision of various services using such a web page (web browser). Various services can be provided based on communication. Regardless of which mode the service is provided, the server 10 can send and receive various data necessary for providing various services (including data necessary for screen display) to and from the terminal device 30. Further, the server 10 can store various data for each identification information (for example, user ID) that identifies each user, and can manage the provision status of various services for each user. Although detailed description is omitted, the server 10 can also have a function of performing user authentication processing, billing processing, and the like.

The terminal device 30 in the embodiment is an arbitrary information processing device that displays a web page of a website provided by the server 10 on a web browser and implements an execution environment for executing an application, such as a smartphone or a tablet terminal. , A wearable device, a personal computer, a game terminal, and the like, but not limited thereto.

The terminal device 30 is configured as a general computer, and as shown in FIG. 1, a CPU (computer processor) 31, a main memory 32, a user I/F 33, a communication I/F 34, and a storage (storage device) 35. , And each of these components is electrically connected to each other via a bus 37.

The CPU 31 loads the operating system and various other programs from the storage 35 into the main memory 32, and executes the instructions included in the loaded program. The main memory 32 is used to store the program executed by the CPU 31, and is composed of, for example, a DRAM or the like.

The user I/F 33 includes, for example, an information input device such as a touch panel, a keyboard, a button, and a mouse that receives user input, and an information display device such as a liquid crystal display that outputs the calculation result of the CPU 31. The communication I/F 34 is implemented as hardware, firmware, communication software such as a TCP/IP driver or a PPP driver, or a combination thereof, and is configured to communicate with the server 10 via the communication network 20.

The storage 35 is composed of, for example, a magnetic disk drive or a flash memory, and stores various programs such as an operating system. Further, the storage 35 may store various applications received from the server 10.

The terminal device 30 includes, for example, a web browser for interpreting an HTML format file (HTML data) and displaying the screen, and interprets and receives the HTML data acquired from the server 10 by the function of the web browser. The web page corresponding to the HTML data can be displayed. In addition, plug-in software capable of executing various types of files associated with HTML data may be incorporated in the web browser of the terminal device 30.

When the user of the terminal device 30 uses the service provided by the server 10, for example, HTML data, an animation instructed by an application, an operation icon, or the like is displayed on the screen of the terminal device 30. The user can input various instructions using the touch panel or the like of the terminal device 30. The instruction input by the user is transmitted to the server 10 via the function of the web browser of the terminal device 30 and the application execution environment such as NgCore (trademark).

Next, the function of the system 1 according to the embodiment configured as described above will be described. As described above, the system 1 according to the embodiment can provide various internet services to the user, and in particular, can provide the electronic commerce service and the content distribution service. Hereinafter, the function of the system 1 according to the embodiment will be described by taking a function of providing an electronic commerce service as an example.

FIG. 2 is a block diagram schematically showing the functions of the system 1 (server 10 and terminal device 30). First, the function of the server 10 according to the embodiment will be described. As illustrated, the server 10 includes an information storage unit 41 that stores various pieces of information, and an image information control unit for presenting a specific image to a user and selecting and presenting an image similar thereto, in one embodiment. 42. Although an image is described as an example in the embodiment, the target of similarity determination is not limited to this, and may include, for example, a signal such as text or voice. In the present specification, these shall be defined as objects. Therefore, the image information control unit 42 can be read as the object information control unit 42. Hereinafter, for convenience of description, an image will be described as an example of the object of similarity determination. These functions are realized by the hardware such as the CPU 11 and the main memory 12 and various programs and tables stored in the storage 15 operating in cooperation with each other, and, for example, the instructions included in the loaded program. Is realized by the CPU 11 executing. Further, some or all of the functions of the server 10 illustrated in FIG. 2 may be realized by the terminal device 30 or by the server 10 and the terminal device 30 cooperating with each other.

The information storage unit 41 in one embodiment is realized by the storage 15 or the like, and as shown in FIG. 2, an image information management table 41a that manages image information of products provided in the electronic commerce service, and an image similar to the images of the products. A similar image information management table 41b that manages image information related to the image of the product to be processed.

Next, the function of the image information control unit 42 for presenting a specific image to the user and selecting and presenting an image similar thereto will be described in an embodiment. The image information control unit 42 uses a machine-learned multilayered neural network to represent an image as a multidimensional vector, and finally determines a similar image by approximating the vector or comparing distances between the vectors. To do. The similar image thus extracted is stored in the similar image information management table 41b.

More specifically, FIG. 3 shows a similar image determination method which is one function of the image information control unit 42. In the similar image determination method according to the embodiment, first, the feature amount is extracted from the target image (input layer). After that, the convolution layers (also referred to as convolution layers) 100 to 140 are passed through five layers, and as a sixth layer, a fully-connected layer 150 is passed.

The above-mentioned first to fifth convolution layers to sixth to sixth fully coupled layers will be described with reference to FIG. FIG. 4 shows the architecture of the AlexNet convolution network (FIG. 4 corresponds to FIG. 2 disclosed in Non-Patent Document 1). As illustrated, the AlexNet convolution network is composed of five convolution layers (convolutional layers) and three fully connected layers. The output of the final fully connected layer is subjected to 1000 kinds of softmax and classified into 1000 classes. As shown in FIG. 2, each of the kernels of the second, fourth and fifth convolution layers is only combined with the kernel of the same GPU layer of the previous layer. The kernel of the third convolution layer is combined with all the kernels of the second layer.

The neurons in all connected layers are connected to all the neurons in all connected layers. The first and second convolution layers have a configuration in which a normalization layer (Response-normalization layer) follows. Further, the Max-pooling layer is configured to follow the normalization layer and the fifth convolution layer. ReLU (Rectified Linear Unit) is also applied to the outputs of all convolutional layers and fully connected layers.

The first convolution layer filters the input image of size 224x224x3 with 96 kernels of size 11x11x3 (4 pixel slides). The second convolution layer then takes the output (after normalization and pooling) of the first convolution layer as input and filters it with 256 kernels of size 5x5x48. The third, fourth, and fifth convolution layers are connected to each other with no normalization layer or pooling layer interposed between them. The third convolution layer has 384 kernels of size 3x3x256 combined with the output (after normalization and pooling) of the second convolution layer. The fourth convolution layer has 384 kernels of size 3x3x192 and the fifth convolution layer has 256 kernels of size 3x3x192. In addition, each fully connected layer has 4096 neurons.

The invention according to one embodiment is characterized by utilizing the architecture of the existing AlexNet convolution network shown in FIG. However, if the final output value of the convolution network is used as it is, the feature amount for categorizing the object of each image is extracted too much. It has been found that it is difficult to determine the degree of similarity between images including. Therefore, in the invention according to the embodiment, as a result of repeating the experiment, the output value of the fully coupled layer of the sixth layer after the convolution layers of the first layer to the fifth layer of AlexNet, that is, the category of the object is determined. By using the output value in which the influence of the feature amount, which is more suitable for classification, is relatively low and the influence of the other feature amount of the target object is relatively high, it is possible to perform the operation regardless of the target category. We have found that the determination of the similarity between the images including the object can be effective.

Here, in the invention according to one embodiment, the architecture of the existing convolution network of AlexNet is used, but it is not intended to limit the number of layers of the convolution layer or the total coupling layer, and the cost and efficiency are not limited. Needless to say, it can be changed as appropriate in consideration of the change.

As described above, in the invention according to the embodiment, the convolution first layer 100, the convolution second layer 110, the convolution third layer 120, the convolution fourth layer 130, and the convolution fifth layer 140, The output value of the fully-connected layer of the sixth layer after that is used. However, since the output value of the sixth layer takes a value range from −∞ to ∞, the output value is set to a range of 0 to 1 by using a sigmoid function in order to keep the value range within a predetermined range. You can The output value of the sigmoid layer 160 of the seventh layer can be set in the range of 0 to 1 by applying the sigmoid function shown by the solid line in FIG. On the other hand, when the sigmoid function shown by the dotted line in FIG. 5 is applied, the output value can be in the range of -1 to 1.

By changing the output value from 0 to 1 through the sigmoid layer at this stage, subsequent approximation and comparison of distance scales can be performed easily and efficiently. Further, by limiting the output value in this way, the classification accuracy of the category classification of the target object included in the image is slightly lowered, but the similar image determination in one embodiment is similar to the image including the target object of the same category. It is also intended to extract images that include objects that have similar characteristics even if they are objects of different categories, so that it is more accurate and efficient. It was clarified by various experiments that this is an image extraction method.

Next, through the sigmoid layer 160, the approximation/distance comparison layer 170 determines the degree of similarity between the plurality of images based on the converted output value in the range of 0 to 1. As a method for determining the similarity between the plurality of images, there are a hash method and an approximate nearest neighbor search method using a step function. Specifically, Local Sensitive Hashing (LSH) can be used as an approximate nearest neighbor search method using the hash method. LSH can extract an approximate nearest neighbor point in a vector space by using a hash function that is locally sensitive, that is, a hash value that has a high probability of taking a closer hash value as the distance is shorter, and the data space is linearly divided. The distance is calculated by extracting the points that entered the same area as the query. Note that such a hash function refers to a hash function having a characteristic that inputs with a short distance collide with a high probability, and it is possible to create a hash table in which data with a short distance is mapped to the same value with high probability. By using the hash function of, it is possible to reduce the collision probability significantly when the distance is equal to or more than a certain distance. As a result, the degree of similarity between the plurality of images is determined, and whether or not the images are similar is determined.

On the other hand, in the approximation/distance comparison layer 170, as another method of determining the similarity between a plurality of images, there is a method of obtaining a distance between points corresponding to each image in the feature amount space, such as a Euclidean distance and a Hamming distance. Or cosine distance etc. are used for this purpose. This method is characterized by comparing distance measures, and shows that a plurality of images located at close positions in the feature amount space are similar to each other. With this method, it is possible to estimate the degree of similarity between images by calculating the mutual distances in the feature amount space for a plurality of images. A two-dimensional feature amount space composed of two types of feature amounts A and B will be described as an example, but the following concept can be extended and applied to a higher-dimensional feature amount space. As an example, consider a case where ten images (P=10) are plotted according to the value of the feature amount in a two-dimensional feature amount space having the feature amounts X1 and X2 as coordinate axes. In FIG. 6, circles with numbers inside indicate the positions of the respective images in the feature amount space, and the numbers indicate the image numbers of the respective images.

In the example in FIG. 6, it was determined that images 1, 6 and 9 were similar, and images 5, 8 and 10 were also similar. Further, although the images 3 and 7 are similar, it is determined that the images 2 and 4 have no images similar to them.

In this way, the image similar to the specific image is finally determined through the approximation/distance comparison layer. In the learning stage, the features are learned from a large number of input data generated by the sensor to build a convolution network. The constructed convolution network is represented as a weighting coefficient used in each arithmetic unit in the image information control unit 42. For example, when input data corresponding to an image in which a certain number “x” is drawn is input, Find a weighting factor that outputs that the input data is "x". By receiving a lot of input data, the accuracy of the neural network improves. In the present embodiment, the image information control unit 42 constructs the convolution network by a known method.

The functions of the server 10 have been described above. Next, functions of the terminal device 30 according to the embodiment will be described. As shown in FIG. 2, the terminal device 30 includes an information storage unit 51 that stores various kinds of information and a terminal-side control unit 52 that executes control for displaying image information in the embodiment on the terminal side. Have. These functions are realized by the hardware such as the CPU 31 and the main memory 32, and various programs and tables stored in the storage 35 working in cooperation with each other. For example, the instructions included in the loaded program. Is realized by the CPU 31 executing. Further, some or all of the functions of the terminal device 30 illustrated in FIG. 2 may be realized by the server 10 and the terminal device 30 cooperating with each other, or may be realized by the server 10.

The information storage unit 51 in one embodiment is realized by the main memory 32, the storage 35, or the like. The terminal-side control unit 52 in one embodiment controls execution of various terminal-side processes such as a request for image information reception and display of received image information. For example, the terminal-side control unit 52
In order for the user to purchase a product such as clothes or glasses, an image that is a candidate for the product is searched and the result is received from the server 10 and displayed, or an image similar to the image received from the server 10. Can be displayed together.

By doing so, in a service such as electronic commerce or distribution of digital contents, if there is an image similar to the image to be traded or the image included in the content to be distributed, the server 10 displays the image. The information can be transmitted to the user's terminal 30 and displayed. By doing so, the user can efficiently find and purchase products similar to the product to be purchased, and also introduce contents including images similar to the contents desired to be distributed. As a result, the user can more easily understand the image information that is close to his/her taste, and in some cases, it becomes possible to purchase and distribute such image information together. As described above, the image has been described as an example in the embodiment, but the present invention is not limited to this, and it can be widely applied to objects including signals such as text and voice.

As another example of determining the similarity of an object, it can be applied to the determination of the similarity of a dialogue sentence. In one embodiment, it is assumed that a user who is close to a persona image (a female in his 30s is an example) has said, “I like Keisuke Honda”. Suppose another user is A and B, and A says “I like Keisuke Honda” and B says “I like Shinji Kagawa”. In such a case, generally speaking, in the conventional natural language processing, the judgment of the similarity of the dialogue sentence is such that the utterance of A in which the low-frequency word "Keisuke Honda" matches is closer to the original utterance. Is evaluated. However, if you want to find not only the “content of the statement” but also the statement of another user close to the “user's statement of the targeted persona image” by repeating learning in advance, use the above-mentioned multilayer neural network. Then, it was confirmed that it can be applied to the task of extracting a user's utterance content and taste of a target persona image, and the utterance of another user who utters an utterance with a close character, in the interactive example search. There is. In such an interactive example search, not only infrequent words, but also relatively frequent words such as "Ore" and "I" are effective for such classification, and differences between such words are By constructing a metric space that is regarded as important, it is effective in similarity determination for searching not only the above-mentioned image but also other objects such as “sentences with similar taste and character”.

The processes and procedures described in the present specification are implemented by software, hardware, or any combination thereof other than those explicitly described in the embodiments. More specifically, the processes and procedures described in this specification are implemented by implementing a logic equivalent to the process in a medium such as an integrated circuit, a volatile memory, a non-volatile memory, a magnetic disk, and an optical storage. Will be realized. In addition, the processes and procedures described in this specification can be implemented by various computers by mounting the processes and procedures as a computer program.

Even though it is described that the processes and procedures described herein are performed by a single device, software, component, or module, such processes or procedures may be implemented by multiple devices, multiple software, or multiple software. Components and/or multiple modules. Also, even though the data, tables, or databases described herein are described as being stored in a single memory, such data, tables, or databases may be provided in a single device. It may be stored in a distributed manner in a plurality of memories or a plurality of memories arranged in a plurality of devices. Furthermore, the software and hardware elements described herein may be implemented by integrating them into fewer components or breaking them down into more components.

In the present specification, when the constituent elements of the invention are described as either a singular or a plural number, or even when the constituent elements are described without being limited to either a singular or plural number, a case where it is to be understood separately in context Except for the above, the constituent may be singular or plural.

10 server 20 communication network 30 terminal device 41 information storage unit 42 image information control unit 51 information storage unit 52 terminal side control unit 100 convolution first layer 110 convolution second layer 120 convolution third layer 130 convolution fourth layer 140 Convolution 5th layer 150 Fully coupled layer 160 Sigmoid layer 170 Approximation/distance comparison layer

Claims (13)

On one or more computers,
A first extraction step of extracting a feature amount from one image;
A second extraction step of extracting an output value of a fully connected layer after one or a plurality of convolution layers based on the feature amount;
A determination step of determining a similar image similar to the one image based on the output value;
A program to execute. Causing the one or more computers to perform a conversion step of converting the output value of the fully connected layer into a range within a predetermined range,
The program according to claim 1. The similar image is stored in the information storage unit,
The program according to claim 1 or 2. The convolution neural network (CNN) comprises a plurality of convolution layers,
The program according to any one of claims 1 to 3. The convolutional neural network (CNN) comprises five convolution layers,
The program according to claim 4. The convolutional neural network (CNN) comprises one fully connected layer,
The program according to claim 4 or claim 5. The converting step is performed using a sigmoid function,
The program according to claim 2. The conversion step is executed so that the range of the output value is in the range of 0 to 1.
The program according to claim 7. The determination step includes an approximation step of approximating the output value,
The program according to any one of claims 1 to 8. In the approximating step, the output value is approximated by LSH,
The program according to claim 9. The determining step determines a similar image by calculating a distance measure based on a Euclidean distance, a cosine distance, or a Hamming distance between the output value and the candidate for the similar image, and comparing the distance measures.
The program according to any one of claims 1 to 10. A first extraction step of extracting a feature amount from one image;
A second extraction step of extracting an output value of a fully connected layer after one or a plurality of convolution layers based on the feature amount;
A determination step of determining a similar image similar to the one image based on the output value;
An image processing method comprising: A system comprising one or more computers,
The one or more computers are
A first extraction step of extracting a feature amount from one image;
A second extraction step of extracting an output value of a fully connected layer after one or a plurality of convolution layers based on the feature amount;
A determination step of determining a similar image similar to the one image based on the output value;
A system that runs.

Images