JP2020091828A - Information processing device and similar image search program - Google Patents

Abstract

To provide an information processing device capable of reducing computational load, and to provide a similar image search program.SOLUTION: An information processing device provided herein is configured to: retain a plurality of data patterns obtained through machine learning; upon reception of input of data to be processed, count the number of portions of the entered data determined to be similar to each of the data patterns; generate feature vector information based on a counting result; and perform a predetermined process associated with the data using the feature vector information.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing apparatus and a similar image search program that perform processing such as searching for similar images.

In recent years, information processing using deep learning has been widely studied. For example, in order to classify images, there is deep learning using a convolutional neural network (CNN) that can recognize the same image even if the position where the object is imaged changes.

Patent Document 1 discloses a technique of classifying images by a decision tree using a plurality of types of feature quantities extracted by the CNN intermediate layer.

JP, 2018-5639, A

However, in the above-mentioned conventional technique, firstly, the calculation load of the neural network cannot be reduced because the calculation of the neural network is required in order to use the result of the machine learning obtained by the neural network.

The present invention has been made in view of the above circumstances, and one of the objects thereof is to provide an information processing apparatus and a similar image search program capable of reducing the calculation load.

The present invention for solving the above-mentioned problems of the conventional example is an information processing apparatus, which holds a pattern of a plurality of data acquired by machine learning, and an input of data to be processed. A feature that counts the number of portions of the processing target data that are determined to be similar to each of the data patterns for each pattern of the plurality of data and generates feature vector information based on the result of the count. A quantity generation means and an execution means for executing a predetermined process relating to the processing target data by using the feature vector information are included.

It is a block diagram showing the example of composition of the information processor concerning an embodiment of the invention. It is a functional block diagram showing the example of the information processing apparatus which concerns on embodiment of this invention. It is explanatory drawing showing the example of the base image data which the information processing apparatus which concerns on embodiment of this invention hold|maintains. It is explanatory drawing showing the example of the content of the search database which the information processing apparatus which concerns on embodiment of this invention hold|maintains.

Embodiments of the present invention will be described with reference to the drawings. The information processing apparatus 1 according to the embodiment of the present invention is configured to include a control unit 11, a storage unit 12, an operation unit 13, a display unit 14, and a communication unit 15, as illustrated in FIG. 1, for example. .. Further, the information processing apparatus 1 is communicably connected to server devices 2a, 2b, etc. such as various web servers via communication means such as the Internet.

The control unit 11 is a program control device such as a CPU, and operates according to a program stored in the storage unit 12. In the example of the present embodiment, the control unit 11 executes a machine learning process of acquiring a plurality of image patterns by machine learning and storing the acquired image patterns in the storage unit 12. Further, the control unit 11 receives the input of the image data to be processed, and for each of the plurality of image patterns, it is determined that the region included in the image data to be processed is similar to the image pattern. The number of regions is counted, and feature vector information based on the result of the count is generated. Then, the control unit 11 uses the feature vector information to execute a predetermined process related to the image data to be processed.

In an example of the present embodiment, the control unit 11 crawls the web pages provided by the respective web servers 2a, 2b,... And the image data provided by the web pages is used as a processing target of the plurality of images. For each pattern, the number of regions that are determined to be similar to the image pattern among the regions included in the image data that is the processing target is counted, feature vector information based on the result of the count is generated, and the generation is performed. The feature vector described above is associated with the information (which may be the URL of the web page) that specifies the web page that provides the image data to be processed, and is stored in the storage unit 12 as a search database.

Then, the control unit 11 receives the input of the image data to be searched from the user, and sets the input image data as the processing target for each of the above-described plurality of image patterns. Among the areas included in, the number of areas that are determined to be similar to the image pattern is counted, and feature vector information based on the result of the count (in order to distinguish from the one already held in the search database, the search target (Called feature vector information). The control unit 11 compares the search target feature vector information generated here with the feature vector information held in the search database, and sets the feature vectors held in the search database in the order close to the search target feature vector information (related Extract a predetermined number in order. The control unit 11 displays and outputs a list of information identifying web pages, which is held in the search database in association with the extracted feature vector, in the order of the relevance.

However, this example is an example of processing, and the control unit 11 of the present embodiment may perform other processing. The details of the processing of the control unit 11 will be described later.

The storage unit 12 is a memory device, a disk device, or the like, and holds a program executed by the control unit 11. The program may be stored in a computer-readable non-transitory storage medium, provided, and stored in the storage unit 12. In addition, in the example of the present embodiment, the storage unit 12 holds a search database. The storage unit 12 also operates as a work memory for the control unit 11.

The operation unit 13 is a keyboard, a mouse, or the like, and in response to a user's instruction operation, outputs information indicating the content of the instruction operation to the control unit 11. The display unit 14 is a display or the like, and displays and outputs the instructed information according to an instruction input from the control unit 11.

The communication unit 15 is a network interface or the like, and transmits and outputs the instructed information to the instructed destination according to the instruction input from the control unit 11. The communication unit 15 also receives information from the web server 2 or the like and outputs it to the control unit 11.

Next, the operation of the control unit 11 according to the example of the present embodiment will be described. Here, first, as an example, an example in which the control unit 11 performs an operation of searching for image data provided by the web server 2 will be described.

By executing the machine learning processing program stored in the storage unit 12, the control unit 11 in this example functions as the machine learning processing unit 20 and the search processing unit 30 as illustrated in FIG. 2. FIG. 2 is a functional block diagram of the control unit 11 according to an example of the present embodiment.

The control unit 11 that executes the machine learning processing program operates as the machine learning processing unit 20 as illustrated in FIG. At this time, the control unit 11 is functionally configured to include a reception unit 21, a machine learning unit 22, and an intermediate layer extraction unit 23.

Further, the control unit 11 that operates as the search processing unit 30 functionally includes a pattern holding unit 31, an information collecting unit 32, a feature amount calculating unit 33, a feature amount holding unit 34, a receiving unit 35, and a searching unit. 36 and an information output unit 37.

The receiving unit 21 receives input of data to be machine-learned. In the example of the present embodiment, since the operation of searching for image data is performed, the data received by the receiving unit 21 here is image data. In an example of the present embodiment, the image data may be obtained by crawling a website. Alternatively, this image data may include an image to be searched, for example, if it is used for searching a Japanese landscape, a collection of image data obtained by photographing Japan.

The machine learning unit 22 is, for example, an auto encoder, and executes machine learning processing of a neural network including at least one intermediate layer. The machine learning unit 22 resizes the image data received by the receiving unit 21 to a predetermined size and inputs the resized image data to the input layer of the neural network. Further, the machine learning unit 22 sets the difference between the output of the neural network and the input image data so that the output of the neural network when the image data is input becomes the original (resized to a predetermined size) image data. Based on the above, the weight information between the input layer and the intermediate layer and the weight information between the intermediate layer and the output layer in the neural network are set. Since this processing is the same as the machine learning processing of a general neural network, detailed description thereof is omitted here.

The intermediate layer extraction unit 23 extracts and outputs the information of the intermediate layer of the neural network machine-learned by the machine learning unit 22. Specifically, when there are N nodes in the intermediate layer and M nodes in the input layer (assuming that the size of the input image data is x pixels in the horizontal direction and y pixels in the vertical direction, M=x×y), the intermediate Focusing on one node j in the layer, there are M weights wij (i=1, 2,..., M). Therefore, the intermediate layer extraction unit 23 arranges these M weights for each node (j=1, 2,..., N) of the intermediate layer in the image data size (horizontal x pixel, vertical y pixel). Image data (N pieces of image data) is generated and output as information of the intermediate layer.

Although the intermediate layer extraction unit 23 extracts the information of the intermediate layer from the information of the weight between the input layer and the intermediate layer here, instead of or together with this, the intermediate layer and the output layer are extracted. The information of the intermediate layer may be extracted from the information of the weight of. That is, for example, when the neural network machine-learned by the machine learning unit 22 is an auto encoder, the intermediate layer for extracting information is the last layer on the encoder side (the deepest intermediate layer connected to the first layer on the decoder side). , A layer that represents a so-called latent variable). Further, the intermediate layer extracting unit 23 may extract the information of the intermediate layer from the information of the weight between arbitrary adjacent intermediate layers. Further, the intermediate layer extraction unit 23 selects a node (or an artificial node) from among the nodes in each intermediate layer, the information of the weight (the weight between each node in the adjacent layer) related to the node matches a predetermined criterion. It is also possible to select (the node designated by the above) and generate the base image data based on the information of the weight related to the node. That is, the information of the intermediate layer may be extracted from a specific intermediate layer such as the deepest intermediate layer described above, or the information of the intermediate layer may be extracted from a plurality of intermediate layers.

In this case, when there are L nodes in the output layer, focusing on one node j in the intermediate layer, there are L weights wij (i=1, 2,..., L). Therefore, the intermediate layer extracting unit 23 arranges the L weights for each node (j=1, 2,..., N) of the intermediate layer in a matrix having a ratio closest to the matrix of x×y. Is generated, and the temporary base image data for each node is resized into image data of x×y pixels by a predetermined method to obtain base image data. Then, the intermediate layer extraction unit 23 outputs this base image data (N pieces of image data) as intermediate layer information.

The base image data (N pieces of image data) includes, for example, as shown in FIG. 3, patterns (line segment patterns, color patterns, etc.) in the image data input in the machine learning process. (For example, Dumitru Erhan, et.al., “Understanding Representations Leaned in Deep Architectures”, inet: https://www.researchgate.net/publication/265745969_Understanding_Representations_Learned_in_Deep_Architectures (2018 Search on the 10th of the month)). In this embodiment, this base image data is used as an image pattern.

The pattern holding unit 31 of the search processing unit 30 holds the base image data output by the intermediate layer extraction unit 23 of the machine learning processing unit 20. The pattern holding unit 31 may hold all of the base image data output by the intermediate layer extraction unit 23 as an image pattern, or may set a predetermined condition among the base image data output by the intermediate layer extraction unit 23. Only satisfying base image data may be held as an image pattern. As a condition here, for example, when an image of a line segment is used as an image pattern, well-known line segment detection processing such as Hough transform and RANSAC is performed, and the number of detected line segments is equal to or more than a predetermined number. It may be on condition that it has become. According to this example, only the base image data having the feature as a line segment is selectively held as an image pattern.

In another example, the pattern holding unit 31 may select and hold, as the image pattern, one of the base image data whose predetermined feature amount satisfies a predetermined condition. Here, the characteristic amount and the above-mentioned condition are, for example, widely known using a Haar-Like characteristic amount and a calculation pattern (edge pattern, line segment pattern, peripheral pattern, etc.) used for calculating the Haar-like characteristic amount. May be used)), and the condition that the degree of coincidence with the value is higher than a predetermined threshold value.

Here, the degree of coincidence is the number of pixels in which the brightness of the pixel value is lower than a predetermined threshold value in the region in the base image data corresponding to the region in which the calculation pattern is significant pixels (for example, black). PBN and the number of pixels PWN in which the brightness of the pixel value is higher than a predetermined threshold value in the region in the base image data corresponding to the region in which the calculation pattern is not a significant pixel (for example, white) Sum of PBN+PWN, etc. The areas are associated with each other by enlarging or reducing the calculation pattern to the size of the base image data (if the ratios do not match, enlarging or reducing the circumscribed rectangle of the calculation pattern to match the size of the base image data. ), and the pixels are associated with each other.

Further, as the above feature amount, a HOG (Histograms of Oriented Gradients) feature amount may be used. In this case, when the HOG feature amount is extracted from the base image data and the condition that the number of pixels corresponding to the extracted boundary line of the brightness change exceeds a predetermined threshold value is satisfied, the base image data is extracted. May be selected and held as an image pattern.

As a method for selecting an image pattern from the base image data, various methods such as a method selected by the user can be adopted in addition to this.

The information collecting unit 32 crawls the web page based on a predetermined rule, and outputs the image data provided on the acquired web page to the feature amount computing unit 33 as the processing target image data. Here, as a crawling rule, a method of following a link and various other methods are widely known, and therefore a detailed description thereof will be omitted here. In addition, the information collecting unit 32 outputs the reference information of the image data (the URL from which the image data is acquired) of which the feature amount is calculated by the feature amount calculating unit 33 to the feature amount holding unit 34, and outputs it to the search database. Instruct to record.

The feature amount calculator 33 receives input of image data to be processed, and calculates the degree of similarity between the image data (image data to be processed) and each of the plurality of base image data. Specifically, the calculation of the similarity here is performed by raster-scanning a window having the same size as the base image data in the processing target image data by a predetermined number of dots, for example, and extracting a partial image in the window to obtain the base image. This can be realized by performing processing such as pattern matching by a widely known method such as calculating the cross-correlation between the data and the partial image.

Here, it is preferable that the window for extracting the partial image is scanned while allowing overlap in at least one of the vertical and horizontal directions. That is, the partial image for calculating the mutual layer and the like with the base image data may be extracted from the processing target image data while allowing duplication. When extraction is performed while allowing overlap, it is possible to detect even if the pattern to be compared with the base image data is vertically or horizontally shifted.

Further, the degree of similarity can be obtained, for example, as follows. That is, the feature amount calculation unit 33 obtains the direction line element feature amount for each partial image. Specifically, the feature amount calculation unit 33 further divides the partial image into blocks of a predetermined size, and obtains a directional line element feature amount for each block. The direction line element feature amounts here are not limited to those in four directions. For an example of the direction line element feature amount in the case of multiple directions exceeding 4, the unit direction vector of the line segment between the feature points forming the stroke and the unit vectors (line element patterns) of various predetermined directions are used. A widely known method such as an example in which the inner product is obtained and used as the directional line element feature amount can be adopted, and thus detailed description thereof is omitted here.

The feature amount calculation unit 33 refers to the direction line element feature amount obtained for each block, and for each angle of the line segment represented by each predetermined line element pattern, the direction line element associated with the line element pattern of the corresponding angle. Data that associates a list of information that identifies blocks that are feature quantities is generated. Hereinafter, this data will be referred to as comparison target phase sequence data.

The feature amount calculation unit 33 also divides each of the base image data into blocks of the above-described predetermined size in advance as in this example, obtains the directional line element feature amount of each block, and determines each of the predetermined line elements. For each angle of the line segment represented by the pattern, data is generated and stored in association with a list of information that identifies blocks that are directional line element feature amounts related to the line element pattern of the corresponding angle. Hereinafter, this data is referred to as search phase sequence data.

The feature amount calculation unit 33 calculates the difference between the list (each piece of information specifying blocks) of the comparison target phase order data and the list included in the search phase order data for each base image data. As a specific example, the comparison target phase order data for the angle θ includes information (dxi, dyi) (here, i=0, 1, 2,...) That identifies a block, and is used for searching certain base image data. When information (xj, yj) (here, j=0, 1,...) Which specifies a block is included in the phase sequence data,
ΣiΣj|(xj, yj)-(dxi, dyi)|
Is calculated, and the sum of these calculation results for the angle θ is taken as the distance d between the partial image and the base image data. It should be noted that Σi means to take the sum for i, and Σj means to take the sum for j. Further, in (dxi, dyi) or (xj, yj), the i (or j)th of the list is dxth in the x-axis direction, dy-th in the y-axis direction (or x-th in the x-axis direction, in the y-axis direction). It means that it is the information that identifies the block at the y-th position.

Further, the feature amount calculation unit 33 may give a weight to the distance d by using the color information of each pixel included in the extracted partial image and the pixel adjacent thereto.

For example, the feature amount calculation unit 33 may give a weight based on the difference between the pixel in the partial image (the same size as the base image data) and the corresponding pixel in the base image data. In this case, |e|, which is the sum of the absolute values of the pixel value differences for each of the corresponding pixels of the partial image and the base image data, is used, and this |e| The correction distance D=α|e|+βd may be obtained by weighting and adding to the distance d with the image data (α and β are positive weight values experimentally determined).

Further, the difference in the pixel values may be used instead of the difference in the pixel values. In this example, the feature amount calculation unit 33 sequentially selects the pixels included in the extracted partial image as the target pixel. Then, the feature amount calculation unit 33 calculates the absolute value (distance in the color space) of the color difference between the pixel adjacent to the target pixel and the target pixel. This color difference may be due to, for example, the pixel of interest in the raster scan order (from the pixel on the upper left of the video data, one line is scanned from left to right, then to the next lower line, and from left to right... When scanning and selecting in the sequential scanning order), the difference between the pixel value of the immediately preceding target pixel and the pixel value of the current target pixel may be used, or the average of the differences between the four neighboring pixels may be used. Good.

Note that the feature amount calculation unit 33 determines the difference (color difference) between the pixel value (color) of the pixel in the partial image and the pixel value (color difference) of the base image data with respect to the distance d (color difference). ; Preliminarily obtained by the same method as for the partial image). Specifically, when the value of the color difference between each corresponding pixel in the partial image and certain base image data with respect to each peripheral pixel is cpz and cpdz, the feature amount calculation unit 33 determines the difference f=cpz-cpdz. calculate.

Using the absolute value |f| of this value f, the feature amount calculation unit 33 weights and adds the correction distance D to the distance d together with the absolute value |e| of the difference in pixel value,
D=α|e|+βd+γ|f|
(Α, β, γ are positive weight values determined experimentally).

According to this, it becomes possible to influence the correction distance D when the portions where the color difference is significant coincide with each other, and it is possible to reduce the influence of the portion where the color difference is not generally large such as the background.

In these examples, the feature amount calculation unit 33 uses the reciprocal of the distance d (or the corrected distance D) obtained here (or the reciprocal may be obtained by adding a predetermined positive constant to d or D) to the partial image and the base. The degree of similarity with the image data.

According to this example of the present embodiment, an effect that exceeds the case where the convolution widely used in deep learning is adopted is shown.

The feature amount calculation unit 33, for each base image data, the similarity between the target image data and the base image data exceeds a predetermined threshold value (a pattern that matches the pattern of the base image data Count) how many partial images were found.

Then, the feature amount calculation unit 33 obtains the count value for each base image data, and the histogram information (vectors) in which the count values are arranged in the order of the predetermined base image data (the order in which the base image data is added may be sufficient). Information) is output as the feature amount of the image data to be processed.

When the feature amount storage unit 34 is instructed by the information gathering unit 32 to record the feature amount, the feature amount holding unit 34 obtains from the reference information output by the information gathering unit 32, the address specified by the reference information, and the like. The image data thus output is added to the search database and recorded in association with the feature amount output by the feature amount calculation unit 33 (FIG. 4). In the following description, the feature amount stored in the search database will be called a search code.

One of the features of the present embodiment is that at this time, it is not necessary to record the processing target image data (image data acquired from the website) in the search database. As a result, the storage capacity for holding the image data can be reduced, and there is no problem of copyright related to the image data.

The acceptance unit 35 accepts an input of image data to be searched from the user, and outputs the image data to the feature amount calculation unit 33 as processing target image data.

The search unit 36 receives the information of the feature amount (referred to as a search target feature amount) calculated by the feature amount calculation unit 33 for the image data output by the reception unit 35, and outputs a search code similar to the search target feature amount. Search from the search database. Here, the similarity between the feature amounts can be calculated as follows, for example.

In this example of the present embodiment, the feature amount is information of the histogram for each base image data, and is therefore a vector amount having dimensions as many as the number of base image data. Therefore, for example, the similarity of the feature amounts can be obtained by the inner product between the normalized feature amounts (the size is “1”) (between the search target feature amount and the search code) (so-called cosine similarity). ..

The search unit 36 calculates the degree of similarity between each of the search codes stored in the search database and the search target feature amount, and associates a predetermined number of search codes with the highest degree of similarity with the search codes. Get the reference information and output it. Note that the search unit 36 may acquire the reference information associated with the search code related to the similarity only when the similarity exceeds a predetermined threshold value.

The information output unit 37 searches the input image data for a list of reference information output by the search unit 36 (a list in which reference information is arranged in descending order of similarity between the associated search code and the search target feature amount). Output as a result. If the reference information is included in this list only when the similarity exceeds a predetermined threshold value, the list may not include the reference information. In this case, the information output unit 37 may output information indicating that image data similar to the input image data could not be found.

[motion]
The present embodiment basically has the above configuration and operates as follows. In the information processing device 1 according to the present embodiment, first, a machine learning process is performed to obtain an image pattern.

That is, the control unit 11 of the information processing device 1 accepts input of data to be machine-learned. Here, it is assumed that the data to be machine-learned is image data. The control unit 11 executes machine learning processing of a neural network including at least one intermediate layer such as an auto encoder. This neural network does not necessarily have to be unsupervised learning like an automatic encoder, and machine learning processing for image classification using CNN (convolutional network) may be executed. When performing machine learning with a teacher, it is assumed that teacher data corresponding to input image data is prepared. The neural network that is the target of this machine learning may have a plurality of intermediate layers.

The information processing apparatus 1 resizes the received image data to a predetermined size and inputs the resized image data to the input layer of the neural network, so that the output of the neural network becomes the target data and the difference between the target data and the output is obtained. Based on, the weight information between the input layer and the intermediate layer in the neural network, and the weight information between the intermediate layer and the output layer (if multiple intermediate layers are included, further between the intermediate layers). Weight information). Since this processing is the same as the machine learning processing of a general neural network, detailed description thereof is omitted here.

When the auto encoder is used, the target data here is the input image data itself.

The information processing apparatus 1 performs machine learning based on a plurality of image data and then extracts and outputs the machine learned information of the intermediate layer of the neural network. As described above, in this example, for each node of the intermediate layer, information of a plurality of weights input to or output from the node is set to a predetermined size (horizontal x pixel, vertical y pixel). ) To generate base image data, and output the base image data obtained for each node as information of the intermediate layer.

Further, the information processing device 1 that performs the search process operates as follows. One of the characteristic features of the present embodiment is that the information processing apparatus 1 that performs a search process does not need to use a neural network, needs to hold a neural network, updates its weight, or has a neural network. It is not necessary to input information into and to calculate the output.

The information processing device 1 that performs the search process according to the example of the present embodiment holds the base image data. Then, a search database generation process and a search database search process are executed.

First, the information processing apparatus 1 that performs the process of generating the search database crawls the web page based on a predetermined rule, and sets the image data provided on the acquired web page as the processing target image data.

Then, the information processing device 1 receives the input of the image data to be processed (the vertical and horizontal sizes are larger than the basic image data), and receives the image data (processing target image data) and the plurality of basic image data. The degree of similarity with each is calculated. In the example of the present embodiment, the similarity calculation is performed by raster-scanning a window having the same size as the base image data in the processing target image data, extracting a partial image in the window, and extracting the partial image in the window. A widely known method such as calculating a cross-correlation with a partial image is used to perform processing such as pattern matching.

As described above, the window for extracting the partial image may be scanned while allowing overlap in at least one of the vertical and horizontal directions.

The information processing apparatus 1 has, for each piece of base image data, the similarity with the base image data in the processing target image data exceeds a predetermined threshold value (has a pattern that matches the pattern of the base image data). ) Count how many partial images were found.

The information processing apparatus 1 obtains the count value for each piece of base image data, and histogram information (vector information) in which the count values are arranged in the order of predetermined base image data (the order in which the base image data is added may be used). Is stored as a search database in association with the URL (reference information) from which the image data input as the processing target is acquired, as the feature amount of the processing target image data.

By this processing of the information processing apparatus 1, as shown in FIG. 4, reference information (URL) of image data and vector information enumerating the number of each image pattern which is the base image data included in the image data (search) And a code) are stored in the storage unit 12 as a search database.

The information processing device 1 may repeatedly execute the process of generating the search database at every predetermined timing.

Next, the information processing apparatus 1 that executes the search process receives the input of the image data to be searched from the user and operates as follows.

The information processing apparatus 1 calculates the similarity between the image data to be searched and each of the plurality of base image data. Then, the information processing apparatus 1 has, for each base image data, the similarity between the target image data and the base image data exceeds a predetermined threshold value (a pattern that matches the pattern of the base image data is Count) how many partial images were found.

The information processing device 1 obtains the count value for each base image data and retrieves the information (vector information) of the histogram in which the count values are arranged in a predetermined base image data order (the same order as the search code). The feature amount of the target image data (search target feature amount) is set.

Then, the information processing device 1 searches the search database for a search code similar to the search target feature amount. Here, the similarity between the feature amounts may be an inner product of the normalized feature amounts (the size is “1”) (between the search target feature amount and the search code).

The information processing apparatus 1 calculates the degree of similarity between each of the search codes stored in the search database and the search target characteristic amount, and associates a predetermined number of search codes with the highest degree of similarity with the search code. Get the reference information. Then, the information processing apparatus 1 uses the acquired list of reference information (a list in which reference information is arranged in descending order of similarity between the associated search code and the search target feature amount) as the search result of the input image data. Output.

Note that, in the above description, an example in which one information processing device 1 performs a machine learning process, a search database generation process, and a search process has been described, but the present embodiment is not limited to this. For example, the information processing device 1 that performs the machine learning process and the information processing device 1 that performs other processes may be different. In this case, the information processing device 1 that performs the search database generation process and the search process acquires the information of the base image data from the information processing device 1 that performs the machine learning process. As described above, the information processing apparatus 1 that performs the process of generating the search database and the process of searching at this time has information on the neural network that is the source of the base image data. Information on the weights of 1) is not always necessary, and only the base image data need be held.

Further, the information processing device 1 that performs the search database generation process and the information processing device 1 that performs the search process may be separate entities. In this case, the information processing apparatus 1 that performs the search process receives the input of the search database from the information processing apparatus 1 that performs the process of generating the search database, and holds the search database.

According to the present embodiment, since the base image data that is the image pattern is used in the process of generating the search database and the process of the search, there is no need for the calculation of the neural network, and each process is performed by a relatively lightweight process such as pattern matching. Can be executed.

[Match with known data]
Further, in the example of the present embodiment, matching may be performed with known data, for example, image data such as tires, headlights, windows, etc., in which what is captured is known.

In this example, image data of a known object (such as the tire) is input in advance as image data to be processed, and the operation of the characteristic amount calculation unit 33 is performed to obtain the characteristic amount as a search code. deep. Hereinafter, the search code based on the image data of this known object will be referred to as a known code. In the present embodiment, this known code is stored and held in the storage unit 12 in association with the name of the imaged object (information for identifying the known object).

Then, in this example, the control unit 11 further determines, for the image data (processing target image data) to be processed by the feature amount calculating unit 33, a partial region of the processing target image data that is determined by a predetermined method. At least one is extracted. This partial area may be, for example, image data divided into a matrix of 32×32, or may be an area of a predetermined size including four corners of the image data (in this case, each partial area includes overlapping portions). It may be).

The control unit 11 causes the feature amount calculation unit 33 to calculate the feature amount (referred to as a partial feature amount) in each partial region extracted here. Then, the known code stored in the storage unit 12 is compared with the partial feature quantity, and if there is a known code whose similarity to the partial feature quantity exceeds a predetermined threshold value, the object associated with the known code Gets the name information of.

When the information collection unit 32 gives an instruction to record the feature amount for the processing target image data as a list of object names obtained for each partial area and an object list related to the processing target image data, the control unit 11 returns the information The reference information output by the collection unit 32 and the feature amount output by the feature amount calculation unit 33 for the image data acquired and output by the information collection unit 32 from the address specified by the reference information and the like are obtained here. The object list is associated and added to the search database and recorded. If no object name is obtained from any of the partial areas, an empty list is recorded as the object list.

Further, as the processing of the search unit 36, the control unit 11 acquires the search target feature amount calculated by the feature amount calculation unit 33 for the received image data, and generates the object list based on the received image data. The object list that is the target of this search is hereinafter referred to as the search target list.

The control unit 11 searches the search database for a search code that is similar to the acquired search target feature amount, acquires the object list associated with the reference information together with the search code in descending order of similarity, and searches for the search target. Compare with list.

Then, the control unit 11 associates each of the search codes similar to the search target feature quantity (search code in which the inner product between the feature quantities with the search target feature quantity exceeds a predetermined threshold value) to the search code. The number n of object names included in the object list associated with the search code is a value n obtained by counting the number of object names included in the search target list among the object names included in the object list. The value n/N divided by is obtained as the target degree of coincidence.

Among the search codes similar to the search target feature amount, the control unit 11 determines, for the search codes whose target matching degree exceeds a predetermined threshold value, a predetermined number or less in descending order of similarity to the search target feature amount. , Obtains and outputs the reference information associated with the search code.

According to this example of the present embodiment, for example, when image data in which a tire of an automobile is imaged is input as a search target, it has a circular shape similar to the tire (the classification patterns match), but the tire is It is possible to reduce the chance that the reference information of the image data including different objects is output as a result of the search.

[Modification]
In the above description, the data to be processed is image data, but the present embodiment is not limited to this, and may be a character string or other data. Also in this case, the number of each data pattern included in the data to be processed is counted for the data to be processed using the information of the weight of the intermediate layer of the neural network learned by machine learning as the data pattern.

Then, the count values for each data pattern are arranged in a predetermined order (predetermined order for the data pattern) to generate vector information representing a histogram, and this vector information is used as a feature amount (feature vector information), The predetermined process may be executed.

[Modified example in which processing is performed for each color component]
Further, the information processing apparatus 1 according to the present embodiment described so far may divide image data into color components and perform machine learning and search processing.

Specifically, when the control unit 11 receives an input of image data to be machine-learned, the control unit 11 resizes this image data to a predetermined size, and further, for example, image data of each component of RGB (hereinafter, for the sake of distinction, (Called image data). Here, a method of dividing image data into component image data for each color component such as R component, G component, and B component is widely known, and therefore detailed description thereof will be omitted. Also, although an example of separating into RGB will be described below, it may be decomposed into HSV (hue, saturation, lightness) instead of RGB components, or into other color space components. May be.

The control unit 11 holds information on the model of the neural network for each color component, and machine-learns these neural networks. That is, the control unit 11 inputs the component image data obtained by the decomposition into the input layer of the neural network corresponding to the color components of the component image data, so that the output of this neural network becomes the target data. Based on the difference between the target data and the output, information on the weight between the input layer and the intermediate layer in the neural network, and information on the weight between the intermediate layer and the output layer (including multiple intermediate layers) If it is set, the weight information between the intermediate layers is set.

The information processing apparatus 1 performs machine learning of a neural network corresponding to each component based on each component image data of a plurality of image data, and then extracts information on the intermediate layer of each neural network subjected to machine learning. Output. As described above, in this example, for each node in the intermediate layer, information on a plurality of weights input to or output from the node is arranged in a predetermined size to obtain base image data. The base image data generated and obtained for each node is output as information of the intermediate layer.

In this example of the present embodiment, since the information of the intermediate layer is obtained for each color component, the base image data is also obtained for each color component.

Then, the information processing apparatus 1 that performs the search process holds the base image data for each color component (corresponding to each color component) obtained as described above. Then, a search database generation process and a search database search process are executed.

First, the information processing apparatus 1 that performs the process of generating the search database crawls the web page based on a predetermined rule, and sets the image data provided on the acquired web page as the processing target image data.

Then, the information processing apparatus 1 receives the image data to be processed (the vertical and horizontal sizes are larger than the basic image data), and receives the image data (processing target image data) as a color component (basic image data). (The same color component as each color component of) is obtained to obtain the processing target component image data.

The information processing apparatus 1 calculates the degree of similarity between the processing target component image data obtained corresponding to each color component and each of the plurality of base image data relating to the corresponding color component. In the example of the present embodiment, the similarity calculation is performed by raster-scanning a window having the same size as the base image data in the processing target component image data for each color component, and extracting a partial image in the window, Processing such as pattern matching is performed and executed by a widely known method such as calculating the cross-correlation between the base image data corresponding to the color component of the processing target component image data and the partial image.

Also in this case, the window for extracting the partial image may be scanned while allowing overlap in at least one of the vertical and horizontal directions. That is, the partial image for calculating the inter-layer and the like with the base image data may be extracted from the processing target image data while allowing duplication. When extraction is performed while allowing overlap, it is possible to detect even if the pattern to be compared with the base image data is vertically or horizontally shifted.

The information processing apparatus 1 includes, for each piece of base image data relating to the corresponding color component, the similarity with the base image data in the processing target component image data exceeds a predetermined threshold value (the base image data Count how many partial images were found (having a pattern that matches the pattern).

The information processing apparatus 1 obtains the count value for each color component and for each base image data, and obtains information (vector information) of a histogram in which the count values are arranged in order of the base image data of each predetermined color component, As the feature amount of the image data to be processed, it is accumulated and stored as a search database in association with the URL (reference information) from which the image data input as the process target is acquired.

By this processing of the information processing device 1, similar to the example illustrated in FIG. 4, reference information (URL) of the image data and vector information listing the number of each image pattern that is the base image data included in the image data. (Search code) is associated and stored as a search database in the storage unit 12.

Also in this example, the information processing apparatus 1 may repeatedly execute the process of generating the search database at every predetermined timing.

Further, in this example, the information processing apparatus 1 that executes the search process receives the input of the image data to be searched from the user and operates as follows.

The information processing device 1 divides the image data to be searched into image data of the same color components as the components of the color space used for machine learning (for example, component image data of RGB components).

Then, the information processing device 1 calculates the degree of similarity between the component image data obtained by division and each of the plurality of base image data relating to the corresponding color component. In the example of the present embodiment, the similarity calculation is performed by raster-scanning a window having the same size as the base image data in the component image data, extracting a partial image in the window, and calculating the color of the component image data. Processing such as pattern matching is performed and executed by a widely known method such as calculating the cross-correlation between the base image data corresponding to the component and the partial image.

The information processing device 1 includes, for each of the base image data relating to the corresponding color component, the similarity between the component image data and the base image data exceeds a predetermined threshold value (in the pattern of the base image data, Count how many sub-images (with matching patterns) were found.

The information processing device 1 obtains the count value for each base image data and retrieves the information (vector information) of the histogram in which the count values are arranged in a predetermined base image data order (the same order as the search code). The feature amount of the target image data (search target feature amount) is set.

Then, the information processing device 1 searches the search database for a search code similar to the search target feature amount. Here, the similarity between the feature amounts may be an inner product of the normalized feature amounts (the size is “1”) (between the search target feature amount and the search code).

The information processing apparatus 1 calculates the degree of similarity between each of the search codes stored in the search database and the search target characteristic amount, and associates a predetermined number of search codes with the highest degree of similarity with the search code. Get the reference information. Then, the information processing apparatus 1 uses the acquired list of reference information (a list in which reference information is arranged in descending order of similarity between the associated search code and the search target feature amount) as the search result of the input image data. Output.

DESCRIPTION OF SYMBOLS 1 information processing device, 11 control part, 12 memory part, 13 operation part, 14 display part, 15 communication part, 20 machine learning processing part, 21 acceptance part, 22 machine learning part, 23 middle class extraction part, 30 search processing part , 31 pattern holding unit, 32 information collecting unit, 33 feature amount calculating unit, 34 feature amount holding unit, 35 receiving unit, 36 searching unit, 37 information output unit.

Claims (4)

Holding means for holding a pattern of a plurality of data acquired by machine learning,
In response to the input of the data to be processed, for each of the plurality of data patterns, the number of portions of the data to be processed that are determined to be similar to each of the data patterns is counted, and Feature amount generating means for generating feature vector information based on the result,
An execution unit that executes a predetermined process relating to the process target data by using the feature vector information;
Information processing device including. The information processing apparatus according to claim 1, wherein the data is image data,
The holding unit holds a plurality of image patterns acquired by machine learning,
The feature amount generation means receives the input of the image data to be processed, and counts the number of regions determined to be similar to the image pattern among the regions included in the image data for each of the plurality of image patterns. Then, the feature vector information based on the result of the count is generated,
The information processing apparatus, wherein the execution unit uses the feature vector information to execute a predetermined process related to the image data. The information processing apparatus according to claim 2, wherein
For a plurality of image data accessible via a network using unique network addresses, holds a database that holds the network address of the image data and the feature vector information generated by the feature amount generating means in association with each other. Then
The executing means receives the input of the image data to be searched, and the characteristic vector information generated by the characteristic amount generating means for the image data to be searched, and the characteristic vector information included in the database. And an information processing apparatus that extracts the network address of the image data that is the search result based on the comparison result. A computer that holds multiple image patterns acquired by machine learning
Upon receiving the input of the image data to be processed, for each of the plurality of image patterns, the number of regions determined to be similar to the image pattern among the regions included in the image data is counted, and the result of the count is calculated. Feature amount generating means for generating feature vector information based on
Executing means for executing a predetermined process relating to the image data by using the feature vector information;
Similar image search program to function as.

Images