JP7046239B2 - Methods and systems for generating neural networks for object recognition in images - Google Patents

Description

The present disclosure relates to the field of neural networks. In particular, but not limited to, the present disclosure relates to methods and systems for generating neural networks for object recognition in images.

In recent years, the need for real-time video analysis has increased significantly. Video analysis or video content analysis involves object recognition of one or more objects in a video, determining in real time a particular action or action performed by one or more objects, and based on the determined action or action. Provide insights or alerts to users. For example, video analysis is used to perform visual surveillance from a crowded area to a factory to monitor the movement and activity of one or more objects. An important aspect in video analysis is object recognition in images of video equipment. Object recognition involves detecting and identifying one or more objects in an image ranging from humans to animals, cars to safety devices, and the like. Object recognition performed using neural networks achieves higher accuracy compared to traditional image processing techniques. Existing techniques use a class of deep neural networks, or convolutional neural networks, to perform object recognition. A convolutional neural network is trained by providing multiple images corresponding to one or more recognized objects and comparing the output of the convolutional neural network with the class labels associated with the multiple images of the convolutional neural network. One or more parameters are modified using supervised learning techniques. A trained convolutional neural network is used to recognize one or more objects in a video image.

Problems with existing techniques require that the convolutional neural network be trained by a large number of images in order for each object to be recognized by the convolutional neural network. Therefore, the time required to train a convolutional neural network increases significantly with the number of recognized objects and the number of images in the training dataset. In addition, many images may not be available for each object in training convolutional neural networks.

Another problem with existing techniques is that training a convolutional neural network with fewer images leads to overfitting and thus reduces the accuracy of object recognition.

The information disclosed in this background art in the Disclosure section is solely to enhance the understanding of the general background of the invention and tolerate or acknowledge that this information forms prior art already known to those of skill in the art. It should not be interpreted as any form of suggestion.

Additional features and advantages are realized through the techniques of the present disclosure. Other embodiments and embodiments of the present disclosure are described in detail herein and are considered part of the claimed disclosure.

Disclosed herein is a method of generating a neural network for object recognition in an image. The method comprises receiving information about a hierarchical relationship between one or more objects. Further, the method comprises providing one or more images of the training data set corresponding to each object of one or more objects as an informed input to the base neural network, the base neural network being one or more. Associated with the parameters of. Further, the method comprises determining the loss value of the base neural network for each input image based on the output of the base neural network and the class label corresponding to each input image, the output being to one or more objects. Shows the corresponding similarity value. Finally, the method updates one or more parameters for each input image based on at least one of the output, the loss value of the base neural network, and the second user input to generate the neural network. Neural networks are used for object recognition.

Further, the present disclosure discloses a processor and a training system including a memory communicatively coupled to the processor, which, at run time, causes the processor to receive information about hierarchical relationships between one or more objects. To store. Further, the processor is configured to provide one or more images of the training data set corresponding to each object of one or more objects as input to the base neural network based on the information, and the base neural network is one. It is associated with the above parameters. In addition, the processor is configured to determine the loss value of the base neural network for each input image based on the output of the base neural network and the class label corresponding to each input image, the output corresponding to one or more objects. Indicates a similar value to be used. Finally, the processor updates one or more parameters for each input image based on at least one of the output, the loss value of the base neural network, and the second user input to generate the neural network. The neural network is used for object recognition.

The outline of the above invention is for illustration purposes only and is not intended to be limiting. In addition to the embodiments, embodiments, and features described above, additional embodiments, embodiments, and features may be revealed by reference to the drawings and subsequent embodiments for carrying out the invention.

The novel features and features of this disclosure are described in the appended claims. However, the present disclosure itself and preferred embodiments, its additional objectives and advantages, when read in conjunction with the accompanying drawings, can best be understood with reference to embodiments of the embodiments for carrying out the following inventions. The accompanying drawings incorporated into this disclosure and forming part of this disclosure provide exemplary embodiments and, along with explanations, serve to explain the disclosed principles. In the figure, the number on the far left of the reference number (s) identifies the figure in which the reference number is displayed first. One or more embodiments will be described herein by way of example only, with reference to the accompanying figures in which similar reference numbers represent similar elements.

FIG. 6 illustrates an exemplary environment for generating a neural network for object recognition in an image according to some embodiments of the present disclosure. It is a detailed block diagram of the training system which concerns on some embodiments of this disclosure. It is a figure which shows the exemplary information received from the user which concerns on some embodiments of this disclosure. It is a flowchart which shows the method for generating a neural network for object recognition in an image according to some embodiments of this disclosure. It is a figure which shows the exemplary training data set which concerns on some embodiments of this disclosure. It is a diagram showing exemplary information represented using a tree structure according to some embodiments of the present disclosure. It is a figure which shows one or more sorted exemplary images which concerns on some embodiments of this disclosure. It is an exemplary block diagram of a base neural network according to some embodiments of the present disclosure. FIG. 3 illustrates an exemplary convolution of grayscale input images using one or more kernels according to some embodiments of the present disclosure. FIG. 3 illustrates an exemplary convolution of a color input image using one or more kernels according to some embodiments of the present disclosure. It is a figure which shows the exemplary convolution layer of the base neural network which concerns on some embodiments of this disclosure. It is a figure which shows the exemplary mechanism of the normalized linear unit layer of the base neural network which concerns on some embodiments of this disclosure. It is a figure which shows the exemplary mechanism of the pooling layer of the base neural network which concerns on some embodiments of this disclosure. FIG. 3 illustrates an exemplary fully connected layer of a base neural network according to some embodiments of the present disclosure. FIG. 3 illustrates an exemplary first and second layer of a base neural network with one or more connections according to some embodiments of the present disclosure. It is a figure which shows the example kernel activation list of the base neural network which concerns on some embodiments of this disclosure. FIG. 3 illustrates an exemplary one or more kernels of a base neural network with a plurality of values according to some embodiments of the present disclosure. It is a figure which shows the exemplary modification of one or more values of one or more kernels based on the output of a base neural network, according to some embodiments of this disclosure. FIG. 3 illustrates an exemplary base neural network with one or more layers according to some embodiments of the present disclosure. FIG. 5 illustrates an exemplary addition of one or more kernels to a base neural network according to some embodiments of the present disclosure. FIG. 5 illustrates an exemplary modification of one or more connections of a base neural network according to some embodiments of the present disclosure. FIG. 5 illustrates an exemplary addition of one or more objects to the kernel activation list of a base neural network according to some embodiments of the present disclosure. FIG. 5 illustrates an exemplary addition of one or more objects to the kernel activation list of a base neural network according to some embodiments of the present disclosure. It is a figure which shows the exemplary modification of the index of one or more kernels of the kernel activation list of a base neural network which concerns on some embodiments of this disclosure. It is a figure which shows the exemplary modification of the information received based on the 1st user input which concerns on some embodiments of this disclosure. FIG. 5 shows an exemplary generated neural network and kernel activation list for some embodiments of the present disclosure. It is a figure which shows the exemplary test data set which concerns on some embodiments of this disclosure. It is a figure which shows the exemplary output of the first layer of the neural network generated for the test image which concerns on some embodiments of this disclosure. FIG. 6 illustrates exemplary propagation of test images through one or more identified kernels according to some embodiments of the present disclosure. It is a figure which shows the general-purpose computer system for generating a neural network for object recognition in an image which concerns on some embodiments of this disclosure. It is a figure which showed various matrix which concerns on some embodiments of this disclosure. It is a figure which showed various mathematical formulas which concerns on some embodiments of this disclosure. It is a figure which showed various mathematical formulas which concerns on some embodiments of this disclosure.

It should be understood by those skilled in the art that any block diagram herein represents a conceptual diagram of an exemplary system that implements the principles of the subject. Similarly, any flow chart, flow diagram, state transition diagram, pseudo-code, etc. may be represented substantially on a computer-readable medium, regardless of whether the computer or processor is exemplified by the computer or processor. It can be understood that it represents a variety of processes that can be performed.

As used herein, the word "exemplary" is used herein to mean "to serve as an example, example, or example." Any embodiment or embodiment of the subject described herein as "exemplary" should not necessarily be considered preferred or more advantageous than other embodiments.

The present disclosure is susceptible to a variety of modified and alternative forms, the specific embodiments of which are shown by way of illustration in the drawings and may be described in detail below. However, it is not intended to limit this disclosure to any particular form disclosed, and conversely, this disclosure includes all modifications, equivalents, and alternatives that fall within the scope of this disclosure. Please understand.

The terms "comprises", "includes", "comprising", "inclusion" or any other variant thereof are intended to cover comprehensive inclusion. , Thus a setup, device, or method that includes a list of components or steps does not include only those components or steps, but is not explicitly indicated or is specific to such setup or device or method. May include the components of. In other words, one or more elements of a system or appliance preceded by "comprises ... a" or "includes ... a" is more. It does not rule out the presence of other or additional elements of the system or appliance without many restrictions.

The present disclosure describes how to generate a neural network for object recognition in an image. The method comprises receiving information about a hierarchical relationship between one or more objects. Further, the method comprises providing one or more images of the training data set corresponding to each object of one or more objects as an informed input to the base neural network, the base neural network being one. It is associated with the above parameters. Further, the method comprises determining for each input image the output of the base neural network and the loss value of the base neural network based on the class label corresponding to each input image, the output corresponding to one or more objects. Indicates a similar value to be used. Finally, the method updates one or more parameters for each input image based on at least one of the output, the loss value of the base neural network, and the second user input to generate the neural network. Neural networks are used for object recognition.

In the following detailed description of embodiments of the present disclosure, reference is made to the accompanying drawings that form part of the specification and show, by way of example, specific embodiments that the present disclosure may practice. These embodiments will be described in sufficient detail to allow one of ordinary skill in the art to practice the present disclosure, and that other embodiments may be utilized and that modifications may be made without departing from the scope of the present disclosure. I want to be understood. Therefore, the following explanation should not be construed in a limited sense.

FIG. 1 shows an exemplary environment for generating a neural network for object recognition in an image according to some embodiments of the present disclosure.

In one embodiment, the user (101) interacts with the training system (102) via the user interface (103) to generate a neural network for object recognition in the image. The training system (102) may be implemented on a server (not shown) to generate a neural network. The training system (102) is connected to the user interface (103) through one of the wired or wireless interfaces. The user interface (103) includes at least one of a display unit, a touch screen, a keypad, a microphone, a speaker, and the like. The user (101) may use the user interface (103) to provide an input to the training system (102) and receive an output (503) from the training system (102). The training system (102) trains the base neural network when it receives information about the hierarchical relationship between one or more objects from the user (101). The information is in an array of one or more objects in the hierarchy, for example a tree structure (ie, the first object is "above", "below", or "at the same height" as the second object. )including. The information received by the training system (102) may be modified by the user (101) using the first user input via the user interface (103). A base neural network is a convolutional neural network associated with one or more parameters. One or more parameters of the base neural network may be configured by the user (101) via the user interface (103). One or more parameters are, for example, one or more kernels in each of one or more layers, one or more layers, one or more kernel values, and one or more kernels in one or more layers. Includes data about connections between. The training system (102) trains the base neural network by providing one or more images corresponding to each object of the one or more objects as input to the base neural network. One or more images are stored in the training data set (104). In addition, the training dataset (104) stores the class labels associated with each of the one or more images. Class labels correspond to the names of one or more objects. For example, the first image in one or more images stored in the training data set (104) is an image of a car and the class label is a "sedan" indicating the type of car body. The training data set (104) may be implemented in a database connected to the training system (102) shown in FIG. 1, or the training data set (104) may be stored in the training system (102). be.

In one embodiment, for each input image provided as an input to the base neural network, the training system (102) determines the loss value of the base neural network based on the output of the base neural network and the class label corresponding to each input image. decide. The loss value indicates the misclassification rate by the base neural network. In addition, the training system (102) has one or more parameters associated with the base neural network for each input image based on at least one of the output, the loss value of the base neural network, and the second user input. To update. The user (101) may use the second user input via the user interface (103) to modify one or more parameters of the base neural network.

In one embodiment, the base neural network provides one or more images for each object as input from the training data set (104) and updates one or more parameters until the loss value is less than a predetermined threshold. Trained by doing. For example, the predetermined threshold may be 0.01. When the loss value is less than a predetermined threshold for each of the one or more objects, the training of the base neural network ends and a neural network containing one or more updated parameters from the base neural network is generated. The generated neural network is used for object recognition in the image.

In one embodiment, the user (101) provides the training system (102) with a test image stored in a test data set (105) to recognize an object in the test image. The test data set (105) includes one or more images without class labels, and the class labels or names in the one or more images need to be recognized. The training system (102) provides a test image as an input to the neural network, and based on the output of the neural network, the class label of the test image is determined or the object in the test image is recognized. The test data set (105) may be implemented in a database connected to the training system (102) as shown in FIG. 1, or the test data set (105) is stored in the training system (102). In some cases.

FIG. 2A shows a detailed block diagram of the training system (102) according to some embodiments of the present disclosure.

The training system (102) may include a central processing unit (“CPU” or “processor”) (203), and a memory (202) that stores instructions that can be executed by the processor (203). The processor (203) may include at least one data processor for executing program components for executing a user request or a system generation request. The memory (202) may be communicatively coupled to the processor (203). The training system (102) further includes an input / output (I / O) interface (201). The I / O interface (201) may be coupled with a processor (203) through which an input signal and / and an output signal can be communicated. In one embodiment, the training system (102) may receive input images, test images, first user inputs, and second user inputs through the I / O interface (201).

In some embodiments, the training system (102) may include data (204) and modules (209). As an example, the data (204) and the module (209) may be stored in a memory (202) configured by the training system (102). In one embodiment, the data (204) may include, for example, relationship data (205), parameter data (206), loss value data (207), and other data (208). In FIG. 2A shown, data (204) is described in detail herein.

In one embodiment, the relationship data (205) includes information about hierarchical relationships between one or more objects. One or more objects are determined based on the class label associated with the one or more images stored in the training data set (104). One or more objects are arranged in a hierarchy (eg, tree structure, chart, etc.) based on the relationships between the one or more objects. Relationships between one or more objects are determined based on the class label. For example, one or more objects "bike", "scouter", "sedan", "hatchback", "sport utility vehicle (SUV)", "bus", and one or more objects in one or more images of the training dataset (104). Include "track". As shown in FIG. 2B, the information (217) is arranged in the form of a tree structure having one or more objects represented by the nodes of the tree structure, and the edges represent the hierarchical relationship of the tree structure. Further, as shown in FIG. 2B, one or more objects "motorcycles" and "scooters" are classified as "vehicles with two wheels" and one or more objects "sedans", "hatchbacks", and. A "sports utility vehicle (SUV)" is classified as a "vehicle with four wheels" and one or more objects "bus" and "truck" are classified as a "vehicle with more than four wheels". Further, one or more objects "vehicle with two wheels", "vehicle with four vehicles", and "vehicle with more than four vehicles" are classified as "vehicles".

In one embodiment, the parameter data (206) comprises one or more parameters associated with the base neural network. One or more parameters are one or more kernels of each layer of the base neural network, one or more kernels of the first layer for one or more layers of the base neural network, and the first after the first layer. Includes one or more connections between one or more kernels in two layers, multiple values in one or more kernels, and at least one of the kernel activation lists. In addition, the kernel activation list contains the output of the first layer of the base neural network, one or more kernel indexes, and at least one of the class labels. For example, considering a base neural network with four layers, one or more kernels in each layer of the base neural network are seven kernels in the first layer, twelve kernels in the second layer, and three layers. There are 15 kernels and 18 kernels in the fourth layer. In another embodiment, consider a first kernel from one or more kernels having a size of 3x3. The plurality of values of the first kernel are represented using the matrix shown in FIG. 9A (matrix 1).

In one embodiment, the loss value data (207) comprises a loss value determined for the base neural network based on the loss function. The loss function is based on the predicted output of the base neural network and the class label associated with each input image. The loss value may indicate the rate of misclassification of the base neural network (ie, predicting the wrong class label for a given input image). Higher loss values indicate a higher misclassification rate and lower loss values indicate a lower misclassification rate. For example, a loss value of 2.4 indicates a higher misclassification rate and a loss value of 0.1 indicates a lower misclassification rate.

In one embodiment, the other data (208) is associated with a predetermined threshold, a first user input, a second user input, a base neural network and a generated neural network zeroed, stride, output 1 It may contain one or more values, a loss function, and so on.

In some embodiments, the data (204) may be stored in memory (202) in the form of various data structures. Further, the data (204) may be organized using a data model such as a relational data model or a hierarchical data model. The other data (208) may store data, including temporary data and temporary files, generated by the module (209) to perform various functions of the training system (102).

In some embodiments, the data (204) stored in the memory (202) may be processed by the module (209) of the training system (102). The module (209) may be stored in the memory (202). In one example, the module (209) communicatively coupled to the processor (203) configured within the training system (102) resides outside the memory (202) and is implemented as hardware, as shown in FIG. 2A. May be done. As used herein, the term module (209) is an application specific integrated circuit (ASIC), FPGA (field programmable gate array), electronic circuit, processor (203) (shared, dedicated, or group). And may refer to memory (202) running one or more software or firmware programs, combined logic circuits, and / or other suitable components that provide the described functionality. In some other embodiments, the module (209) may be implemented using at least one of an ASIC and an FPGA.

In one embodiment, the module (209) may be, for example, a communication module (210), an information correction module (211), an input module (212), a loss determination module (213), an update module (214), a recognition module (215). , And other modules (216). It can be understood that such above-mentioned module (209) may be represented as a single module or a combination of different modules (209).

In one embodiment, the communication module (210) is used to receive a first user input and a second user input from the user (101) via the user interface (103). The first user input includes modification of information (217) regarding hierarchical relationships between one or more objects. The second user input involves updating one or more parameters of the base neural network. The communication module (210) is connected to the user interface (103) through one of the wired or wireless interfaces. In addition, the communication module (210) outputs the base neural network to the user (101) via the user interface (103), one or more parameters of the base neural network, the generated neural network, the training data set (104). , And used to provide at least one of the test data sets (105).

In one embodiment, the information modification module (211) has information about a hierarchical relationship between one or more objects based on a first user input received from the user (101) via the user interface (103). Used to modify 217). The first user input is to add a new object to the information (217) based on the training dataset (104) and based on the total number of images for one or more objects in the training dataset (104). Swap the positions of one or more objects in the information (217) and delete at least one of the one or more objects in the information (217) based on the loss value of the base neural network, etc. include.

In one embodiment, the input module (212) is used to retrieve one or more images and the class labels corresponding to one or more images from the training data set (104). In addition, the input module (212) is used to sort one or more images based on the class label. Those skilled in the art may understand the use of one or more techniques including bubble sort, selection sort, merge sort, insertion sort, quicksort, etc. for sorting one or more images based on class labels. Further, the input module (212) is used to provide one or more images corresponding to each object as input to the base neural network based on the position of each object in the information (217). For example, one or more images corresponding to the first object "bike" of information (217) shown in FIG. 2B may be provided as input to the base neural network and one or more images such as "scooters". Follows.

In one embodiment, the loss determination module (213) is used to determine the loss value of the base neural network. The loss value is determined for each input image provided to the base neural network. In addition, the loss value is determined using the loss function based on the output of the base neural network and the class label corresponding to each input image. For example, the loss function may be at least one of mean squared error, cross entropy, hierarchical cross entropy techniques, and the like.

In one embodiment, the update module (214) is used to update one or more parameters of the base neural network. One or more parameters are the loss value of the base neural network, the output of the base neural network corresponding to the input image, and the second user input received from the user (101) via the user interface (103). Updated based on at least one. In addition, one or more parameters include adding one or more kernels at least one of the one or more layers of the base neural network and one or more of the one or more layers of the base neural network. Modifying one or more of the values in the kernel and after one or more kernels and the first layer of the first layer for one or more layers of the base neural network It is updated by performing at least one of modifying one or more connections to one or more kernels in the second layer and modifying the kernel activation list. For example, the connection from the third kernel of the first layer of the base neural network to the second kernel of the second layer of the base neural network after the first layer is the third kernel of the first layer. It may be modified to connect the kernel to the 4th kernel in the 2nd layer. In addition, new connections may be added between the fifth kernel in the first layer and the first kernel in the second layer.

In one embodiment, the recognition module (215) uses the generated neural network and is used to recognize object recognition in the test image. The user (101) may select a test image stored in the test data set (105) via the user interface (103). The recognition module (215) takes the test image from the test data set (105) and provides the test image as input to the first layer of the neural network. In addition, the recognition module (215) compares the output of the first layer of the neural network with the kernel activation list. In addition, the recognition module (215) identifies one or more kernels from the kernel activation list based on comparisons. The recognition module (215) then propagates the test image through one or more identified kernels of one or more layers of the neural network. Finally, the recognition module (215) recognizes the object in the test image based on the output of the neural network.

In one embodiment, the other module (210) adds one or more images to the training data set (104) and test data set (105) received from the user (101) via the user interface (103). Responsible for doing. In addition, another module (210) is used to generate information (217) based on at least one of the training data set (104) and the first user input.

FIG. 3 shows a flow chart showing a method of generating a neural network for object recognition in an image, according to some embodiments of the present disclosure.

The order in which method (300) can be described is not intended to be construed as a limitation, and any number of described method blocks may be combined in any order to implement the method. Moreover, individual blocks may be removed from the method without departing from the spirit and scope of the subject matter described herein. Further, the method may be implemented with any suitable hardware, software, firmware, or a combination thereof.

At step 301, the training system (102) receives information (217) about the hierarchical relationship between one or more objects. Hierarchical relationships indicate one or more objects that are classified into one or more classes and subclasses. In one embodiment, the information (217) is manually generated by the user (101). In one embodiment, the information may be automatically generated by the training system (102) using the training data set (104). Information (217) may be represented using at least one of a tree structure, charts, tables, and the like.

In one embodiment, the training data set (104) is one or more images (401A, 401B, ... 401N) of each of one or more objects, and one or more, as shown in FIG. 4A. Stores class labels (402A, 402B, ... 402N) corresponding to images (401A, 401B ... 401N). The one or more images (401A, 401B, ... 401N) of the training data set (104) is at least one of a black-and-white image, a grayscale image, and a color image. The class label (402A, 402B, ... 402N) is represented as at least one of text, vector, and the like. For example, the class label (402A, 402B, ... 402N) represented as text is a "beagle" and is a vector corresponding to one or more images (401A, 401B, ... 401N) of the "beagle". The class label represented as is shown in (matrix 2) of FIG. 9A.

The one or more images (401A, 401B, ... 401N) and class labels (402A, 402B, ... 402N) shown in FIG. 4A are for illustration purposes only and should not be treated as restrictions. Further, the training system (102) corresponds to one or more objects in the training data set (104) and each object in the one or more objects based on the first input from the user (101). Determine the total number of one or more images (401A, 401B, ... 401N) in the training data set (104). One or more objects are determined by identifying some unique class labels in the training data set (104). One or more images (401A, 401B, ... 401N) corresponding to each of the uniquely identified class labels are one or more images (401A, 401B, ... 401N) corresponding to each object. Integrated to determine the total number.

In one embodiment, the training system (102) is a total number of one or more images (401A, 401B, ... 401N) corresponding to each object, a first user input, and a relationship between the one or more objects. Build a hierarchical tree structure with one or more nodes and one or more edges based on at least one of them. In a hierarchical tree structure, one or more nodes represent one or more objects, and one or more edges represent relationships between one or more objects. The training system (102) may arrange one or more objects having similar relationships as nodes at the same level of the hierarchy (ie, the first level (403)). For example, one or more objects, namely "Labrador", "Shiba Inu", "Beagle", "Pitbull", "Russian Blue", "Siamese Cat", and "Persian Cat", are as shown in FIG. 4B. It is placed at the first level (403) of the hierarchy. In addition, one or more objects at the first level (403) of the hierarchy are integrated into the second level (404) (ie, with higher categories) using one or more edges. It is placed above one or more objects at the first level (403) of the hierarchy. For example, one or more objects "dog" and "cat" are the first level (403) as a subcategory of the corresponding one or more objects of the second level (404), as shown in FIG. 4B. Placed at the second level (404) of the hierarchy by edges representing one or more objects of. Further, one or more objects of the second level (404) are integrated into the third level (405) and the like. One of ordinary skill in the art can understand a hierarchical tree structure with one or more levels, and the three levels of the hierarchy shown in FIG. 4B are for illustration purposes only and should not be construed as restrictions. Information (217) includes a hierarchical tree structure generated by arranging one or more objects, as shown in FIG. 4B.

In one embodiment, the training system (102) receives from the user (101) a hierarchical tree structure showing information (217) about hierarchical relationships between one or more objects.

At step 302, the training system (102) is based on the information (217) with one or more images (401A,) of the training data set (104) corresponding to each object of the one or more objects as input to the base neural network. 401B, ... 401N) is provided. The base neural network is associated with one or more parameters.

In one embodiment, the training system (102) retrieves one or more images (401A, 401B, ... 401N) and class labels (402A, 402B, ... 402N) from the training data set (104). Further, the training system (102) sorts one or more images (401A, 402B, ... 401N) based on the class label (402A, 402B, ... 402N). One or more sorted images (406) are as shown in FIG. 4C. The training system (102) may sort one or more images (401A, 401B, ... 401N) using one of bubble sort, selection sort, merge sort, insertion sort, quick sort, and the like. .. One of skill in the art can understand the use of one or more existing sorting techniques for sorting one or more images (401A, 401B, ... 401N). Further, the training system (102) produces one or more images (401A, 401B, ... 401N) corresponding to each object as an input to the base neural network based on the position of each object in the information (217). offer. For example, one or more images (401A, 401B, ... 401N) corresponding to the object "Labrador" are objects based on the position of each object in the information (217), as shown in FIG. 4B. One or more images (401A, 401B, ... 401N) corresponding to "Shiba Inu" etc. are provided as inputs to the subsequent base neural network.

In step 303, for each input image, the training system (102) has the output of the base neural network and the loss value of the base neural network based on the class labels (402A, 402B, ... 402N) corresponding to each input image. To determine. The output of the base neural network shows the similarity values corresponding to one or more objects.

In one embodiment, the base neural network (501) is a convolutional neural network with one or more layers, as shown in FIG. 5A. One or more layers of the base neural network (501) are a convolutional layer (505A, 505B), a pooling layer (507), a normalized linear unit layer (506A, 506B), a fully connected layer (508) shown in FIG. 5A. , And at least one of the loss layers. Each of the input images (502) is sent as input to the base neural network (501), and the input image (502) propagates through one or more layers of the base neural network (501), as shown in FIG. 5A. The output (503) of the base neural network (501) indicating the class label (402A, 402B, ... 402N) corresponding to the input image (502) is generated. Further, the convolutional neural network is a preprocessing layer (504) of one or more images including at least one of noise reduction operation, clipping operation, morphology operation, resizing operation or scaling operation, normalization, dimension reduction, and the like. May include. One of skill in the art can understand the use of one or more existing image preprocessing operations on the input image (502). Further, the base neural network (501) is one or more kernels of each layer of the base neural network (501), one or more kernels of the first layer for one or more layers of the base neural network (501). And one or more connections between one or more kernels in the second layer after the first layer, multiple values in one or more kernels, and at least one of the kernel activation lists. Associated with one or more parameters including. In addition, the kernel activation list contains at least one of the output of the first layer of the base neural network (501), one or more kernel indexes, and class labels (402A, 402B, ... 402N). include.

In one embodiment, the training system (102) may initialize each plurality of values of one or more kernels to random values having a zero mean and a given variance. The predetermined variance is provided by the user (101) via the user interface (103). In another embodiment, the training system (102) may receive each plurality of values of one or more kernels from the user (101) via the user interface (103). Further, each plurality of values of one or more kernels is represented as a matrix. For example, the first kernel, represented as a matrix, is shown as shown in FIG. 9A (matrix 3).

In one embodiment, the convolutional layer (505A, 505B) comprises one or more kernels, each kernel of one or more kernels having multiple values of each kernel and an input image (502) or an output of the previous layer. By calculating the dot product between one or more values of, it is convolved over the width and height of the input image (502) or the output of the previous layer. Convolution of the input image (502) or the output of the previous layer Convolution of each kernel of the layers (505A, 505B) produces a two-dimensional output (ie, activation map or feature space) corresponding to each kernel. The output of the convolutional layer (505A, 505B) is used to identify one or more kernels that are activated when a feature is detected at some spatial location in the input image (502) or the output of the previous layer. .. For example, as shown in FIG. 5B, the input image (502) corresponding to the grayscale image is represented as a matrix of dimensions 6 × 6 and is the first in one or more kernels (509) of the convolution layer (505A). The kernel of 1 is represented as a matrix of dimensions 3x3, and the output of the convolution layer (505A) (ie, the activation map (510)) is represented as a matrix of dimensions 4x4. The input image (502) corresponding to the color image is represented using three matrices. The three matrices represent the color space. For example, the three matrices may correspond to the "red", "green", "blue" in the color space RGB, and the three matrices are "cyan", "magenta", and "yellow" in the CMYK color space. The three spaces may correspond to the YUV color spaces "Luma", "Chrominance-U", and "Chrominance-V". The three matrices of the input image (502) corresponding to the color image are convolved, the first kernel is duplicated into the three matrices, and the three matrices after convolution are simply as shown in FIG. 5C. Added to generate the output (ie, activation map (510)) of the convolutional layer (505A) represented as a single matrix. Further, as shown in FIG. 5D, the input image (502) is provided as input to the convolution layer (505A) containing one or more kernels (509) and corresponds to each of one or more kernels (509). The output of the convolutional layer (505A) (ie, the activation map (510)) is shown in FIG. 5D.

In one embodiment, the normalized linear unit layer (506A, 506B) is an activation showing a mapping between the input and output (ie, activation map (510)) of the normalized linear unit layer (506A, 508A). Includes functions. The normalized linear unit layer (506A, 506B) includes an activation function such as a normalized linear function, a hyperbolic tangent function, and a sigmoid function. The rectified linear function is expressed using the equation shown below.
Output_relu = maximum (0, input_relu)… (Formula 1)
In the above equation, input_relu indicates the input to the normalized linear unit layer (506A, 506B) and output_relu indicates the output (ie, activation map (510)) of the normalized linear unit layer (506A, 506B). The output of the normalized linear unit layer (506A) corresponding to the output of the input image (502) or the previous layer (511) with the normalized linear function as the activation function (ie, the activation map (510)) is shown in the figure. Shown in 5E.

In one embodiment, the pooling layer (507) implements a non-linear function to reduce the spatial size of the output of the previous layer (511). For example, the pooling layer of the base neural network (501) may use one of maximum pooling, average pooling, region of interest pooling, and the like. The maximum pooling technology pooling layer (507) with a 2x2 kernel size divides the output of the previous layer (511) into one or more blocks of 2x2 with a stride of 2 and one or more. Identify the maximum value for each of the blocks. As shown in FIG. 5F, the maximum values of each of the one or more blocks are concatenated to form a pooling layer (output of 507 (ie, activation map (510))).

In one embodiment, as shown in FIG. 5G, the fully connected layer (508) comprises one or more neurons (512) and each output of the previous layer (511) is one or more neurons (512). It is provided as an input for each neuron in. One or more neurons (512) in the fully connected layer (508) is equal to the number of unique class labels (402A, 402B, ... 402N) in the training data set (104). In addition, one or more neurons (512) in the fully connected layer (508) include an activation function that indicates a mapping between the inputs and outputs of the fully connected layer (508). For example, the activation function may include one of a sigmoid function, a softmax function, a Gaussian function, and the like. For example, the softmax function is expressed using the equation shown in Equation 2 in FIG. 9B.

In Equation 2 of FIG. 9B, K represents the number Z _i of neurons (512) in the fully connected layer (508) and Z _j represents the output of one or more neurons (512). The output of the fully connected layer (508) is the output (503) of the base neural network (501). The output (503) of a base neural network (501) with a softmax function is a vector with one or more values in the range zero to one. One or more values in the vector indicate similarity values corresponding to one or more objects in the training database (104). For example, a vector having one or more values and one or more corresponding objects is as shown in Equation 3 of FIG. 9B.

The output (503) of the base neural network (501) is "as an object present in the input image (502) provided to the base neural network (501)" in one or more objects of the training data set (104). The object "Shiba dog" having the highest similarity value of "0.85" is shown.

In one embodiment, for each input image (502), the loss value of the base neural network (501) is the class label (402A,, corresponding to the output (503) of the base neural network (501) and each input image (502). It is determined based on 402B, ... 402N). The loss layer of the base neural network (501) is used to determine the loss value. The loss value indicates the predictive ability of the base neural network (501) to produce an output (503) equal to the class label (402A, 402B, ... 402N) corresponding to each input image (502). The training system (102) determines the loss value based on the hierarchical cross entropy technique. One of skill in the art can understand the use of other loss calculation techniques such as softmax loss, Euclidean loss. The loss value of the base neural network (501) based on the hierarchical cross entropy technique is determined using the equation shown in Equation 4 of FIG. 9C.

In the above equation, M is the number of objects in the information (217) and N is the number of unique class labels (402A, 402B, ... 402N) in the training data set (104) or the base neural network. The number of neurons (512) in the fully connected layer (508) of (501) is shown.

In one embodiment, as shown in FIG. 5H, the training system (102) is with one or more kernels (509) of the first layer (513) for one or more layers of the base neural network 501. Initialize one or more connections (515A, 515B, ... 515N) between one or more kernels (509) in the second layer (514) after the first layer (513). good. In another embodiment, the training system (102) has one or more connections (515A, 515B, ...) Based on a second user input received from the user (101) via the user interface (103). 515N) may be initialized.

In one embodiment, the training system (102) has at least one of a second user input received from the user (101) via the user interface (103) and an output (503) of the base neural network (501). You may initialize the kernel activation list based on one. The output of the base neural network (501) (ie, the activation map (510)), the index (517) of one or more kernels (509), and the input image corresponding to the object "Labrador" in the base neural network (501). A kernel activation list (516) containing at least one of the class labels (402A, 402B, ... 402N) initialized when providing (502) is as shown in FIG. 5I. The index (517) of one or more kernels (509) in one or more layers of the base neural network (501) has greater similarity compared to one or more kernels corresponding to the input image (502). It is shown to generate an activation map (510) with. In one embodiment, the index (517) of one or more kernels (509) is based on the output of one or more layers of the base neural network (501) (ie, the activation map (510)). It may be selected by the user (101) via 103).

In step 304, the training system (102) is based on at least one of an output (503), a loss value of the base neural network (501), and a second user input to generate the neural network, respectively. Update one or more parameters for the input image (502). Neural networks are used for object recognition.

In one embodiment, updating one or more parameters means adding one or more kernels (509) in at least one layer in one or more layers of the base neural network (501). Modifying one or more of the values in one or more kernels (509) in one or more layers of the base neural network (501) and one or more of the base neural network (501). One or more connections (515A, 515A,) between one or more kernels (509) in the first layer for the layer and one or more kernels (509) in the second layer after the first layer. Includes at least one of modifying 515B, ... 515N) and modifying the kernel activation list (516). To provide each of the one or more objects in the information (217) with one or more images (401A, 401B, ... 401N) as inputs to the base neural network (501), and the base neural. A method step involving updating one or more parameters of the network (501) is shown as training of the base neural network (501).

In one embodiment, the training system (102) is one or more kernels (509) of one or more layers of the base neural network (501) based on the output (503) and the loss values of the base neural network (501). One or more parameters may be updated, including modifying one or more values among multiple values of. The training system (102) uses backpropagation techniques to update one or more values among multiple values in one or more kernels (509) in one or more layers. For example, consider an input image (502) in which the dimensions H × W and the respective dimensions of one or more kernels (509) are k1 × k2. The change of one or more values of each of the one or more kernels (509) is determined using the equation shown in Equation 5 of FIG. 9C.

In the above equation, E represents the loss calculation technique used to determine the loss value, and the term with respect to x is one or more values of the output of the input image (502) or the previous layer (511). Shown, the term relating to W indicates one or more values of one or more kernels (509) of one or more layers of the base neural network (501), and the left side is one or more of the base neural network (501). Indicates a change in the value added to one or more values in one or more kernels (509) in the layer of. One or more of the plurality of values of one or more kernel (509) values before modification is shown in FIG. 6A and one or more after modification by the training system (102) based on backpropagation. One or more of the plurality of kernel (509) values is shown in FIG. 6B.

In one embodiment, the training system (102) comprises information (217), one or more input images (401A, 401B, ..., 401N) stored in the training data set (104), and a class label (402A). , 402B, ... 420N), one or more test images stored in the test data set (105), the base neural network (501), the loss value, and one or more associated with the base neural network (501). Display at least one of the parameters and so on. The training system (102) of the base neural network (501) is based on a second user input received from the user (101) via the user interface (103) and a loss value of the base neural network (501). One or more parameters may be updated, including adding one or more kernels (509) in at least one layer in one or more layers. User (101) may add one or more kernels (509) after selecting a new object from one or more objects in information (217) to train the base neural network (501). .. In addition, the user (101) adds one or more kernels (509) based on the output of one or more layers (ie, the activation map (510)) and the loss value of the base neural network (501). It's okay. For example, in the case of a base neural network (501) containing three layers and seven kernels shown in FIG. 6C, the user (101) has the output of one or more layers (ie, the dotted line shown in FIG. 6D). One or more kernels (509) may be added based on the activation map (510)).

In one embodiment, the user (101) is one of the base neural networks (501) based on the output of one or more layers (ie, the activation map (510)) and the loss value of the base neural network (501). One or more connections between one or more kernels (509) in the first layer for one or more layers and one or more kernels (509) in the second layer after the first layer. (515A, 515B, ... 515N) may be modified. For example, in the case of a base neural network (501) containing three layers and eight kernels shown in FIG. 6D, the user (101) has one or more kernels (509) shown in dotted lines, as shown in FIG. 6E. ) One or more connections (515A, 515B, ... 515N) may be modified. Modification of one or more connections (515A, 515B, ... 515N) involves adding one or more connections (515A, 515B, ... 515N) and one or more of the base neural networks (501). One or more connections between one or more kernels (509) in the first layer for the first layer and one or more kernels (509) in the second layer after the first layer. 515A, 515B, ... 515N) includes at least one of the deletions.

In one embodiment, the user (101) has kernel activity based on the output of one or more layers of the base neural network (501) (ie, the activation map (510)) and the loss value of the base neural network (501). The conversion list (516) may be modified. When the user (101) adds a kernel list and selects a new object from one or more objects of information (217) to train the base neural network (501), the output (ie, activity) of the first layer. The kernel map (510)) and class labels (402A, 402B, ... 402N) may be output. For example, after training the base neural network (501) with one or more images (401A, 401B, ... 401N) of the object "Labrador", the kernel activation list (516) is as shown in FIG. 5I. Yes, when the user (101) provides one or more images (401A, 401B, ... 401N) of the object "pitbull" to train the base neural network (501), the kernel activation list (516) is displayed. , May be updated as shown in FIG. 6F. Further, when the user (101) provides one or more images (401A, 401B, ... 401N) of the object "Beagle" to train the base neural network (501), the kernel activation list (516) is displayed. , May be updated as shown in FIG. 6G. In one embodiment, the user (101) has at least one of the outputs of one or more layers (ie, the activation map (510)) and the loss value of the base neural network (501), as shown in FIG. 6H. One or more, as shown in FIG. 6G, by adding one or more kernel (509) indexes (517) to one or more objects in the kernel activation list (516) based on one. The index (517) of the kernel (509) of the above may be modified. Modification of one or more kernel (509) indexes (517) adds and removes one or more kernel (509) indexes (517) for one or more objects in the kernel activation list (516). Includes at least one of.

In one embodiment, the user (101) has the loss value of the base neural network (501), the output of the first layer of the base neural network (501) (ie, the activation map (510)), and the training data set (ie). Based on at least one of the additions of one or more images (401A, 401B, ... 401N) to 104), a hierarchical tree via the user interface (103) using the first user input. Modify the structure. Modifications include adding one or more objects to the hierarchical tree structure and modifying the position of one or more objects in the hierarchical tree structure. For example, when one or more images (401A, 401B, ... 401N) of a new object in the training data set (104) are identified, the user (101) is informed (217) as shown in FIG. 6I. You may add new objects (ie, "Congolion") and further train the base neural network (501).

In one embodiment, the training system (102) provides one or more images (401A, 401B, ... 401N) corresponding to each object as inputs to the base neural network (501), with loss values and predetermined. Update one or more parameters of the base neural network (501) based on the result of comparison with the threshold. For example, the predetermined threshold may be 0.01. Training of the base neural network (501) for each object in one or more objects is performed until the loss value falls below a predetermined threshold. After training of the base neural network (501) is complete, the training system (102) uses one or more parameters associated with the base neural network (501) to generate the neural network. For example, the generated neural network is a trained base neural network (501). In another example, the kernel activation list (516) obtained after training the base neural network (501) for the generated neural network and one or more objects of class "dog" is shown in FIG. 6J. As you can see. In addition, the generated neural network (601) is used for object recognition. Object recognition is performed for one or more images (401A, 401B, ... 401N) of the test data set (105). One or more test images in the test data set (105) do not include class labels (402A, 402B, ... 402N) for one or more test images, as shown in FIG. 7A.

In one embodiment, in order to perform object recognition using the generated neural network (601), the training system (102) receives the user (101) as an input in the first layer of the neural network (601). ) May provide a test image among one or more test images stored in the test data set (105). In addition, the training system (102) compares the output of the first layer of the neural network (601) (ie, the activation map (510)) with the kernel activation list (516). Further, based on the comparison, the training system (102) identifies one or more kernels (509) from the kernel activation list (516). The training system (102) then propagates the test image through one or more identified kernels (509) of one or more layers of the neural network (601). Finally, the training system (102) recognizes the object of the test image (701) based on the output (503) of the neural network (601). For example, given the test image (701) provided as an input to the first layer of the neural network (601), the output of the first layer of the neural network (601) (ie, the activation map (510)) is. As shown in FIG. 7B. The output of the first layer of the neural network (601) (ie, the activation map (510)) is compared to the kernel activation list (516) associated with the neural network (601) shown in FIG. 6J. Makes the output of the first layer (ie, the activation map (510)) similar to the object "Beagle". The neural network (601) propagates the test image (701) through one or more kernels (509) having indexes (517) "7" and "11". Further, the output (503) of the neural network (601) recognizes the object of the test image (701) as a "beagle".

The method for generating the neural network (601) for object recognition in the image is by the training system (102) one or more images (401A, 401B, ... 401N) of the training data set (104). Includes training the base neural network (501) in to generate the neural network (601). For similar object classes (eg, "dog"), the base neural network (501) is based on each class of similar objects (ie, "labrador", "pitbull", "beagle", etc.) for training purposes. Does not require a large training data set (104). The feature used to recognize similar objects may have one or more kernels (509), so one or more trained kernels (509) have a smaller training dataset (104). Can be used for objects with. Using information (217), a trained base neural network (501) allows for faster training. In addition, the test image (701) is propagated through one or more subsets of the kernel (509) of the neural network (601), so that the neural network (601) performs object recognition in less time and with higher accuracy. ..

Computer System FIG. 8 shows a block diagram of an exemplary computer system (800) for implementing embodiments consistent with the present disclosure. In one embodiment, a computer system (800) may be used to implement the method. The computer system (800) may include a central processing unit (“CPU” or “processor”) (802). The processor (802) may include at least one processor for executing program components for dynamic resource allocation at run time. The processor (802) may include a special processing unit such as an integrated system (bus) controller, a memory (805) management control unit, a floating point unit, a graphics processing unit, a digital signal processing unit, and the like.

The processor (802) may be arranged in communication with one or more I / O devices (not shown) via an input / output (I / O) interface (801). The I / O interface (801) includes audio, analog, digital, monaural, RCA, stereo, IEEE- (1394), serial bus, universal serial bus (USB), infrared, PS / 2, BNC, coaxial, component, Composites, Digital Visual Interface (DVI), High Definition Multimedia Interface (HDMI®), RF Antenna, S-Video, VGA, IEEE (802). n / b / g / n / x, Bluetooth, cellular (eg code division multiple access (CDMA), high speed packet access (HSPA +), global system for mobile communications (GSM), long term evolution (LTE), WiMax, etc.) However, communication protocols / methods that are not limited to this may be used.

Using the I / O interface (801), the computer system (800) may communicate with one or more I / O devices. For example, the input device (810) is an antenna, keyboard, mouse, joystick, (infrared) remote control device, camera, card reader, facsimile, dongle, biometric reader, microphone, touch screen, touch pad, trackball, stylus, scanner. , Storage device, transceiver, video device / source, etc. The output device (811) includes a printer, a facsimile, a video display (for example, a cathode line tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED), a plasma, a plasma display panel (PDP), an organic light emitting diode display (OLED)). Etc.), may be a voice speaker or the like.

In some embodiments, the computer system (800) is connected to the service provider through a communication network (809). The processor (802) may be arranged in communication with the communication network (809) via the network interface (803). The network interface (803) may communicate with the communication network (809). The network interface (803) is a direct connection, Ethernet (eg, twisted pair 10/100/1000 base T), transmission control protocol / Internet protocol (TCP / IP), Token Ring, IEEE802.11a / b / g / n / x. Etc., but not limited to, connection protocols may be used. Communication networks (809) include direct interconnects, electronic commerce networks, peer-to-peer (P2P) networks, local area networks (LANs), wide area networks (WANs), wireless networks (eg, using wireless application protocols), the Internet, Wii. -Fi may be included, but the present invention is not limited to this. The computer system (800) may communicate with one or more service providers using the network interface (803) and the communication network (809).

In some embodiments, the processor (802) may be arranged in communication with memory (805) (eg, RAM, ROM, etc. not shown in FIG. 8 via storage interface (804)). The storage interface (804) is a connection protocol such as Serial Advanced Technology Attachment (SATA), Integrated Drive Electronics (IDE), IEEE-1394, Universal Serial Bus (USB), Fiber Channel, Peripheral Interface for Small Computers (SCSI). May be used to connect to a memory (805) including, but not limited to, a memory drive, a removable disk drive, and the like. The memory drive may further include a drum, a magnetic disk drive, a magnetic optical drive, an optical drive, a RAID, a solid state memory device, a solid state drive, and the like.

The memory (805) may store a collection of programs or database components including, but not limited to, a user interface (806), an operating system (807), a web server (808), and the like. In some embodiments, the computer system (800) may include user / application data such as the data, variables, records described herein. The database may be implemented as a fault tolerant, relational, scalable safety database such as Oracle or Sybase.

The operating system (807) may facilitate resource management and operation of the computer system (800). Examples of operating systems include APPLE® MACINTOSH® OS X®, UNIX®, system distributions such as UNIX (eg, BERKELEY SOFTWARE DISTRIBUTION® (BSD), FREEBSD (registered trademark), NETBSD (registered trademark), OPENBSD, etc.), LINUX (registered trademark) DISTRIBUTIONS (for example, RED HAT (registered trademark), UBUNTU (registered trademark), KUBUNTU (registered trademark), etc.), IBM (registered trademark) ) OS / 2 (registered trademark), MICROSOFT (registered trademark) WINDOWS (registered trademark) (XP (registered trademark), VISTA (registered trademark) 7/8/10, etc.), APPLE (registered trademark) IOS (registered trademark), Includes, but is not limited to, GOOGLE® ANDROID®, BLACKBERRY® OS, and the like.

In some embodiments, the computer system (800) may implement a web browser (not shown) storage program component. Web browsers include MICROSOFT (registered trademark) INTERNET EXPLORER (registered trademark), GOOGLE (registered trademark) CHROME (registered trademark), MOZILLA (registered trademark) FIREFOX (registered trademark), APPLE (registered trademark) SAFARI (registered trademark), etc. It may be a hypertext viewing application. Secure web browsing may be provided using Secure Hypertext Transfer Protocol (HTTPS), Secure Sockets Layer (SSL), Transport Layer Security (TLS), and the like. The web browser (808) can utilize functions such as AJAX, DHCP, ADOBE (registered trademark) FLASH (registered trademark), JAVASCRIPT (registered trademark), JAVA (registered trademark), and application programming interface (API). In some embodiments, the computer system (800) may implement a mail server storage program component. The mail server may be an internet mail server such as Microsoft Exchange. The mail server is an active server page (ASP), ACTIVEX (registered trademark), ANSI (registered trademark) C ++ / C #, MICROSOFT (registered trademark) ,. Functions such as NET, CGI SCRIPTS, JAVA (registered trademark), JAVASCRIPT (registered trademark), PERL (registered trademark), PHP, PYTHON (registered trademark), and WEBOBJECTS (registered trademark) can be used. The mail server may utilize communication protocols such as Internet Message Access Protocol (IMAP), Messaging Application Programming Interface (MAPI), MICROSOFT® Exchange, Post Office Protocol (POP), Simple Mail Transfer Protocol (SMTP). .. In some embodiments, the computer system (800) may implement mail client storage program components. The mail client is a mail viewing application such as APPLE (registered trademark) MAIL, MICROSOFT (registered trademark) ENTOURAGE (registered trademark), MICROSOFT (registered trademark) OUTLOOK (registered trademark), MOZILLA (registered trademark) THUNDERBIRD (registered commercial method). You can do it.

In addition, one or more computer-readable storage media can be utilized in implementing embodiments consistent with the present invention. Computer-readable storage medium refers to any type of physical memory (805) in which information or data readable by a processor (802) may be stored. Accordingly, the computer-readable storage medium stores instructions for execution by one or more processors, including instructions for causing the processor to perform a step or step consistent with the embodiments described herein. good. The term "computer-readable medium" should be understood to include tangible items and exclude carrier and transient signals, i.e., non-transient. Examples are random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, compact discs (CDs) ROMs, digital video discs (DVDs), flash drives, disks, and any. Includes other known physical storage media.

In one embodiment, the computer system (800) may include a remote device (812). The computer system (800) has one or more images (401A, 401B, ... 401N), a first user input, a second user input, and one from the remote device (812) through the communication network (809). You may receive the above test image.

The terms "an embodiment", "embodied", "multiple embodiments", "the embodied", "the embodiments". , "One or more embodied", "some embodied" and "one embodied" are "the present invention (one embodied)" unless otherwise specified. Means one or more (but not all) embodiments (which may be more than one).

The terms "include", "comprising", "having", and variants thereof mean "including, but not limited to," unless otherwise specified. ..

The enumerated list of items does not imply that any or all of the items are mutually exclusive, unless otherwise specified. The terms "is", "one", and "the" mean "one or more" unless otherwise specified.

The description of one embodiment having several components communicating with each other does not imply that all such components are required. Conversely, various optional components are described to illustrate a wide variety of possible embodiments of the invention.

When a single device or product is described herein, one or more devices / products may be used in place of a single device / product (whether or not they cooperate). It can be easily revealed. Similarly, if multiple devices or products are described herein (whether or not they cooperate), a single device / product may be used in place of multiple devices or products, or It can be easily apparent that different numbers of devices / products may be used in place of the indicated number of devices or programs. The functionality and / or characteristics of the device may be performed by one or more other devices that are not explicitly described as having the functionality / characteristics of the alternative. Therefore, other embodiments of the invention need not include the device itself.

The behavior shown in FIG. 3 indicates that certain events occur in a particular order. In alternative embodiments, the particular actions may be performed, modified, or deleted in a different order. In addition, steps may be added to the logic described above and may still be compatible with the embodiments described. In addition, the actions described herein may occur consecutively, or certain actions may be processed in parallel. Moreover, the operation may be performed by a single processing unit or by a distributed processing unit.

Finally, the language used herein has been selected primarily for readability and educational purposes, and the language may not have been selected to elaborate or limit the subject matter of the invention. .. Accordingly, it is intended that the scope of the invention is not limited by the embodiments for carrying out the invention, but rather by any claims arising from an application based herein. Accordingly, the disclosure of embodiments of the invention is intended to be exemplary rather than limiting in the scope of the invention as described in the claims that follow.

Various embodiments and embodiments have been described herein, but other embodiments and embodiments may be apparent to those of skill in the art. The various aspects and embodiments disclosed herein are for illustration purposes only and are not intended to be restrictive, the true scope and spirit of which is set forth by the claims below. ing.

101 User 102 Training system 103 User interface 104 Training data set 105 Test data set 201 I / O interface 202 Memory 203 Processor 204 Data 205 Relationship data 206 Parameter data 207 Loss value data 208 Other data 209 Module 210 Communication module 211 Information correction module 212 Input module 213 Loss determination module 214 Update module 215 Recognition module 216 Other modules 217 Information 401A, 401B, ... 401N One or more images 402A, 402B, ... 402N Class label 403 1st level 404 2nd level 405th Level 3 406 Sorted one or more images 501 Base Neural Network 502 Input Image 503 Base Neural Network Output 504 Pretreatment Layers 505A, 505B Convolution Layers 506A, 506B Normalized Linear Unit Layers 507 Pooling Layers 508 Fully Connected Layers 509 One or more kernels 510 activation map 511 previous layer outputs 512 neurons 513 first layer 514 second layers 515A, 515B, ... 515N one or more connections 516 kernel activation list 517 index 601 neural network 701 test Image 800 Computer system 801 I / O interface 802 Processor 803 Network interface 804 Storage interface 805 Memory 806 User interface 807 Operating system 808 Web server 809 Communication network 810 Input device 811 Output device 812 Remote device

Claims (19)

A method of generating a neural network for object recognition in an image.
Receiving information about hierarchical relationships between one or more objects through a training system,
The training system provides one or more images of a training dataset corresponding to each object of the one or more objects as input to the base neural network based on information about the hierarchical relationship between the one or more objects. To provide such that the base neural network is associated with one or more parameters.
For each input image, the training system determines the loss value of the base neural network based on the output of the base neural network and the class label corresponding to each input image, wherein the output is said. Making decisions that indicate similarity values for one or more objects,
One or more of the input images based on at least one of the output, the loss value of the base neural network, and a second user input for generating the neural network by the training system. To update one or more parameters such that the neural network is used for the object recognition .
The training system allows the recognition of the object in the neural network.
To provide a test image received from the user as an input to the first layer of the neural network.
Comparing the output of the first layer of the neural network with the kernel activation list ,
Identifying one or more kernels from the kernel activation list based on the comparison.
Propagating the test image through the identified kernel of one or more layers of the neural network.
Recognizing the object in the test image based on the output of the neural network
How to include. Receiving information about hierarchical relationships between one or more of the objects
Determining the total number of the one or more objects in the training data set and the one or more images in the training data set corresponding to each object.
One or more nodes and one or more based on at least one of the total number of the one or more images corresponding to each object, the first user input, and the relationship between the one or more objects. By constructing a hierarchical tree structure with edges, such that the one or more nodes indicate the one or more objects and the one or more edges indicate the relationships between the one or more objects. , Building a hierarchical tree structure ,
Receiving from the user the hierarchical tree structure showing information about the hierarchical relationship between the one or more objects regarding the hierarchical relationship between the one or more objects.
The first user input based on at least one of the loss value of the base neural network, the output of the first layer of the base neural network, and the addition of one or more images to the training data set. The method of claim 1, comprising modifying the hierarchical tree structure via a user interface. Providing one or more of the above images
Retrieving the one or more images and the class label from the training dataset
Sorting the one or more images based on the class label,
The first aspect of claim 1, wherein the one or more images corresponding to the respective objects are provided based on the position of the respective objects in the information regarding the hierarchical relationship between the one or more objects . Method. The method of claim 1, wherein determining the loss value is based on a hierarchical cross entropy technique. The method of claim 1, wherein providing the one or more images corresponding to each of the objects is based on the result of comparison between the loss value and a predetermined threshold. The base neural network and the neural network are convolutional neural networks having one or more layers, and the one or more layers are a convolutional layer, a pooling layer, a rectified linear unit layer, a fully connected layer, and a loss layer. The method according to claim 1, which is at least one of. The one or more parameters are one or more kernels of each layer of the base neural network, one or more kernels of the first layer for one or more layers of the base neural network and the first layer. Includes one or more connections between the one or more kernels in the second layer after, multiple values of the one or more kernels, and at least one of the kernel activation lists. The method according to claim 1. 7. The method of claim 7, wherein the kernel activation list comprises the output of the first layer of the base neural network, the index of one or more kernels, and at least one of the class labels. Updating the one or more parameters adds one or more kernels in at least one layer of the one or more layers of the base neural network and the one or more of the base neural network. Modifying one or more of the values of one or more kernels in one of the layers and said one or more of the first layer for the one or more layers of the base neural network. At least one of modifying one or more connections between the kernel and the one or more kernels in the second layer after the first layer and modifying the kernel activation list. The method of claim 8, comprising: The method of claim 2, wherein the first user input and the second user input are received via a user interface. A training system for generating neural networks for object recognition in images.
With the processor
With a memory coupled to the processor by communication
When the memory is executed, the processor
Receive information about hierarchical relationships between one or more objects,
Based on the information about the hierarchical relationship between the one or more objects, the input to the base neural network is to provide one or more images of the training data set corresponding to each object of the one or more objects, in which case the base. Neural networks associate with one or more parameters,
For each input image, the loss of the base neural network is determined based on the output of the base neural network and the class label corresponding to each input image, in which case the output corresponds to the one or more objects of similarity. Show value,
The one or more parameters are updated for each of the input images based on at least one of the output, the loss value of the base neural network, and the second user input for generating the neural network. In that case, the neural network is used for the object recognition,
Stores processor instructions ,
The processor
It is configured to receive information about hierarchical relationships between one or more of the objects .
Determining the total number of the one or more objects in the training data set and the one or more images in the training data set corresponding to each object.
One or more nodes and one or more based on at least one of the total number of the one or more images corresponding to each object, the first user input, and the relationship between the one or more objects. By constructing a hierarchical tree structure having the edges of, such that the one or more nodes indicate the one or more objects and the one or more edges indicate the relationships between the one or more objects. Well, doing the above construction and
Receiving from the user the hierarchical tree structure showing information about the hierarchical relationship between the one or more objects regarding the hierarchical relationship between the one or more objects .
The first user input based on at least one of the loss value of the base neural network, the output of the first layer of the base neural network, and the addition of one or more images to the training data set. Including modifying the hierarchical tree structure via the user interface using
The processor is configured to recognize the object in the neural network.
To provide the first layer of the neural network with a test image received from the user.
Comparing the output of the first layer of the neural network with the kernel activation list ,
Identifying one or more kernels from the kernel activation list based on the comparison.
Propagating the test image through the identified kernel of one or more layers of the neural network.
Recognizing the object in the test image based on the output of the neural network
Training system , including . The processor is configured to provide the one or more images.
Retrieving the one or more images and the class label from the training data set.
Sorting the one or more images based on the class label,
11. The training according to claim 11 , comprising providing the one or more images corresponding to each object based on the position of each object in the information about the hierarchical relationship between the one or more objects . system. 11. The training system of claim 11 , wherein the processor is configured to determine the loss value based on hierarchical cross entropy technology. 11. The training system of claim 11 , wherein the processor is configured to provide one or more images corresponding to each of the objects based on the result of comparison of the loss value with a predetermined threshold. The processor is configured to generate the neural network as a convolutional neural network having one or more layers, the one or more layers being a convolutional layer, a pooling layer, a normalized linear unit layer, a fully connected layer, and the like. And the training system according to claim 11 , which is at least one of the loss layers. The one or more parameters are one or more kernels of each layer of the base neural network, one or more kernels of the first layer for one or more layers of the base neural network, and the first. Includes one or more connections between the one or more kernels in the second layer after the layer, multiple values of the one or more kernels, and at least one of the kernel activation lists. The training system according to claim 11 . 16. The training system of claim 16 , wherein the kernel activation list comprises the output of the first layer of the base neural network, the index of one or more kernels, and at least one of the class labels. .. The processor is configured to update the one or more parameters, adding one or more kernels in at least one layer of the one or more layers of the base neural network, and the base neural. Modifying one or more of the values of one or more kernels of the one or more layers of the network and the first layer for the one or more layers of the base neural network. Modifying one or more connections between the one or more kernels of the first layer and the one or more kernels of the second layer after the first layer, and modifying the kernel activation list. The training system according to claim 11 , comprising at least one of the above. 11. The training system of claim 11 , wherein the processor is configured to use a user interface to receive the first user input and the second user input.
The processor is configured to recognize the object in the neural network.
To provide the first layer of the neural network with a test image received from the user.
Comparing the output of the first layer of the neural network with the kernel activation list,
Identifying one or more kernels from the kernel activation list based on the comparison.
Propagating the test image through the identified kernel of one or more layers of the neural network.
11. The training system of claim 11 , comprising recognizing the object of the test image based on the output of the neural network.

Images