TWI643137B - Object recognition method and object recognition system - Google Patents

Abstract

The invention provides an object identification method and an object identification system. The method includes: acquiring multimedia data; inputting the multimedia data to the deep learning model to identify the object in the multimedia data; and outputting output information corresponding to the object according to the identified object in the multimedia data.

Description

Object identification method and object identification system

The invention relates to an object identification method and an object identification system for object recognition using a deep learning model.

In general, the augmented reality can be divided into two main phases: the stage of object recognition and the stage of superimposing the corresponding augmented reality content on the image and displaying it according to the result of the recognition. However, the ability to identify objects greatly affects the experience of augmented reality.

In particular, the stage of object identification can be divided into two phases: the feature extraction phase and the classification phase. Figure 1 is a schematic flow chart of object recognition. Referring to FIG. 1 , the general object identification method is to input the image 10 into the object recognition module 100 . When the image 10 is input into the object recognition module 100, the feature extraction phase is first performed via step S101. In step S101, the object recognition module 100 can perform feature extraction on the image 10, and generate a feature vector in step S102. Each of the feature vectors is used to represent a certain image 10 feature. Then, in step S103, the object recognition module 100 can input the feature vector captured in step S102 to a classifier, and the classifier classifies the feature vector according to the feature vector, thereby identifying the target object in the image 10. In particular, the feature vector obtained through the feature capture stage usually determines the quality of the object recognition result.

2A is a schematic diagram of conventional object recognition. Referring to FIG. 2A, in the conventional augmented reality, the feature extraction phase of object recognition (ie, step S101 in FIG. 1) generally uses an algorithm associated with a color turning point (Corner). The algorithm uses the concept of differentiation to find points with large color changes in the image, and uses the color turning points to generate feature vectors belonging to the image (such as the color turning point 20 shown in FIG. 2A). Finally, according to the feature vector, classification and prediction of the size and position of the object are performed.

However, the object recognition method using the color turning point algorithm has the following disadvantages: the same object photographed at different angles cannot be recognized. For example, FIG. 2B is a schematic diagram of an image taken by the same object through different angles. Referring to FIG. 2B, since the image 22, the image 24, and the image 26 are the same object photographed through different angles respectively, the distribution of the color turning points may also change according to the shooting angle, so that it may not be classified as the same object. Therefore, in the current augmented reality of image recognition, it is usually only possible to recognize planar marks (eg, two-dimensional codes, simple pictures); and it is difficult to effectively identify complex three-dimensional objects.

In addition, the object recognition method using the color turning point algorithm has the following disadvantages: different objects cannot be identified, but belong to the same type of object. For example, the volcanoes of the A and the volcanoes of the B are slightly different in shape, but they cannot be classified as volcanoes, and superimposed and displayed the relevant augmented reality content.

Furthermore, the object recognition method using the color turning point algorithm has the following disadvantages: when the target to be recognized increases, the probability of misidentification increases. In detail, when the recognition target is increased, the probability of representing the distribution of the color turning points is increased. If only the color turning point is taken as the feature vector, then the target A is easily mistaken for the B target, and finally the wrong augmented reality content is placed.

Finally, the object recognition method using the color turning point algorithm has the following disadvantages: the speed is greatly reduced when the high resolution picture is recognized. In detail, due to the high resolution of the picture, the number of pixels to be calculated is increased, and the number of color turning points generated is also large, resulting in an increase in the time for amplifying the real target and the solid image.

The invention provides an object identification method and an object identification system for object recognition using a deep learning model, which can improve the accuracy and elasticity of the identification object in the multimedia data, and can also be applied in the technology of augmented reality and provides better. User experience.

The invention provides an object identification method. The method includes: acquiring multimedia data; inputting the multimedia data to the deep learning model to identify the object in the multimedia data; and outputting output information corresponding to the object according to the identified object in the multimedia data.

In an embodiment of the invention, wherein the deep learning model comprises a convolutional layer-like neural network.

In an embodiment of the invention, the convolutional layer-like neural network includes at least one convolution layer and at least one pooling layer, and the object identification method further includes: capturing, by the convolution layer and the pooling layer The eigenvalue of the multimedia material.

In an embodiment of the invention, wherein the deep learning model further comprises a fully connected layer or a machine learning algorithm, the step of outputting the output information corresponding to the object comprises: according to the full connection layer or the machine learning algorithm according to the feature The value classifies the object and obtains the object information corresponding to the object.

In an embodiment of the invention, the object information includes a center point of the object in the multimedia material, a length of the bounding box for circled the object, a width of the bounding box, and a rotation angle of the object in the multimedia material.

In an embodiment of the invention, the item information includes coordinates for enclosing a plurality of vertices of the bounding box of the object.

In an embodiment of the invention, the item information includes the type of the item.

In an embodiment of the present invention, the deep learning model includes a plurality of layers, and the object identification method further includes: inputting a plurality of multimedia materials to be trained and solutions corresponding to the multimedia materials to be trained to the deep learning model; The plurality of weights of each layer in the deep learning model are adjusted according to the multimedia materials and solutions to be trained to train the deep learning model.

In an embodiment of the invention, the deep learning model includes a loss layer in the training phase, and the steps of training the deep learning model by adjusting the weight of each layer in the deep learning model according to the multimedia data and the solution to be trained include: And outputting, by the deep learning model, a plurality of output solutions respectively corresponding to the multimedia material to be trained according to the multimedia data to be trained; and outputting the solution and the solution corresponding to the multimedia material to be trained by the penalty layer, and adjusting each layer in the deep learning model according to the penalty function the weight of.

In an embodiment of the present invention, before the step of acquiring a multimedia material, the object identification method further comprises: obtaining current geographic information; determining whether the current geographic information meets specific geographic information; and when the current geographic information meets the specific geographic information. When performing the steps of obtaining multimedia materials.

In an embodiment of the present invention, in the step of outputting output information corresponding to the object, the method comprises: acquiring a superimposed object corresponding to the object according to the object; superimposing the superimposed object on the multimedia material; and outputting the multimedia of the superimposed superimposed object data.

In an embodiment of the present invention, in the step of superimposing the superimposed object on the multimedia material, the method comprises: rotating the superimposed object according to the rotation angle of the object in the multimedia material, and superimposing the superimposed object on the multimedia material, wherein the object is in the multimedia The angle of rotation in the data is identified by the deep learning model.

In an embodiment of the present invention, in the step of superimposing the superimposed object on the multimedia material, the method comprises: superimposing the superimposed object on a position of a central point in the multimedia material according to a central point of the object in the multimedia material, wherein the object The central point in the multimedia material is identified by the deep learning model.

In an embodiment of the invention, in the step of superimposing the superimposed object on the multimedia material, the method comprises: scaling the superimposed object according to the length of the bounding box for circled the object, and width of the bounding box and superimposing the superimposed object To the multimedia material, the length of the bounding box of the object and the width of the bounding box are recognized by the deep learning model.

In an embodiment of the invention, the multimedia material includes at least one of an image, a point cloud, a voxel, and a mesh.

In the present invention, an object recognition system is provided, the system comprising an input device, a processor and an output device. The input device is used to obtain multimedia materials. The processor is configured to input the multimedia material into the deep learning model to identify the objects in the multimedia material. The output device is configured to output output information corresponding to the object according to the identified object in the multimedia material.

In an embodiment of the invention, wherein the deep learning model comprises a convolutional layer-like neural network.

In an embodiment of the invention, the convolutional layer-like neural network includes at least one convolution layer and at least one pooling layer, and the processor is further configured to extract the feature values of the multimedia material by using the convolution layer and the pooling layer.

In an embodiment of the invention, the deep learning model includes a fully connected layer or a machine learning algorithm, wherein in the operation of outputting the output information corresponding to the object, the processor is further used by the fully connected layer or the machine The learning algorithm classifies the object according to the feature value and obtains the object information corresponding to the object.

In an embodiment of the invention, the object information includes a center point of the object in the multimedia material, a length of the bounding box for circled the object, a width of the bounding box, and a rotation angle of the object in the multimedia material.

In an embodiment of the invention, the item information includes coordinates for enclosing a plurality of vertices of the bounding box of the object.

In an embodiment of the invention, the item information includes the type of the item.

In an embodiment of the present invention, the deep learning model includes a plurality of layers, and the input device is further configured to input a plurality of to-be-trained multimedia materials and solutions corresponding to the multimedia materials to be trained to the deep learning model, and the processor is further configured to The multimedia materials and solutions to be trained adjust multiple weights of each layer in the deep learning model to train the deep learning model.

In an embodiment of the present invention, wherein the deep learning model includes a penalty layer, the processor is further used in the operation of training the deep learning model by adjusting the weight of each layer in the deep learning model according to the multimedia data and the solution to be trained. The deep learning model outputs a plurality of output solutions respectively corresponding to the multimedia data to be trained according to the multimedia data to be trained, and the processor is further configured to compare the output solution and the solution corresponding to the multimedia material to be trained by the penalty layer and adjust the deep learning model according to the penalty function. The weight of each layer in the middle.

In an embodiment of the present invention, before the operation of the multimedia material is obtained, the processor is further configured to obtain the current geographic information, and the processor is further configured to determine whether the current geographic information conforms to the specific geographic information, and when the current geographic information meets the specific geographic information. In the case of information, the operation of obtaining multimedia materials is performed.

In an embodiment of the present invention, the object recognition system further includes a storage device, wherein in the operation of outputting the output information corresponding to the object, the processor is further configured to: obtain the superimposed object corresponding to the object from the storage device according to the object, and process The device is further configured to superimpose the superimposed object on the multimedia material, and the output device is further configured to output the multimedia material of the superimposed superimposed object.

In an embodiment of the present invention, in the operation of superimposing the superimposed object on the multimedia material, the processor is further configured to rotate the superimposed object and superimpose the superimposed object to the multimedia material according to the rotation angle of the object in the multimedia material, wherein The angle of rotation of the object in the multimedia material is identified by the deep learning model.

In an embodiment of the present invention, in the operation of superimposing the superimposed object on the multimedia material, the processor is further configured to superimpose the superimposed object to the position of the central point in the multimedia material according to the central point of the object in the multimedia material. The center point of the object in the multimedia material is identified by the deep learning model.

In an embodiment of the present invention, in the operation of superimposing the superimposed object on the multimedia material, the processor is further configured to scale the superimposed object according to the length of the bounding box for circled the object and the width of the bounding box and The superimposed object is superimposed on the multimedia material, wherein the length of the bounding box of the object and the width of the bounding box are recognized by the deep learning model.

In an embodiment of the invention, the multimedia material includes at least one of an image, a point cloud, a voxel, and a mesh.

Based on the above, the object identification method and the object identification system of the present invention can improve the accuracy of the identification object in the multimedia material, and can also be applied in the technology of augmented reality and provide a better user experience.

The above described features and advantages of the invention will be apparent from the following description.

FIG. 3 is a schematic diagram of an object identification system according to an embodiment of the invention. Referring to FIG. 3, the object recognition system 300 can include a subsystem 302 and a subsystem 304. Subsystem 302 includes input device 30, output device 32, processor 34a, and storage device 36. Subsystem 304 includes a processor 34b.

The input device 30 is, for example, a charge coupled device (CCD) lens, a complementary metal oxide semiconductor transistor (CMOS) lens, or a depth camera (Depth Camera, Time-Of-Flight Camera). Stereo Camera.

The output device 32 is, for example, a display device that provides a display function such as a liquid crystal display (LCD), a light-emitting diode (LED), or a field emission display (FED).

The processor 34a and the processor 34b may be a central processing unit (CPU), or other programmable general purpose or special purpose microprocessor (Microprocessor), digital signal processor (Digital Signal Processor, DSP). ), a programmable controller, an Application Specific Integrated Circuit (ASIC) or other similar component or a combination of the above.

The storage device 36 can be any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory or the like. Or a combination of the above elements.

In the present exemplary embodiment, the input device 30, the output device 32, and the storage device 36 can be coupled to the processor 34a by wire or wirelessly, respectively. In the present exemplary embodiment, subsystem 302 is, for example, a handheld electronic device, and subsystem 304 is, for example, a server. Subsystem 302 can be coupled to subsystem 304 in a wired or wireless manner, and subsystem 302 can pass portions of the operations to processor 34b of subsystem 304 for implementation of cloud computing functions. It should be noted that the present invention is not intended to limit the actual configuration architecture of the subsystems in the object recognition system 300. In an embodiment, storage device 36 may also be external to subsystem 302.

It should be noted that the present invention uses a deep learning model to identify objects in a multimedia material. The multimedia material is, for example, at least one of a two-dimensional image, a three-dimensional point cloud, a voxel, and a mesh. In the present exemplary embodiment, the multimedia material is a two-dimensional image.

In the present exemplary embodiment, the deep learning model is implemented by a Convolution Neural Network (CNN). 4 is a schematic diagram of a convolutional layer-like neural network according to an embodiment of the invention. Referring to FIG. 4, in the present exemplary embodiment, the convolutional layer-like neural network 400 is composed of at least one Convolution Layer 401, at least one Pooling Layer 402, and at least one fully connected layer. (Fully connected layer) 403. The convolutional layer-like neural network 400 is generally composed of a convolutional layer 401 and a pooling layer 402. It is generally used as a feature extraction of the image to obtain the feature value of the input multimedia material 40. This feature value can be a multi-dimensional array and is generally considered to be a feature vector representing the multimedia material 40. It should be noted, however, that in another embodiment, the convolution layer 401 and the pooling layer 402 may also be combined in a series connection in parallel. The present invention is not used to limit the combination or arrangement of the convolution layer 401 and the pooling layer 402. the way.

In the latter stage of the convolutional layer-like neural network 400, a fully-connected layer 403 is used. The fully-connected layer 403 classifies the objects in the multimedia material 40 according to the feature values generated by the convolutional layer 401 and the pooling layer 402. And the object information corresponding to the identified object can be obtained.

In this exemplary embodiment, the object information includes a bounding box for circled the identified object, and the object information further includes a center point of the identified object in the multimedia material 40, and a length of the bounding box. And the width and the angle of rotation of the object in the multimedia material 40. In another exemplary embodiment, the object information further includes a coordinate box whose coordinates are a plurality of vertices in the multimedia material. In another exemplary embodiment, the item information further includes the type of the identified object. In particular, the classification function of the fully connected layer 403 can also be replaced by a conventional machine learning algorithm. However, if you want to get the above object information (for example, center coordinates, length and width, rotation angle), etc., you still need to rely on the neural network of the fully connected layer. In addition, the above conventional machine learning methods are, for example, Support Vector Machine (SVM), Joint Bayesian , Regression Analysis, and the like. It should be noted here that in the above conventional algorithms, the conventional algorithms are generally more efficient in classifying objects than the fully connected layer 403. Therefore, if the result is more accurate, the coordinates, size, and rotation angle of the object can be obtained by using the full connection layer, and then the input of the fully connected layer 403 is input into the traditional algorithm for classification.

It should be noted that the convolution layer 401 is composed of a plurality of sets of filters (also referred to as filters). For each set of filters, it is composed of a plurality of sliding windows.

Specifically, FIG. 5 is a schematic diagram of a filter and a sliding window thereof according to an embodiment of the invention. Referring to FIG. 5, in the exemplary embodiment, the multimedia material 501 is first disassembled into a red channel image 501_R, a green channel image 501_G, and a blue channel image 501_B. In the case of the filter 503, the filter 503 includes a sliding window 503_R for convolution operation with the image 501_R, a sliding window 503_G for convolution calculation with the image 501_G, and for performing with the image 501_B. The sliding window 503_B of the convolution calculation. The sliding window 503_R, the sliding window 503_G, and the sliding window 503_B respectively contain a plurality of weights. When the multimedia material 501 is recognized, the weights in the sliding window 503_R, the sliding window 503_G, and the sliding window 503_B are respectively applied to the image 501_R, the image 501_G, and the image 501_B.

6A-6E are schematic diagrams showing the operation mode of a sliding window according to an embodiment of the invention. Referring to FIG. 6A to FIG. 6E, in FIG. 6A, starting from a certain corner of the image 501_R (in the present exemplary embodiment, starting from the upper left corner), the value of each weight in the sliding window 503_R is multiplied by the image. The corresponding pixel (Pixel) value in the block 6001 in 501_R, and finally the product result of each weight and the pixel value is added to obtain an output value R _out .

Similarly, in FIG. 6B, starting from a certain corner of the image 501_G (in the present exemplary embodiment, starting from the upper left corner), the value of each weight in the sliding window 503_G is multiplied by the block 6002 in the image 501_G. The corresponding pixel (Pixel) value is added, and finally the product of each weight and the pixel value is added to obtain an output value G _out .

Similarly, in FIG. 6C, starting from a certain corner of the image 501_B (in the present exemplary embodiment, starting from the upper left corner), the value of each weight in the sliding window 503_B is multiplied by the block 6003 in the image 501_B. The corresponding pixel (Pixel) value, and finally the product of each weight and the pixel value is added to obtain an output value B _out .

After calculating the output value R _out , the output value G _{out ,} and the output value B _out , as shown in FIG. 6D, the output value R _out , the output value G _{out ,} and the output value B _out and the bias value of the filter 503 may be used. (Bias) is added, and the summed sum is input to a function (usually a Sigmoid or ReLU function, represented by a function f in Fig. 6D), and finally output to a corresponding position in the two-dimensional array 600. Then, the sliding window 503_R, the sliding window 503_G, and the sliding window 503_B respectively shift a specific pixel (for example, one pixel) to the right in the image 501_R, the image 501_G, and the image 501_B, and repeat the foregoing operation, and connect the output to the previous output. Right, as shown in Figure 6E.

After the horizontal direction is completed (ie, the sliding window 503_R, the sliding window 503_G, and the sliding window 503_B respectively scan a column of the image 501_R, the image 501_G, and the image 501_B), the sliding window 503_R, the sliding window 503_G, and the sliding window 503_B move downward to the specific The pixels are repeated from left to right, and their outputs are connected below the output of the previous column in the two-dimensional array 600. Finally, the sliding image 503_R, the sliding window 503_G, and the sliding window 503_B sweep the entire image 501_R, the image 501_G, and the image 501_B, respectively. When the filter 503 is activated by the image 501_R, the image 501_G, and the image 501_B, the result is output as a two-dimensional array 600. Similarly, other filters may also output a two-dimensional array 601 and a two-dimensional array 602. Finally, these two-dimensional arrays can be cascaded and sent to the next layer of neural networks, such as the pooling layer.

The output of the convolutional layer is typically a three-dimensional array in which many element values are close to zero. Therefore, in order to reduce the amount of calculation, the pooling layer is usually connected after the convolutional layer. It acts like a convolutional layer; however, the sliding window of the pooled layer acts to average or maximize the data it contains. This effect is similar to the reconciliation of local features in the data. It is usually sent to the next layer of convolution. The input array depth of this convolutional layer is no longer as R, G, B as the first layer; it may be a very deep array (the depth is equal to the number of filters of the first convolutional layer). But the principle of action is still the same as before.

It should be noted that the combination of the convolutional layer and the pooled layer may be referred to as a convolutional layer-like neural network, and the convolutional layer-like neural network has various characteristics. For example, a convolutional layer-like neural network can derive the weight of the sliding window in each filter from the training data. Traditional object recognition must be derived from a specific algorithm (for example, the calculation of color turning points) or a filter designed by an expert to find the feature vector of the input image. Convolutional neural networks, by inputting a large number of images during their training phase, and their corresponding solutions (ie, classification results, object boxes), allow the network to automatically find suitable filters. Often the feature extraction algorithms or filters that can be artificially designed are limited, and it is inevitable that there are omissions. But deep learning allows the computer to find a large number of suitable filters by using a large number of convolutional layers, paralleling, and adding a large number of filters to each convolutional layer. Therefore, it not only saves the time for the human design filter, but also finds a large number of filters that humans can't think of, to obtain a feature vector suitable for representing images.

In addition, the convolutional layer-like neural network also has the characteristics of weight sharing. Specifically, in a traditional neural network, each neuron must be fully connected to the previous layer (or input data). This often results in a huge amount of calculations. For example, the input picture is 500*300 pixels, and assuming that the first layer of neural network has 700 neurons; then there are 500*300*700 links in the layer, or 500*300*700 weights. Need to adjust during training. In the end, it usually causes the computer to be unable to afford such a large amount of calculations. Each picture (or input array) in the convolutional layer shares the weight of the same set of sliding windows. This can greatly reduce the amount of computation of the fully connected network.

Convolutional-like neural networks also have pre-perceived characteristics. Specifically, as previously mentioned, since a filter is typically smaller than the entire image (or input array), the filter typically only perceives the local area of the image. This feature effectively supports the translation of the target to be identified in the image. Assuming that a 5*5 filter can effectively find a face, no matter where the face is in the image, the face can be detected as long as the filter is translated there. This method handles the translation of the target more efficiently than a fully connected neural network.

Convolutional-like neural networks also have the characteristics of filters that automatically acquire different levels of abstraction. Specifically, although in the convolutional layer, the weights in the filter are initially random; then, with a large amount of data training, the appropriate weights are finally learned. However, in the rule of thumb, after each layer of convolutional layer (or pooling layer) of deep learning is mapped back to the multimedia data (or image) input, it will be found that the previous layers are usually detected in the local area or Low level of abstraction features (eg, color turning points, boundary lines, outlines); while back layers typically detect features that are broader or higher in abstraction (eg, faces, cars, buildings, etc.). This feature is difficult to implement with manually designed filters.

Referring again to FIG. 4, immediately following the convolutional layer-like neural network 400 is a fully-connected layer, such as a fully-connected neural network or a traditional classifier (eg, SVM). Because the convolutional layer usually only considers the characteristics of the local area, the fully connected layer can comprehensively consider the features of all localities, and classify and predict the bounding box for circled objects.

FIG. 7 is a schematic diagram of a training convolutional layer-like neural network according to an embodiment of the invention. Referring to FIG. 7, after the convolutional layer-like neural network 400 of the deep learning model is planned, a large amount of multimedia data to be trained 700 must be input and the solution 703 to the deep learning model 701 of each multimedia material to be trained 700 is also marked. . The deep learning model 701 includes the aforementioned convolutional layer-like neural network 400 and a penalty layer immediately following it. That is, in the present exemplary embodiment, the deep learning model 701 also includes a penalty layer during the training phase. The method of calculating the error is defined in the penalty layer. The deep learning model 701 can adjust the weight of each layer network according to this error. Developers can define methods for calculating errors as needed. For example, if the object type is successfully predicted, a function can be designed. When the deep learning model predicts the wrong type of object, it will produce a large error (for example, the fourth power of the difference between the predicted result and the solution). If the width and height of the object are successfully predicted to be secondary, then the general calculation error can be used (for example, the square of the difference between the prediction result and the solution).

In the present exemplary embodiment, the penalty layer compares the output 702 of the convolutional layer-like neural network 400 with the multimedia material 700 solution 703 and calculates the error. The deep learning model 701 then adjusts the weights of each layer in its network to train the deep learning model based on the error. When the weight is adjusted to a certain extent, the output 702 of the convolutional layer-like neural network 400 is very close to the input solution 703 of the multimedia material 700 to be trained, which is referred to as learning completion, or the network has converged.

That is to say, in the present exemplary embodiment, the deep learning model 701 adds a penalty layer (also referred to as a loss layer) at the end of the convolutional layer-like neural network 400 during the training phase. The penalty layer compares the output of the convolutional layer-like neural network 400 with the output of the multimedia material to be trained according to the output of the multimedia material to be trained, and calculates an error. Then the convolutional layer-like neural network adjusts the weight of each layer in the network by using this error and by backwards, one by one, from the back to the front. The way in which the error is calculated (ie, the penalty function) is, for example, square difference, Softmax, and the like. In this exemplary embodiment, the penalty layer is only used during the training phase. The penalty layer will be removed when the training phase is completed.

When the study is complete, you can enter the Deploy phase. FIG. 8 is a schematic diagram of a subordinate stage according to an embodiment of the invention. Referring to FIG. 8, as long as the multimedia material 801 is input, the deep learning model 803 (same as the above-described deep learning model 701) can obtain the object information 805 of the detected object. In the present exemplary embodiment, the multimedia material 801 is the Japanese city, and the object information 805 may include the depth learning model 803 determining the probability that the multimedia material 801 is the Japanese city, and obtaining the coordinates of the bounding box as the center point in the multimedia material 801 ( That is, the X value and the Y value) and the length and width of the bounding box for circled the Japanese city and the rotation angle of the Japanese city in the multimedia material 801.

In an exemplary embodiment of the invention, the deep learning model will be applied to the technique of augmented reality. In the example of FIG. 8 above, after the object information 805 is obtained, the corresponding superimposed object (also referred to as augmented reality content) may be taken out from the storage device 36 according to the obtained object information 805. And rotating the superimposed object according to the rotation angle described above, scaling the superimposed object according to the length and width of the above-mentioned bounding box, and placing the superimposed object to the position of the coordinate of the center point.

That is to say, by using the output of the deep learning model (such as the probability of the object identified above, the bounding box), the position, size and rotation angle of the object to be identified can be found in the input multimedia material. In this way, the augmented reality images (eg, pictures, animated models, graphic information) can be scaled according to the size and placed in a relatively suitable position to achieve the augmented reality effect.

Through the above characteristics of the deep learning model, the shortcomings of traditional object recognition can be solved by the following methods: (1) By inputting various angles of the training materials of the target and classifying them into the same category, the traditional augmented reality cannot be solved. Identify the shortcomings of different angles. (2) If the training target is to identify the type of an object, the information of the same type of object is added to the training data (for example, the volcanoes of A and B are thrown into the same category) to achieve the same type of identification. The ability of the object. If you want to distinguish between different objects of the same type (for example, you want to distinguish between the volcanoes of A and B), divide the data of different objects into different categories and input them to the deep learning model training. In this way, it has higher flexibility than the conventional object recognition method. (3) The traditional object recognition can only take out the color turning points of the low abstraction level. By comparing the distribution of these turning points, it is easy to cause misjudgment (for example, when the distribution of color transition points between the mountain and the house is similar, it is easy to misjudge). The deep learning model can refine the feature vectors of low abstract levels into higher abstract features, thus avoiding the situation of misjudgment. (4) Because of the deep learning model, it is not determined by the color turning point whether the two images are similar, so the high-resolution image can be reduced to a low-resolution image, and then judged. Because the deep learning model uses the feature vector of higher abstraction level to judge whether the two images are similar, when the image is reduced, the accuracy of the judgment is not lowered.

It should be noted that the above training and subordinates regarding the deep learning model can be performed by the processor 34a or 34b of the object recognition system 300 of FIG. The object recognition system 300 of FIG. 3 can use a deep learning model to achieve augmented reality techniques.

For example, FIG. 9 is a schematic diagram of a display flow of an augmented reality according to an embodiment of the invention. Referring to FIG. 9, a description will be given with a subsystem 302 (eg, a handheld device) in the object recognition system 300 of FIG. In step S901, the subsystem 302 can capture and retrieve the multimedia material whose content is "bridge" through the input device 30, for example. Next, in step S902, the processor 34a of the subsystem 302 can, for example, input the multimedia material whose content is "bridge" to the deep learning model. Then, in step S903, the processor 34a of the subsystem 302 can obtain the category of the object in the multimedia material as a "bridge" by using the deep learning model, and the center point coordinates and the bounding box of the bounding box of the object. The length, the width of the bounding box, and the angle of rotation of the above "bridge" in the multimedia material. Then, in step S904, the processor 34a of the subsystem 302 can obtain the superimposed object corresponding to the object from the storage device 36, for example, according to the identified category of the object. Finally, in step S905, the processor 34 of the object recognition system 300 may superimpose the superimposed objects according to the center point coordinates of the bounding box, the length of the bounding box, the width of the bounding box, and the rotation angle of the "bridge" in the multimedia material. To the multimedia material, and outputting (or displaying) the multimedia material of the superimposed superimposed object by the output device 32.

It should be noted that the above steps S902 and S903 can be implemented by using cloud computing. For example, the user may, for example, transfer the multimedia material retrieved in step S901 from subsystem 302 to subsystem 304, and processor 34b of subsystem 304 performs the steps by the deep learning model located in subsystem 304. S902, and the corresponding identification result is returned to the subsystem 302 in step S903. Finally, the subsystem 302 performs step S904 and step S905 to complete the display of the augmented reality. Thereby, the deep learning model can be run by using cloud computing, thereby reducing the amount of computation of the device used by the user (ie, subsystem 302).

FIG. 10 is a schematic diagram of a display method of augmented reality according to an embodiment of the invention. The above display method of augmented reality can be explained by FIG. Referring to FIG. 10, in step S1001, the user can use the input device of the subsystem 302 to acquire the multimedia material. In step S1003, the processor 34a of the subsystem 302 can determine whether to perform cloud computing. When the cloud computing is to be performed, in step S1005, the processor 34a of the subsystem 302 can transmit to the subsystem 304 through the communication unit (not shown), and the multimedia data is input to the processor 34b of the subsystem 304. The deep learning model in subsystem 304 identifies the objects in the multimedia material and passes the identification results back to subsystem 302. When there is no cloud computing to be performed, in step S1007, the processor 34a of the subsystem 302 can directly input the multimedia material into the deep learning model located in the subsystem 302 to identify the objects in the multimedia material. After the object in the multimedia material is identified, in step S1009, the processor 34a of the subsystem 302 can execute the augmented reality image processing module to obtain the corresponding superimposed object according to the identified object, and superimpose the superimposed object. Objects are superimposed on the multimedia material. Finally, in step S1011, the processor 34a of the subsystem 302 can output (or display) the multimedia material of the superimposed overlay object through the output device 32.

In an exemplary embodiment of the present invention, the display mode of the augmented reality may also be combined with a deep learning model and geographic information.

FIG. 11 is a schematic diagram of a method for displaying geographic information to an augmented reality according to an embodiment of the invention.

Referring to FIG. 11, in step S1101, the processor 34a of the subsystem 302 can obtain the current geographic location of the subsystem 302 through a positioning device (not shown). Next, in step S1103, the processor 34a of the subsystem 302 determines whether the current geographic location conforms to a particular geographic information. If the current geographic location does not meet the specific geographic information, the process returns to step S1101. If the current geographic location conforms to the specific geographic information, step S1101 of FIG. 10 may be performed, and the subsequent steps in FIG. 10 are continued.

It should be noted that in the prior art, it is only possible to determine whether the user is near a specific geographic coordinate, and depending on the network signal, a very large error may be generated, so that it is impossible to know exactly whether the user's mobile device is true. Reach a specific location and capture multimedia material (ie, image) with the target. In the current popular game "Bao Ke Meng", the player does not even need to open the camera of the mobile phone to play.

However, with the deep learning module of the present invention, the application can determine whether the user has reached the designated location based on the current geographic information, and further determine whether the user uses the input device to obtain the multimedia material (ie, image) with the target object. And determine whether to allow the augmented reality content to interact with the player. In this way, the user's interaction with the real environment can be further enhanced, and the augmented reality content can be brought to life. For example, the application requires users not only to go to Sun Moon Lake, but also to capture the pool of Sun Moon Lake with a mobile phone, in order to further generate augmented reality and interact with it.

In particular, the combination of geographic information and deep learning model object recognition can overcome the effects that can only be achieved by using only geographic information. For example, if the user wants to use the mobile device to scan the product and display the corresponding augmented reality introduction or advertisement in the store, if the geographic information is used, the geographic coordinates of each product in the store are very close and cannot be effectively Achieve the display of augmented reality. Moreover, traditional object recognition does not effectively identify three-dimensional objects, and it is difficult to achieve accurate object recognition goals.

Combined with the deep learning model, it can also be used to determine if an event has occurred and to have the application react specifically to the event.

FIG. 12 is a schematic diagram of a method for displaying geographic information to an augmented reality according to another embodiment of the invention.

Referring to FIG. 12, in step S1201, the processor 34a of the subsystem 302 can obtain the current geographic location of the subsystem 302 through a positioning device (not shown). Next, in step S1203, the processor 34a of the subsystem 302 determines whether the current geographic location conforms to a particular geographic information. If the current geographic location does not meet the specific geographic information, the process returns to step S1201. If the current geographic location meets the specific geographic information, then in step S1205, the user can use the input device of the subsystem 302 to obtain the multimedia material. In step S1207, the processor 34a of the subsystem 302 can input the multimedia material into a deep learning model located in the subsystem 302 to identify the objects in the multimedia material. After identifying the object in the multimedia material, in step S1209, the processor 34a of the subsystem 302 can determine whether to capture a particular object. If the specific object is not captured, the process returns to step S1207. If a specific item is captured, it is determined in step S1211 whether the object is in a state of a special event. If the object is not in a special event state, then in step S1213, the processor 34a of the subsystem 302 may superimpose the general overlay information to display the general augmented reality content. If the object is in the state of a special event, then in step S1215, the processor 34a of the subsystem 302 can superimpose the special overlay information to display the particular augmented reality content.

The above method can be applied to the following situations, for example, if the application reacts to the water body, in addition to the geographical database such as the lake, pool, etc. originally established by the application, the small pool suddenly generated after the rain will also make the application and use. Interaction. For example, when the application captures the image of "Mountain", it can play some kind of augmented reality content; but when the volcano erupts, another special augmented reality content can be played. In this way, you can achieve the ability to surprise and give the application a random response.

FIG. 13 is a flow chart of an object identification method according to an embodiment of the invention. Referring to FIG. 13, in step S1301, the input device 30 is configured to acquire multimedia materials. In step S1303, the processor 34a or the processor 34b inputs the multimedia material to the deep learning model to identify the object in the multimedia material. Finally, in step S1305, the output device 32 outputs output information corresponding to the object according to the identified object in the multimedia material.

In summary, the present invention provides an object identification method and an object identification system for object recognition using a deep learning model, which can improve the accuracy of identifying objects in multimedia data, and can also be applied in augmented reality technology. Provide a better user experience.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

10, 22, 24, 26, 501_R, 501_G, 501_B: image 100: object recognition module step S101: step of feature extraction step S102: step of generating feature vector step S103: step 20 of classifier classifying according to feature vector Color Inflection Point 300: Object Identification System 302, 304: Subsystem 30: Input Device 32: Output Device 34a, 34b: Processor 36: Storage Device 400: Convolutional Layer Neural Network 40: Multimedia Data 401: Convolution Layer 402: Pooling layer 403: Fully connected layer 501: Multimedia material 503: Filters 503_R, 503_G, 503_B: sliding windows 6001, 6002, 6003: Blocks 600, 601, 602: Two-dimensional array 700: To-be-trained multimedia materials 701, 803 : Deep Learning Model 702: Output 703: Solution 801: Multimedia Data 805: Object Information Step S901: Step of Obtaining Multimedia Material by Input Device Step S902: Step of Inputting Multimedia Data into Deep Learning Model Step S903: Model by Deep Learning Get the category of the object in the multimedia material, the center point coordinate of the bounding box used to circle the object, and the bounding box Length and width and the rotation angle of the object in the multimedia material. Step S904: Step of obtaining the superimposed object corresponding to the object according to the identified category of the object. Step S905: superimposing the superimposed object on the multimedia material, and outputting the Step S1001: Step of acquiring multimedia data Step S1003: Step of determining whether to perform cloud computing Step S1005: transmitting multimedia material, and inputting multimedia material to a deep learning model by another subsystem to identify multimedia The object in the data, and the step of returning the identification result, step S1007: the step of inputting the multimedia material into the deep learning model to identify the object in the multimedia material, step S1009: executing the augmented reality image processing module according to the identified The step of acquiring the corresponding superimposed object and superimposing the superimposed object on the multimedia material, step S1011: outputting the multimedia material of the superimposed superimposed object, step S1101: step of obtaining the current geographical position, step S1103: determining that the current geographical position is Step S1201: Step of Retrieving Current Geographical Position Step S1203: Step of Determining Whether the Current Geographical Location Meets Specific Geographical Information Step S1205: Step of Obtaining Multimedia Data Step S1207: Input multimedia material into the deep learning model to identify Step of the object in the multimedia material Step S1209: Step of determining whether to capture a specific object Step S1211: Step of determining whether the object is in a state of a special event Step S1213: superimposing general superimposed information to display a general augmented reality Step of content step S1215: Step of superimposing special superimposed information to display special augmented reality content Step S1301: Step of acquiring multimedia material Step S1303: Step of inputting multimedia material into the deep learning model to identify objects in the multimedia material Step S1305: Step of outputting output information corresponding to the object according to the identified object in the multimedia material

Figure 1 is a schematic flow chart of object recognition. 2A is a schematic diagram of conventional object recognition. 2B is a schematic view of an image obtained by photographing the same object through different angles. FIG. 3 is a schematic diagram of an object identification system according to an embodiment of the invention. 4 is a schematic diagram of a convolutional layer-like neural network according to an embodiment of the invention. FIG. 5 is a schematic diagram of a filter and a sliding window thereof according to an embodiment of the invention. 6A-6E are schematic diagrams showing the operation mode of a sliding window according to an embodiment of the invention. FIG. 7 is a schematic diagram of a training convolutional layer-like neural network according to an embodiment of the invention. FIG. 8 is a schematic diagram of a subordinate stage according to an embodiment of the invention. FIG. 9 is a schematic diagram showing a display flow of an augmented reality according to an embodiment of the invention. FIG. 10 is a schematic diagram of a display method of augmented reality according to an embodiment of the invention. FIG. 11 is a schematic diagram of a method for displaying geographic information to an augmented reality according to an embodiment of the invention. FIG. 12 is a schematic diagram of a method for displaying geographic information to an augmented reality according to another embodiment of the invention. FIG. 13 is a flow chart of an object identification method according to an embodiment of the invention.

Claims (10)

An object identification method includes: acquiring a multimedia material; inputting the multimedia data to a deep learning model to identify an object in the multimedia material; and outputting, according to the identified object in the multimedia material, an output corresponding to the object An output information of the object, wherein, in the step of outputting the output information corresponding to the object, the method comprises: obtaining, according to the object identified by the deep learning model, a superimposed object corresponding to the object; Rotating the superimposed object at a rotation angle in the multimedia material, and scaling the superimposed object according to a length of a bounding box for circled the object, and adding the superimposed object to the superimposed object a multimedia material; and outputting the multimedia material superimposed with the superimposed object, wherein the rotation angle of the object in the multimedia material, the length of the bounding box of the object, and the width of the bounding box are learned by the depth The model recognizes it. The object identification method according to claim 1, wherein the deep learning model comprises a Convolution Neural Network (CNN), the convolutional layer-like neural network comprising at least one convolutional layer And the at least one pooling layer, the object identifying method further includes: capturing, by the at least one convolution layer and the at least one pooling layer, a feature value of the multimedia material. The object identification method of claim 2, wherein the deep learning model further comprises a fully connected layer or a machine learning algorithm, and the step of outputting the output information corresponding to the object comprises: The fully connected layer or the machine learning algorithm classifies the object based on the feature value and obtains an object information corresponding to the object. The object identification method of claim 3, wherein the object information includes coordinates for enclosing a plurality of vertices of the bounding box of the object. The object identification method of claim 3, wherein the object information includes the type of the object. The object identification method of claim 1, wherein the deep learning model comprises a plurality of layers, and the object identification method further comprises: inputting a plurality of multimedia materials to be trained and respectively corresponding to the multimedia materials to be trained. And a plurality of solutions to the deep learning model; and adjusting the plurality of weights of each of the layers in the deep learning model to train the deep learning model according to the multimedia materials to be trained and the solutions. The object identification method according to claim 6, wherein the deep learning model includes a loss layer, and the multimedia to be trained according to the And the step of adjusting the weights of each of the layers in the deep learning model to train the deep learning model includes: the depth learning model according to the multimedia data output to be trained respectively corresponding to the Training a plurality of output solutions of the multimedia material; and comparing the output solutions and the answers to the multimedia materials to be trained by the penalty layer and adjusting the respective layers of the deep learning model according to a penalty function Some weights. The object identification method of claim 1, wherein before the step of obtaining a multimedia material, the object identification method further comprises: obtaining a current geographic information; determining whether the current geographic information conforms to a specific geographic information; When the current geographic information conforms to the specific geographic information, the step of obtaining the multimedia material is performed. The object identification method of claim 1, wherein the multimedia material comprises at least one of an image, a point cloud, a voxel, and a mesh. An object identification system comprising: an input device for acquiring a multimedia material; a processor for inputting the multimedia data to a deep learning model to identify an object in the multimedia material; and an output device for Outputting an output information corresponding to the object according to the identified object in the multimedia material; a storage device, wherein in the operation of outputting the output information corresponding to the object, the processor further uses the object recognized by the deep learning model to obtain a superimposed object corresponding to the object from the storage device The processor is further configured to rotate the superimposed object according to a rotation angle of the object in the multimedia material, and according to a length of a bounding box for circled the object, the width of the bounding frame is superimposed on the width of the bounding frame The object is zoomed and the superimposed object is superimposed on the multimedia material, and the output device is further configured to output the multimedia material that has been superimposed on the superimposed object, wherein the rotation angle of the object in the multimedia material, the boundary of the object The length of the frame and the width of the bounding box are identified by the depth learning model.