TWI757965B - Deep learning method for augmented reality somatosensory game machine - Google Patents

Abstract

A deep learning method for augmented reality somatosensory game machine includes the following operations: obtaining, by a processor, a depth map according to a depth image and a first mask, in which the depth image and the first mask are generated separately according to the image; inputting, by the processor, the depth map and the image to a neural network model so as to generate a second mask; and generating, by the processor, a matting image according to the second mask and the image.

Description

Deep Learning Method for Augmented Reality Somatosensory Game Machine

This case is about a deep learning method and device for an augmented reality somatosensory game machine, especially a deep learning method and device for an augmented reality somatosensory game machine with a deep learning neural network model.

Among the deep learning methods of augmented reality somatosensory game consoles, the real-time character removal technology based on deep learning is a commonly used technology in current social software, and there are also a considerable number of back removal software tools on the market. For example, poor accuracy, resulting in abnormally corrupted backing images, or requiring a lot of performance and operating time, cannot be used for instant backing applications on low-performance platforms, and may cause other heavy loads to run synchronously. app, so usability drops drastically.

An aspect of the present application is to provide a deep learning method for an augmented reality somatosensory game console, comprising the following steps: obtaining a depth map by a processor according to a depth image and a first mask, wherein the depth image and the first mask are respectively generated according to the image; the processor inputs the depth map and the image to the neural network model to generate a second mask; and the processor generates a background image according to the second mask and the image.

Another aspect of the present application is to provide a deep learning device for an augmented reality somatosensory game machine, including a processor and a memory. The processor is used for obtaining a depth map according to the depth image and the first mask, inputting the depth map and the image to the neural network model to generate a second mask, and generating a background image according to the second mask and the image, wherein the depth image and Masks are generated separately from images. Memory is used to store neural network models.

The following disclosure provides many different embodiments or illustrations for implementing different features of the invention. The elements and configurations of the particular instances are used to simplify the present case in the following discussion. Any examples discussed are for illustrative purposes only and do not in any way limit the scope and meaning of the invention or its examples.

See Figure 1. FIG. 1 is a schematic diagram of a deep learning device 100 of an augmented reality somatosensory game machine. As depicted in FIG. 1 , the deep learning device 100 of the augmented reality somatosensory game machine includes a processor 110 and a memory 130 . The processor 110 is coupled to the memory 130 . In some embodiments, the deep learning device 100 of the augmented reality somatosensory game machine further includes a camera 150 . The camera 150 is coupled to the processor 110 .

See Figure 2. FIG. 2 is a schematic diagram of a deep learning method 200 of an augmented reality somatosensory game machine according to some embodiments of the present invention. Embodiments of the present invention are not limited thereto.

It should be noted that the deep learning method 200 of the augmented reality somatosensory game machine can be applied to a system with the same or similar structure as the deep learning apparatus 100 of the augmented reality somatosensory game machine in FIG. 1 . In order to simplify the description, the following will take FIG. 2 as an example to describe the operation method, but the present invention is not limited to the application of FIG. 1 .

It should be noted that, in some embodiments, the deep learning method 200 of the augmented reality somatosensory game machine can also be implemented as a computer program and stored in a non-transitory computer-readable medium, so that the computer, The electronic device or the processor 110 in the deep learning device 100 of the augmented reality somatosensory game machine in FIG. 1 reads the recording medium and executes the operation method. The processor may be composed of one or more chips. composition. The non-transitory computer-readable recording medium can be read-only memory, flash memory, floppy disk, hard disk, optical disk, pen drive, magnetic tape, a database accessible through a network, or a person skilled in the art can easily think of it. and a non-transitory computer-readable recording medium with the same function.

In addition, it should be understood that the operations of the deep learning method of the augmented reality somatosensory game machine mentioned in this embodiment can be adjusted according to actual needs unless the order is specifically stated, or even Can be executed concurrently or partially concurrently.

Furthermore, in different embodiments, these operations may be adaptively added, replaced, and/or omitted.

See Figure 2. The deep learning method 200 of the augmented reality somatosensory game machine includes the following steps.

In step S210 : obtaining a depth map according to the depth image and the first mask. Please refer to FIG. 1 together. In some embodiments, step S210 may be performed by the processor 110 in FIG. 1 . The detailed operation of step S210 will be described below with reference to FIGS. 3 to 6 together.

In some embodiments, the camera 150 captures images. The depth image is generated by the NUITRACK software according to the image captured by the camera 150 . In some embodiments, the NUITRACK software is stored in the memory 130 and executed by the processor 110 .

See Figure 3. FIG. 3 is a schematic diagram of an image 300 captured by a camera 150 according to some embodiments of the present invention. It should be noted that the images captured by the camera 150 are continuous images, and the image 300 shown in FIG. 3 is one frame of the continuous images. In some embodiments, the image 300 is an RGB color image.

See Figure 4. FIG. 4 is a schematic diagram of a first mask 400 according to some embodiments of the present invention. The first mask 400 shown in FIG. 4 is generated by the NUITRACK software according to the image 300 . As can be seen from FIG. 4 , the first mask 400 generated by the NUITRACK software contains flaws and leaks, and the effect of directly using it to remove the back of a character is not good.

See Figure 5. FIG. 5 is a schematic diagram of a depth image 500 according to some embodiments of the present invention. Referring back to FIG. 1, in some embodiments, the camera 150 is a depth camera. The depth image 500 is one frame of the depth image obtained by the camera 150 . The depth image 500 corresponds to the image 300 in FIG. 3 .

See Figure 6. FIG. 6 is a schematic diagram of a depth map 600 according to some embodiments of the present invention. The depth map 600 is generated by the processor 110 in FIG. 1 according to the depth image 500 in FIG. 5 and the first mask 400 in FIG. 4 . In some embodiments, the processor 110 in FIG. 1 combines the depth image 500 in FIG. 5 and the first mask 400 in FIG. 4 to generate the depth map 600 . After processing the depth image 500 in FIG. 5 with the first mask 400 in FIG. 4 , only the depth data of the target person is left in the depth image 500 to generate the depth image 600 .

Please refer back to Figure 2. In step S230, the depth map and the image are input to the neural network model to generate a second mask. Please refer to FIG. 1 together. In some embodiments, step S230 may be performed by the processor 110 in FIG. 1 . The detailed operation of step S230 will be described below with reference to FIGS. 7 to 8 together.

See Figure 7. FIG. 7 is a schematic diagram of a neural network model 700 according to some embodiments of the present invention. In step S230 , the processor 110 in FIG. 1 inputs the image 300 in FIG. 3 and the depth map 600 in FIG. 6 to the neural network model 700 to generate a second mask.

As shown in FIG. 7 , the neural network model 700 includes an encoding unit 710 and a decoding unit 730 . In some embodiments, in step S230 , the encoding unit 710 generates the first data D1 and the second data D2 according to the depth map 600 and the image 300 . In some embodiments, the neural network model 700 is DeepLabv3+.

Specifically, in some embodiments, the encoding unit 710 includes a MobileNetV2 unit 712 and an ASPP (Atrous Spatial Pyramid Pooling) unit 714 . The MobileNetV2 unit 712 performs a feature compression operation on the depth map 600 and the image 300 to generate the first data D1. On the other hand, after the MobileNetV2 unit 712 performs feature compression on the depth map 600 and the image 300, the ASPP unit 712 performs feature extraction on the depth map 600 and the image 300 to generate the second data D2.

As shown in FIG. 7 , after the MobileNetV2 unit 712 performs feature compression on the depth map 600 and the image 300 , the ASPP unit 712 performs multiple convolution operations 715A to 715D on the depth map 600 and the image 300 to generate a plurality of first features Data C1 to C4, in addition, ASPP unit 712 performs a pooling operation 718 on depth map 600 and image 300 to generate second feature data C5. Next, the ASPP unit 712 synthesizes the plurality of first feature data C1 to C4 and the second feature data C5 to generate a data set CC1.

In ASPP unit 712, convolution operations 715A-715D include convolution parameters and scale parameters, respectively. The scale parameters between convolution operations 715A to 715D are coprime.

For example, in some embodiments, the convolution parameter of the convolution operation 715A is 1×1×256. The convolution parameter of the convolution operation 715B is 3×3×256, and the scale parameter is 1. The convolution parameter of the convolution operation 715C is 3×3×256, and the scale parameter is 2. Convolution Operation 715D Convolution The parameter system is 3×3×256, and the scale parameter system is 3. The scale parameters between convolution operations 715B to 715D are coprime.

The ASPP unit uses atrous convolution to increase the neural network's receptive field around each pixel, so that it does not only recognize linear features, which can greatly improve the results of neural network inference, and make the obtained mask more complete. high. The feature of hole convolution is that it will extract features according to the ratio parameter (rate), and extract features with a large field of view with a small amount of convolution. If the ratio parameters between different convolution operations share a common factor of more than 1, it will cause grid The gate effect problem, so that the features of some pixels are not extracted. Therefore, if the scale parameters between different convolution operations are relatively prime, the problem of grid effect can be improved.

Next, the encoding unit 710 performs a 1×1 convolution operation on the data set CC1 to generate the second data D2.

After receiving the first data D1 and the second data D2 from the encoding unit 710, the decoding unit 730 synthesizes the first data D1 and the second data D2 to generate a data set CC2, and the decoding unit 730 generates a second mask according to the data set CC2.

See Figure 8. FIG. 8 is a schematic diagram of a second mask 800 according to some embodiments of the present invention. Compared with the first mask 400 in FIG. 4 , the accuracy of the second mask 800 is higher.

Specifically, in some embodiments, the decoding unit 730 performs a 1×1 convolution operation on the first data D1 to generate the first feature data D1C. The decoding unit 730 performs a dilation operation with a parameter of 4 on the second data D2 to generate the second feature data D2C. The decoding unit 730 synthesizes the first feature data D1C and the second feature data D2C to generate a data set CC2. decoding The unit 730 then performs a 3×3 convolution operation on the data set CC2 to generate a third feature data D3C, and then performs a dilation operation on the third feature data D3C to generate a second mask.

In some embodiments, the neural network model 700 includes a loss function. The formula for the loss function is as follows:

為推論所得的遮罩像素透明度，

為正解的像素透明度，

為常數，以防止透明度皆為0時，神經網路無梯度可更新。 in

is the inferred mask pixel transparency,

is the pixel transparency of the correct solution,

is a constant to prevent the neural network from having no gradients to update when the transparency is all 0.

Please refer back to Figure 2. In step S250, a background-removed image is generated according to the second mask and the image. Please refer to FIG. 1 together. In some embodiments, step S250 may be performed by the processor 110 in FIG. 1 . See Figure 9. FIG. 9 is a schematic diagram of a background-removed image 900 according to some embodiments of the present invention. In some embodiments, the background-removed image 900 is generated by combining the second mask 800 shown in FIG. 8 and the image 300 shown in FIG. 3 .

In some embodiments, the processor 250 may be a server or other device. In some embodiments, the processor 110 may be a server, a circuit, a central processing unit (CPU), a microprocessor having functions of storing, computing, reading data, receiving signals or messages, and transmitting signals or messages. controller (MCU) or other devices with equivalent functions. In some embodiments, the camera 150 may be capable of capturing images, taking pictures, etc. Functional circuit Other devices with the same function. In some embodiments, the memory 130 may be an element with a storage function or a similar function element.

There are relatively mature products for character detection and segmentation on the market, such as NuiTrack SDK, the masks obtained from segmentation often have various flaws and leaks, and the effect of directly using them to remove the backs of characters is not good. It can be seen from the above-mentioned embodiments that the embodiment of this application provides a deep learning method for an augmented reality somatosensory game machine and an augmented reality device thereof, and uses a neural network to refer to the broken masks generated by products such as NuiTrack SDK and so on. information, deduce the complete mask, which can then be used to generate the correct de-backed image for a particular person. In addition, because the network structure of DeepLabv3+ requires a large amount of computation, each inference takes a long time. Therefore, replacing the network for the initial feature extraction of the neural network from Xception_101 to MobileNetV2 can greatly reduce the time-consuming of this stage. Speed boost, instant application of moving images to the back.

To sum up, the mask obtained by the neural network inference in the implementation of this case is more accurate than the original segmentation mask, and the final image of the specific person without the back is more complete, and it can still be compared with the heavyweight on the low-performance platform. Load applications simultaneously.

In addition, the above illustrations include sequential exemplary steps, but the steps do not have to be performed in the order shown. It is within the contemplation of this disclosure to perform the steps in a different order. These steps may be added, replaced, changed order and/or omitted as appropriate within the spirit and scope of the embodiments of the present disclosure.

Although this case has been disclosed above in terms of implementation, it is not intended to limit this case. Anyone who is familiar with this technique can make various changes and modifications without departing from the spirit and scope of this case. Therefore, the scope of protection in this case should be considered later. The scope of the attached patent application shall prevail.

100: Deep Learning Device for Augmented Reality Somatosensory Game Console

110: Processor

130: Memory

150: Camera

200: Deep Learning Methods for Augmented Reality Somatosensory Game Consoles

S210 to S250: Steps

300: Video

400: first mask

500: Deep Image

600: Depth Map

700: Neural Network Models

710: Coding unit

730: Decoding unit

712: MobileNetV2 unit

714: ASPP unit

D1, D2: Data

715A to 715D: Convolution Operation

718: Pooling operation

C1 to C5: Characteristic data

CC1,CC2: Datasets

D1C to D3C: Characterization Information

800: Second mask

900: remove back image

In order to make the above and other objects, features, advantages and embodiments of the present disclosure more clearly understood, the accompanying drawings are described as follows: Figure 1 is a schematic diagram of a deep learning device of an augmented reality somatosensory game console; FIG. 2 is a schematic diagram of a deep learning method for an augmented reality somatosensory game console according to some embodiments of the present invention; FIG. 3 is a schematic diagram of an image captured by a camera according to some embodiments of the present invention. ; FIG. 4 is a schematic diagram of a first mask according to some embodiments of the present invention; FIG. 5 is a schematic diagram of a depth image according to some embodiments of the present invention; FIG. 6 is a schematic diagram of a depth map according to some embodiments of the present invention; FIG. 7 is a schematic diagram of a neural network model according to some embodiments of the present invention; FIG. 8 is a schematic diagram of a second mask according to some embodiments of the present invention; FIG. 9 is a schematic diagram of a background-removed image according to some embodiments of the present invention.

200: Deep Learning Methods for Augmented Reality Somatosensory Game Consoles

S210 to S250: Steps

Claims (7)

A deep learning method for an augmented reality somatosensory game machine, comprising: obtaining a depth map by a processor according to a depth image and a first mask, wherein the depth image and the first mask are based on an image are generated respectively; the processor inputs the depth map and the image to a neural network model to generate a second mask; and the processor generates a background image according to the second mask and the image; wherein the The neural network model includes an encoding unit and a decoding unit, wherein the encoding unit includes a MobileNetV2 unit and an ASPP unit, wherein the processor inputs the depth map and the image to the neural network model to generate the second mask The mask includes: generating a first data and a second data by the coding unit according to the depth map and the image; and synthesizing the first data and the second data by the decoding unit to generate a first data set, and decoding the first data set to generate the second mask; wherein generating the first data and the second data by the coding unit according to the depth map and the image includes: generating by the MobileNetV2 unit according to the depth map and the image the first data; and the MobileNetV2 unit and the ASPP unit to generate the second data according to the depth map and the image. The deep learning method for an augmented reality somatosensory game machine according to claim 1, wherein the MobileNetV2 unit and the ASPP unit generate the second data according to the depth map and the image, comprising: according to a plurality of convolution parameters and a plurality of scale parameters to perform a plurality of convolution operations to generate a plurality of first feature data; perform a pooling operation to generate a second feature data; collect the first feature data and the second feature data to generate a second feature a data set; and generating the second data according to the second data set. The deep learning method for an augmented reality somatosensory game machine according to claim 2, wherein the proportional parameters are mutually prime. The deep learning method for an augmented reality somatosensory game machine according to claim 1, wherein synthesizing the first data and the second data by the decoding unit to generate the first data set comprises: performing the first data processing on the first data set. The convolution operation is performed to generate a first feature data; the expansion operation is performed on the second data to generate a second feature data; and the first feature data and the second feature data are synthesized. The depth of the augmented reality somatosensory game machine as described in claim 1 A degree learning method, wherein decoding the first data set to generate the second mask comprises: performing a convolution operation on the first data set to generate a third feature data; and performing a dilation operation on the third feature data to generate the second mask. The deep learning method for an augmented reality somatosensory game console according to claim 1, further comprising: generating the first mask by a NUITRACK software; and generating the depth image by a camera. A deep learning device for an augmented reality somatosensory game machine, comprising: a processor for obtaining a depth map according to a depth image and a first mask, and inputting the depth map and an image to a neural network a model to generate a second mask, and generate a back image according to the second mask and the image, wherein the depth image and the mask are respectively generated according to the image; and a memory for storing the nerve A network model; wherein the neural network model includes a MobileNetV2 unit and an ASPP unit, wherein the processor is further configured to generate a first data according to the MobileNetV2 unit, and generate a second data according to the MobileNetV2 unit and the ASPP unit data, and generate the second mask according to the first data and the second data.

Images