TWM596391U - Situation awareness system - Google Patents

Abstract

A situation awareness system includes a server, a first augmented reality (AR) device and a second AR device. A first digital image is extracted by the first AR device. The first AR device receives a class label corresponding to an object to be labeled through a user interface, and uploads the class label and the first digital image to the server which trains a machine learning model accordingly. A second digital image is extracted by the second AR device and a situation object is detected according to the machine learning model. The second AR device displays a graphic object on a transparent display according to the location of the situation object.

Description

Situational awareness system

The present invention relates to a system for automatically generating artificial intelligence applications through augmented reality devices, where the applications can be shared with other users.

Most people make the wrong decision because of a lack of awareness of the situation, which depends on personal experience, life habits and talents. Some talented people can recognize specific situations based on instinct, such as identifying specific objects. Today's technology is sufficient to convert these experiences and talents into digital artificial intelligence models through machine learning. However, a lot of labeling work and complex model training make artificial intelligence models slow to produce and difficult to spread. In recent years, artificial intelligence has become the hottest emerging industry because of the substantial improvement in the computing power of graph processing units (GPUs). However, the application of artificial intelligence today is independent. There is no systematic and structural ecological chain, which is difficult for ordinary people to contact.

An embodiment of the present invention proposes a situational awareness system, including a server, a first augmented reality device, and a second augmented reality device. First augmented reality The device is communicatively connected to the server. The first augmented reality device includes a first image sensor and a first transparent display for capturing the first digital image through the first image sensor. The second augmented reality device is communicatively connected to the server. The second augmented reality device includes a second image sensor and a second transparent display. The first augmented reality device receives the class label about an object to be classified through the user interface, and uploads the class label and the first digital image to the server. The server trains a machine learning model based on the class label and the first digital image. The second augmented reality device captures the second digital image through the second image sensor. The second augmented reality device or server detects the contextual object in the second digital image according to the machine learning model. The second augmented reality device The reality device displays the corresponding illustrated object on the second transparent display according to the position of the situation object.

In some embodiments, the aforementioned user interface includes a voice interface or a gesture interface.

In some embodiments, the first augmented reality device or server is used to detect at least one first recommended object in the first digital image. The first augmented reality device is used to display a bounding box about the first recommended object on the first transparent display, and receive a command from the user through the user interface to adjust the position and size of the bounding box.

In some embodiments, the first augmented reality device or server detects the first recommended object according to the first convolutional neural network. The first convolutional neural network is used to execute the inference process only once to output multiple The position of each bounding box and the confidence values of multiple categories, where the size of the bounding box is fixed.

In some embodiments, the server is used to obtain multiple training images and labeled data about each training image, where the labeled data includes training edges The size of the bounding box. The server is used to execute an unsupervised grouping algorithm to divide the size of the training bounding box into multiple groups, and obtain the default bounding box size from each group. The server uses the default bounding box size in the one-time inference procedure described above.

In some embodiments, the aforementioned machine learning model is a second convolutional neural network, and the operation of detecting the context object in the second digital image includes: detecting the second digital image according to the first convolutional neural network The second recommended object and the category confidence value of the second recommended object; and output the situation confidence value of the second recommendation object according to the second convolutional neural network, and multiplying the category confidence value of the second recommendation object by the situation confidence value to The result confidence value is obtained, and if the result confidence value is greater than the critical value, it is determined that the second recommended object is a situation object.

In some embodiments, the first recommended object belongs to the first category, the category label belongs to the second category, and the second category is covered in the first category.

In the above system, users can create machine learning models through augmented reality devices without writing programs and share them with other users.

In order to make the above-mentioned features and advantages of the present invention more obvious and understandable, the embodiments are specifically described below and described in detail in conjunction with the accompanying drawings.

100‧‧‧Congnition System

102‧‧‧Server

110, 120‧‧‧ augmented reality device

111、121‧‧‧Image sensor

112、122‧‧‧Transparent display

130‧‧‧ Internet

201‧‧‧ Creator

210, 220, 230, 232, 240 ‧‧‧ steps

211‧‧‧Parameter

226‧‧‧Training sample

231‧‧‧Application

221~225‧‧‧Step

300, 400‧‧‧ digital images

301~304‧‧‧Recommended items

310‧‧‧ Augmented reality scene

311~314‧‧‧Boundary box

401‧‧‧Situational object

410‧‧‧ Augmented reality scene

411‧‧‧Boundary box

412‧‧‧Text

421~424‧‧‧Icon

502, 504, 506‧‧‧ steps

501‧‧‧ training image

503‧‧‧Preset bounding box size

505‧‧‧Enhanced image

507‧‧‧ label

508‧‧‧ machine learning model

[FIG. 1] is a schematic diagram illustrating a situation recognition system according to an embodiment.

[Fig. 2A] is a schematic diagram illustrating the operation of a situational awareness system according to an embodiment.

[FIG. 2B] is a flowchart illustrating step 220 according to an embodiment.

[FIG. 3A] A schematic diagram illustrating digital images captured by an augmented reality device according to an embodiment.

[FIG. 3B] A schematic diagram illustrating an augmented reality scene seen by a user according to an embodiment.

FIG. 4A is a schematic diagram illustrating digital images captured by an augmented reality device according to an embodiment.

[FIG. 4B] A schematic diagram illustrating an augmented reality scene seen by a user according to an embodiment.

[Fig. 5] is a flowchart illustrating training a machine learning model according to an embodiment.

With regard to the "first", "second", ... etc. used in this article, it does not specifically mean the order or order, it is only to distinguish the elements or operations described in the same technical terms.

The technology proposed in this disclosure can be called Situation Awareness Magnifier (SAM). On the basis of artificial intelligence (AI) technology, SAM uses lightweight smart glasses as a medium to allow creation with a high degree of situational awareness. Through their experience, people can mark the situation that ordinary people can't realize, and through automated machine learning in the cloud, these experiences can be converted into AI applications (applications). Ordinary users can connect with the SAM cloud system through smart glasses to obtain the situational awareness provided by the creator.

FIG. 1 is a schematic diagram illustrating a situational awareness system according to an embodiment. The situational awareness system 100 includes a server 102, augmented reality (augmented reality) reality (AR) device 110 and augmented reality device 120, wherein the augmented reality devices 110, 120 can be connected to the server 102 via the network 130 communication. In this embodiment, the augmented reality devices 110, 120 are implemented as smart glasses, but in other embodiments, they can also be implemented as face masks, transparent tablets, etc. The present invention does not limit the size of the augmented reality devices 110, 120 With appearance. In this embodiment, the user wearing the augmented reality device 110 is called a creator, and the creator can use the server 102 to implement his own situational awareness ability as an application, thereby sharing the situational awareness ability to the augmentation The user of the reality device 120.

The augmented reality device 110 includes an image sensor 111 and a transparent display 112. The augmented reality device 120 includes an image sensor 121 and a transparent display 122. The transparent displays 112 and 122 can also be referred to as see-through displays (See-through display), which is a device that allows users to simultaneously view external images and project images, so as to achieve the effect of superimposing real and virtual images without obscuring the field of view. The transparent displays 112, 122 may include liquid crystal displays, organic light emitting diodes, projectors, lenses, light guides, etc. The present invention does not limit the components included in the transparent displays 112, 122. The image sensors 111 and 121 may include a charge-coupled device (CCD) sensor, a complementary metal-oxide semiconductor (Complementary Metal-Oxide Semiconductor) sensor, or other suitable photosensitive elements. For simplicity, not all components of the augmented reality device 110, 120 are shown in FIG. 1, for example, the augmented reality device 110, 120 may further include a processor, a microphone, a sound player, and an inertial measurement unit , Communication modules, physical buttons, etc., this new model is not limited to this. The above-mentioned communication module may be a cellular network (or mobile network), near field communication, infrared communication, Bluetooth or wireless fidelity (WiFi), etc., for connecting to the network 130.

2A is a schematic diagram illustrating the operation of a situational awareness system according to an embodiment. Please refer to FIG. 2A. Here, creator 201 refers to a user of augmented reality device 110. In step 210, creator 201 is on server 102. Create a project on the provided system platform. This system platform can have interfaces such as web pages, applications, and network services. The above-mentioned project is used to manage the machine learning model created by the creator 201 according to his own situational cognitive ability. The creator 201 can input a plurality of parameters 211, and these parameters 211 are sent to the server 102. The parameter 211 includes the category to be recognized, image resolution, type of machine learning model, various parameters of the machine learning model, and so on. For example, the above categories may include human emotions, whether people lie, species of animals and plants, whether machinery malfunctions, whether they are sick, etc. The server 102 may provide multiple preset categories, or the creator 201 may create new category. In some embodiments, the server 102 can also provide a category tree created by experts, which records the affiliation of each category. For example, the category of "face" will cover the category of "angry", so "human The "face" node will be the parent of the "angry" node, and so on. The affiliation of each category in the category tree can be used to help the creator mark the objects in the image. For example, if the creator wants to create a "angry" category, the server 102 can provide an object detector for the face When you want to mark the "angry" category in an image, you can first detect the face based on this object detector for the creator to judge whether it is "angry" category.

After creating the project, in step 220, the creator will capture digital images through the augmented reality device 110 and mark the objects to be recognized to generate training samples. Specifically, FIG. 2B is a flowchart illustrating step 220 according to an embodiment. Please refer to FIG. 2B. In step 221, the augmented reality device 110 first captures a digital image through its own image sensor 111 (as shown in FIG. 3A. Like 300), in this example, the creator wants to mark objects in the digital image 300 that belong to the "angry" category.

In step 222, the augmented reality device 110 or the server 102 detects the recommended objects 301-304 (in this example, human faces) in the digital image 300. In some embodiments, the server 102 may provide an object detector that recommends objects 301 to 304 to the augmented reality device 110, or the object detector is built into the augmented reality device 110. In some embodiments, the augmented reality device 110 may also send the digital image 300 to the server 102, and after the server detects the recommended objects 301-304, the detection result is returned to the augmented reality device 110.

In step 223, the icon object is displayed according to the detection result. Specifically, the augmented reality device 110 generates a corresponding icon object (such as a bounding box) according to the positions of the recommended objects 301-304 and displays the icon object on the transparent display 112. FIG. 3B shows the creation The augmented reality scene 310 seen by the author, in which the bounding boxes 311 to 314 are virtual images, and the rest are real scenes, the two will be mixed together.

In step 224, the bounding box to be marked is selected. The creator can input one or more commands through a user interface, thereby selecting the objects to be marked from the recommended objects 301-304. The user interface is, for example, a voice interface or a gesture interface, which means that the augmented reality device 110 can receive the audio signal from the creator through the microphone and recognize the words spoken by the creator or recognize the creator through the image sensor 111 Gesture to analyze the instructions that the creator wants to give. Or in some embodiments, the user interface described above may also be implemented with a mouse, keyboard, tablet, physical keys, or other hardware devices. In this example, the author selects the bounding box 313 (recommended object 303).

At step 225, a category label of the recommended item is provided. The creator can enter one or more commands through the above user interface to further classify the bounding box 313 (recommended object 303) as the "angry" category. The category label provided by the creator can use text, numbers or binary The code is recorded. In some embodiments, the creator can also issue commands through the user interface to adjust the position and size of the bounding boxes 311-314, and then provide the category label. In some embodiments, if the object to be marked by the creator is not within the range of the bounding boxes 311-314, the creator can create a new bounding box through the user interface and provide the corresponding category label. The position and size of the digital image 300, the bounding box 313 (or the position and size of the newly created bounding box) and the category label can be combined to be called a training sample, and then steps 222~225 can be repeated to generate more training samples .

In some embodiments, the recommended object may not be generated, and the augmented reality device 110 will directly receive the category label about an object to be classified through the user interface. For example, the augmented reality device 110 can set a specific position (eg, image center) in the digital image corresponding to the user's gaze and obtain an object (also referred to as an object to be classified) at this specific position, the augmented reality device 110 can determine whether the user's gaze stays on the object to be classified for more than a preset time (for example, 5 seconds), and if so, wait for the user to say the category label of the object to be classified. In the present disclosure, the above-mentioned recommended objects can also be referred to as objects to be classified, so regardless of whether or not to generate recommended objects, as long as the category label of an object to be classified is generated through the user interface of the augmented reality device 110, this category Within the scope of disclosure.

Please refer back to FIG. 2A. Next, the augmented reality device 110 transmits the collected one or more training samples 226 to the server 102. In step 220 In, because the system provides recommended items first, the creator can quickly create category tags. It is worth noting that the category of recommended objects must cover the category that the creator wants to create. For example, the category of "face" covers the category of "angry". From another perspective, if the recommended object belongs to the first category and the category label provided by the creator belongs to the second category, the second category will be covered in the first category, but the above category is just an example. It does not limit what the first category and the second category are. In addition, the creator creates the category label through the augmented reality device 110, so the creator can move indoors/outdoors arbitrarily. When he finds a suitable object, he can generate new training samples, compared with the Labels are constantly created in front of the computer, and the system provided by this embodiment is more friendly to creators.

After the augmented reality device 110 sends the training samples 226 to the server 102, the server 102 can determine whether the number of training samples exceeds a critical value, if so, it will start to perform step 230, if not, it will notify the creator Continue to collect training samples. In step 230, the server 102 trains a machine learning model based on the collected class labels and digital images. The machine learning model can be a decision tree, random forest, multi-level neural network, convolutional neural network, support vector machine Wait, the new model is not limited to this. In some embodiments, the server 102 may also perform some pre-processing on the collected digital images, such as brightness adjustment, noise reduction, etc. The present invention does not limit the content of these pre-processing. In some embodiments, the server 102 may also create an application 231 for the user to use the trained machine learning model through the application 231. After creating the application 231, the server 102 will publish the application 231 on a platform accessible to the user.

In some embodiments, before publishing the application 231, The server 102 may first send the application 231 to the creator for testing (step 232), and after the test, the creator may return to step 220 to collect more training samples, or the creator may accept the training results, and then the server 102 Only then will the application 231 be released.

In step 240, the user downloads and installs the application program 231 through the augmented reality device 120, and then can recognize the new situation according to the trained machine learning model. Specifically, the augmented reality device 120 can capture a digital image through its own image sensor 121, as shown in the digital image 400 of FIG. 4A, the augmented reality device 120 can detect according to the trained machine learning model Measure the situation object 401 in the digital image 400 (in this example, an angry face). In some embodiments, the augmented reality device 120 may send the digital image 400 to the server 102, and the server 102 detects the situation object 401 and returns the detection result to the augmented reality device 120. Alternatively, the augmented reality device 120 can also download the trained machine learning model from the server 102 through the application program 231, and detect the situation object 401 in the digital image 400 through its own processor.

The augmented reality device 120 will also display the corresponding graphical object on the transparent display 121 according to the position of the situation object 401, as shown in the augmented reality scene 410 shown in FIG. 4B, the illustrated object includes a bounding box 411 and "angry" Of text 412. However, if 4B is just an example, the above illustrated objects may also include any patterns, symbols, numbers, etc., and the present invention is not limited thereto. In some embodiments, multiple icons 421 to 424 can also be displayed in the augmented reality scene 410. These icons 421 to 424 are used to switch to other situational awareness, that is, to detect other objects, such as a car , Flowers, trees, children, etc., the new model is not limited to this.

In some embodiments, the object detection used in step 220 The machine and the machine learning model trained in step 230 are both convolutional neural networks. In the conventional object detection method, a window is used to scan the entire digital image. For each window, it is determined whether it is the object to be detected. If so, set this window as a bounding box, and then adjust the window The size of is rescanned again, that is to say, it needs to be inferred multiple times in such an algorithm. However, in this embodiment, a plurality of bounding boxes are preset first, and each bounding box has a fixed size, and the convolutional neural network only executes the inference process once to output the positions of multiple bounding boxes and the confidence of multiple categories value. More specifically, each bounding box corresponds to at least 3+N parameters, including X coordinate, Y coordinate, the probability of whether there is an object P(Object), N class confidence values, and if there are M total bounding boxes, Then the convolutional neural network will output at least Mx(3+N) values, where M and N are positive integers. The above-mentioned N category confidence values correspond to N categories such as dogs, cats, faces, and cars. Multiplying the above probability P(Object) by the corresponding category confidence value indicates that there is a corresponding category probability in this bounding box Why, it can be expressed as the following equation (1).

P(C _i )=P(C _i |Object)×P(Object)...(1)

Where P(C _i |Object) is the above-mentioned category confidence value, C _i is the i-th category, i=1...N, and P(C _i ) is the probability that there is a corresponding category C _i in this bounding box. It is worth noting that during the training phase, if the number of objects in the image is greater than the above positive integer M, there are no objects in some bounding boxes, so the probability P(Object) of these bounding boxes should be 0. In some embodiments, the above Mx(3+N) values may be output after the fully connected layer, or may be output after the convolution layer in some embodiments, however, those with ordinary knowledge in the art can understand convolution The architecture of the neural network will not be described in detail here.

Since the size of the above bounding box is fixed, there must be a Mechanism to determine this size (including width and height). FIG. 5 is a flowchart illustrating training a machine learning model according to an embodiment. Please refer to FIG. 5, wherein each step is executed by the server 102. First, obtain multiple training images 501 and labeled data about the training images from the database. The labeled data is also called ground truth, including the size and position of the bounding box (hereinafter referred to as the training bounding box) that has been marked . In step 502, an unsupervised clustering algorithm is executed to divide the size of these training bounding boxes into multiple groups, and the default bounding box size 503 is obtained from each group, for example, taking the center of each group The position (averaged) is taken as the preset bounding box size 503. The unsupervised clustering algorithm is, for example, K-means algorithm (K-mcans), where K may be 5, 6 or any suitable positive integer. These preset bounding box sizes 503 can be expressed as vectors [height, width], for example, [64,44], [88,75], and a total of K vectors. The present invention does not limit the values of these preset bounding box sizes 503 . In step 504, the aforementioned pre-processing may be performed on the training image 501 to obtain an enhanced image 505. Next, in step 506, a supervised machine learning algorithm is executed according to the label 507, the preset bounding box size 503 and the enhanced image 505 to obtain the machine learning model 508.

The preset bounding box size 503 is used in the above-mentioned inference procedure only in addition to the training phase. The convolutional neural network of this implementation directly predicts the position of the bounding box, so there is no need to use the window to scan the entire image, so only one inference procedure needs to be executed, which can reduce the time of inference.

In the above example, step 220 uses a convolutional neural network to detect faces, and step 230 is to train a convolutional neural network to detect angry faces. In some embodiments, in step 230, the server 102 may not use a convolutional neural network to detect human faces, and retrain a machine learning "angry face" The model, that is to say, the input image is the entire image containing the background, which has the mark of the angry face. Or in some embodiments, a "angry" machine learning model can be trained separately, and this machine learning model can be combined with a "human face" machine learning model. Specifically, the convolutional neural network used in step 220 is called the first convolutional neural network, and the convolutional neural network trained in step 230 is called the second convolutional neural network. The input of the two convolutional neural network is a face image, and the output is a value indicating whether it is "angry". In step 240, the operation of detecting the context object may first detect the second recommended object (same as the context object 401) in the digital image 400 according to the first convolutional neural network and obtain the class confidence value P of the second recommended object (face), according to the above description can be expressed as the following equation (2).

P(face)=P(face|Object)×P(Object)...(2)

Then output the situational confidence value P(angry|face) of the second recommended object according to the second convolutional neural network, and multiply the category confidence value P(face) of the second recommended object by the situational confidence value P(angry|face) In order to obtain the result confidence value P(angry), it is expressed as the following equation (3).

P(angry)=P(angryface)×P(face)...(3)

If the result confidence value P(angry) is greater than a critical value, it is determined that the second recommended object is a situation object. In this embodiment, since the second convolutional neural network judges whether it is angry according to the human face, the complexity of the second convolutional neural network will be relatively low, which can reduce the training time. It is worth noting that one reason for reducing the training time is that the category of the recommended object used in step 220 covers the category to be marked by the user, which not only reduces the time required for the creator to mark, but also reduces the training time.

The new model uses refined smart glasses and an automated AI model training system to refine the quality of the samples and shorten the marking work and model training time. With the perfect cloud system, ordinary people can obtain these precious cognitive experiences and apply them in daily life. This new model is based on the awareness of creators and users to share the situation, and creates an ecosystem with AI apps as the main body. With automated AI training and recognition services, the entire ecosystem is created so that ordinary people can participate in AI. In addition, through the above system, users do not need to write programs, just mark the status through smart glasses and user interface, and the cloud platform automatically performs machine learning, which greatly reduces the entry threshold. In addition to being used in the general consumer market, this tool can be applied to any occasion that requires a large number of marked objects for future machine learning. The personnel only need to mark the object through the glasses and upload it to the SAM platform. The subsequent training will be completed automatically by the platform. The training model generated by it can be further connected and reformed according to the needs to form a multi-functional AI App.

Although the present invention has been disclosed as above with examples, it is not intended to limit the present invention. Anyone who has ordinary knowledge in the technical field can make some changes and retouching without departing from the spirit and scope of the present invention. The scope of protection of this new model shall be subject to the scope defined in the appended patent application.

100: situational awareness system

102: server

110, 120: Augmented reality device

111, 121: image sensor

112, 122: Transparent display

130: Internet

Claims (7)

A situational awareness system, including: A server; A first augmented reality device is communicatively connected to the server. The first augmented reality device includes a first image sensor and a first transparent display for capturing through the first image sensor Take a first digital image; and A second augmented reality device, communicatively connected to the server, the second augmented reality device includes a second image sensor and a second transparent display, The first augmented reality device receives a category label about an object to be classified through a user interface, and uploads the category label and the first digital image to the server, The server trains a machine learning model based on the category label and the first digital image, Wherein the second augmented reality device captures a second digital image through the second image sensor, and the second augmented reality device or the server detects the second digital image according to the machine learning model A situation object, the second augmented reality device displays the corresponding icon object on the second transparent display according to the position of the situation object. The situation recognition system as described in item 1 of the patent application scope, wherein the user interface includes a voice interface or a gesture interface. For the situational awareness system described in item 1 of the patent application scope, The first augmented reality device or the server is used to detect at least one first recommended object in the first digital image as the object to be classified, and the first augmented reality device is used in the first A transparent display displays a bounding box on the at least one first recommended object, and receives a command from the user through the user interface to adjust the position and size of the bounding box. The situational awareness system as described in item 3 of the patent application scope, wherein the first augmented reality device or the server detects the at least one first recommended object according to a first convolutional neural network, the first A convolutional neural network is used to execute the inference procedure only once to output the positions of multiple bounding boxes and multiple class confidence values, wherein the sizes of the bounding boxes are fixed. The condition recognition system as described in item 4 of the patent application scope, wherein the server is used to obtain a plurality of training images and a piece of labeled data about each of the training images, wherein the marked data includes the size of at least one training bounding box , The server is used to execute an unsupervised grouping algorithm to divide the size of the at least one training bounding box of the training images into multiple groups, and obtain a default boundary from each of the groups Frame size, The server uses the preset bounding box sizes in the one-time inference process. The condition recognition system as described in item 5 of the patent application scope, wherein the machine learning model is a second convolutional neural network to detect the second number The operations of the situation object in the bit image include: Detecting a second recommended object in the second digital image and the class confidence value of the second recommended object according to the first convolutional neural network; and Output a context confidence value of the second recommended object according to the second convolutional neural network, and multiply the category confidence value of the second recommended object by the context confidence value to obtain a result confidence value, if the result confidence If the value is greater than a critical value, the second recommended object is determined to be the situation object. The situation recognition system as described in item 6 of the patent application scope, wherein the first recommended object belongs to a first category, the category label belongs to a second category, and the second category is covered in the first category.

Images