KR20210048385A - Electronic device and method for controlling the electronic device thereof - Google Patents

Abstract

Provided are an electronic device and a control method thereof. In accordance with the present invention, the electronic device comprises: a camera; a non-volatile memory which stores one or more instructions and a plurality of object recognition models; a volatile memory; and a processor connected to the non-volatile memory, the volatile memory, and the camera to control the electronic device. The processor is able to execute the one or more instructions, load one or more object recognition models having a hierarchy structure corresponding to a determined operation mode among the plurality of object recognition models on the volatile memory when the operation mode of the electronic device is determined, acquire information on the object by entering an object image acquired by the camera into the loaded object recognition models, and determine the operation of the electronic device based on the information on the object. The present invention aims to provide an electronic device and a control method thereof, which are able to reduce memory consumption.

Description

Electronic device and its control method {ELECTRONIC DEVICE AND METHOD FOR CONTROLLING THE ELECTRONIC DEVICE THEREOF}

The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an object recognition model having a hierarchical structure that can be configured in various ways according to an operation mode of the electronic device.

BACKGROUND ART As the functions of electronic devices using object recognition technology have recently been advanced, the number of objects to be recognized by the electronic device is increasing. Accordingly, various object recognition technologies using artificial intelligence models are being researched and developed.

However, in the past, a technology for recognizing an object through a single artificial intelligence model has been used. As the number of objects to be recognized increases, the required memory size and computational consumption continue to increase. In addition, in the case of the existing technology, when performing object recognition, not only a specific model according to the situation is loaded onto the memory, but the entire artificial intelligence model is loaded onto the memory, so there is a disadvantage that the memory is used inefficiently. .

As a result, there is a limit to solving the object recognition ability of a user device, which has limitations in memory, computational capability, and communication capability, only by improving the function of one artificial intelligence model itself.

The present disclosure was devised according to the above-described necessity, and the present disclosure is that at least one recognition model having a hierarchical structure corresponding to an operation mode determined in an electronic device is selected and combined in various ways and loaded into a volatile memory, and the loaded It is to provide an electronic device and a control method thereof that determine an operation to be performed through a selected and combined hierarchical recognition model.

In an electronic device according to an embodiment of the present disclosure for achieving the above object, a camera, a nonvolatile memory for storing at least one instruction and a plurality of object recognition models, a volatile memory, and the nonvolatile memory, And a processor connected to the volatile memory and the camera to control the electronic device, the processor, when the operation mode of the electronic device is determined by executing the at least one instruction, the plurality of object recognition models At least one object recognition model having a hierarchical structure corresponding to the determined operation mode is loaded into the volatile memory, and the object image obtained through the camera is input to the loaded object recognition model, Information may be obtained, and an operation of the electronic device may be determined based on the information on the object.

Meanwhile, in a nonvolatile memory storing a plurality of object recognition models and a control method of an electronic device storing a volatile memory according to an embodiment for achieving the above object, when the operation mode of the electronic device is determined, the plurality of Loading at least one object recognition model having a hierarchical structure corresponding to the determined operation mode among object recognition models of, into the volatile memory, inputting an object image acquired through a camera into the loaded object recognition model, and the object It may include obtaining information on the object and determining an operation of the electronic device based on the information on the object.

In a method of constructing a recognition model according to an embodiment of the present disclosure, an electronic device identifies level information according to a requested operation mode, and includes a layered overall model structure having a plurality of levels stored in a nonvolatile memory and each operation. Using level information corresponding to a mode, a recognition model corresponding to a hierarchical model structure from the highest level to the identified level among the entire model structures is loaded into a volatile memory, and the operation is performed using the loaded recognition model. Mode can be performed.

In a method of configuring a recognition model according to an embodiment of the present disclosure, an electronic device identifies hierarchical template information and hierarchical configuration information corresponding to a requested operation mode, and Using the hierarchical structure template, a plurality of sub-models, and the hierarchical configuration information corresponding to each operation mode, a specific model is at a specific level of the hierarchical structure according to the identified hierarchical template information and the configuration information of the hierarchical structure. A recognition model corresponding to a layered model structure configured to be linked may be loaded into a volatile memory, and the operation mode may be performed using the loaded recognition model.

In the method of constructing a recognition model according to an embodiment of the present disclosure, the electronic device is loaded with a layered recognition model composed of a plurality of levels in a volatile memory, and the sub-models of each level extract feature values of input source data. It consists of a feature extraction unit and a classifier that classifies the feature values according to a specific criterion.The feature extraction unit of the upper level model is activated and the feature extraction unit of the lower level model is deactivated, so that the extracted value output from the submodel at the higher level is It can be used as it is in the model.

In a method of constructing a recognition model according to an embodiment of the present disclosure, when a new class is added in a layered recognition model composed of a plurality of levels, the new class may be defined by a user-defined or feature similarity comparison method. A specific sub-model of a specific location level to be added is determined, and the determined sub-model and at least a higher-level and a highest-level sub-model linked in association with the sub-model may be updated.

As described above, according to various embodiments of the present disclosure, the electronic device loads only the layered object recognition model corresponding to the determined operation mode into the volatile memory, and obtains information on the object through the loaded object recognition model. It is possible to reduce the amount of computation and memory consumption.

In addition, the electronic device according to the present disclosure learns only a model associated with an object recognition model corresponding to a new object in order to learn a new object, so that a user can learn and utilize an object recognition technology to which an artificial intelligence model is applied more quickly and efficiently. .

1A, 1B, 1C, and 1D are diagrams for explaining the configuration and operation of an electronic device for loading an object recognition model having a hierarchical structure into a volatile memory according to an embodiment of the present disclosure;
2 is a diagram illustrating in detail the configuration of an electronic device according to an embodiment of the present disclosure;
3A is a diagram for explaining the configuration and operation of an object recognition model according to an embodiment of the present disclosure;
3B is a diagram for explaining the configuration and operation of an object recognition model having a hierarchical structure according to an embodiment of the present disclosure;
4 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure;
5 is a diagram illustrating an operation of an electronic device when an operation mode of the electronic device is a first operation mode, according to an embodiment of the present disclosure;
6 is a diagram for explaining an operation of the electronic device when the operation mode of the electronic device is a second operation mode, according to an embodiment of the present disclosure;
7 is a diagram illustrating an operation of an electronic device when an operation mode of the electronic device is a third operation mode, according to an embodiment of the present disclosure;
8 is a diagram illustrating an operation of an electronic device when an operation mode of the electronic device is a first operation mode, according to an embodiment of the present disclosure;
9 is a diagram illustrating an operation of an electronic device when an operation mode of the electronic device is a second operation mode according to an embodiment of the present disclosure;
10 is a diagram illustrating an operation of an electronic device when an operation mode of the electronic device is a third operation mode, according to an embodiment of the present disclosure;
11 is a flowchart illustrating a learning method of an electronic device according to an embodiment of the present disclosure;
12 is a flowchart illustrating a learning method of an electronic device according to an embodiment of the present disclosure.

Hereinafter, embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present disclosure.

1A is a diagram illustrating a configuration and operation of an electronic device 100 for loading an object recognition model having a hierarchical structure into a volatile memory 130 according to an embodiment of the present disclosure.

As shown in FIG. 1A, according to an embodiment of the present disclosure, the electronic device 100 may include a camera 110, a nonvolatile memory 120, a volatile memory 130, and a processor 140. . However, the configuration illustrated in FIG. 1A is an exemplary diagram for implementing the embodiments of the present disclosure, and appropriate hardware and software configurations, which are obvious to those of ordinary skill in the art, may be additionally included in the electronic device 100.

The camera 110 is a component for acquiring one or more images of the surroundings of the electronic device 100. In one embodiment of the present disclosure, the camera 110 may acquire an image of the object by photographing an object existing around the electronic device 100. That is, in describing the present disclosure, the'image of an object' is an image of an object around the electronic device 100 obtained through the camera 110, and is used to refer to an image of an object input to the object recognition model. Is used.

Further, the camera 110 may be a configuration including a plurality of cameras, and may be variously implemented as an RGB camera, a 3D camera, a depth camera, and the like. Further, the camera 110 may be located in front of the electronic device 100, but this is only an example, and may be located behind or above the electronic device 100. In addition, the camera 110 may be located outside the electronic device 100 and electrically connected.

The nonvolatile memory 120 refers to a memory capable of maintaining stored information even when power supply is stopped. For example, the nonvolatile memory 120 may include at least one of a flash memory, a programmable read-only memory (PROM), a magnetic random-access memory (MRAM), and a resistive RAM (RRAM).

Volatile memory 130 refers to a memory requiring continuous power supply in order to maintain stored information. For example, the volatile memory 130 may include at least one of a dynamic random-access memory (DRAM) and a static RAM (SRAM).

In the following description of the present disclosure, a description will be made on the premise that the volatile memory 130 is a separate configuration from the processor 140, but this is only for clearly describing the operation of the electronic device 100 according to the present disclosure. , The volatile memory 130 according to the present disclosure may be implemented in a form included in the processor 140 as a component of the processor 140.

In particular, the nonvolatile memory 120 according to an embodiment of the present disclosure may store at least one instruction, a plurality of object recognition models 10-1 to 10-N, and operation mode data 20. . Meanwhile, an instruction is a programming language, which is an action statement for the electronic device 100 and is a minimum unit of a program that the electronic device 100 can directly execute.

In addition, each of the plurality of object recognition models 10-1 to 10-N is an artificial intelligence model capable of outputting information about an object using an image of an object acquired through the camera 110. Specifically, the object recognition model may output feature data of an object using an object image, and output information about an object based on the output feature data of the object. The information on the object may include information on which class the object is classified into among preset classes. A class is a collection of objects with the same or similar properties. Each of the plurality of object recognition models 10-1 to 10-N may store data on a class for classifying an object into a preset class. The configuration and operation of the object recognition model will be described in detail with reference to FIG. 3A.

Meanwhile, as shown in FIG. 1A, a plurality of object recognition models may be individually stored in the nonvolatile memory 120, but this is only an example, and a plurality of fixed hierarchical structures in the nonvolatile memory 120 An object recognition model having a (hierarchy structure) can be stored. The object recognition model having a fixed hierarchical structure is a model capable of classifying a specific object into one of preset classes while at least one individual object recognition model forms a hierarchical structure, and can be implemented in various ways according to the type of the electronic device 100. have. In addition, the hierarchical structure (or tree structure) refers to a data structure in which an upper-level node may have one or more lower-level nodes, but a lower-level node necessarily has one higher-level node.

Also, the operation mode data 20 may include information related to a hierarchical structure corresponding to each of a plurality of operation modes of the electronic device 100. In an embodiment of the present disclosure, the operation mode data 20 includes the total number of levels of the hierarchical structure corresponding to each operation mode, and the type of object recognition model that can be arranged at each level of the hierarchical structure corresponding to each operation mode. And information on the relationship between object recognition models. The association relationship between object recognition models may mean a connection relationship between a specific level in the hierarchical structure and an object recognition model that may be disposed at a lower level of the specific level. Accordingly, the information on the association relationship between the object recognition models may include information on a model that can be connected as a child node of the object recognition model of a specific level.

In addition, the operation mode data 20 may be constructed as a relational database, but this is only an embodiment and may be constructed in various ways. For example, the operation mode data 20 may be constructed as shown in Table 1 below.

3rd mode of operation Total number of levels 3 Types of models that can be placed on each level {L1:(A), L2:(B,C), L3:(D,E)} Association between each model Model connected to B as a child node: D
Model connected to C as a child node: E

Specifically, in the operation mode data 20 constructed as shown in Table 1, the total number of levels of the hierarchical structure corresponding to the third operation mode among the plurality of operation modes of the electronic device is 3, and the first level is A and the second level. Object recognition models such as B and C in the level and D and E in the third level may be arranged, and the model connected as a child node of B is D, and the model connected as a child node of C may include information such as E. . That the total number of levels is 3 may mean that the hierarchical structure is configured from the first level to the third level, which is the root level. On the other hand, in another embodiment of the present disclosure, the operation mode data 20 is a plurality of hierarchical templates, an index number capable of identifying each hierarchical structure template, and an operation mode corresponding to the operation mode, as shown in FIG. 1B. It may include configuration information 70 of a hierarchical structure. The hierarchical structure template is a template having a hierarchical structure so that a plurality of object recognition models can be linked to a specific node. For example, as shown in FIG. 1B, the operation mode data 20 has an index number of a hierarchical structure template configured by connecting two second level nodes 50-1 and 50-2 to a first level node. Information of number 1, of the hierarchy template composed of two third level nodes (60-1, 60-2) connected to the right node (50-2) among the second level nodes of the hierarchy template with index number 1. It may include information that the index number is number 2, and the like. Meanwhile, the hierarchical structure template shown in FIG. 1B (the hierarchical structure template with index numbers 1 to 3) is only an embodiment and can be implemented in various ways. In addition, it goes without saying that the hierarchical structure template can be added/deleted/changed by a user command. Meanwhile, a link may mean an operation of arranging an object recognition model at a specific node location of a specific level among the hierarchical structure templates.

Meanwhile, the hierarchical configuration information 70 includes information on the index number of the hierarchical structure template corresponding to each operation mode and the type of object recognition model that can be linked to each level of the hierarchical template having the index number. can do. The configuration information of the hierarchical structure may be constructed as a relational database as shown in FIG. 1B, but this is only an embodiment and may be constructed in various ways.

For example, as shown in FIG. 1B, the hierarchical configuration information is the index number of the hierarchy template corresponding to the first operation mode is 1, and the first level node of the first hierarchical template is A. The object recognition model can be linked, and B is to the node 60-1 of the second level connected to the left branch of the first level, and C is the node 60-2 of the second level connected to the right branch. May include information that is linked. As another example, as for the configuration information of the hierarchy, the index number of the hierarchy template corresponding to the second operation mode is 2, the object recognition model of A is used for the first level node, and the second level nodes 60-1,60 -2) may include information that object recognition models B and C may be loaded from the left, and object recognition models F and G may be loaded from the left to the third level nodes 70-1 and 70-2, respectively. .

Meanwhile, in an embodiment, the processor 140 may control the overall operation of the electronic device by executing at least one instruction. Specifically, the processor 140 may be connected to the camera 110, the nonvolatile memory 120, and the volatile memory 130 to control the overall operation of the electronic device 100. In describing the present disclosure, the processor 140 may be configured with one or a plurality of processors. At this time, the one or more processors 140 may be a general-purpose processor such as a CPU (Central Processing Unit), an AP (Application Processor), a graphics-processing unit (GPU), a graphics-processing unit (VPU), or an NPU ( It may be a processor dedicated to artificial intelligence such as Neural Processing Unit).

In particular, the processor 140 uses the operation mode data 20 stored in the nonvolatile memory 120 to generate at least one object recognition model having a hierarchical structure corresponding to the determined operation mode among a plurality of object recognition models in various ways. Configurable. In addition, as shown in FIG. 1A, the processor 140 may load at least one object recognition model having a structured hierarchical structure into the volatile memory 130. Meanwhile, loading refers to an operation of loading and storing data stored in the nonvolatile memory 120 into the volatile memory 130 so that the processor 140 can access it.

In an embodiment of the present disclosure, the processor 140 provides a hierarchical structure based on a template of a hierarchical structure identified through the operation mode data 20, an index number capable of identifying each hierarchical structure template, and configuration information of the hierarchical structure. You can construct an object recognition model with. For example, when the operation mode of the electronic device 100 is determined as the first operation mode, the processor 140 uses a hierarchy template corresponding to the first operation mode through the hierarchical configuration information among the operation mode data 20 It can be identified that the index number of is the first, and object recognition models of A to the first level and B and C to the second level of the first hierarchical template may be linked. Accordingly, as shown in FIG. 1B, the processor 140 links the model A to the first level of the first hierarchical structure template, and the models B and C based on the left node at the second level to create a hierarchical structure. The branch may configure an object recognition model and load the configured object recognition model into the volatile memory 130.

In another embodiment, the processor 140 includes the total number of levels of the hierarchical structure corresponding to the operation mode identified through the operation mode data 20 stored in the nonvolatile memory 120, and each level of the hierarchical structure. An object recognition model having a hierarchical structure may be configured based on information on the types of object recognition models that can be arranged and the association relationship between the object recognition models. For example, as disclosed in Table 1, when the operation mode of the electronic device 100 is determined as the third operation mode, the processor 140 performs a layer corresponding to the third operation mode through the operation mode data 20. It is identified that the total number of levels of the structure is 3, and in the third operation mode, the types of object recognition models that can be placed in the first to third levels and the association information of each object recognition model (e.g., Information on the object recognition model connected as a child node of each of the two-level object recognition models) can be identified.

In addition, as shown in (a) of FIG. 1C, the processor 140 includes a total number of levels (eg, 3) corresponding to the identified third operation mode and an object recognition model that can be disposed at each level. An object recognition model may be arranged for each level by using the type of (eg, A at the first level, B and C at the second level, and D and E at the third level). And, as shown in (b) of FIG. 1C, the processor 140 has a hierarchical structure by connecting the object recognition models arranged for each level by branches using information on the association relationship between the identified object recognition models. You can construct an object recognition model. In addition, the processor 140 may load the configured object recognition model into the volatile memory 130.

On the other hand, in another embodiment, when the operation mode of the electronic device 100 is determined, the processor 140 stores an object recognition model and operation mode data 20 having a plurality of fixed hierarchical structures stored in the nonvolatile memory 120. ), an object recognition model having a hierarchical structure corresponding to an operation mode may be loaded into the volatile memory 130. Specifically, when the operation mode of the electronic device 100 is determined, the processor 140 identifies the total number of levels of the hierarchical structure corresponding to the determined operation mode through the operation mode data 20 constructed as shown in Table 1 above. can do. In addition, the processor 140 may load an object recognition model layered from the first level to a level equal to the total number of identified levels among the object recognition models having a fixed hierarchical structure into the volatile memory 130. On the other hand, when the root node is implemented at the 0th level, the processor 140 is an object recognition model that is layered from the first level to a level that is one less than the total number of levels among object recognition models having a fixed hierarchical structure. May be loaded into the volatile memory 130. In one embodiment, as illustrated in FIG. 1D, the object recognition model 80 having a fixed hierarchical structure may be stored in the nonvolatile memory 120. In addition, when the operation mode of the electronic device 100 is determined as the third operation mode, the processor 140 determines that the total number of levels of the hierarchical structure corresponding to the third operation mode is 3 through the operation mode data 20. Can be identified. In addition, the processor 140 may load the recognition model layered from the first level to the identified third level among the hierarchical structure consisting of a total of four levels into the volatile memory 130. In the case of the above method, if only the total number of levels corresponding to the operation mode is identified without having to separately configure a hierarchical structure, the processor 140 generates an object recognition model having a hierarchical structure corresponding to the operation mode to the volatile memory 130 Can be loaded into

An embodiment of loading an object recognition model having a hierarchical structure corresponding to various operation modes of the electronic device 100 into the volatile memory 130 will be described in detail with reference to FIGS. 5 to 10.

Meanwhile, in an embodiment, the processor 140 may acquire feature data for the object by inputting the object image acquired through the camera 110 into the loaded object recognition model. Specifically, the processor 140 inputs an object image to an object recognition model of the highest level among the object recognition models having a hierarchical structure (or, an object recognition model corresponding to a root node in the hierarchical structure), and features of the object. Data can be acquired. The feature data of the object may be implemented in the form of a vector, but this is only an example and may be variously implemented as a matrix, a graph, or the like. In addition, the processor 140 may obtain information on the object based on feature data of the object. The information on the object may include information on which class the object is classified into among preset classes.

In addition, the processor 140 may identify an object recognition model to which feature data of an object is to be input among lower-level object recognition models based on the acquired information on the object. Specifically, when obtaining information that an object is classified as a first class through a higher-level object recognition model, the processor 140 determines that the lower-level object recognition model for inputting feature data of the object corresponds to the first class. It can be identified that it is an object recognition model. For example, referring to FIG. 1A, information about an object in the object recognition model 30-1 of the first level among the object recognition models having a hierarchical structure loaded in the volatile memory 130 (for example, Is classified as a first class), the processor 140 uses the acquired object information to correspond to the first class among the plurality of second level object recognition models 40-1 and 40-2. Object recognition model can be identified. In addition, the processor 140 may input feature data of the object into the object recognition model corresponding to the identified first class.

Meanwhile, the processor 140 may determine an operation of the electronic device 100 based on information on an object. Specifically, if the object recognition model that outputs information on an object is identified as a model located at an end node (or leaf node) in the hierarchy, the processor 140 is configured to perform an electronic device based on the information on the object. It is possible to determine the operation to be performed of (100). When only the object recognition model of the first level (or the root node) is loaded into the volatile memory 130, the processor 140 performs an electronic device based on information on the object acquired through the object recognition model of the first level. Of course, it is possible to determine the operation of (100). An embodiment in which the processor 140 controls the operation of the electronic device 100 based on information on objects acquired through a plurality of object recognition models will be described in detail with reference to FIGS. 5 to 10.

Meanwhile, according to an embodiment of the present disclosure, when the operation mode is determined as the learning mode, the processor 140 may train a plurality of object recognition models having a hierarchical structure in various ways.

In one embodiment, when the operation mode is the learning mode, when a new object image is acquired through the camera 110, the processor 140 may acquire feature data of the new object using at least one of a plurality of object recognition models. I can. The'new object' may include an object that cannot be classified with an accuracy exceeding a threshold value when the plurality of object recognition models classify the object into a preset class.

Further, the processor 140 may determine an object recognition model corresponding to the new object from among the plurality of object recognition models based on feature data of the new object and information on the plurality of object recognition models. Specifically, the processor 140 may obtain a similarity value between feature data of a new object and data for a class included in each of the plurality of object recognition models. In addition, the processor 140 may identify an object recognition model corresponding to data for a class having the highest similarity value as an object recognition model corresponding to a new object.

Further, the processor 140 may train an object recognition model that can be connected as a higher level of an object recognition model corresponding to a new object based on feature data of the new object. Specifically, the processor 140 may identify an object recognition model that can be connected as an ancestor node of an object recognition model corresponding to a new object through the operation mode data 20. In addition, the processor 140 may train an object recognition model that can be connected as an identified ancestor node. That is, the processor 140 may reduce the number of models to be trained by learning only an object recognition model corresponding to a new object and an object recognition model that can be connected as a higher level of the model, rather than training the entire object recognition model. . Meanwhile, an embodiment of learning an object recognition model based on a similarity value will be described in detail with reference to FIG. 11.

Meanwhile, in another embodiment, when the operation mode is a learning mode, the processor 140 may control the display 160 to display a user interface (UI) representing a plurality of stored object recognition models. And, while a new object image is acquired through the camera 110, if an object recognition model corresponding to the new object is selected by the user through the UI, the processor 140 inputs the new object image to the selected object recognition model and generates a new object. Image feature data can be obtained. Further, the processor 140 may learn an object recognition model selected based on feature data of a new object and an object recognition model that can be connected as a higher level of the selected object recognition model. An embodiment of training an object recognition model based on user selection will be described in detail with reference to FIG. 12.

Meanwhile, functions related to artificial intelligence according to the present disclosure are operated through the nonvolatile memory 120, the volatile memory 130, and the processor 140.

One or more processors 140 control to process input data according to a predefined operation rule or an artificial intelligence model stored in the nonvolatile memory 120 and the volatile memory 130. A predefined motion rule or an artificial intelligence model is characterized by being created through learning. Here, being made through learning means that a predefined operation rule or an artificial intelligence model of a desired characteristic is created by applying a learning algorithm to a plurality of training data. Such learning may be performed in a device on which artificial intelligence according to the present disclosure is performed, or may be performed through a separate server/system.

The artificial intelligence model may be composed of a plurality of neural network layers. Each layer has a plurality of weight values, and a layer operation is performed through the operation result of a previous layer and a plurality of weights. Examples of neural networks include CNN (Convolutional Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), BRDNN (Bidirectional Recurrent Deep Neural Network) and deep There are Q-Networks (Deep Q-Networks), and the neural network in the present disclosure is not limited to the above-described example except for the case where it is specified.

The learning algorithm is a method in which a predetermined target device (eg, a robot) is trained using a plurality of pieces of learning data so that a predetermined target device can make a decision or make a prediction by itself. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithm in this disclosure is specified. It is not limited to the above-described example except for.

2 is a diagram illustrating a detailed configuration of an electronic device 100 according to an exemplary embodiment of the present disclosure. As shown in FIG. 2, the electronic device 100 includes a camera 110, a nonvolatile memory 120, a volatile memory 130, a processor 140, a communication unit 150, a display 160, and a driving unit 170. ), a speaker 180, an input unit 190, and a sensor 195. Meanwhile, since the camera 110, the non-volatile memory 120, the volatile memory 130, and the processor 140 have been described in FIG. 1, redundant descriptions will be omitted.

The communication unit 150 includes a circuit and may perform communication with a server (not shown) or an external device (not shown). Specifically, the processor 140 may receive various data or information from a server (not shown) or an external device (not shown) connected through the communication unit 150, and the server (not shown) or an external device (not shown) It is also possible to transmit various data or information.

Meanwhile, the communication unit 110 may include various communication modules to perform communication with an external device. For example, the communication unit 110 may include a wireless communication module, for example, LTE, LTE-A (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications). system), WiBro (Wireless Broadband), 5G (5th generation), or GSM (Global System for Mobile Communications). As another example, the wireless communication module may include at least one of, for example, wireless fidelity (WiFi), Bluetooth, Bluetooth low power (BLE), and Zigbee.

The display 160 may display various types of information under the control of the processor 140. In an embodiment, the display 150 may display a UI representing a plurality of object recognition models under the control of the processor 140.

The display 160 may be implemented as a Liquid Crystal Display Panel (LCD), Organic Light Emitting Diodes (OLED), or the like, and may be implemented as a flexible display or a transparent display in some cases. In addition, the display 160 may be implemented as a touch screen together with a touch panel. However, it is not limited to the above-described implementation, and the display 160 may be implemented differently according to the type of the electronic device 100.

The driving unit 170 is a component that moves the electronic device 100 and may include a motor and a plurality of wheels. In particular, the driving unit 170 may move the electronic device 100 according to the control of the processor 140.

The speaker 160 is a component that outputs various notification sounds or voice messages as well as various audio data on which various processing tasks such as decoding, amplification, and noise filtering have been performed by an audio processing unit (not shown). When the operation of the electronic device 100 is completed, the speaker 160 may output a notification message indicating that the operation has been completed. In another embodiment, when the operation mode is the third operation mode and the human face included in the object image through the object recognition model of the third level is not a registered person, the speaker 160 controls the processor 140 You can print a notification message by

However, the speaker 160 is only an exemplary embodiment, and may be implemented as another output terminal capable of outputting audio data.

The input unit 190 includes a circuit, and the processor 140 may receive a user command for controlling the operation of the electronic device 100 through the input unit 190. The input unit 190 may include a touch sensor, a (digital) pen sensor, a pressure sensor, a key, or a microphone. The touch sensor may use at least one of, for example, a capacitive type, a pressure sensitive type, an infrared type, or an ultrasonic type.

In an embodiment, when the operation mode is a learning mode, the input unit 190 may receive an input for selecting an object recognition model corresponding to a new object from a user. In addition, the input unit 190 may transmit the received input to the processor 140.

The sensor 195 may detect various state information of the electronic device 100. For example, the sensor 195 is a sensor capable of detecting various physical quantities such as the presence and distance of a user or an object (e.g., an ultrasonic sensor, a proximity sensor, an optical sensor, an IR sensor, a UWB sensor, a LiDAR sensor, etc.) ), sensors that can detect surrounding environment information (e.g., temperature sensor, humidity sensor, barometric pressure sensor, etc.), sensors that can detect location information (e.g., GPS (Global Positioning System) sensor), etc. I can.

3A is a diagram illustrating a configuration and operation of an object recognition model 300 stored in a nonvolatile memory 120 according to an embodiment of the present disclosure. As shown in FIG. 3A, the object recognition model 300 may include a feature data extraction module 320 and a classifier module 330. In addition, each of the feature data extraction module 320 and the classifier module 330 may be controlled by the processor 140.

The feature data extraction module 320 may output feature data of an object through the input object image 310. However, this is only an exemplary embodiment, and the feature data extraction module 330 may output feature data for voice, text, and the like.

Meanwhile, the feature data extraction module 320 may be implemented as a CNN, but this is only an embodiment and may be implemented through various artificial neural networks such as DNN and RNN. In addition, the feature data of the object acquired by the feature data extraction module 320 may be implemented as a vector, a matrix, a graph, or the like.

The classifier module 330 may output information 340 on the object based on the feature data of the object obtained through the feature data extraction module 320. In an embodiment, the information on the object 340 may include information on which class the object is classified into among preset classes. That is, when classifying characteristic data of an object according to a preset condition, the Classifier module 330 may output information on which class the object belongs. In one embodiment, the classifier module 330 included in the object recognition model for identifying whether the object is an avoidance target is information on the result of classifying the object as an avoidance target or a non-avoidance target based on the acquired feature data of the object. Can be printed.

Meanwhile, the classifier module 330 may include data (not shown) for a class. The data for a class is a set of data that can classify an object into a plurality of classes according to a preset condition. Since the classifier module 330 included in each object recognition model has different preset conditions, data for classes may also be different, but this is an embodiment and may include data for overlapping classes. In an embodiment, a Classifier module included in an object recognition model for identifying whether an object is an avoidance target may include data capable of classifying an object into one of an avoidance target or a non-avoidance target. 3B is a diagram illustrating a configuration and operation of an object recognition model having a hierarchical structure according to an embodiment of the present disclosure.

As an example, as shown in FIG. 3B, the first level object recognition model 300-1 and the second level object recognition models 300-2 and 300-3 are layered by the electronic device 100 Thus, when loaded into the volatile memory 130, the object recognition model 300-1 of the first level may output information (not shown) on the object based on the input object image 310. Specifically, the object recognition model 300-1 of the first level extracts feature data of an object through an object image, and classifies the extracted feature data according to a preset condition. Can be printed.

In addition, the electronic device 100 may identify a model corresponding to information on the acquired object from among the object recognition models 300-2 and 300-3 of the second level. In addition, the electronic device 100 may identify a model corresponding to information on the acquired object as a model to input feature data of the object acquired through the object recognition model of the first level.

For example, when the first-level object recognition model 300-1 is a model that identifies whether an object is a target to be avoided, and outputs information that the object is classified as a target to be avoided, the electronic device 100 Among the object recognition models 300-2 and 300-3 of, an object recognition model capable of classifying an object to be avoided into a preset class may be identified. In addition, the electronic device 100 identifies an object recognition model capable of classifying the object to be avoided into a preset class as a model to input feature data of the object obtained through the object recognition model 300-1 of the first level. can do. In addition, the electronic device 100 may input feature data of an object to an object recognition model corresponding to a class in which the object is classified among the object recognition models 300-2 and 300-3 of the second level.

Meanwhile, in the case of an object recognition model having a hierarchical structure including at least two levels, the electronic device 100 activates only the feature data extraction module included in the top-level model, and features included in the remaining lower-level models. The data extraction module can be deactivated. The operation of deactivating the module may include an operation of controlling the module not to perform a specific operation. For example, referring to FIG. 3B, the electronic device 100 activates the feature data extraction module 320-1 included in the object recognition model 300-1 of the first level, and activates the object of the second level. The feature data extraction modules 320-2 and 320-3 included in the recognition models 300-2 and 300-3 may be deactivated.

Accordingly, the classifier modules 330-2 and 330-3 included in the model into which the object feature data is input among the second level object recognition models 300-2 and 300-3 Information 340-1 and 340-2 may be output. That is, the object recognition models 300-2 and 300-3 of the second level extract information on the object by using the feature data of the object output from the object recognition model of the first level, and feature included in each model. The data extraction modules 320-2 and 320-3 may be deactivated and may not perform an operation of extracting feature data of an object.

In addition, in an embodiment, when information on an object is output from an object recognition model located at an end node, the electronic device 100 may determine an operation based on the information on the object. In an embodiment, as shown in FIG. 3B, when the object recognition model of the second level, which outputs information on objects 340-1 and 340-2, is a model located at an end node in a hierarchical structure, the electronic device 100 may determine an operation based on information on an object.

4 is a diagram for describing a method of controlling the electronic device 100 according to an embodiment of the present disclosure.

First, when an operation mode of the electronic device 100 is determined, the electronic device 100 loads at least one object recognition model having a hierarchical structure corresponding to the determined operation mode among a plurality of object recognition models into the volatile memory 130 It can be done (S410). Specifically, the electronic device 100 constructs an object recognition model having a hierarchical structure corresponding to the operation mode in various ways by using a plurality of object recognition models and operation mode data stored in the nonvolatile memory 120, and The recognition model may be loaded into the volatile memory 130.

According to an embodiment of the present disclosure, the electronic device 100 includes the total number of levels of the hierarchical structure corresponding to the operating mode through the operation mode data stored in the nonvolatile memory 120, to be disposed at each level of the hierarchical structure. It is possible to identify the types of object recognition models that can be used and information on the relationship between the object recognition models. In addition, the electronic device 100 may construct an object recognition model having a hierarchical structure corresponding to an operation mode through each of the identified information and load it into the volatile memory 130. Since a specific embodiment according to the above method has been described with reference to Table 1 and FIG. 1D, redundant descriptions will be omitted.

Meanwhile, in another embodiment, the electronic device 100 may identify a plurality of hierarchical structure templates among operation mode data, an index number for identifying each hierarchical structure template, and configuration information of a hierarchical structure corresponding to the operation mode. have. In addition, the electronic device 100 may construct an object recognition model having a hierarchical structure corresponding to an operation mode through each of the identified information and load it into the volatile memory 130. Since a specific embodiment according to the above method has been described in detail with reference to FIG. 1B, redundant descriptions will be omitted.

In another embodiment, the electronic device 100 may identify the total number of levels corresponding to the operation mode through operation mode data. In addition, the electronic device 100 volatile the object recognition model layered from the first level to the total number of levels identified among the plurality of object recognition models having a fixed hierarchical structure stored in the nonvolatile memory 120. It can be loaded into the memory 130. Since the above embodiment has been described in detail with reference to FIG. 1D, redundant descriptions will be omitted.

Further, the electronic device 100 may obtain information on the object by inputting the object image acquired through the camera 110 into the loaded object recognition model (S420). In more detail, the electronic device 100 may acquire feature data of the object by inputting the acquired object image into the object recognition model of the highest level. In addition, the electronic device 100 may obtain information on the object based on feature data of the object. When the object recognition model that outputs information on an object is not a model corresponding to an end node in the hierarchical structure, the electronic device 100 inputs feature data of the object among the object recognition models at a lower level based on the information on the object. You can identify the object recognition model to be used.

Further, the electronic device 100 may determine an operation based on information on the object (S430). Specifically, when the object recognition model outputting information on an object is a model corresponding to an end node in a hierarchical structure, the electronic device 100 may determine an operation based on the information on the object.

5 to 10 are diagrams for explaining the structure and operation of an object recognition model having a hierarchical structure loaded into the volatile memory 130 according to each operation mode according to an embodiment of the present disclosure. A process of loading a plurality of object recognition models stored in the nonvolatile memory 120 by the electronic device 100 into the volatile memory 130 has been described in detail in FIG. 1, and thus, a redundant description will be omitted.

5 to 7 are examples when the electronic device 100 is implemented as a cleaning robot, and FIGS. 8 to 10 are examples when the electronic device 100 is implemented as a retail robot.

Meanwhile, the electronic device 100 according to an embodiment of the present disclosure may include at least one of a smartphone, a tablet PC, a desktop PC, a laptop PC, a netbook computer, a server, a PDA, a medical device, or a wearable device. . In some embodiments, the electronic device 100 may include at least one of a television, a refrigerator, an air conditioner, an air purifier, a set-top box, various robots, and a media box (eg, Samsung HomeSyncTM, Apple TVTM, or Google TVTM). It is limited to this.

The plurality of operation modes may be variously implemented according to the type of the electronic device 100, and object recognition models corresponding to each operation mode may also be variously implemented. Accordingly, the plurality of object recognition models having a hierarchical structure are not limited to those illustrated in FIGS. 5 to 10.

5 is a diagram for explaining an object recognition model loaded into the volatile memory 130 when the electronic device 100 is implemented as a cleaning robot and an operation mode is determined as a first operation mode, according to an embodiment of the present disclosure to be.

When the operation mode is determined to be the first operation mode, which is the general cleaning mode, the electronic device 100 uses the first level object recognition model to generate an avoidance identification model 500 for identifying whether an object is an object to be avoided, as a volatile memory 130. ) Can be loaded. The electronic device 100 may input the object image 310 acquired through the camera 110 into the avoidance identification model 500. The avoidance identification model 500 may output feature data of an object based on the input object image. In addition, the avoidance identification model 500 may output information as a result of classifying the object as an avoidance target or a non-avoidance target based on feature data of the object. Accordingly, the electronic device 100 may identify whether an object existing around the electronic device 100 is an avoidance target through result information output from the avoidance identification model 500.

In addition, when the operation mode is the general cleaning mode, the object recognition model loaded in the volatile memory 130 is only one avoidance identification model 500, which is a first-level object recognition model, so that the electronic device 100 is the avoidance identification model. The operation may be determined based on the result information obtained from 500. If the result information includes information that the object is the object to be avoided, the electronic device 100 may avoid the object and move (510). Meanwhile, when the result information includes information that the object is a non-avoidant target, the electronic device 100 may start cleaning while moving around the object without avoiding the object (520 ).

6 is a diagram for explaining an object recognition model loaded into the volatile memory 130 when the electronic device 100 is implemented as a cleaning robot and an operation mode is determined as a second operation mode, according to an embodiment of the present disclosure to be.

When the operation mode is determined to be a specific location cleaning mode, which is the second operation mode, the electronic device 100 loads the avoidance identification model 500 for identifying whether the object is an avoidance target as an object recognition model of the first level, and As the object recognition model of the second level as a lower level of the first level, the object type identification models 600 and 610 capable of identifying the type of the object may be loaded into the volatile memory 130. In an embodiment, as shown in FIG. 1B, when the second operation mode is determined, the electronic device 100 determines the hierarchy template corresponding to the second operation mode through the hierarchical configuration information among the operation mode data. It can be identified that it is the No. 1 template. In addition, the electronic device 100 includes an avoidance identification model 500 at the first level of the first template, and a model 600 for identifying the type of the object to be avoided at the second level through the configuration information of the hierarchical structure. It can be identified that a model identifying the type of object to be avoided can be linked. Accordingly, the electronic device 100 may load the object recognition model having a hierarchical structure into the volatile memory 130 as illustrated in FIG. 6 based on the identified information. On the other hand, since the embodiment related to the avoidance identification model 500 has been described with reference to FIG. 5, redundant descriptions will be omitted.

The electronic device 100 includes the characteristics of the object obtained through the avoidance identification model 500 among the plurality of second level object recognition models 600 and 610 based on the result information output from the avoidance identification model 500. It is possible to identify an object recognition model into which data is to be input. In one embodiment, upon obtaining information that an object is classified as a class called an object to be avoided, the electronic device 100 selects the type of the object to be avoided corresponding to the classified class among the object recognition models 600 and 610 of the second level. The identifying model 600 may be identified as a model to which feature data of an object is to be input. As another example, when obtaining information that an object is classified as a non-avoidant object, the electronic device 100 determines the type of the non-avoidable object corresponding to the classified class among the object recognition models 600 and 610 of the second level. The model 610 that identifies the object may be identified as a model to which feature data of an object is to be input.

According to an embodiment of the present disclosure, each object type identification model 600 and 610 may output information as a result of classifying an object type based on the input feature data of the object. Accordingly, the electronic device 100 may identify the type of the object through the respective object type identification models 600 and 610.

Meanwhile, while the operation mode is determined as the specific location determination mode, the electronic device 100 may obtain information on the specific location from the user. In an embodiment, the electronic device 100 may receive a user's voice (eg,'clean around the sofa') including information on a specific location from the user. In another embodiment, the electronic device 100 may display a UI for selecting a specific location, and receive a signal for selecting a specific location from a user through the displayed UI.

And, as an embodiment, when the identified object corresponds to a specific location cleaning mode, the electronic device 100 may perform an operation corresponding to the specific location cleaning mode. That is, when the identified object corresponds to the object requested to be cleaned by the user, the electronic device 100 may start cleaning around the identified object. For example, when the operation mode is determined as a specific location cleaning mode and a command to clean the surroundings of the sofa is input from the user, if the object is identified as a sofa, the electronic device 100 may start cleaning at the location of the sofa (620). ). As another example, if the object is identified as an object other than the sofa, the electronic device 100 may avoid the object and move until the sofa object is identified (630 ). FIG. 7 illustrates an embodiment of the present disclosure. Accordingly, a diagram for explaining an object recognition model loaded in the volatile memory 130 when the electronic device 100 is implemented as a cleaning robot and the operation mode is determined as the third operation mode.

According to an embodiment of the present disclosure, when the operation mode is determined to be the security mode, which is the third operation mode, the electronic device 100 determines whether an object, which is a first-level object recognition model, is an avoidance object. 500), an object type identification model that can identify the type of an object as a second level object recognition model that is a lower level of the first level, and a third level object recognition model that is a lower level of the second level. A face recognition model 700 capable of recognizing a face may be loaded into the volatile memory 130. Since the embodiments related to the avoidance identification model 500 and the object type identification model 600 have been described with reference to FIGS. 5 and 6, overlapping descriptions will be omitted.

Meanwhile, when the operation mode is determined as the security mode, the electronic device 100 is a second-level object recognition model among a plurality of object recognition models having a hierarchical structure, and the model 610 for identifying the type of the non-avoidable object is volatile. It may not be loaded into the memory 130. Specifically, the security mode may be a mode for identifying whether the electronic device 100 is a registered person through the face recognition model 700, which is a third-level object recognition model. In the case of a non-avoidable object whose type cannot be a person, it is not necessary to identify the type, so when the operation mode is a security mode, the model 610 that identifies the type of the non-avoidable object is layered. As an object recognition model of the second level in the structure, the volatile model 130 may not be loaded.

As an embodiment, as shown in FIG. 7, when the type of the object is identified as human through the object recognition model 600 for identifying the type of the object to be avoided, the electronic device 100 determines whether the avoidance identification model 500 The feature data of the object obtained in) may be input to the face recognition model 700, which is a third-level object recognition model.

Meanwhile, the face recognition model 700 according to an embodiment may output information as a result of classifying a human face included in an object image as a pre-registered human face or an unregistered human face based on feature data of the object. . Accordingly, the electronic device 100 may identify whether a human face included in the object image is a pre-registered human face through the face recognition model 700.

In addition, as an embodiment, when it is identified that the human face included in the object image is not a pre-registered human face, the electronic device 100 may provide a notification message (720). For example, the electronic device 100 may transmit a message indicating that the face of an unregistered person has been identified to a preset person or institution (eg, a person registered in advance, a security company, a police officer, etc.). As another example, the electronic device 100 may output an alarm sound indicating that an unregistered person has been identified.

Meanwhile, as an embodiment, when a human face included in the object image is identified as a pre-registered human face, the electronic device 100 may perform the security mode while moving while avoiding the identified human face (operation 710).

And, as shown in FIG. 7, when the object is identified as a non-avoidant object through the avoidance identification model 500 or the object type identification model 600 identifies the type of the object to be avoided other than a person, the electronic device Reference numeral 100 may perform the security mode while moving while avoiding the identified object (730).

FIG. 8 is a diagram for explaining an object recognition model loaded into the volatile memory 130 when the electronic device 100 is implemented as a retail robot and an operation mode is determined as a first operation mode, according to an embodiment of the present disclosure to be.

When the operation mode is determined as the first operation mode, the busy mode, the electronic device 100 uses an object recognition model of a first level to determine whether an object is an object to be avoided, and the evasion identification model 800 is used in the volatile memory 130. Can be loaded into

In an embodiment, the electronic device 100 may identify whether the object is a non-avoidant guest or an avoidance target through the avoidance identification model 800. Specifically, the electronic device 100 may input the image 310 of the object into the avoidance identification model 800. In addition, the avoidance identification model 800 may output feature data of the object based on the image 310 of the object. In addition, the avoidance identification model 800 may output information as a result of classifying the object as a non-avoidant target guest or an avoidance target based on feature data of the object. In an embodiment, when it is identified that the object is not a person, the avoidance identification model 800 may output information as a result of classifying the object as an object to be avoided. In another embodiment, when the object is identified as a registered person such as a store employee, the avoidance identification model 800 may classify the object as an object to be avoided and output information as a result of classifying the object. Accordingly, the electronic device 100 may identify whether the object is a non-avoidable guest or an avoided target based on result information output from the avoidance identification model 800.

In addition, when the operation mode is the Busy mode, the object recognition model loaded in the volatile memory 130 is one avoidance identification model 800, which is a first-level object recognition model, so the electronic device 100 is the avoidance identification model. An operation to be performed may be determined based on result information obtained from 800. When the result information includes information that the object is a non-avoidant customer, the electronic device 100 performs a customer response operation (for example, an operation of displaying a UI providing information on a store location, price information of an item, etc.) Can do it (810). Meanwhile, when the result information includes information indicating that the object is the object to be avoided, the electronic device 100 may avoid the object and move (820).

9 is a diagram for explaining an object recognition model loaded into the volatile memory 130 when the electronic device 100 is implemented as a retail robot and an operation mode is determined as a second operation mode, according to an embodiment of the present disclosure to be.

When the operation mode is determined to be the second operation mode, the general mode, the electronic device 100 loads the avoidance identification model 800 for identifying whether the object is an avoidance target as a first level object recognition model, and the first As an object recognition model of a second level, which is a lower level of the level, a model capable of identifying a type of an object to be non-avoidable may be loaded into the volatile memory 130. The model capable of identifying the type of object that is to be avoided may be implemented as an age group recognition model 910 capable of recognizing the age of the customer, but this is only an example, and the model for recognizing the height of the customer and the customer's belongings It can be implemented in various ways such as models. Since the embodiment related to the avoidance identification model 800 has been described with reference to FIG. 5, redundant descriptions will be omitted.

As shown in FIG. 9, since the second level object recognition model is one age group recognition model 910, when an object is identified as a non-avoidant customer, the electronic device 100 uses the avoidance identification model 800. The acquired feature data of the object may be input into the age group recognition model 910.

In addition, the electronic device 100 may recognize the age group of the customer, which is an object, through the age group recognition model 910. As an embodiment, as shown in FIG. 9, the age group recognition model 910 may output information as a result of classifying the age group of a guest into elderly, adult, child, etc. based on the input feature data of the object. Accordingly, the electronic device 100 may recognize the age group of the customer based on the result information obtained through the age group recognition model 910.

And, in the case of the general mode, since the age group recognition model 910, which is the second level object recognition model, corresponds to an end node in the hierarchical structure, the electronic device 100 results in a result obtained through the age group recognition model 910. Actions can be determined based on the information. In an embodiment, when the age of the customer is identified as the elderly, the electronic device 100 may perform an easy and detailed customer response operation (920 ). As another example, when the age of the guest is identified as an adult, the electronic device 100 may perform a preset general customer response operation (930 ). As another example, when the age group of the guest is identified as a child, the electronic device 100 may perform a guest response operation with an agitated voice (940 ). Meanwhile, if the object is identified as the object to be avoided through the avoidance identification model 800, the electronic device 100 may move by avoiding the object (820 ).

10 illustrates a hierarchical object recognition model loaded in the volatile memory 130 when the electronic device 100 is implemented as a retail robot and an operation mode is determined as a third operation mode, according to an embodiment of the present disclosure It is a drawing to do.

As an embodiment of the present disclosure, as shown in FIG. 10, when the operation mode is determined to be the VIP mode, which is the third operation mode, the electronic device 100 identifies whether the object is an object to be avoided using a first-level object recognition model. The avoidance identification model 800, which is a second level object recognition model that is a lower level of the first level, and a first VIP recognition model 1000 that identifies whether a guest is a pre-registered VIP guest, and a lower level of the second level. As an object recognition model of a third level, which is a level, a second VIP model 1010 and an age group recognition model 910 capable of identifying individual VIP guests may be loaded into the volatile memory 130.

As shown in FIG. 10, since the object recognition model of the second level is one first VIP recognition model 1000, when the object is identified as a non-avoidant customer, the electronic device 100 is the evasion identification model 800. The feature data of the object acquired through may be input into the first VIP model 1000.

And, in detail, the electronic device 100 may identify whether the guest is a pre-registered VIP guest through the first VIP model 1000. In an embodiment, the first VIP recognition model 1000 may output information as a result of classifying a guest, which is an object, as a VIP guest or a non-VIP guest based on the input feature data of the object. Accordingly, the electronic device 100 may identify whether the guest is a pre-registered VIP guest based on result information obtained through the first VIP recognition model 1000.

And, as an embodiment, the electronic device 100 is an object acquired through the avoidance identification model 800 among the object recognition models 910 and 1010 of the third level based on whether the guest is a pre-registered VIP guest. It is possible to identify an object recognition model to which the feature data of is to be input. In an embodiment, when information indicating that a guest, which is an object, has been classified into a class called a pre-registered VIP guest, is obtained through the first VIP recognition model 1000, the electronic device 100 is configured with the object recognition model 910 of the third level. , 1010), the feature data of the object may be identified as the second VIP recognition model 1010. As another example, when information indicating that a guest, which is an object, has been classified as a class called a general guest, not a previously registered VIP guest, is obtained through the second VIP recognition model 1010, the electronic device 100 recognizes the object of the third level. Among the models 910 and 1010, feature data of an object may be identified as an age group recognition model 910. In addition, the electronic device 100 may input feature data to a model that has identified that the feature data is an object recognition model to be input.

Meanwhile, in an embodiment, when the guest is identified as a VIP, the electronic device 100 may identify individual VIP guests by inputting feature data into the second VIP recognition model 1010. Specifically, the second VIP recognition model 1010 may output result information on which customer is classified as a customer based on the input feature data of the object. Accordingly, the electronic device 100 may identify which of the previously registered VIP guests is a guest included in the object image based on the result information obtained from the second VIP recognition model 1010.

Further, since the second VIP recognition model 1010 is a model corresponding to an end node in a hierarchical structure, the electronic device 100 may determine an operation based on result information obtained through the second VIP recognition model 1010. . In an embodiment, the electronic device 100 may perform a response operation corresponding to the identified pre-registered VIP guest (1020, 1030, and 1040). For example, when the object is identified as a first VIP guest, the electronic device 100 performs an operation corresponding to the first VIP guest (for example, an operation in which the first VIP displays information on an item related to a recently purchased item, etc.). ) Can be performed (1020).

Meanwhile, in an embodiment, when a guest, which is an object, is identified as a general guest rather than a registered VIP guest, the electronic device 100 recognizes the age of the guest by inputting characteristic data of the object into the age group recognition model 910. And, it is possible to perform an operation corresponding to the recognized age group. Since the embodiment related to the age group recognition model 910 has been described with reference to FIG. 9, redundant descriptions will be omitted.

11 illustrates an embodiment in which the electronic device 100 additionally learns an object recognition model based on class data that each object recognition model can classify when an image of an object is input according to an embodiment of the present disclosure. This is a flow chart for explanation.

First, when a new object image is acquired through a camera when the operation mode is a learning mode, the electronic device 100 may acquire feature data of the new object (S1110). Specifically, the electronic device 100 may acquire feature data of a new object by using one of a plurality of object recognition models.

Further, the electronic device 100 may determine an object recognition model corresponding to the new object from among the plurality of object recognition models based on feature data of the new object and information on the plurality of object recognition models. Specifically, the electronic device 100 may acquire a similarity between feature data of a new object and data for a class included in each of a plurality of object recognition models stored in the nonvolatile memory 120 (S1120). ). Further, the electronic device 100 may identify and determine an object recognition model including data on a class having the highest similarity among the plurality of object recognition models as an object recognition model corresponding to the new object (S1130).

Further, the electronic device 100 may learn an object recognition model corresponding to a new object and an object recognition model that can be connected as a higher level of the model based on feature data of the new object (S1140). That is, the electronic device 100 may increase the number of classes capable of classifying an object by learning the determined object recognition model. Specifically, the electronic device 100 may identify an object recognition model that can be connected as a higher level of an object recognition model corresponding to a new object through operation mode data stored in the nonvolatile memory 120. In addition, the electronic device 100 may learn not the entire object recognition model, but only an object recognition model corresponding to a new object and an object recognition model that can be connected as a higher level of the model.

Meanwhile, in another embodiment of the present disclosure, the electronic device 100 simultaneously or thresholds an object recognition model that can be connected as a higher level and a lower level of an object recognition model corresponding to a new object based on feature data of a new object. It can be learned within a range of time. For example, if an object recognition model corresponding to a new object is not a model to be placed in a leaf node in the hierarchical structure, the electronic device 100 may determine the object recognition model corresponding to the new object through operation mode data. An object perception model that can be connected as a higher level and a lower level that can be connected may be identified, and the identified object recognition model may be trained based on feature data of the object.

12 is a flowchart illustrating a process in which the electronic device 100 learns an object recognition model selected by a user based on feature data of a new object according to an embodiment of the present disclosure.

First, when the operation mode is a learning mode, the electronic device 100 may display a UI representing a plurality of object recognition models stored in the volatile memory 120 (S1210). Specifically, in a UI representing a plurality of object recognition models, each object recognition model may be implemented in the form of an icon or text, but is not limited thereto. In addition, as an embodiment, the electronic device 100 may display a UI representing a plurality of individual object recognition models, but this is only an example, and the electronic device 100 may display a plurality of fixed object recognition models. UI can be displayed.

In addition, when an object recognition model corresponding to a new object among a plurality of object recognition models is selected by the user while a new object image is acquired through the camera, the electronic device 100 selects the new object image to the selected object recognition model. By inputting, feature data of a new object may be obtained (S1220). In addition, when the name of the class corresponding to the new object is input by the user, the electronic device 100 may identify the acquired feature data as data on the class corresponding to the new object for which the name is input (S1230). For example, when a model identifying the type of an object to be avoided is selected from the user, and the name of the class corresponding to the new object is input as'air purifier', the electronic device 100 selects data corresponding to the new object. It is acquired through a model, and the acquired data can be identified as data for a class named'air purifier'.

Further, the electronic device 100 may learn an object recognition model selected based on data on a class corresponding to a new object and an object recognition model that can be connected as a higher level of the selected object recognition model (S1240). Specifically, the electronic device 100 identifies an object recognition model that can be connected as a higher level of the selected object recognition model through operation mode data, and matches the selected object recognition model and the identified object recognition model to the new object. It can be trained based on the data for the class being used.

On the other hand, in another embodiment, when the selected object recognition model is not a model to be placed at the end node, the electronic device 100 recognizes an object that can be connected as a higher level and a lower level of the selected object recognition model through operation mode data. Model can be identified. In addition, the electronic device 100 may learn the selected object recognition model and an object recognition model that can be connected as upper and lower levels of the model based on data on a class corresponding to a new object.

Meanwhile, the drawings attached to the present disclosure are not intended to limit the technology described in the present disclosure to a specific embodiment, and various modifications, equivalents, and/or alternatives of the embodiments of the present disclosure are provided. It should be understood to include. In connection with the description of the drawings, similar reference numerals may be used for similar elements.

In the present disclosure, expressions such as "have," "may have," "include," or "may include" are the presence of corresponding features (eg, elements such as numbers, functions, actions, or parts). And does not exclude the presence of additional features.

In the present disclosure, expressions such as "A or B," "at least one of A or/and B," or "one or more of A or/and B" may include all possible combinations of items listed together. . For example, "A or B," "at least one of A and B," or "at least one of A or B" includes (1) at least one A, (2) at least one B, Or (3) it may refer to all cases including both at least one A and at least one B.

Expressions such as "first," "second," "first," or "second," used in the present disclosure may modify various elements regardless of order and/or importance, and It is used to distinguish it from other components, but does not limit the components.

Some component (eg, the first component) is “(functionally or communicatively) coupled with/to)” to another component (eg, the second component) or “ When referred to as "connected to", it should be understood that the certain component may be directly connected to the other component or may be connected through another component (eg, a third component). On the other hand, when a component (eg, a first component) is referred to as being “directly connected” or “directly connected” to another component (eg, a second component), the component and the It may be understood that no other component (eg, a third component) exists between the different components.

The expression "configured to (configured to)" used in the present disclosure is, for example, "suitable for," "having the capacity to" depending on the situation. ," "designed to," "adapted to," "made to," or "capable of". The term "configured to (or set)" may not necessarily mean only "specifically designed to" in hardware. Instead, in some situations, the expression "a device configured to" may mean that the device "can" along with other devices or parts. For example, the phrase "a subprocessor configured (or configured) to perform A, B, and C" refers to a dedicated processor (eg, an embedded processor) for performing the operation, or executing one or more software programs stored in a memory device. By doing so, it may mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

An electronic device according to various embodiments of the present disclosure may include, for example, at least one of a smartphone, a tablet PC, a desktop PC, a laptop PC, a netbook computer, a server, a PDA, a medical device, or a wearable device. In some embodiments, the electronic device may include at least one of, for example, a television, a refrigerator, an air conditioner, an air purifier, a set-top box, and a media box (eg, Samsung HomeSyncTM, Apple TVTM, or Google TVTM).

Meanwhile, the term user may refer to a person using an electronic device or a device (eg, an artificial intelligence electronic device) using an electronic device. Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium (eg, a computer). The device receives instructions stored from the storage medium. A device capable of calling and operating according to the called command, may include an electronic device (eg, the electronic device 100) according to the disclosed embodiments. When the command is executed by a processor, the processor directly, or A function corresponding to the instruction may be performed using other components under the control of the processor, and the instruction may include a code generated or executed by a compiler or an interpreter. It can be provided in the form of a non-transitory storage medium, where the term'non-transitory storage medium' means that it does not contain a signal and is tangible, but the data is semi-permanent in the storage medium. Or, it does not distinguish that the data is temporarily stored, for example, the'non-transitory storage medium' may include a buffer in which data is temporarily stored.

According to an embodiment, a method according to various embodiments disclosed in the present disclosure may be provided by being included in a computer program product. Computer program products can be traded between sellers and buyers as commodities. The computer program product may be distributed online in the form of a device-readable storage medium (eg, compact disc read only memory (CD-ROM)) or through an application store (eg, Play StoreTM). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) is at least temporarily stored in a storage medium such as a server of a manufacturer, a server of an application store, or a memory of a relay server, or temporarily Can be created with

Each of the constituent elements (eg, modules or programs) according to various embodiments may be composed of a singular or plural entity, and some sub-elements of the above-described sub-elements are omitted, or other sub-elements are various. It may be further included in the embodiment. Alternatively or additionally, some constituent elements (eg, a module or a program) may be integrated into a single entity, and functions performed by each corresponding constituent element prior to the consolidation may be performed identically or similarly. Operations performed by modules, programs, or other components according to various embodiments may be sequentially, parallel, repetitively or heuristically executed, or at least some operations may be executed in a different order, omitted, or other operations may be added. I can.

110: camera 120: non-volatile memory
130: volatile memory 140: processor

Claims (20)

In the electronic device,
camera;
A nonvolatile memory for storing at least one instruction and a plurality of object recognition models;
Volatile memory; And
A processor connected to the nonvolatile memory, the volatile memory, and the camera to control the electronic device; and
The processor, by executing the at least one instruction,
When the operation mode of the electronic device is determined, at least one object recognition model having a hierarchy structure corresponding to the determined operation mode among the plurality of object recognition models is loaded into the volatile memory,
Inputting the object image acquired through the camera into the loaded object recognition model to obtain information on the object,
An electronic device that determines an operation of the electronic device based on information on the object. The method of claim 1,
The processor,
When the operation mode is determined as the first operation mode, an object recognition model of a first level for identifying whether the object is an avoidance object among the plurality of object recognition models is loaded into the volatile memory,
An electronic device that identifies whether the object is a target to be avoided based on feature data of the object obtained by inputting the object image to the first level object recognition model. The method of claim 2,
The processor,
When the operation mode is determined as the second operation mode, the object recognition model of the first level and at least one of the second level identifying the type of the object as a lower level of the first level among the plurality of object recognition models Load an object recognition model into the volatile memory,
An electronic device for identifying an object recognition model to which feature data of the object is to be input from among at least one object recognition model of the second level, based on whether the object is a target to be avoided. The method of claim 3,
The processor,
An electronic device for identifying the type of the object by inputting feature data of the object acquired through the first level object recognition model into the determined second level object recognition model. The method of claim 4,
The processor,
When the identified object corresponds to a second operation mode, the electronic device controls the electronic device to perform an operation corresponding to the second operation mode. The method of claim 4,
The processor,
When the operation mode is determined as the third operation mode, the first and second level object recognition models among the plurality of object models and at least a third level capable of recognizing a human face as a lower level of the second level An electronic device that loads one object recognition model into the volatile memory. The method of claim 6,
The processor,
When the type of the object is identified as human through the second-level object recognition model, a person whose human face included in the object image is pre-registered by inputting feature data of the object into the third-level object recognition model An electronic device that identifies whether it is a face. The method of claim 7,
The processor,
When it is identified that the human face is not a pre-registered human face, a notification message is provided,
When the human face is identified as a pre-registered human face, the electronic device controls the electronic device to move while avoiding the identified person. The method of claim 1,
The processor,
When the operation mode is a learning mode, when a new object image is obtained through the camera, feature data of the new object is obtained,
Determine an object recognition model corresponding to the new object from among the plurality of object recognition models based on feature data of the new object and information on the plurality of object recognition models,
An electronic device that trains the determined object recognition model based on feature data of the new object and an object recognition model that can be connected as a higher level of the determined object recognition model. The method of claim 1,
Including; a display;
The processor,
When the operation mode is a learning mode, controlling the display to display a user interface (UI) representing the plurality of stored object recognition models,
When an object recognition model corresponding to the new object is selected by the user through the UI while the image of a new object is acquired through the camera, the new object image is input to the selected object recognition model, and feature data of the new object To get
When a name of a class corresponding to the new object is input from the user, feature data of the new object is identified as data on a class corresponding to the new object with the name input,
An electronic device that trains the selected object recognition model and an object recognition model that can be connected as a higher level of the selected object recognition model based on data on a class corresponding to the new object. In the nonvolatile memory storing a plurality of object recognition models and a control method of an electronic device storing the volatile memory,
When an operation mode of the electronic device is determined, loading at least one object recognition model having a hierarchical structure corresponding to the determined operation mode among the plurality of object recognition models into the volatile memory;
Inputting an object image acquired through a camera into the loaded object recognition model to obtain information on the object; And
And determining an operation of the electronic device based on the information on the object. The method of claim 11,
The loading step,
If the operation mode is determined as the first operation mode, loading an object recognition model of a first level to identify whether the object is an avoidance object among the plurality of object recognition models into the volatile memory; And
And identifying whether the object is a target to be avoided based on feature data of the object obtained by inputting the object image to the object recognition model of the first level. The method of claim 12,
The loading step,
When the operation mode is determined as the second operation mode, the object recognition model of the first level and at least one of the second level identifying the type of the object as a lower level of the first level among the plurality of object recognition models Loading an object recognition model into the volatile memory; And
Identifying an object recognition model to which feature data of the object is to be input from among at least one object recognition model of the second level based on whether the object is a target to be avoided; The method of claim 13,
The determining step,
And identifying the type of the object by inputting feature data of the object acquired through the first level object recognition model into the determined second level object recognition model. The method of claim 14,
The identifying step,
And when the identified object corresponds to a second operation mode, controlling the electronic device to perform an operation corresponding to the second operation mode. The method of claim 14,
The loading step,
When the operation mode is determined as the third operation mode, the first and second level object recognition models among the plurality of object models and at least a third level capable of recognizing a human face as a lower level of the second level And loading one object recognition model into the volatile memory. The method of claim 16,
The determining step,
When the type of the object is identified as human through the second-level object recognition model, a person whose human face included in the object image is pre-registered by inputting feature data of the object into the third-level object recognition model The method of controlling an electronic device comprising: identifying whether it is a face or not. The method of claim 17,
The determining step,
When it is identified that the human face is not a pre-registered human face, a notification message is provided,
And if the human face is identified as a pre-registered human face, controlling the electronic device to move while avoiding the identified person. The method of claim 11,
The loading step,
Acquiring feature data of the new object when a new object image is acquired through the camera when the operation mode is a learning mode;
Determining an object recognition model corresponding to the new object from among the plurality of object recognition models based on feature data of the new object and information on the plurality of object recognition models; And
And learning an object recognition model that can be connected as a higher level of the determined object recognition model and the determined object recognition model based on the feature data of the new object. The method of claim 11,
The loading step,
When the operation mode is a learning mode, displaying a user interface (UI) representing the stored plurality of object recognition models;
When an object recognition model corresponding to the new object is selected by the user through the UI while the image of a new object is acquired through the camera, the new object image is input to the selected object recognition model, and feature data of the new object Obtaining a;
When a name of a class corresponding to the new object is input from the user, identifying characteristic data of the new object as data on a class corresponding to the new object to which the name is input; And
And learning an object recognition model that can be connected as a higher level of the selected object recognition model and the selected object recognition model based on data on a class corresponding to the new object.

Images