KR20210052153A - Electronic apparatus and method for controlling thereof - Google Patents

Abstract

Disclosed is a method of controlling an electronic device. According to the present disclosure, the control method includes the steps of: obtaining a neural network model trained to detect an object corresponding to at least one class; obtaining a user command for detecting a first object corresponding to a first class; and obtaining the new neural network model based on the neural network model and information on the first object if the first object does not correspond to at least one class. The present invention provides the electronic device capable of obtaining the new neural network model by adding a new class corresponding to a user command to the neural network model, and recognizing an object corresponding to a new class based on the obtained neural network model.

Description

Electronic device and its control method {ELECTRONIC APPARATUS AND METHOD FOR CONTROLLING THEREOF}

The present disclosure relates to an electronic device for customizing a neural network model and a control method thereof, and more particularly, to an electronic device for obtaining a new neural network model by adding a new class according to a user command, and a control method thereof.

Conventional artificial intelligence (AI)-based object recognition models are learned offline based on data for a predetermined category or class, and when learning is completed, a smartphone, robot/robot device, or other image and/or voice recognition system Applies to devices such as

On the other hand, it is difficult to change the object recognition model, such as adding a new class (or category) that can be recognized, to the object recognition model after the object recognition model that has been learned is applied to the device. In order to add a new class to the object recognition model, a large number of samples for the added new class and cloud computing for retraining the object recognition model are required, because this is not appropriate in terms of time and cost.

Recently, as the needs of consumers to customize an object recognition model applied to a smartphone or the like increase, the need for a technology for customizing an object recognition model applied to a product has emerged as learning is completed.

One technical problem to be solved by the present invention is to obtain a new neural network model by adding a new class corresponding to a user command to a neural network model, and to recognize an object corresponding to the new class based on the obtained neural network model. It is to provide the device.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

According to an exemplary embodiment of the present disclosure for solving the above technical problem, there is provided a method for controlling an electronic device, the method comprising: acquiring a neural network model trained to detect an object corresponding to at least one class; Obtaining a user command for detecting a first object corresponding to the first class; And if the first object does not correspond to the at least one class, obtaining a new neural network model based on the neural network model and information on the first object.

According to another exemplary embodiment of the present disclosure for solving the above technical problem, there is provided an electronic device comprising: a memory including at least one instruction; And a processor; wherein the processor obtains a neural network model trained to detect an object corresponding to at least one class, obtains a user command for detecting a first object corresponding to a first class, and the If the first object does not correspond to the at least one class, an electronic device for obtaining a new neural network model based on the neural network model and information on the first object may be provided.

The solution means of the subject of the present disclosure is not limited to the above-described solution means, and solutions that are not mentioned will be clearly understood by those of ordinary skill in the art from the present specification and the accompanying drawings. I will be able to.

According to various embodiments of the present disclosure as described above, the electronic device may recognize an object corresponding to a new class added to the neural network model according to a user command.

In addition, effects that can be obtained or predicted by the embodiments of the present disclosure will be disclosed directly or implicitly in the detailed description of the embodiments of the present disclosure. For example, various effects predicted according to an embodiment of the present disclosure will be disclosed within a detailed description to be described later.

1 is a diagram for describing an electronic device according to an exemplary embodiment of the present disclosure.
2 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.
3A is a diagram illustrating a conventional object recognition model using very few learning.
3B is a diagram illustrating a method of learning a conventional object recognition model.
4A is a diagram illustrating a method of obtaining a new neural network model according to an embodiment of the present disclosure.
4B is a diagram illustrating a method of learning a neural network model according to an embodiment of the present disclosure.
5A is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.
5B is a diagram illustrating a method of obtaining a new neural network model according to an embodiment of the present disclosure.
6A is a diagram illustrating a method of customizing a neural network model using image samples according to an embodiment of the present disclosure.
6B is a diagram illustrating a customizing a neural network model using video frame samples according to an embodiment of the present disclosure.
7 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.
8 is a flowchart of an example of checking whether a user request class already exists in a neural network model.

The terms used in the present specification will be briefly described, and the present disclosure will be described in detail.

Terms used in the embodiments of the present disclosure have selected general terms that are currently widely used as possible while considering functions in the present disclosure, but this may vary according to the intention or precedent of a technician working in the field, the emergence of new technologies, etc. . In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning of the terms will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the overall contents of the present disclosure, not a simple name of the term.

Since the embodiments of the present disclosure may apply various transformations and may have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the specific embodiment, it is to be understood to include all conversions, equivalents, or substitutes included in the disclosed spirit and technical scope. In describing the embodiments, if it is determined that a detailed description of a related known technology may obscure the subject matter, a detailed description thereof will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by terms. The terms are used only for the purpose of distinguishing one component from other components.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "consist" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other It should be understood that the presence or addition of features, numbers, steps, actions, components, parts, or combinations thereof, does not preclude the possibility of preliminary exclusion.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted in order to clearly describe the present disclosure, and similar reference numerals are attached to similar parts throughout the specification.

1 is a diagram for describing an electronic device according to an exemplary embodiment of the present disclosure. The electronic device 100 may acquire the image 30 by capturing the surrounding environment. For example, the electronic device 100 may be a robot device. The electronic device 100 may recognize an object included in the acquired image 30. In more detail, the electronic device 100 may recognize an object included in the image 30 by inputting the acquired image 30 to a neural network model trained to recognize an object included in the image. For example, the electronic device 100 may recognize a cup included in the image 30.

Meanwhile, the electronic device 100 may obtain a command from the user 10 to find the cup 20 of the user 10. In this case, in order for the electronic device 100 to recognize the cup 20 of the user 10, the electronic device 100 uses a neural network model learned to recognize the cup 20 of the user 10 ) Of the cup (20) should be recognized. However, in general, the electronic device 100 distributed to the user 10 uses a trained neural network model suitable for the most common and general purpose (eg, identifying whether an object is a cup or a book). Accordingly, although the electronic device 100 can recognize a plurality of cups included in the image 30 using a conventional object recognition model (or neural network model), the cup 20 of the user 10 among the plurality of cups is ) Cannot be recognized.

Meanwhile, the electronic device 100 according to the present disclosure searches for a command related to the cup 20 of the user 10, which is an object that cannot be identified by the user 10 with a neural network model previously stored. Command), it is possible to obtain a new neural network model by extracting a feature for the cup 20 and changing a weight vector value of a previously stored neural network model based on the extracted feature. In addition, the electronic device 100 may perform a function corresponding to a user command (for example, a function of providing a location of the cup 20 to the user) by using the acquired new neural network model.

Hereinafter, a method of controlling the electronic device 100 for performing such a function will be described.

2 is a flowchart illustrating a method of controlling the electronic device 100 according to an embodiment of the present disclosure.

The electronic device 100 may obtain a trained neural network model to detect an object corresponding to at least one class (S210). In this case, the electronic device 100 may acquire a neural network model learned from an external server or an external device. Further, the electronic device 100 may detect an object corresponding to at least one class by using the obtained neural network model. For example, the electronic device 100 may acquire an image of a surrounding area, and the neural network model may input the acquired image to obtain type information and location information about an object included in the image. Meanwhile, in the present disclosure, a class is a term for classifying a plurality of objects and may be used interchangeably with "classification" and "category".

Meanwhile, a neural network model is a classification that obtains a classification value for an object based on a feature extraction module (or feature extractor) that extracts a feature value (or feature) for an object and a feature value obtained from the feature extraction module. It may include a value acquisition module (or classifier). In addition, the classification value acquisition module may include a weight vector including at least one column vector.

Neural network models can be trained in a variety of learning methods. In particular, the neural network model may be trained based on a machine learning method. In this case, the neural network model may be trained based on a loss function including a predefined normalization function to prevent overfitting. In particular, a neural network model can be trained based on orthogonal constraints.

The electronic device 100 may obtain a user command for detecting a first object corresponding to the first class (S220). In this case, the user command may be made in various forms. For example, the user command may include a command for storing the first class as a new class in the electronic device 100 or a command for detecting a first object corresponding to the first class. Also, the electronic device 100 may obtain an image of the first object.

When a user command is obtained, the electronic device 100 may determine whether the first object corresponds to at least one pre-learned class that can be identified by the neural network model. In more detail, the electronic device 100 may acquire an image of the first object and input the image of the first object into a neural network model to obtain a first feature value of the first object. The electronic device 100 may compare the first feature value with the weight vector of the learned neural network model to determine whether it corresponds to at least one previously learned class.

If the first object does not correspond to at least one class, the electronic device 100 acquires a new neural network model based on the neural network model and information on the first object (S230), and the electronic device 100 The first object may be detected using a neural network model. In this case, the electronic device 100 may acquire a new neural network model based on few-shot learning. Very few learning means learning using a small amount of learning data compared to general learning. Meanwhile, hereinafter, the neural network model refers to the neural network model obtained in step S210 unless otherwise specified, and the neural network model obtained through step S230 is referred to as a new neural network model.

The electronic device 100 may obtain a first feature value for the first object by inputting an image of the first object into a feature extraction module of the neural network model. Further, the electronic device 100 may acquire a new classification value obtaining module based on the first feature value and the classification value obtaining module of the neural network model. Specifically, the electronic device 100 may obtain a new classification value acquisition module by generating a first column vector based on the average value of the first feature value and adding the first column vector as a new column vector of the weight vector. . Also, the electronic device 100 may normalize a new classification value acquisition module based on a predefined normalization function.

Hereinafter, a method of learning a neural network model and a method of acquiring a new neural network model based on the learned neural network model will be described in more detail.

3A is a diagram illustrating a conventional object recognition model using very few learning. When a new class is added to the previously learned object recognition model, the object recognition model can only detect objects corresponding to the new class, and cannot detect objects corresponding to the previously learned class before the new class is added. That is, the classifier 302 of the conventional object recognition model includes only the weight vector for the new class and does not include the weight vector for the existing class. In addition, in order to recognize an object corresponding to a new class, a process of learning the classifier 302 based on learning data corresponding to the new class was required.

3B is a diagram illustrating a method of learning a conventional object recognition model. Referring to FIG. 3B, the object recognition model includes a feature extractor 300 and a classifier 302. The feature extractor 300 extracts features for the input training sample, and the classifier 302 outputs a classification value for the extracted features. The classifier 302 includes a weight vector W. The object recognition model was trained by an error backpropagation that minimizes an error calculated based on a loss function defined based on the output classification value and labeling data corresponding to the training sample. .

4A is a diagram illustrating a method of obtaining a new neural network model according to an embodiment of the present disclosure. The electronic device 100 may train a neural network model based on very few learning. The neural network model may include a feature extraction module 410 and a classification module (or a classification value acquisition module) 420. The classification module 420 may include a weight vector composed of a plurality of column vectors or row vectors. Each column vector of the weight vector may correspond to each class. The classification module 420 may include a base portion 421 and a novel portion 422. The base region 421 may include a vector corresponding to a previously learned class, and the new region 422 may include a vector corresponding to a new class according to a user input. The new area 422 may be used interchangeably as a local portion.

The electronic device 100 may obtain a feature value for the object 41 by inputting an object 41 corresponding to a previously learned class (ie, included in the base region) into the feature extraction module 410. The electronic device 100 may obtain a classification value for the object 41 by inputting the obtained feature value into the classification module 420. In this case, the electronic device 100 may perform a dot product calculation between the feature value of the object 41 and the weight vector included in the classification module 420. In addition, the neural network model may output a classification value based on a vector for a feature value and a cosine distance of a weight vector of the classification module 420.

Meanwhile, the electronic device 100 may receive a request from a user for a first object 42 that does not correspond to a previously learned class (ie, is not included in the base region 421 ). In this case, the electronic device 100 may obtain the first feature value 43 by inputting the first object 42 into the feature extraction module 410. The electronic device 100 may obtain a new weight vector (or column vector) to be allocated or stored in the new region 422 based on the first feature value 43. For example, the electronic device 100 may average the first feature values 43 and store the averaged first feature values 43 in the new area 422. Accordingly, the new area 422 may include a weight vector corresponding to a new class corresponding to the first object 42. In this way, the electronic device 100 may obtain a new neural network model by updating the new area 422 of the classification module 420 based on the first feature value 43.

Meanwhile, referring again to FIG. 3A, when a conventional neural network model (or an electronic device to which a neural network model is applied) is trained to recognize an object corresponding to a new class, the classifier 302 includes a weight vector corresponding to the new class. ) Is updated, the weight vector corresponding to the previously learned class is not included. Accordingly, the conventional neural network model could no longer recognize an object corresponding to a previously learned class. On the other hand, the new neural network model according to the present disclosure includes both the base region 421 and the new region 422 even though it is learned to recognize an object corresponding to a new class. Accordingly, the electronic device 100 may recognize not only the first object 42 but also the previously learned object 41 by using the new neural network model.

4B is a diagram illustrating a method of learning a neural network model according to an embodiment of the present disclosure.

The neural network model may include a feature extraction module 410 and a classification value acquisition module 420. The feature extraction module 410 is trained to extract a feature (or feature value) for an input training sample. The classification value acquisition module 420 is trained to output a classification value for a training sample based on the feature value output from the feature extraction module 410. The neural network model may be trained based on an orthogonality score calculated based on a predefined function, a classification value, and a loss function defined based on labeling data corresponding to a training sample. The neural network model can be trained according to the error inversion method that minimizes the error calculated based on the loss function. Here, the orthogonality score may be a value in which the weight vector of the classification value acquisition module 420 is normalized.

Meanwhile, since the conventional object recognition model according to FIG. 3B is not trained based on the orthogonality score, normalization for the classifier 302 is not performed. Accordingly, the conventional object recognition model was over-fitting with respect to the previously learned class, so that the object for the new class could not be recognized. In contrast, since the neural network model according to the present disclosure is normalized based on the orthogonality score, it is not overfitting for a specific class. In addition, compatibility between weight vectors included in the base region 421 and weight vectors added to the new region 422 may be improved. That is, if the neural network model is trained based on orthogonality constraint, the user can more easily customize the neural network model. Also, the electronic device 100 may normalize weight vectors added to the new region 422 based on orthogonality constraints.

5A is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

The electronic device 100 may include a camera 110, a microphone 120, a communication interface 130, a memory 140, a display 150, a speaker 160, and a processor 170. For example, the electronic device 100 is a smartphone, a tablet, a notebook, a computer or computing device, a virtual assistant device, a robot and a robot device, a consumer goods/home appliance device (eg, a smart refrigerator), an Internet of Things device, or an image capturing system/device. It may be any user device such as, but is not limited thereto. Hereinafter, each configuration will be described.

The camera 110 may acquire an image by capturing an image around the electronic device 100. In addition, the camera 110 may obtain a user command. For example, an image of an object provided by a user or an image of a user's gesture may be acquired. The camera 110 may be implemented with various types of cameras. For example, the camera 110 may be any one of a 2D-based RGB camera and an IR camera. Alternatively, the camera 110 may be any one of a 3D-based Time of Flight (ToF) camera and a stereo camera.

The microphone 120 is a configuration for receiving a user's voice input, and may be provided in the electronic device 100, but this is only an embodiment, and the electronic device 100 and wired or Can be connected wirelessly. In particular, the microphone 120 may receive a user voice for searching for a specific object.

The communication interface 130 includes at least one circuit and may communicate with various types of external devices. For example, the communication interface 130 may perform communication with an external server or a user terminal. In addition, the communication interface 130 may communicate with an external device according to various types of communication methods. The communication interface 130 may perform data communication wirelessly or wired. When performing communication with an external device in a wireless communication method, the communication interface 130 includes a Wi-Fi communication module, a cellular communication module, a 3G (3rd generation) mobile communication module, a 4G (4th generation) mobile communication module, and a 4th generation LTE (Long It may include at least one of a Term Evolution) communication module and a 5G (5th generation) mobile communication module. Meanwhile, according to an embodiment of the present disclosure, the communication interface 130 may be implemented as a wireless communication module, but this is only an embodiment, and may be implemented as a wired communication module (eg, LAN, etc.). .

The memory 140 may store an operating system (OS) for controlling the overall operation of the components of the electronic device 100 and commands or data related to the components of the electronic device 100. To this end, the memory 140 may be implemented as a nonvolatile memory (eg, a hard disk, a solid state drive (SSD), a flash memory), a volatile memory, or the like.

For example, the memory 140 stores an instruction that causes the processor 150 to acquire type information and location information for an object included in the image when an image is acquired from the camera 110 when executed. I can. In addition, the memory 140 may store a neural network model for recognizing an object. In particular, the neural network model may be executed by an existing general-purpose processor (eg, CPU) or a separate AI-only processor (eg, GPU, NPU, etc.). In addition, the memory 140 may store data on an application that the user may request to add a new class to the neural network model.

The display 150 may display various screens. For example, the display 150 may display an application execution screen so that the electronic device 100 may display a screen through which a user can input a request for a new class using the application. In addition, the display 150 may display an object requested by the user, or may display a prompt or notification generated by the electronic device 100. Meanwhile, the display 150 may be implemented as a touch screen. In this case, the processor 170 may obtain a user's touch input through the display 150.

The speaker 160 may be a component that outputs not only various audio data received from the outside, but also various notification sounds or voice messages. In this case, the electronic device 100 may include an audio output device such as the speaker 160, but may include an output device such as an audio output terminal. In particular, the speaker 160 may provide a response result and an operation result to the user's voice in a voice form.

The processor 170 may control the overall operation of the electronic device 100. For example, the processor 170 may acquire a trained neural network model to detect an object corresponding to at least one preset class. The obtained neural network model may be a neural network model trained to acquire type information on an object included in the image. The neural network model may include a feature extraction module for extracting a feature value for an object, and a classification value acquisition module for obtaining a classification value for an object based on the feature value obtained from the feature extraction module.

The processor 170 may obtain a user command for detecting a first object corresponding to the first class. The processor 170 inputs an image of the first object into the learned neural network model to obtain a first feature value for the first object, and compares the first feature value with a weight vector of the learned neural network model to obtain a first object. It can be determined whether is corresponding to at least one class.

If the first object does not correspond to at least one class, the processor 170 may acquire a new neural network model based on the neural network model and information on the first object. Specifically, the processor 170 may acquire an image of the first object and input the acquired image of the first object to the feature extraction module to obtain a first feature value of the first object. Further, the processor 170 may acquire a new classification value obtaining module based on the first feature value and classification value obtaining module. In this case, the classification value acquisition module may include a weight vector including a plurality of column vectors. The processor 170 may obtain a new classification value acquisition module by generating a first column vector based on the average value of the first feature values and adding the first column vector as a new column vector of the weight vector. The processor 170 may normalize the acquired new classification value acquisition module based on a predefined normalization function.

The processor 170 may customize the neural network model. The processor 170 may receive a user request for a new class and determine whether the new class is new and should be added to the neural network model. If it is determined that the new class is new, the processor 170 may obtain at least one sample representing the new class. In this case, the processor 170 may acquire at least one of an image, an audio file, an audio clip, a video, and a frame of a video as a sample.

The processor 170 may acquire at least one feature extracted from at least one sample from a neural network model including a basic region of the feature extraction module and the classification value acquisition module. In this case, the processor 170 may transmit a user request and at least one sample to an external server including a neural network model through the communication interface 130. The processor 170 may receive features of at least one sample from an external server. The processor 170 may store the extracted at least one feature as a representative of the new class. In this case, the processor 170 may store the weight vector of the classification value acquisition module corresponding to the new class in the memory 140.

The processor 170 may acquire at least one keyword related to the new class. The processor 170 may determine whether at least one keyword matches one of a plurality of predefined keywords in the base region of the classification value acquisition module of the neural network model. When at least one keyword matches one of a plurality of predefined keywords, the processor 170 may identify a class corresponding to the matched predefined keyword. The processor 170 may control the display 150 or the speaker 160 to output an example sample corresponding to the identified class and a proposal for allocating at least one keyword to the identified class. At this time, when a user confirmation that at least one keyword should be assigned to the identified class is obtained, the processor 170 may allocate at least one keyword to the identified class. On the other hand, when a user input disallowing assignment of at least one keyword to the identified class is obtained, the processor 170 may add a new class to the neural network model.

Meanwhile, the electronic device 100 according to the present disclosure may be effective in terms of protecting a user's privacy. This is because the new class is not stored in the cloud that other users can use/access to or added to the base part of the classifier, but is stored in the electronic device 100. However, you may want to share the new class defined by the user to other devices of the user (eg, from smartphone to laptop, virtual assistant, robot butler, smart refrigerator, etc.). Accordingly, the processor 170 may share the new class stored in the local area of the classification value acquisition module with the external device. The processor 170 may share the new class stored in the local area of the classification value acquisition module with an external server including the base area of the classification value acquisition module. This can be done automatically. For example, when the neural network model is used as part of a camera application, when the neural network model is updated on the user's smartphone, the neural network model can be automatically shared with other devices of the user running the same camera application. Thus, sharing can be part of the synchronization of software applications between many devices.

In particular, functions related to artificial intelligence according to the present disclosure are operated through the processor 170 and the memory 140. The processor 170 may be composed of one or a plurality of processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, or Digital Signal Processor (DSP), a graphics-only processor such as a GPU, a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as an NPU. One or more processors control to process input data according to a predefined operation rule or an artificial intelligence model stored in the memory 140. Alternatively, when one or more processors are dedicated AI processors, the AI dedicated processor may be designed with a hardware structure specialized for processing a specific AI model.

A predefined motion rule or artificial intelligence model is characterized by being created through learning. Here, to be made through learning means that a basic artificial intelligence model is learned using a plurality of learning data by a learning algorithm, so that a predefined motion rule or artificial intelligence model set to perform a desired characteristic (or purpose) is created. Means Jim. Such learning may be performed in a device on which artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and/or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, generative adversarial network, or reinforcement learning. , Is not limited to the above-described example.

The artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation result of a previous layer and a plurality of weights. The plurality of weights of the plurality of neural network layers can be optimized by the learning result of the artificial intelligence model. For example, a plurality of weights may be updated to reduce or minimize a loss value or a cost value obtained from the artificial intelligence model during the learning process. The artificial neural network may include a deep neural network (DNN), for example, CNN (Convolutional Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), BRDNN (Bidirectional Recurrent Deep Neural Network), Deep Q-Networks (Deep Q-Networks), and the like, but is not limited to the above-described example.

Meanwhile, the electronic device 100 according to the present disclosure also ensures the accuracy of the neural network model and allows the neural network model to be customized with efficient time, resources, and cost. This is accomplished by locally expanding the classification value acquisition module of the neural network model in the electronic device 100. In other words, the electronic device 100 does not change the entire pre-learned classification value acquisition module. This means that the neural network model does not need to be retrained from scratch, and the neural network model can be quickly updated. In addition, this means that there is no need to use expensive cloud computing to update/customize neural network models.

Meanwhile, the electronic device 100 may obtain a neural network model from an external server.

5B is a diagram illustrating a method of obtaining a new neural network model according to an embodiment of the present disclosure. The electronic device 100 may obtain a neural network model 180 including a feature extraction module 181 and a classification value acquisition module 182 from the external server 500. The classification value acquisition module 182 may include a base area 182-1. The electronic device 100 may obtain a new classification value obtaining module from the classification value obtaining module 182 obtained from the external server 500. Specifically, the electronic device 100 may average feature values for objects corresponding to the new class and store them in the local area 182-2. Accordingly, the electronic device 100 may detect an object corresponding to a new class. Meanwhile, the operation of obtaining a feature value for an object corresponding to a new class may be performed by the external server 500. In this case, the electronic device 100 obtains information on the object corresponding to the new class from the user and transmits it to the external server 500, and information on the object corresponding to the new class extracted by the external server 500 Can be received.

6A is a diagram illustrating a method of customizing a neural network model using image samples according to an embodiment of the present disclosure. In this embodiment, the user may try to find a picture of the user's dog on the electronic device 100. The electronic device 100 may be a smartphone. However, in addition to the user's dog, images of numerous other dogs may be stored in the electronic device 100. The image gallery application of the electronic device 100 can search the location of all photos including dogs using text-based keyword search, but there is no keyword or class corresponding to the user's dog, so the location of the image including the user's dog You may not be able to search.

In operation S600, the electronic device 100 may obtain a user command for selecting an image gallery application installed in the electronic device 100. In addition, the user may enter the setting field of the image gallery application installed in the electronic device 100. The electronic device 100 may display a screen for "adding a new search category". The user may add a "new search category" on the screen displayed on the electronic device 100. Accordingly, the electronic device 100 may perform an operation of customizing the neural network model. The electronic device 100 may induce a user to input a request for a new class. The electronic device 100 may induce the user to input a new category keyword. In this case, in step S602, the user may input keywords "German Shepherd", "My Dog", and "Leica" (dog name) related to the new category.

In step S604, the user may take an image of the dog using the camera or may add a photo of the user's dog to the application from the image gallery. Accordingly, the electronic device 100 may store a new category input by the user. Thereafter, when the user inputs the keyword "my dog" in the search function of the image gallery application, the electronic device 100 may display an image of the user's dog and provide it to the user (S606).

In addition, the user can use the settings field of the image gallery application to delete categories that are no longer requested by deleting keywords associated with the category. As a result, the classifier weight vector associated with the keyword may be deleted from the local part of the classifier.

6B is a diagram illustrating a customizing a neural network model using video frame samples according to an embodiment of the present disclosure. In this embodiment, the user may not like the default palm gesture (such as S610) used on the user's smartphone to give a command to the smartphone's camera when taking a selfie. The user may wish to register a new gesture for "taking a selfie". In step S612, the user enters the camera setting field and selects "new selfie shooting gesture setting". The user starts recording a user-defined gesture moving his or her head from left to right (S614). In step S616, the user can confirm the user's selection. Accordingly, the user can use the user's new gesture to operate the selfie photographing operation (S618).

As described above, only if the base part of the classifier (or the local part if it already exists) does not already contain a similar or substantially identical class, the new local part of the classifier of the machine learning model is It may be desirable to register the class. If a similar or substantially identical class already exists in the classifier, the user can announce the existence of the class and suggest linking the keyword to the existing class. The user can accept or reject this offer, and in case of rejection, the process of adding the user-provided class to the model continues.

7 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.

The electronic device 100 may obtain a user command for the first class (S710). The user can make this request in any suitable way, such as installing an application that allows the user to interact with the neural network model. The application may be, for example, an application related to the camera of the electronic device 100 or an application used to match a video and an image captured by the camera.

The electronic device 100 may determine whether the first class is a new class and should be added to a neural network model previously stored in the electronic device 100 (S720). This determination can be performed to avoid class duplication of the model, which can lead to inefficient model operation. An example of a method for determining whether the first class requested by the user is a new class will be described below with reference to FIG. 8.

If the first class is determined to be a new class in step S720, the electronic device 100 may obtain at least one sample representing the first class in step S730. The at least one sample may be one or more of an image, an audio file, an audio clip, a video, and a frame of a video. In general, at least one sample may all be a set of images representing the same object (or feature) to be used to define the first class. For example, if the user wants the neural network model to identify the user's puppy in images and videos, the user may provide one or more pictures of the user's puppy as input samples representing the first class. When multiple samples are acquired, all of the samples may be of the same type/file type (eg, image) or different types (eg, image and video). In other words, the user can provide both the user's dog photo and video as input samples.

One sample representing the first class may be sufficient to customize the neural network model. However, like other machine learning techniques, the more samples you usually get, the better or better you get. If the obtained sample is of poor quality or is insufficient to define the first class and add it to the neural network model, the electronic device 100 may output a message requesting the user to input more samples.

In some cases, the user request in step S710 may include a sample representing a new class (ie, a first class). In this case, in step S730, the electronic device 100 may simply use a sample that has already been received. In some cases, the user request in step S710 may not include any samples. In this case, in step S730, the electronic device 100 may include a guide message for inducing a user to provide/input a sample. Alternatively, the sample may be received in step S720, and accordingly, in step S730, the electronic device 100 may use the sample acquired in step S720.

The electronic device 100 may obtain a feature value for the obtained sample (S740). In this case, the electronic device 100 may acquire a feature value for the sample by inputting the acquired sample to the feature extraction module included in the neural network model. All or part of the neural network model may be implemented in the electronic device 100 or a remote server/cloud server.

The electronic device 100 may store the acquired feature value in a local area of the classification value obtaining module (S750). Accordingly, the electronic device 100 may obtain a neural network model capable of recognizing an object corresponding to the first class.

8 is a flowchart of an example of checking whether a user request class already exists in a neural network model.

The electronic device 100 may receive a user request for a new class (S800) and receive at least one keyword associated with the new class (S802). The electronic device 100 may determine whether the received at least one keyword matches a predefined keyword (S804). As a result of the determination, if the local area of the classifier already exists, the electronic device 100 may match the base area of the classifier and the keyword related to the local area.

When the received keyword matches a predefined keyword, the electronic device 100 may identify a class corresponding to the matched keyword (S806). Further, the electronic device 100 may output a proposal for allocating the received keyword to the identified existing class (S808). In this case, the electronic device 100 may also output an example sample corresponding to the identified class to explain why the new class is similar/same as the identified existing class.

The electronic device 100 may determine whether the user has accepted the proposal (S810). In this case, the electronic device 100 may determine whether to accept the user's proposal based on the user response. If it is determined that the user has accepted the proposal, the electronic device 100 may allocate the keyword to the identified class (S812). On the other hand, if it is determined that the user has not accepted the proposal, the electronic device 100 may perform an operation for adding a new class to the neural network model, and this operation may lead to step S730 of FIG. 7.

If the keyword received in step S804 does not match any of the predefined keywords, the electronic device 100 may receive at least one sample representing the received class (S814). Further, the electronic device 100 may obtain a feature value of the received sample and determine whether the obtained feature value matches an existing class (S818). The electronic device 100 may determine by calculating the dot product of the feature vector extracted from the received sample and each classifier weight vector of the classifier.

If it is determined that the feature value of the sample matches the existing class, the electronic device 100 may output a proposal to allocate the received keyword to the identified class (S808). On the other hand, if it is determined that the feature value of the sample does not match the existing class, the electronic device 100 may perform an operation to add a new class to the neural network model, and this operation may lead to step S730 of FIG. 7. have.

Meanwhile, the various embodiments described above may be implemented in a recording medium that can be read by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. According to software implementation, embodiments such as procedures and functions described in the present specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, the various embodiments described above may be implemented in a recording medium that can be read by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. According to software implementation, embodiments such as procedures and functions described in the present specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, computer instructions for performing a processing operation according to various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. When a computer instruction stored in such a non-transitory computer-readable medium is executed by a processor, a specific device may cause the processing operation according to the above-described various embodiments to be performed.

The non-transitory computer-readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short moment, such as registers, caches, and memory. Specific examples of non-transitory computer-readable media may include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

Meanwhile, a storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, the term'non-transitory storage medium' is a tangible device and only means that it does not contain a signal (e.g., electromagnetic wave), and this term refers to the case where data is semi-permanently stored in the storage medium and temporary. It does not distinguish the case where it is stored as. For example, the'non-transitory storage medium' may include a buffer in which data is temporarily stored.

According to an embodiment, a method according to various embodiments disclosed in the present document may be provided by being included in a computer program product. Computer program products can be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store ^TM ) or two user devices It can be distributed (e.g., downloaded or uploaded) directly between, e.g. smartphones), online. In the case of online distribution, at least a part of the computer program product (e.g., downloadable app) is at least in a device-readable storage medium such as the manufacturer's server, the application store's server, or the relay server's memory. It can be temporarily stored or created temporarily.

In the above, preferred embodiments of the present disclosure have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and is generally used in the technical field belonging to the disclosure without departing from the gist of the disclosure claimed in the claims. Various modifications are possible by those skilled in the art of course, and these modifications should not be individually understood from the technical idea or perspective of the present disclosure.

Claims (15)

In the control method of an electronic device,
Obtaining a trained neural network model to detect an object corresponding to at least one class;
Obtaining a user command for detecting a first object corresponding to the first class; And
If the first object does not correspond to the at least one class, obtaining a new neural network model based on the neural network model and information on the first object; including;
Control method. The method of claim 1,
The trained neural network model,
A feature extraction module that extracts feature values for an object, and
Comprising a classification value acquisition module for obtaining a classification value for the object based on the feature value obtained from the feature extraction module
Control method. The method of claim 2,
Obtaining the user command,
Including the step of obtaining an image for the first object,
Obtaining the new neural network model,
Inputting the acquired image of the first object into the feature extraction module to obtain a first feature value for the first object,
Comprising the step of obtaining a new classification value obtaining module based on the first feature value and the classification value obtaining module
Control method. The method of claim 3,
The classification value acquisition module includes a weight vector including a plurality of column vectors,
Obtaining the new classification value acquisition module,
Generate a first column vector based on the average value of the first feature value,
Acquiring the new classification value acquisition module by adding the first column vector as a new column vector of the weight vector
Control method. The method of claim 4,
Normalizing the obtained new classification value acquisition module based on a predefined normalization function; further comprising
Control method. The method of claim 1,
The trained neural network model,
It is learned based on a loss function that includes a predefined regularization function to prevent overfitting.
Control method. The method of claim 1,
When the user command is obtained, determining whether the first object corresponds to the at least one class; further comprising,
The determining step,
Obtaining an image for the first object,
Inputting an image of the first object into the learned neural network model to obtain a first feature value of the first object,
Comparing the first feature value and the weight vector of the learned neural network model to determine whether the first object corresponds to the at least one class
Control method. In the electronic device,
A memory including at least one instruction; And
Including; a processor;
The processor,
Acquire a neural network model trained to detect an object corresponding to at least one class,
Obtaining a user command for detecting a first object corresponding to the first class,
If the first object does not correspond to the at least one class, obtaining a new neural network model based on the neural network model and information on the first object
Electronic device. The method of claim 8,
The trained neural network model,
A feature extraction module that extracts feature values for an object, and
Comprising a classification value acquisition module for obtaining a classification value for the object based on the feature value obtained from the feature extraction module
Electronic device. The method of claim 9,
The processor,
Obtaining an image for the first object,
Inputting the obtained image of the first object into the feature extraction module to obtain a first feature value of the first object,
Acquiring a new classification value obtaining module based on the first feature value and the classification value obtaining module
Electronic device. The method of claim 10,
The classification value acquisition module includes a weight vector including a plurality of column vectors,
The processor,
Generate a first column vector based on the average value of the first feature value,
Acquiring the new classification value acquisition module by adding the first column vector as a new column vector of the weight vector
Electronic device. The method of claim 11,
The processor,
Normalizing the acquired new classification value acquisition module based on a predefined normalization function
Electronic device. The method of claim 8,
The trained neural network model,
It is learned based on a loss function that includes a predefined regularization function to prevent overfitting.
Electronic device. The processor,
Obtaining an image for the first object,
Inputting an image of the first object into the learned neural network model to obtain a first feature value of the first object,
Comparing the first feature value and the weight vector of the learned neural network model to determine whether the first object corresponds to the at least one class
Electronic device. A recording medium capable of recording a computer version on which a program for performing the method according to any one of claims 1 to 14 is recorded.

Images